They weren't probably weren't a part of the Lannister family (definitely not apart of the Tywin branch), more likely just members of families allied to the Lannisters.

The Lannisters have (well had) an army of 10s of thousands of people. While those at the top may be corrupt, power hungery, maybe even evil... most of those beneath them have no illusions of power or of ruling. They're just people tryna make their way in the world.