The people convicted as "Variants" aren't necessarily doing anything wrong, just something that wasn't "supposed" to happen based on the Time Keepers idea of what the sacred timeline is. It's not about purposefully doing something wrong or evil, or the Time Keepers would be stopping Thanos and other villains. So end the end it's likely that there's a Wizard of Oz type of reveal for the Time Keepers where they are the real villains, resetting all these innocent people, calling them variants and making them work for the TVA.

>He picked up the tesseract and boom, he was a criminal.
>But he didn't purposely break the timeline.