Talking to tyrion isn't the problem.

But Varys is not the type to PUBLICLY take sides. Now, this could change - maybe something happens on the way - but there is simply no way Varys will just show up in Mereen and take her side. King's Landing would find out. Vary will continue to play both (or all) sides until the victory is complete and his involvement will only be known to a select few.

Same with Littlefinger. Nobody trusts him but people also don't know how many things he is responsible for.

The show cannot convey these kind of subtleties well - so they seem to have abandoned trying to maintain it and simply just made them known 'players.'

Those two men were so powerful because they worked in secret.

And Varys' typically claims to serve the 'realm.' He may just be a Targ loyalist because they are the best option in his eyes.