okay so I got this off reddit and it's probably the best explanation I've seen thus far:

"The speed up is only relative to places very close to Gargantua. If you are far away from it - like they are most of the time - like planet 2 and planet 3 are - then in the parameters of the film time dilation is negligible. It makes sense because gravity follows the inverse-square law as explained.

When is time dilation not negligible?

When Cooper and Miller go on planet 1. They 'lose' 23 years during that attempt. Meaning they literally travel 23 years into earth's future. Yes, time travel is possible.

In the whole time once they regroup with Romilly you can neglect time dilation for the entire movie until

They leave Mann's planet and do a sling-shot along Gargantua. During that scene they are really close to the black hole. Closer than before. Time dilation is massive. However, they also are really, really quick in terms of orbital mechanics. Both Cooper and Brandt 'lose' 60-70 years during that maneuver.

So what you can get from this, there is no significant disparity between Cooper and Brand (at most the order of magnitude of a single decade). The only difference in trips they experience is Brand leaving orbit around Gargantua. Brand instead travels toward the singularity. You can assume what happens in the Tesseract is beyond time dilation. At that point he is in a place where time does not behave like in our space-time anymore.

So in essence, they share the same time development + a decade or so of time dilation after they separate. Cooper from traveling into the singularity, Brand from traveling away from the singularity.
The whole time travel involved is
23 years + ~60 years

The 60 years are the time that Murph's 35 year old actress ages until she lies as a grandma in bed. During those 60 years, Cooper and Brand literally only had an hour or so pass.

So Brand is also of Cooper's age when earth is 85 years in the future. But thousands if not millions of light years away."