I think maybe the right thing to do is bite the bullet is disallow him from calling me dad. Then spending as much time as I can over the next little while constantly trying to reaffirm our relationship in the interests of damage control.
it may be a long term/short term consideration. It's best that he doesn't call me 'dad' in the long term even though it may hurt him in the short term. It's tuff though