Inviting someone (of whatever gender you sexually prefer) over your place to have wine and cheese, one-on-one, definitely comes off as too intimate.

Add to that, he's a married man AND her boss (whether directly or indirectly).... it's completely inappropriate and I believe he was fishing to see if the intellectual stimulation could go somewhere else.