Maybe pay at an accelerated rate and invest the rest. Depending on how much time is left on your mortgage you could refi from a 30 year into a 15 year which would typically give you an even lower rate.

But if you can invest and make more than that 3-4% you're paying in interest on the mortgage, then investing makes more sense.