I think talking to him would be just the answer. If he loves you and has no intention on leaving for somebody younger without kids, then all is well, and it will settle your insecurities--that's what partners are for after all. If that isn't the case, then as you say yourself, it's best for it to come out in the open now. There are a number of successful relationships where the female is a number of years older than the male, so if both of you are comfortable with it, then that isn't a problem.
I, for one, do not think you're a weak person for wondering what the future may bring.
