as your boss, he should be behaving more appropriately. i would say email him (so you have a written record) about how you found his comment inappropriate. if he disagrees or doesn't apologise and continues, then you have a case for your HR dept and its hopefully catalogued via email.
Yeah, so email him to tell him I feel uncomfortable with his remarks and then should I ask him for a meeting as well?