Whenever something like that happens to me I force myself to just ignore it and keep moving. Most times it turns out it was just the voices in my head.
Never lower yourself to the level of the people around you. Meaning, by even uttering a response, you've given him credibility, which he doesn't have, and when arguing with fools, no one can tell the difference. Walk away.