i guess you've got two choices. Either get the hell out, go cold turkey and just don't speak or see or think about him (hardest one) and then eventually you won't have to worry about it anymore and can concentrate on rest of you r life.
Or you need to talk to him about it, or at least talk to him and find out if he still feels the same way.
Whatever you do don't just leave it, do something or it will probably only get worse.