It's like a guy when asked to say something nice about his gf he said, "She doesn't sweat much for a fat girl." Some people just open their mouths and never know what is coming out. Sounds like he made a very stupid statement and hit a tender spot with you. Not too smart that he did but he might have not meant to be cruel.
You have a couple of things you can do. First is make sure he understands exactly what a stupid comment it was. Next, get rid of your love handles if you are sensitive about them. Refer to his small penis will get even with him very quickly. Watch his reaction to that one.