Hah if worst comes to worst put one of those odor balls in there - don't grind the ball! Just let it sit there. These balls work for lots of things other than shoes.
I have the same problem you have. I'll want to grind some jeera, but it always smells strongly methee. (It's my mom's old spice grinder, and we grind Indian spices - I'm Indian).
Those are honestly hard to clean. I think I might have seen some medium quality grinders where the top chamber actually comes out and I'd like to clean that; however, I still want to use a high quality grinder.