i do agree with you on this not because i'm a drummer but i've heard first hand the same things you mentioned.
although i'm not sure why it is that drummers are appealing, not only to girls, but also to guys(not in a gay manner), you know what i mean. that's actually the same reason why i got into drumming, i saw my cousin play the drums. and when i heard, take note, heard, not seen, heard him playing the drums, i was blown away. i really find drummers cool ever since.
i think, it's because it's the drummer who's giving the beat or groove and that's where the listeners are moved and entertained, mostly. i don't mean to be unfair to the other members of the band. of course, each has it's own appeal in a different way.
just my 2 cents...