I really don't like the idea of being around a bunch of haughty computer geeks and annoying girls trying to be geeky.
Reading about how the MD5 algorithm worked when I was a teen was cool. Playing with Linux rootkits as a teen was cool. Learning how to code "real" malware in x86 ASM when I was bored in college was cool.
But I don't define myself by the computer security knowledge I have (I do a lot of other things), so going to a conference with a bunch of people I can't stand does not interest me.
People like this are why I didn't join Mensa either. I feel out of place with boring, normal office drones, but I don't like freaks either.