Absolutely none. I programmed as a hobby, but got a Master's degree in mathematics. When I realized that there weren't that many jobs that demanded pure math skills, I thought I may as well go for a job in programming.
The upside is that I find the math side of programming a breeze, but the downside is that I normally have to spend a really long time learning how to use a certain language, since I never learned about general programming topics that you would understand if you took a CS course.