Physics: where to start?

Hi /sci/,

I'm a math nerd but of late have become exasperated with how little my field seems to apply to reality.  Plus what I've seen and heard about theoretical physics (NOVA specials lol) is interesting as fuck.

My background is limited: 2 semesters of shitty "general physics" (collegiate plug and chug) and I've read Einstein's "Relativity: The Special & General Theory". 

So my question is, where do I start?  Are there any good, comprehensive sources for actually learning physics?  Any books suitable for self-study (if this is even possible) but still hard science?  I want to learn more about a lot of this stuff - QM, E&M, astrophysics, classical mechanics - but I don't want a Stephen Hawking-esque "Oh lol I understand physics" feeling, I want to understand the motivation for the theory and how and why the experimental data supports it.  I feel like an emphasis on the math would be helpful for me, too.

So yeah, if anyone can help a brother out, I'd appreciate it.  Thanks in advance.

OP here, thanks for all of your suggestions so far.

>>5 My math background: 4 semesters of calc (multivariable and ODEs), 2 of real analysis (up to differentials & integration in Rn, a little measure theory), a taste of complex analysis, some algebra (abstract & linear, up to JC form and spectral mappings), probability (some combinatorics; independent RVs, up to central limit thm), logic, set theory, and topology.  This is all undergrad, but I'll be taking graduate real analysis & algebra this fall.

Something I certainly wouldn't mind starting with (after a review of basics, of course) is QM; I've heard that the book by Griffiths is a classic but honestly don't know if I'm ready to just jump in.

>>3 Would Young & Freeman's "University Physics" be such a text?  Part of my problem is not knowing which basic physics texts have broad coverage without needlessly abstaining from using calculus and thus descending into bullshit.