Why is math so hard for me but everything else is extremely easy?

I'm almost eighteen, and ever since started school as a young kid I have always had considerable difficulty in mathematics. So much so that i was almost completely math illiterate until my freshmen year of high school. I took three years of algebra in high school and never advanced onto a full geometry class. Most freshmen are in geo and most seniors are either in calc or statistics by the time they graduate. Yet i can barely grasp the concepts of algebra. However, I don't even have to try in anything else. Its kind of an oxymoron, but i excel extensively in the sciences. Chemistry was a breeze and i was fascinated by it, and physics was so easy and again i was fascinated by it. I'm an academic freak when it comes to physics and chem, spending hours learning and teaching myself even after I've taken the courses. The supposedly advanced math concepts like vectors in physics were easy for me to learn, i think because I actually knew how i was using the math. Still i can't get pass a dumbo math class course to save my life. I'm also fascinated with history (as i am known to be a history buff) and love to write fiction. Pretty much my life is surrounded by intellectual pursuits. It just frustrates me considerably that people around me come to me for academic help, and I have no problem understanding and even expanding upon complex scientific concepts, but I can't do the stupid class room math. Anyone know why this might be? I have aspergers as well as dyslexia so i've always wondered if it might have something to do with it.