You're going to be walking places. Your mana bar is always recharging. It takes 30 seconds to refill from empty. Why on Nirn wouldn't you be casting spells as you walked, no matter what your preferred skills are? More skill in some magic means that those skills are cheaper to use and more effective. Faster healing, cheaper invisibility, free summoned arrows, underwater breathing. You’re going to pick up the books for free in dungeons, and you've got a lot of wilderness to explore.