Definitely something you need to see in Madrid. I came with a study abroad group, so the fee for the museum came with my tuition - and we had a tour guide who wasn't a regular palace museum guide. But I'd definitely recommend taking a guided tour - people who didn't have one kept trying to sneak onto ours.
The castle is huge, and the tour took at least an hour (our guide was very thorough). I can't imagine having gone in and just wandering around, I learned so much about the history of the palace and the items in it.
Also look at the armory - it was open when I went but I'm not sure if that's year round or not.More