The views from the Castle are amazing and should not be missed on any trip to Budapest.
We visited there in the afternoon and took the funicular up as we didn't fancy the climb in the sun and the views were amazing. We spent a good hour wandering round the outside taking in the architecture and impressive statues but did not go into any of the museums.
Bus no. 16 and 16A take you up the hill and drop you right in front of the Church / Castle.