I had the opportunity to walk from the gardens to the castle in early May 2014. So glad I visited! It's a little pricey €5 for the grounds even if the castle isn't open. I went too late to see the inside of the castle unfortunately. But I was glad to see the grounds and walk around. Well worth the 5 minute walk from the front gate of the gardens.
If you own or manage Keukenhof Castle, register now for free tools to enhance your listing, attract new reviews, and respond to reviewers.