My wife and I (60 yrs old) took a walk from the ferry terminal to the Fort. Perhaps 30 minutes, with manageable uphill most of the way. Beautiful place to relax and see the views. Well manicured with flowers inside the fort. Fort seems rather small until you go underground into the tunnels and the dry moat. I rather liked walking through the moat due to the vegetation making it seem like a jungle canyon.
Return trip was easier and poked along all the shops on Front St until we got back to the ferry.