We have visited Victoria twice in five years (2008 & 2013) and stayed at Helm's Inn both times. We enjoyed both stays. It is clean. It is quiet. It is at most a 10 minute walk from the ferry terminal. The price is excellent for the large rooms and suites. Just right for a family of three and would serve a family of four as well.
Helm's Inn is directly across the street from Beacon Hill Park - Victoria's answer to Central Park. A very pleasant park in which to stroll.
It is an easy walk into town (no reason to take a car on the ferry unless you plan to go well beyond Victoria) and Victoria is clean and pleasant day and night.
The staff at Helm's Inn are knowledgeable about restaurants and activities in the city. No reason to stay anywhere else unless you want a view of the harbor, in which case there are many other hotels.