Recommend you stay away from the Gorge. It's a well known high crime area. Sure there's a nice path farther down West Gorge Road as stated above, but where the hotels are located on East Gorge, there is a LOT of drug dealing and prostitution going on. It's a high transient area, not the most conducive for a peaceful stay. I used to live in the area and moved as soon as my lease was up.
You may wish to try Sidney vs. Victoria - a little quieter but in my opinion more scenic if you want to be near the water.
Try the Cedarwood:
http://thecedarwood.ca
Or if you want upscale but still affordable, try the Sidney Pier Hotel & Spa. They are currently advertising $129/night:
http://sidneypier.com/
If you want to be downtown Victoria, try:
Executive House Hotel - about 2 blocks from the inner harbour:
http://www.executivehouse.com/
Swan's Suite Hotel (half a block from waterfront, right near old town and Wharf/Government Streets)
http://swanshotel.com/
For $100, you won't get waterfront luxury ... but there are lots of nice places around $120 / night.