Don't know the Hotel Holt, but it looks quite smart, and they certainly seem sincere about maintaining standards. It's in a quiet area, not too far from the centre at all, and might be a nice choice.
The Borg is completely swell, very central, quite a treat. I suspect you'd enjoy your stay in either hotel.
Borg is still under construction. Don't stay there unless you have a guarantee that it will be finished. I just came back from Iceland and was very glad that I didn't stay in Borg when I went into the lobby.
Holt has a great lobby but I read that the rooms are small. You might want to consider Radisson 1919 but I read that it might be noisy.
Check out the Hotel Vik in Reykjavit we are staying there and it seems very nice , quiet area and rated 7 on the trip advisor site. Check out www.coupons.is for a hotel vik coupon , maybe you can use it and save 14% off the rate, by booking before it expires in june 30....
Good luck. If you like wine... bring a box of wine with you to save $$$ over there.
Also if renting a car check out Budget.is you can book on line and they were cheaper than anyone else I researched.