As a BIG fan of the sea (oceanography, maritime history etc..), I found this small museum a pleasant surprise.
It is rather compact and should not take more than 1 hour to view.
BUT, they do guided tours of the retired coast guard vessel docked just outside. That was rather fascinating as the guide was a retired Icelandic coast guard officer who has really filled us in on the whole culture and history of the coast guard.
As I am not personally a big art fan (and thus might skip the 3 major Reykjavik art museums), for me personally this was definitely a "must see".