Captain's Cove is a popular boating destination on the Connecticut shore: there's a full service marina. You can also go there by car: there's plenty of parking. You'll find lots of on-shore stuff to amuse the kids and the gift-shop crowd, with a nice view of Black Rock Harbor. There may be a maritime exhibition of some kind - early this month they had a replica of the Pinta and Nina - but check with their website.
There's a seafood restaurant and a bar, and there's usually live music at weekends and evenings. The restaurant is basically just a counter where you can order inexpensive but good seafood (I've noticed a poor review from a couple of years ago - they've got much better since then). While you are waiting for your food, you can stake out a table - most are in the open air, but there are some inside and under cover, in case it rains. No table service - they'll call you when your order is ready.
Be sure to check out the huge model of the Titanic in the upstairs bar.
Good place to take the family, or relatives visiting from an inland state.