In Ogunquit, The Anchorage is right on the Marginal Way and you can walk to the Big Beach, about 10 min. walk. Seacastles Resort across the street from the Anchorage has nice units with Kitchenettes and a pool that has a retractable roof.
Right on Big Beach is the Norseman Motor Inn. I have not stayed there but you cannot get any closer to the beach!
The Sparhawk Resort is next door the Anchorage and also very nice. All of these can be painful for the budget in season. Motels along Rt. 1 are less pricey but then you have to take the trolley or drive to the beach. Colonial Village Resort on Rt. 1 was highly recommended by our friends, nice pool, quiet, kitchenettes. You can walk up rt. 1 and down a side street to the Footbridge Beach (north of Big Beach) from Colonial Village. It might be a bit of a hike for the 4 yr old though.
If you are die-hard beachgoers, check out York Beach as well. Very family friendly.
I agree Goose Rocks is quiet and beautiful but limited in activities for the kids.