For breakfast: If I want to go out for breakfast nine-times out of ten I go to Skylarks (Fairhaven). Sehome Dinner is good if you want hash - it is a hole in the wall type of place in Sehome Village. Other good choices in the downtown area are: The Baglery, Little Cheerful and Old Town Cafe. If you just want sugar and coffee - Rocket Donuts is a must stop! (also downtown).
Little Tokyo (Barkley Village) has a sushi Happy Hour 5 - 7 on Wed.
Dirty Dan's has great food, but I don't know if is well equiped for the baby the space is kinda tight and the prices kinda high.
Someone else will have to speak about dessert since I'm hypoglycemic (I have to "front load" my sugar hence the donut knowledge. I've heard that Mt Bakery (downtown) is very good and is now open late. There is a gelato shop in Fairhaven that looks amazing and the small tastes I've had are quite good.
Don't miss Chocolate Necessity (downtown) for something sweet and/or gelato - it is amazing. And they sell little tasting size cups of the gelato. This is a local chocolate factory's store front shop and their chocolate is amazing!