We've only been once to the races (although we've vacationed there for years and years), and our "kids" were college students at the time. This is one of those things that gets better as kids get older, I think. The early show would be ideal for elementary-age children. Your three-year-old might or might not like it. They might be happier just spending their evening chasing ghost crabs on the beach.
If you are bored by that night and are looking for something to do or it's raining, go for it. The adults will "get" the humor, even if the kids don't. When we were there, even for the early show, it still had a bar vibe, so keep that in mind. But if they are happy doing something else that evening, don't insist on going. There's so much else to do -- from sunsets to exploring the beach by flashlight -- for young children, that you won't have missed anything. Just my opinion, of course!
I guess you know to make sure you walk over to the docks wherever you can -- Jensen's on the bay is a great place to see manatees, and you can often see a variety of wading birds on the dock at The Green Flash. We enjoyed renting bikes on Sanibel and riding them on the trails around the lighthouse, and we used to drive through the refuge there, too. We treasure the memories of our children when they were very young on San-Cap; it didn't take much to keep them occupied!