For someone in good physical condition, and a good pair of walking shoes, it's do-able. I would head over early in the morning, before it gets too hot and the traffic fires up. Once you get out of Gustavia there will be no sidewalks, and some road without much, if any berm. Blind curves, too. Take a look at a map of Gustavia, I would head due east to the highest road out of Gustavia, towards the light house....proceed to the round-about, and then its down hill and the airport is in your sights. It is a tough trek, though, and I would not attempt it.
A better plan if you don't want to take a taxi...hitch hike. It's a common practice on the island and no doubt someone schlepping croissants back to their wife will take pitty on you and drop you off!
Safe travels! Binti