I first heard of this island from a girl working at a coffee shop up in Chaing Mia. I asked "where do you go on vacation" and Ko Samet is the answer I got.
Its a beautiful little island (with horrible roads, or should I say road) with limited access once on the island.
Once you are off the ferry, rent a motor bike (with dirt bike tires) for the young and coordinated. If your older or don't feel like dirt biking to your location, jump on the back of a pick-up tuck tuck.
The further south you go, the less of the standard tourism you will see. For the most part this island is a "local spot". The beaches are ok, nothing breath taking....but refreshing after the yuck that is Pattaya.
Back to the roads....These are serious roads! I'm pretty young, fit and a master moped-er....and some of these spots were treacherous!!
It was worth it though....hardly anyone at the end of the road. It was nice to have a peaceful beautiful little spot to stop and listen to the waves.
As for getting there.....well you can pay pay pay or you can be nuts like me and take a moped! GPS on my phone lets me drive around like a local. It was about an hour, hour and a half on the moped. Mind you, this is highway travel, and I was going faster than the cars (moped power!). Make sure you have goggles or a visor on your helmet.
300b is the best price I found for a moped (you have to rent a new one once you get on the island). Make sure you get one with the dirt bike tires, you will need it!!!
Beware of the 6pm ferry (The last one off the island) if there aren't enough people to make it worth their while (money wise) they just won't show up. We lucked out because a newly made friend happened to speak Thai and got us on a different ferry. Once landed they had a truck waiting to take us back to our original ferry terminal.