We went to Phnom Kulen, and it is indeed a nice day out. It was especially nice for us as many of Ponheary's family were with us and they brought a great picnic lunch for us all to enjoy, and we swam in the waterfalls, in addition to seeing the rather new but impressive temple and the lingas in ther river.
It took us about 2 hours to get there from Banteay Srei, which is 30 - 45 minutes outside of town if I remember correctly. Don't wander off of the paths and you will have no worries about landmines. It is, in fact, a popular place for Cambodian families to go for a day out.
You can go through the entry get heading up only until noon. From noon on, only outgoing downward traffic is permitted. You pay a fee there, one price for Cambodians and another price for foreigners, though that price exactly I have forgotten. The road is unpaved, pretty much uphill the entire time, but not in especially bad shape.
The day we went, we left t he hotel at 7:30, went to Banteiy Srei and headed from there up to Phnom Kulen. We got up there around 11 I think. We saw the lingas, the temple, and walked through a forest path to the picnic pavillions where Ponheary's family had set up a feast. After lunch, everyone swam in a small waterfall area and then a big waterfall area. We got back that day around 5 I think...
We had 6 full days in Siem Reap and no jet lag issues. We didn't come close to seeing all that we wanted to see, not even close. We were lazy, for one thing, and prefer to go at a slow pace, seeing fewer sigihts in detail rather than many sights but just an overview. We also went out only in the morning...by lunchtime, we were all ready for some relaxing by the pool. We had Ponheary as our guide, and she was great about adapting herself to our style and speed (or lack thereof). We certainly would have been much slower without her.
It's up to you if you want to spend one of your days there, really. For us, it was our most memorable day, but that's really because of the great company we were in, though the lingas and waterfalls are very nice too. Just going up there to see the lingas, though, seems like a lot of time to spend for 20 minutes of viewing lingas.
Kbal Spean is supposed to have more intricate and interesting carvings. It is quite far as well, and involves about a 45 minute hike up to see the carvings. It would take the better part of a day to go out there as well, though I'm pretty sure you could combine it with Beng Mealea.