Baphuon is not visited by as many people as the Bayon, which I don't quite understand because it's literally a few hundred metres away. But I guess that people who are on a tight schedule wouldn't probably bother to include most of the other temples of Angkor Thom. Therefore if you are willing to spend a bit of extra time or be thorough with your visit, Baphuon is a nice place. There are still plenty of people, just LESS people.
Baphuon was the Royal Temple of the city, and they are strict about what you're wearing, so ensure your shoulders and knees are covered. They will not hesitate to turn you away if you are not properly covered. There are also a lot of incredibly steep steps but it's worth the climb and the scaffolding is safe, so take your time and enjoy the view once you get up the top.
Also be careful walking through some of the low doorways, as I saw a lady hit her head and it looked quite painful. Remember that the people who used these temples were much more petite than many of us nowadays.
If you own or manage Baphuon Temple, register now for free tools to enhance your listing, attract new reviews, and respond to reviewers.