I have seen many pictures of Kinkaku-ji -- the Temple of the Golden Pavillion. And I have even read Mishima's book about the monk that found it too beautiful, so he burned it down. Finally I was able to get to Kyoto. And Kinkaku-ji lived up to it's reputation. I hit it on a clear, sunny day and it was incredibly beautiful.
Yes, you should expect a crowd. Probably the best time is earler in the morning and midweek in the Winter, Spring, and Fall. But even then, there will be lots of people adoring the temple.
But it's worth it. Just look at the picture!