I agree with the previous post that getting to see all of the sights here in Charlottesville without a car will be a bit of a challenge. However, here are a few thoughts that occurred to me yesterday as I was thinking about your question. The Inn at Monticello is a short walk, without the benefit of a sidewalk from the Monticello Visitor Center. You might be able to make arrangements with the staff at the inn to drive you up the mountain to Mr. Jeffersons', and yes, the locals speak of him as if he were still among us! The very helpful visitor center folks might be able to hook you up with another visitor for a ride up the mountain, at this time I don't believe that they offer a shuttle from the center to the house and gardens. If you stay at the inn though you will be further from all of the dining and shopping that is available downtown and near the university. Another option would be to stay downtown and catch the bus from that area to the local community college, look for the buses that say PVCC, which is adjacent to the Monticello Visitor Center. You still have the dilemma of how to get from the center to the house, but you will have made it most of the way. If you are feeling VERY ambitious there is a walking trail that begins a short distance from the visitor's center and winds its way through beautiful woodlands to the top of the mountain and the house and gardens, but this is really quite a hike.
If you are flexible in your travel dates I might suggest working it around the fourth of July. There is a beautiful and very moving immigration and naturalization ceremony held each fourth at Monticello, usually with a high profile guest speaker, lots of great patriotic music and the feel of a real hometown celebration. This event draws the crowds, but the good news is that most of them go home after the ceremony leaving relatively few visitors to tour the house and gardens.
I hope that these tips help you out and that you enjoy your stay in Charlottesville.