There are critical times to try Avignon and times to avoid it!

Each year the Avignon festival (July) means that rooms are impossible to come by.  This is preceded by a film festival which can make things difficult.  August (in common with the rest of France) is when all the French are on holidays - although there is certainly plenty of shops and cafes left open.  That really leaves early June and September as the best times, with May also nice, although even in May (at least in 2007) it can be too hot for sightseeing and wandering the streets. Outside of this the weather is not as certain and you won't be able to catch some rays by the pool!

From November until May there is the possibility of the Mistral, a cold wind which blows down the Rhone Valley, which can blow for a few days at a time.

If visiting museums is your thing, feel free to go in the low season.  Avignon is a major town and you won't have issues with restaurants being closed etc. that you can get elsewhere.

Planning your trip before you go is essential. There are some  good online sources, like the the official tourist website of Avignon , with all the essential sightseeing and events information and the Provence-Hideaways website, written by expats living in the area, with lots of tips about sightseeing, accommodation and restaurants.