We were in the east of PEI 3-6 May 2008. The very good art gallery in Charlottetown was about the only "attraction" open on the Sunday and as far as we could see, very few "attractions" would be open even during the week until well into June. As hardy walkers with full gear for winter conditions in the UK, we were driven indoors by the combination of 4C temperatures and merciless arctic winds pushing the windchill way below freezing. Then the sun shone and although neither the thermometer nor the wind eased much, the real warmth of the sun and the warm coloured soil made it much easier to get outside and enjoy the scenery. It is gently attractive rather than jaw-droppingly striking but the combination of the red soil, green forest and blue sea gives it great charm.
Our outstanding B&B hosts at Country Charm in Murray Harbour had mentioned the restored Fench settlement at Roma, on the end of the peninsular slightly North of East from Montague. See http://www.romapei.com/ Even though not "open", there were interesting walks around the site with helpful interpretation boards. Definitely worth visiting. Note that the road is unpaved - some hire car contracts forbid use on such routes.
Re-reading, perhaps the rewards of the early season visit don't come through well enough. For us, the May sunshine and the clarity of the light resulting from the strong northerly wind made the views wonderful. We were also the only visitors, so parking was easy and staff in shops and restaurants had time to chat. We would come again out of season, but might try September next time.