Before reaching Villefranche de Confloent you could stop at Ille-sur-Têt and visit the natural site of Les Orgues (hoodoos).
Villefranche is a good stopping poing : it has a fort (Fort Liberia), the town was fortified by Vauban and is on the Unesco World Heritage List, there are 2 abbeys nearby : St Michel de Cuxa near Prades and St Martin du Canigou accessible by foot or taxi from Vernet-les-Bains or Casteill plus a few grottos (Les Grandes Canalettes) and a very good restaurant (Auberge de Saint-Paul). After Villefranche the road becomes winding.
Further up Mont Louis is another Vauban fortified town, also on the Unesco list.
If you really want to drive off the beaten track and are not afraid of very winding, narrow and slow roads, turn right after Olette and take the Garrotxes road via Railleu, Ayguatebia and Sansa, very remote villages. You'll reach Col de la Quillane between Les Angles and Mont-Louis and you could drive down the N116 to get back to Thuir.
If you want to go further up Mont-Louis, there are several interesting places : Can Marcel, an old farm turned into a folks museum in Sainte-Léocadie, Err, Hix and its 11th century church or Font-Romeu with its ermitage and Black Virgin, Llivia, the only Spanish town in France (linked to Spain itself by an international road) which sports the oldes pharmacy in Europe (so they say).
Count 2 hours from Thuir to Font-Romeu via the N116, more via the Garrotxes road.