I understand the feeling, jefhtro! I feel the same way...wish I didn't! If you decide not to drive from Ouray to Silverton, maybe take a jeep tour from Ouray higher into the mountains if you can find a half day in your schedule. And endosunshine's given you another great route which doesn't involve the unnerving parts but still has gorgeous views. I only suggested the Million Dollar Highway because you had mentioned Independence Pass, but maybe you don't want to do either one.
If you don't want to drive Independence Pass, just go from Leadville to Interstate 70 and through Glenwood Canyon (where you had wanted to hike Hanging Lake anyway) and then through Glenwood to Carbondale and Aspen that way. Still an awesome drive and no white knuckles!