Cambria to LAX is a long way - and, in fact, you're more likely to run into race-related traffic leaving from Cambria. Santa Barbara is almost always a bottleneck for traffic (my daughter drives it daily).
The race from Santa Clarita starts in Santa Clarita - about 70 miles away - early that morning. No one will be lining the streets on the morning of the race.
South SB (near the wharf or further south) to LAX: I'd allow 3.5 hours. I live 50 minutes south of SB and we allow 2 hours at a minimum (plus car drop off time). We know alternate routes.
There is *no way* to avoid Santa Barbara from Cambria, unless you use state roads to cut inland to the 5 - which I would not advise as a time-saving alternative (and it would still go through Santa Clarita).
You're over-estimating the effects of a bike race (which will mostly be on roads you'll never see). Cambria is lovely, but I'd allow myself 6 hours just in case, from Cambria (the longer the route, the more variables you could encounter, and there are plenty of bottlenecks on days whether there are bike races or not).
So, SB - to LAX, 3 hours if all is well (with or without bike race, which will not affect you in the morning). Cambria to SB - 2 hours, then, if there's the usual north to south slowdown in SB (which happens with or without bike race), allow an extra hour just in case (can also slow down at multiple other places - if this is an international flight and you need to drop off a car, SB is a much better choice).
Or Carpinteria or Ventura. I'd enjoy my day, but the scenery of the town I'm staying in would not be my top priority on a day when I have to catch a flight at LAX (and Carpinteria and Ventura are pretty, too).