The prices of one-way car rentals vary widely, and you need to do a lot of your own research to find a good deal. Not even the past experiences of other travelers is indicative of what the costs will be.
Renting from off-airport offices can save you a lot of money. However, be sure to stick with a reputable firm. Also keep in mind that off-airport rental facilities have fewer car options.
Considering your late arrival, I see no point in driving to Monterey the first night. I recommend booking a hotel near the airport. The area near the airport is nothing to write home about, but it's perfectly fine for an evening. You could consider something like the San Leandro Marina Inn if you want something a little nicer but still near the airport.
If you spend the night near the airport, you may save some money picking up the rental car the next morning.
Also consider dropping your car off somewhere other than LAX. You could use BUR (Burbank Airport), for example, as it is more convenient in many cases. Another option to consider is dropping the car off downtown or in another neighborhood. LAX has several "Flyaway" buses that run to/from the airport frequently from various locations.