There are no virtually no long distance bus (coach) services in France. Nothing I know of runs between those two cities, even indirectly.
The place to look for train information is www.voyages-sncf.com
You might just make the 1933 TER (Regional train) from the Marseille airport station direct to Montpellier. That costs €23,30 but is the last local train along that line. French regional trains go to bed early.
After that there is one more train, at 2010 from Marseille St Charles station which is a TGV and involving a change of train (and a 1hr 26 minute wait) at Valence TGV. Arrival Montpellier at 2353. That one costs €61,20 according to www.voyages-sncf.com plus of course the cost of getting from the airport into Marseille itself by bus or taxi as the trains into Marseille from the airport are not at the necessary time.
Finally there is a 2337 LUNEA night train from Marseille St Charles arriving in Montpellier at just past 3am if you can get a ticket.
Hope this helps and good luck.