You can easily spend a whole day at the Vatican. You can book entry tickets or a tour directly through them at:
Depending on when you go to the museums you can see the basilica before or after the museums. I prefer to see the museums in the afternoon and the basilica in the morning but it's really up to you. After the vatican you can walk to the Castel Sant Angelo to relax, have a snack and get a great view of the city.
You can easily spend most of the day at the colosseum, forum and palatine hill. I would suggest seeing these as early as possible since they tend to get very hot in the afternoon. There is very little shade. You can book tickets or a tour of the colosseum or underground portion of the colosseum at their official site.
Click on the collect onsite option for the underground tour.
After the colosseum area you could add the circus maximus, Santa Maria in Cosmedin, the baths of caracalla, Santa Maria Maggiore or the St. Peter in Chains church. It really depends on what you want to see.
Another day you could see the Piazza Navona, pantheon, trevi fountain, spanish steps and Borghese gallery. You must make reservations if you want to see this wonderful gallery.
As for tour companies, Viator is a reseller and doesn't do any tours. You'll save money by booking directly with the companies. Personally, I don't do guided tours but do like to use audio guides. There are some really good ones out there including the Rick Steves guides and the ones by pocketvox. They have some really good walking tours of the city that are very easy to follow and allow you to go at your own pace.