There's generally not a problem with dropping off an Irish Republic rented car in Belfast - it's a small island and the additional costs of the one way rental will be minimal. It's also pretty common.
Assuming the rental company will allow an Irish registered car to go to Scotland then you have the costs of putting the car on the ferry plus the costs of the rental company sending someone to Scotland to reclaim the the car plus the ferry charges from Scotland to Ireland. THAT will be expensive.Edited: 3:31 am, May 30, 2012
The cost of one-way rental with drop off in another country is a biggies, no matter the size of the Island. A quick check with Hertz on a weeks rental (small car) gives the following prices (pick up in Dublin): €120 (Drop off in Dublin), €140 (Drop off in Cork, IE) and €250 (Drop off in Belfast).
With a drop off in Edinburgh for the same car is €1650.
Unless there's something in Belfast you want to see/experience get on a plane to Aberdeen (Aer Lingus), Edinburgh (Aer Lingus or Ryanair), Glasgow (Aer Lingus) or Prestwick (Ryanair) and get a car there.
I have a question for someone. I am flying into Belfast. Taking a ferry to Cairnryan, Tour Scotland and ferry back to Belfast and tour Ireland. Depart from Belfast to home. Is it best to rent a car in Belfast and ferry it across both ways? Are you "locked" in your car on this ferry ride? Or do I rent a car in Cairnryan and return it in Cairnryan. Ferry back to Belfast and rent a car car there. Any help would be appreciated, thanks
"Are you "locked" in your car on this ferry ride?"
Definitely not - neither drivers nor passengers are allowed on the vehicle decks during the ferry crossing.
The easiest way for you to tour Scotland and (Northern?) Ireland might be to fly Belfast -> Glascow or Edinburgh and hire a car there for your Scottish tour, return to Belfast by plane and hire a car from the airport - International or City, whichever you fly into - for touring in Ireland.
>>Are you "locked" in your car on this ferry ride?<<
That would be quite dangerous in an emergency, no? The car deck is not easily accessible to where lifeboats are and I suspect that everyone would drown.