Whilst Tangier is an awkward place for the uninitiated with its winding streets, road-side traders, suspect drainage system and very steep roads in places (and I do agree that I too hated my first visit), I did return 12 months later and did not regret it.
My first visit to Tangier was part of a four day trip to Gibraltar where I went over for the day on an organised excursion.
This was not a good experience as it was too much in favour of traders selling those wretched and useless herbs which do you no good what-so-ever unless you bring back a skip full. Another organised visit was to the carpet sellers - very high pressure on the tourist and very frustrating.
The sightseeing part of the tour of Tangier was very good although at the end of the day and on my return to Gibraltar I vowed never to return.
My subsequent returns to Morocco have involved three trips to Agadir (24 days total stay), three trips to Marrakech (10 days total stay) and four days in Tangier, all within a period of four years.
There is a huge difference in how one travels in relation to what one sees and does: I do my own trips to these cities, see the sights mainly unorganised and no longer have any hassle from the traders, I bargain hard and we part amicably.
I now love the place so much that I would live there quite happily until I expire and look forward to my next visit as soon as I return from the present one.
I can understand completely your concerns and would advise you to stay clear of Tangier for the present, it is best done slowly as you will need time to visit the many sites there and would involve at least three or four days if not longer.
Gibraltar is quite easy to navigate although taxis are expensive for short trips. The bus service is cheap and effective and will go to the most popular regions in town.
There will be plenty for you to see in Gibraltar and the organised tours are well worth it, if you can manage any depending on precisely how much time you have free.
You can shop duty free which is cheap for some items but not so for others. I'm planning another visit for a week once the flights have been reintroduced from Manchester direct to Gibraltar.
If you need ideas on what to see, drop me a line.