Safety could be a concern, but only if you are traveling arriving late at night or are alone.  If you are taking a ferry over from Spain, you will might be pressured into getting a "guide" to show you around the city.  Be warned that you will most likely be given a little tour but then taken to a "School of Rugmaking" where you'll be pressured into purchasing overpriced rugs/blankets.  Ideally, you should have a map of the city before you arrive along with accommodations and arrangements on how to get to them. 

Don't feel pressured into getting a guide. If you get one, be prepared to a quick stroll pointing to "this house is from an Italian artist that bought it for X thousand Euros", that one is where Paul Bowles lived, my grandfather was the gate keeper of the Kasbah, blah, blah", all stories that only by accident will be true. Then you'll be asked some money to help his family, "anything you feel it's fair", but don't think they will be happy with just 20 or 30 Dirham for a 15 min. "tour". Perhaps worse is when the tour is bigger or they try to take you to their friends' shops, which happens more frequently than not.

*Allways* negotiate the price and conditions of the guidance before you follow him. Probably it won't be easy to do that, as they can approach you not as guides, but as someone kindly making conversation and then they'll say "you give what you think it's fair". Even after according a price, be prepared to have hassles at the time of paying, as they will be expecting more and will come up with arguments to make you feel guilty of some offence to them or insensible to their hard life.

If your concern is safety and you feel that your attention may be not enough, or you simply like to have guidance, try to get an official guide. They exist, in theory, and they can be easily identified because they have some badges or id. cards and must be dressed in traditional jellabas. And you can allways ask at the hotels, tourism information offices and travel agencies.

As almost everywhere, safety is hardly measured, as one's bad experience can have been only "bad luck" in the most peaceful place on Earth and there are situations where everybody you know that went to place that is famous for its crime rate had no problems. To make thing even more complicated, you have places where the crime rate is high but tourists are rarely involved and the contrary - relatively peaceful places where there is much pitty theft targeting tourists,  so the actual numbers on crime and theft, if they exist also don't help much on clarifying the situation.

As in any large city anywhere in the world, there is always some risk of being robbed, maybe sometimes exploring the natural fear from the unknown that every tourist or even seasoned "traveler" feels in some situations. But it is unlikely. All this said, Tangier is an wonderful place to explore and there are police everywhere downtown and they are alert to any signs of trouble.