Changing money in Tanzania is not a problem in most populated areas. Hotels usually have a worse rate than banks, so you should go to a Forex office (aka Bureau de Change) or bank if possible. You may have to wait in line for a bit. Banks in Tanzania are just like banks in the US or the UK. The main banks include NBC and Standard Chartered.  It should be noted the the bank exchange rates are almost always substancially worse than those at a Forex.

ATMs (cash points) are located in almost all banks with 24-hour access and give the same exchange rate as the bank. They may require a Visa card. Inquire if there is any additional fee for withdrawing using a Visa debit card.

When leaving the country, you need to exchange Tanzania shillings back for USD or other foreign currency before you leave Tanzania. Even in Kenya it is not possible to exchange Tanzania shillings. The Forex office (aka Bureau de Change) has the best Tsh to USD rate. There is one or more Forex offices in each international airport. You can also find them in most cities.

The following table, intended for people traveling to Tanzania from outside Africa, describes the pros and cons of taking various forms of money.

Form of money Notes 
US Dollars - $100 and $50 bills (banknotes) Offers the best exchange rate into Tanzania shillings. The bills should be in good condition, clean, without tears, and without writing on them. Only recent series bills, look on the front for the series, will be accepted. According to information from the bank, only $100-bills dated 2006 or younger are accepted. Older bills are useless in both Tanzania and Uganda.
US Dollars - $20, $10, and $5 bills (banknotes) The exchange rate for these bills will be good but somewhat worse than that for $100 and $50 bills. The bills should be in good condition, clean, without tears, and without writing on them. Only recent series bills, look on the front for the series, will be accepted.
US Dollars - $2 and $1 bills (banknotes) Accepted only at a very poor exchange rate. Consider taking these bills only for tips.
US Dollars - travelers checks Not accepted by most banks. Accepted at a few, but not all, Bureau de Change locations and then only at a very poor exchange rate (the effective cashing charge is often 10% to 15%, or even more, in addition to the normal currency exchange fee).
Kenya Shilling - bills (banknotes) Easy to change into Tanzania shillings. The exchange rate is normally fairly good.
Other bills (banknotes)  (for example: Euro, British Pound, Canadian Dollar, etc.) Most Bureau de Change locations will accept a range of other currencies. However, the exchange rate will be poorer than that for US dollars. How much poorer depends on the specific currency with the Euro and British Pound exchanged at the best rates (close to that of USD).
Debit card Usage is not advised as there is a lot of identity theft at times and you are very open to this if you use a debit card. Use a debit card only if identity theft insurance has been purchased. Best used at ATM machines to receive cash. If the card is accepted by a merchant, for example a hotel, expect a 5% or 6% surcharge to be added to the bill for card usage. This in addition to the 3% foreign usage surcharge added by most card issuers. Debit cards with the Visa logo appear to be accepted most followed by MasterCards.
Credit card Credit cards are not widely accepted outside hotels. Usage is not advised as there is a lot of identity theft at times and you are very open to this if you use a credit card. Use a credit card only if identity theft insurance has been purchased. Best used at ATM machines and for emergencies only. If the card is accepted by a merchant, for example a hotel, expect a 5% or 6% surcharge to be added to the bill for card usage. This in addition to the 3% foreign usage surcharge added by most card issuers. Credit cards with the Visa logo appear to be accepted most followed by MasterCards.