The AIA guide is of course the biggest and best. I have 2 old downtown walking tour books that have a nice amount of historical background as well as architectural info:
Touring Lower Manhattan: 3 Walks in New York's Historic Downtown by Andrew. S. Dolkart and Steven Wheeler; and
Forging a Metropolis: Walking Tours of Lower Manhattan Architecture by Andrew S. Dolkart (covers fewer places than the first one).
Both are easier to carry around than the AIA guide if you're going to be downtown.
Also there are a lot of specialized architecture books; my fave is
Terra Cotta: Don't Take It for Granite: 3 Walks in NY City Neighborhoods by Susan Tunnick.
Terra Cotta is a building material that was very popular at one time and is making a comeback as a modern building material; I love how it gets used in various ways, and it pops up all over NYC. New York has its own association called The Friends of Terra Cotta for us terra cotta cultists.