For me, the flow from Rockefeller Center to Sacks Fifth Avenue, for example, is seamless - both part of the nicest part of Midtown (no, I am not a fan of Times Square for the most part).
I don't find myself doing much shopping on Fifth Avenue, but it is also the avenue of things like the NY Public Library's main branch, the eastern side of Central Park, and Museum Mile, just to name a few of the highlights of what you will find as you travel along the avenue.
There's also St. Thomas' Episcopal Church, at 5th avenue and 53rd street, home to one of the world's greatest boys' choirs. I've not been there in a while, but I recall that their rehearsals are sometimes open to the public, and when they are in town, they do vespers during the week and such.
If you own or manage Fifth Avenue, register now for free tools to enhance your listing, attract new reviews, and respond to reviewers.