Dumbarton Oaks is a worth a visit when wandering around Georgetown. The museum is a very manageable size, with interesting Byzantine and Pre-Columbian artifacts. (It's also free.) The gardens do cost to get into, but it's well worth the money. The terraced garden rooms are arranged on a hillside. There's plenty to wander through, and it's a lovely, quiet little oasis. We went in the middle of summer, but I'm sure it would be just as interesting in the off season, when you can see the underlying architecture.
If you own or manage Dumbarton Oaks, register now for free tools to enhance your listing, attract new reviews, and respond to reviewers.