To answer your question...no....we find the convenience of being close to town and being right on the beach as well as the many amenities of Sugar Beach to far outweigh what some perceive as the negative of being "near" a local housing project. The area in Estate Golden Rock and LaGrande Princess (neighborhoods on St. Croix are named after the original sugar plantation land divisions) contains Club St. Croix, Sugar Beach, Colony Cove, Mill Harbor, The Palms at Pelican Cove, and Hibiscus Beach Hotel. It is a pleasant mix of local residential area and visitor accomodations (condo and hotel).
One of St. Croix's strengths is that there is are choices for everyone. Some prefer larger hotels or villas with little connection to the local community, others prefer accomodations that are closer to the everyday life of St. Croix. Both are perfectly acceptable options and of course have different price tags as well. In most cases St. Croix can provide you with whatever experience you desire to encounter on vacation.
I'd also suggest that you take a moment to go through the review section of Trip Advisor and get first hand opinions on recent stay at any of the options. Feel free to PM me if you'd like further specifics about Sugar Beach.
In any case, you have picked a wonderful island on which to spend your hard-earned vacation!