If you're at Infinity Bay, keep in mind that the cruise ship people don't usually get to West Bay Beach until around 10:00 in the morning, and then they have a brief instructional period. They snorkel around like mad for about an hour or two, and then sun/lounge on the beach, usually leaving for their ship around 3:00. (That's not an absolute schedule, you understand, just a general observation.) So, earlier in the morning, and late in the afternoon, the beach isn't very crowded, even on cruise ship days, for the most part. In the afternoon, no better place to be than at the Infinity Bay pool or lounging on the IB loungers by the beach. I would opt for the pool : )
P.S. Saturday is usually the very best beach day. By Sunday afternoon, locals and their families make the beach fairly crowded again, but hey, they deserve some beach time, too!