The first thing to point out is that this attraction is almost totally without signposts even with a guide it could be difficult to find, there is a small shop in the village of Kirkoswald which is not far from the caves and if it is your first time visiting I recommend popping in to ask for directions.
To get to the caves you have to follow a beautiful route through a wood which is mostly traversable via a long stretch of path which is made out of wooden decking, the walk through the woods and along the river is so lovely it would be worth making in itself even if the caves were not at the end of it. The lush wood is full of wildlife and at the right time of year there is abundant wild garlic growing there it is important to note that is can get quite muddy if there has been heavy rain though the wooden walkway through the wood lifts you up out of most of it.
The caves themselves are surprisingly large with several rooms and corridors running off each other, doors and windows are cut into the rock face with views out over the river, there is a sheer drop from the caves down into the river below. Apparently the caves were carved out of the cliff face by Colonel Lacy who is also famous for attempting to blow up long meg and her daughters which is a large neolithic stone circle not far from the caves, luckily he did not succeed in that ambition though the caves he built as a folly really are a lovely place to visit.
This review is the subjective opinion of a TripAdvisor member and not of TripAdvisor LLC.