Before exploring these caves, you have to talk to: boys front of the tower, then to the guy in the mill (the left one, in Barness), then to the mayor.
Done? OK. Now, If my map is correct...
From the inn in Barness:
-Take the gate left to the inn,
-Go left, take the first cave you'll find
-go down, exit the cave
-then all the way to the right, go to the cave
-you'll find the room with four blocks, push them all in the water to the right, the jump across the water to reach a switch.
-continue your exploration of the caves, you can't be lost from here (only one exit available from here). After two rooms, you'll find a little altar, it will trigger an event.
If you haven't talk to the mayor, nothing will happen.