When a ladder is behind and above a fence, you can jump over the fence, even though the fence should prevent it.
What happens is that you are just inside the "climbable" range in front of the ladder and can climb on it even though the hitbox of the fence is still in between you and the ladder.
The same does NOT work with a wall, because it's slightly wider than a fence. It also doesn't work with any other block, because to have its hitbox in the same block as the ladder, it would have to be in the same block space. But it does work with a boat inside the block that the ladder is on, if the block can't be pushed back (e.g. because of another block behind it).
It also does not work with a vine instead of the ladder, because the vine is slightly thinner than the ladder.
Decreasing the climbable range of a ladder by 0.125 blocks would fix the fence/ladder combination, but not the boat.