You are using the carts vertically, as if it were an elevator. When you get out of the cart, instead of leaving on the nearest block (same as the boat), you leave in the same place as the cart if it is on a ladder.

You are using the carts vertically, as if it were an elevator, but there is a trapdoor on top of the cart, you dont leave on top of it.