When i ride a horse and shift off it it takes some sort of fall damage

i tested a bit and came to the conclusion that it has to do with the either jumping or the hight difference overall