When the <x-rot> argument in /tp [target player] <x> <y> <z> [<y-rot> <x-rot>] is relative (has a ~ ), and is over 90, you look in the wrong direction.

to recreate:
look in the direction 0.0 / 80.0
run the command /tp @p ~ ~ ~ ~ ~-100
you will be looking in the direction -180.0 / 0.0 rather than the expected 0.0 / -20.0

The same thing happens with values under -90

the formula for the actual relative x-rot is (i think) f ( x ) = | x + 90 | - | x - 90 | - x where x = the relative x-rot, and f ( x ) = the actual amount rotated and if | x | > 90, y-rot changes by 180

I think what's happening is that the game looks at the relative /tp as exact, and thus makes it so that any relative value over 90 is changed to be within -90 to 90