As the title said, when going from outer end islands back to the main island through a gateway using elytra and fireworks, this happens. The only way to fix it, is going in and out of your world.