When you're flying using the Elytra, if you open up your inventory, and direct your attention to the black box (at which when not flying your player standing will be displayed) you'll find that your player will be dispositioned at the bottom of the black box, and swaying outside it's borders.

This happens in all instances at which three things are happening:
A: You are flying with the elytra
B: You are in the air
C: You are in your inventory
Gamemode and other settings do not affect this.