I have only experienced this while in the Nether, and when either using a firework to propel myself while using an elytra or attempting to open my inventory, to actually do so took around two to three seconds to complete in-game.