If you set yourself to Spectator Mode and press F3+N, you switch to the previous Game Mode you were in. But if you die in Spectator Mode, pressing F3+N will forget the previous Game Mode, making it impossible to switch with F3+N. It is possible to switch with F3+F4, but notably, Spectator Mode is the first that gets selected. Normally, the previous Game Mode gets selected.