When starting configuration phase (doesn't matter if on login or when already playing), there is a race condition caused by how netty handles auto-read.

Explanation

When the client receives a ClientboundStartConfigurationPacket (PLAY packet), it immediately suspends inbound (= disables auto-reading in netty channel config), as the packet needs to be handled on the main thread.

Disabling auto-read does prevent reading more packets – but the current packet "batch" still gets handled. These packets will get processed in the wrong handler, as the client didn't switch the packet handlers yet.

This causing errors with my custom protocol implementation, but not with vanilla server for some reason (maybe because everything is delayed because of ticks there?).

(If you want some actual reproduction code LMK)

Solution

When configuring serialization in "Connection#configureSerialization", add netty's built-in "FlowControlHandler" at the end of the pipeline. This solves all my problems when testing, as this flow control handler queues all packets (which would otherwise go to the packet handler) and waits for auto-reading to be turned on again.