The client does not accept any server side changes to the offhand slot while a chest and other non-player inventories are open. Client side changes like item swapping (F button), do still work properly.

Modded servers do not have the capability to revert the client's item swap to the offhand till the opened inventory is closed. More specifically, this creates a client/server desync which the server cannot fix till that chest is closed. For creative mode players, this desync fully alters their inventory due to the power given to that gamemode.

This video demonstrates the problem, recreated in a single player 1.16.3 world by having a command block alter the offhand slot on a timer. It is noted that the slot stops updating while the chest is open. And resumes updating when the chest is closed.

 https://youtu.be/KjP6_dx0W6s