Sometimes you can't remove items from your offhand and you have to exit a world or die to get the item out of your offhand like arrows, fireworks, or nautilus shells( ive only seen this in survival)