Steps to reproduce

• Acquire a bow with the Infinity enchantment, for example via fishing. (I reproduced with a bow with Infinity+ Power III)
• Stick arrows in your offhand slot
• Shoot the bow and check your inventory. Notice that an arrow has been taken from your off-hand slot.

This only applies to arrows in the off-hand slot. For arrows in the normal inventory, the normal rules apply.