To reproduce:

1. Place an enchantment table, clear your inventory, put a stack of 64 books in the first hotbar slot (this works with all hotbar slots)
2. Open the enchantment table, put your mouse over its left slot and hit 1 on your keyboard
3. Hit 1 again and observe how the stack of 63 books turns into 62

Note that, as far as my testing goes, other stackable items are not affected. It's only books.