I have a diamond sword with the following enchantments:

• Unbreaking I
• Smite II
• Looting II
• Mending

I have an enchantment book with the following:

• Sharpness III
• Fire Protection III

Presumably, if I put these on an anvil, the Sharpness III enchantment will be added. When I put them in the anvil, it reports that it will have a cost of 32 XP. However, when I hover over the item in the output slot, no enchantment is added. But if I were to pull the item out of the output slot, it appears that it would consume the book and the XP but not actually add the enchantment to the item. If you want me to actually try it, I can do it in a creative world, but someone else would have to reproduce it anyway. The fact is, the anvil should simply not allow this situation to happen; it should not even APPEAR that it's going to consume the XP and the book if it's not going to actually add an enchantment.

I did, of course, search for this bug in your database, and I found one for version 1.6.2 that describes this exact situation, but it's marked "Resolved." So I'm guessing that this bug has since reappeared and is in 1.11.