When I put my very damaged (probably 10-20% left) looting 3 and bane of anthropods 4 sword in the right slot and a full sharpness II in the left slot the dialog tells me the cost is 33 levels. When performing the merge everything seems fine and when the dialog closes both of the original sword drop and my XP-bar seems emptied.
As soon as I get one XP-orb the XP bar goes back up to 33 levels.
There is one more abnormality with the same sword. Just putting it in the anvil boosts the cost to 23 levels, which seems pretty insane for a moot action. Putting one diamond in the right slot boosts the cost to 33, 2 diamonds makes a red cross over the arrow and 3 diamond fixes the sword to about 80-90% at a cost of 38 levels.
I have replicated this in both 1.4.2 and 1.4.4-pre release. It has happened before with the same symptoms and bows. It usually works out after many tries. I can't see any pattern when it works and when it doesn't. It almost seems random.
Attached screenshot is for the mentioned swords the time it did work. The exact same configuration that didn't work on previous tries.