Well, I am playing on my friend's server and I wanted to combine 2 diamong swords. One of them has been repaired, so they are both 100% new, kinda.

1st sword: Sharpness III, Knockback II, Looting II
2nd sword (repaired): Sharpness III, Fire Aspect I, Knockback II

I put 1st sword in the 1st (left) anvil slot, then I put 2nd sword in the 2nd (right) slot then there appears a message saying "Enchantment Cost: 39" for a moment and then disappears. (pic 1)

Then I tried following:
I first put the 2nd sword in the 2nd slot, then I put the 1st sword in the first slot. Message "Enchantment Cost: 39" appears again, but it doesn't disappear. (pic 2)

I thought cool, anvil is buggy, but I can still combine them, so I went to fight mobs to get lvl 39. When I finally got it, I went to the anvil:

2nd sword in 2nd slot, 1st sword in 1st slot, clicking on the new sword, trying to pick it up, but the message "Enchantment Cost: 39" disappears again, leaving me with the both swords and LVL=0! First I was like "oh shi~!" and was about to panic, so I disconnected from the server. Then after a while I connected again to discover that my exp is still there. I tried it a couple more times and could always reproduce it.

I hope my report helps to improve the great game.
Cheers

UPD: I asked my friend to send me the world we were playing on, so I could test the anvil in single player mode and combining worked out just fine. So the anvil seems to be buggy in MP mode only.