You can take away experience levels from players by using the command
/xp -123L Playername
However, you can't take away single experience points by using:
/xp -123 Playername
The following error message will be displayed in chat:
Usage: Cannot give players negative experience points
I expected it to work because it also works with experience levels.