You can take away experience levels from players by using the command
/xp -123L Playername

However, you can't take away single experience points by using:
/xp -123 Playername

The following error message will be displayed in chat:
Usage: Cannot give players negative experience points

I expected it to work because it also works with experience levels.