If you die while holding more than one totem as a stack, all totems in that stack will disappear, rather than one.

EDIT: i have realised that the issue is that they are not meant to be stacked, but they can be stacked through /give