My friend and I were playing on our hardcore* world and we were towering up in the nether with some netherack, when all of the sudden I lost half a heart. I didn't think much of it as it seemed to be a common glitch of getting half a heart off for no reason. But I towered about ten blocks higher and suddenly my totem got destroyed. My friend also lost half a heart, but since they didn't tower as high as I did, he didn't lost his totem as well. There were no aggressive mobs attacking us, no ghasts or piglins, it was all just the game. We created a new world to test it out as well, and the same thing happened around the level y=60-90.

*not a real hardcore world but we "lose" (stop playing on it) the world when we die.

Edit: we were both wearing full netherite except for gold helmets