I was clearing out a gravel patch with my efficiency 4 netherite shovel, and upon hearing some broken stone sounds I saw that I had 2 stone in my inventory.

In the screenshots I have provided, the only item with silk touch on it was the shovel, and I can guarantee all I brought with me were my tools, food, and torches. I didn't switch to my pickaxe at all, not even for the briefest second possible.

I could not replicate this in my very brief testing on a creative world, however I was waving my mouse around very vigorously and randomly when it broke the stone blocks.

I am playing on a realm by myself and didn't notice any lag or anything else unusual on the servers end.