If you switch your item that you're holding while breaking a block, the block breaking restarts. But if you're punching with your fist, and you switch to another empty slot, it goes as normal.