When you break blocks with your hand, the breaking animation won't show up and will not break the block. When you have any item or block in your hand then it works.