When breaking dirt blocks you hear the dirt breaking sound as well as the stone block breaking sound.

Try hitting at a smoothstone block without any tools, and then try the same with the dirt block. It then becomes much easier to hear.

Might be other issues too with breaking wood, where it gives the same "stone block" sound effect in the background.