Originally I was going to put this under the title of "Severe Frame Drops Using Frost Walker Enchantment" however a little bit of testing showed it was due to "blocks" I'll explain..
It seems to do with the block changes eg. from air to whatever block or in the case with frost walker from water to ice and then each stage of the block breaking.
 
Considering that placing a block at a time can drop your(my) frames by 20.. you can imagine when using frost-walker or frost-walker II which creates multiples of blocks at a time that then start changing from one state/block to another, that your frames can quickly drop to 1/0.

I can only imagine this is due to a lack of optimization but figured it's worth mentioning in the case of frost-walker which drops your frames below playable.