If you build something in the air that casts a shadow on the terrain below, if it is even slightly large, and you leave the server render distance of that structure and come back, when you first load it you will get a huge frame drop that varies on the type of structure you built.
(I had all maxed out graphics settings, but the same thing happens with all mined out)
1.) Download example server world with default settings.
2.) Run the server.
3.) Join the server.
4.) Walk past the sign in the world download, if no frame drop, walk back in front of the sign and redo it again. (unloads chunks)
Enjoy frame drop.
No frame drop, or at least a consistent frame drop on both singleplayer and multiplayer.