It's a bit difficult to describe, but the problem comes from skeletons shooting arrows at eachother, and having a mob farm so that more skeletons are pushed to where they're slaughtered. Sometimes they get sight of a player, and so shoot at the player. Missing the player of course, and start shooting at eachother. Then as they're moved along by water, they continue to shoot at eachother, but not hitting each other, instead embedding the arrows into walls.

This happened long enough to result with over 2000 entities in a single chunk. Anyone logging in in this chunk results in crashing the server as the server tries to do ... something with 2000 arrows stuck into walls.

Using a tool to delete the entities: https://github.com/Fenixin/Minecraft-Region-Fixer provided us with a workaround, but it did require a bit of server down time.
It'd be nice if arrows wouldn't spawn too many, at least from skeletons within a certain chunk. Or they despawn relatively quickly, since you can't collect them from the mobs anyway.