This bug has been in Minecraft for a good while. When starting a chain reaction with several hundreds of TNT blocks involved, the game will start to allocate massive amounts of memory until the virtual machine heap space (-Xmx) is fully exhausted, with the well-known "Out of memory" screen as a result.

Steps to reproduce: create a superflat world, preset "2;128x46", ignite any TNT block, wait a few seconds.

This is caused by the dynamic object pooling for vector and bounding box objects combined with very expensive explosion damage/propulsion calculations.

Each explosion potentially requires some thousand vectors and a few hundred bounding boxes, whose instances are all stored in a pool for the duration of the current tick.
Since the pool size isn't imitated in any way, it stores every single instance in its list. (Update: vector lists seem to be controlled in some way since 1.4.x, but it still doesn't stop them from collecting all objects in one single tick) So if many TNT explosions produce 100 million temporary objects during a tick, everything will be kept in memory and unreachable for the garbage collector until the next tick starts.

The solution is pretty simple: set a per-tick limit for the object pool list. If a certain limit is passed (I'd recommend 10-20k), return any additional objects always newly constructed so they can be garbage-collected once they're no longer being used. This also allows the use of fixed-size arrays, which could be a little faster than lists. The attachments below contain some example code that allows me to perform explosions like on a TNT superflat world as above with just 4 GB RAM without any known side effects. (of course, the enormous amount of entities and TNT block meshes will also produce out of memory errors at some point, but much later than in vanilla)

This obviously won't fix the FPS performance issues during mass explosions, but at least you'll be able to see the result after some time without having 32+ GB RAM.