Phantoms are not doing enough damage.

In Easy mode, a Phantom hit does 2 damage (1 heart) instead of 4 damage (2 hearts).

In Normal mode, a Phantom hit does 2 damage (1 heart) instead of 6 damage (3 hearts).

In Hard mode, a Phantom hit does 3 damage (1,5 hearts) instead of 9 damage (4,5 hearts).

In these versions, they actually do exactly the same damage as spiders. This is problematic as this makes phantoms far less of a threat than normally intended.