When the phantoms spawn in my world all they do is fly around way up high and don't attack. They will always be above me but are passive. My settings are on normal and other mobs attack just not phantoms