Phantoms never attack players again after their first swoop at them, even if there are no cats or ocelots nearby. In fact, whenever they try to fly down to attack again, they just go back up.