It seems that phantoms aren't affected by the follow_range attribute. I spawned an unmodified phantom and looked at its attributes, and follow_range is one of the ones that it comes with, however if you set the value of the attribute to zero, it will still attack players.

Steps to recreate:

Run command: /summon phantom ~ ~ ~ {Attributes:[

]}

Go to Survival or Adventure mode

The phantom will still attack you.