When an item is created with the proper components (as per MC-269654) the "minecraft:set_damage" modifier function and "add": true causes strange effects

How to Reproduce:

1. Create an item like /give @s minecraft:iron_axe[minecraft:damage=0,minecraft:max_damage=20]
2. Run /item modify entity @s weapon.mainhand {"function": "minecraft:set_damage","damage": -1,"add": true} and see that it instantly sets damage to max damage.
3. Run /item modify entity @s weapon.mainhand {"function": "minecraft:set_damage","damage": 1,"add": true} and see that it suddenly resets damage entirely.