Quite simple, the check for change in damage is negative (oldDamage - newDamage). Just need to inverse that operation (newDamage - oldDamage).
Example advancement, placed in world/data/advancements/custom/delta.json, showing that losing 5+ durability requires checking for maximum of -5 rather than minimum of 5:
And the following commands to test it at will:
A fishing rod pulling a mob will cause it to lose more than 5 durability, while other uses of a fishing rod causes it to lose less than that.