Quite simple, the check for change in damage is negative (oldDamage - newDamage). Just need to inverse that operation (newDamage - oldDamage).

Example advancement, placed in world/data/advancements/custom/delta.json, showing that losing 5+ durability requires checking for maximum of -5 rather than minimum of 5:

{
    "criteria": {
        "findme": {
            "trigger": "minecraft:item_durability_changed",
            "conditions": {
                "delta": {
                    "max": -5
                }
            }
        }
    }
}

And the following commands to test it at will:

/advancement test @p custom:delta
/advancement revoke @p custom:delta

A fishing rod pulling a mob will cause it to lose more than 5 durability, while other uses of a fishing rod causes it to lose less than that.