Upgrading a world from 1.20.4, empty Attribute Modifiers was removed, I think it should be preserved, because empty Attribute Modifiers can cancel default attribute modifiers.
(Except armor due to MC-271826)

To reproduce:
1. In 1.20.4 get a empty attribute modifiers item with use

/give @s diamond_sword{AttributeModifiers:[]}

2. Upgrade the world
3. Check the item attribute modifiers