When holding an item providing specific attributes, all attributes work as intended. When holding that item and pressing the key to switch to your offhand, all attributes stay and work as intended. However, only when switching the item from your offhand to your mainhand do the attributes not apply until the item is re-selected once again.

 

Recreation steps:

/give @s stone{AttributeModifiers:[

{AttributeName:"generic.movement_speed",Amount:1,Operation:0,UUID:[I;-120711,2288,174939,-4576],Name:"generic.movement_speed"}

]} 1

Hold item.

Press your "Swap Item With Offhand" key.

Notice that attributes are still applied.

Press your "Swap Item With Offhand" key once again.

Notice that attributes are no longer applied, until the item is re-selected.