The Bug

Although players remove items via left-clicking, it shouldn't be thought as damaging the item frames and cancel it.

How to Reproduce

1. Place a item frame.
2. Insert a item into that item frame.
3. Change the invulnerable field to be true:

   /data modify entity <item_frame> Invulnerable set value 1b
   

4. Try to remove the item.

It doesn't affect insertion and rotation.

Code Analysis

This was due to the lack of further detection (decompiled 1.20.2 using vanilla mapping):

public boolean hurt(DamageSource $$0, float $$1) {
    ...
    if (isInvulnerableTo($$0)) return false; 
    ...
}