Items cannot be removed from Item Frames by punching if the gamerule doEntityDrops is set to false. Steps to reproduce:

1. Create a world.
2. Place an item frame and an item into the item frame.
3. Punch it and notice the item is removed.
4. Do /gamerule doEntityDrops false
5. Repeat steps 2 and 3 but notice that the item cannot be removed. However, the item popping sound can still be heard.