When the command:

/scoreboard objectives add power dummy

and the commands:

/scoreboard players add @e[score_power_min=1,score_power=1] power 1 {Inventory:[{id:minecraft:dirt,Slot:-106b}]}

and

/clear @p minecraft:dirt

are plugged into a repeating command block and following chain command block and a dirt item is selected, it cannot be placed in the offhand slot via the 'f' key. It can be dragged there, however. The item cannot by dropped using the 'q' key, and if it is dropped out of the inventory screen, it will appear as a normal item, but when it is picked up, the item will not show up in the inventory.