I tried to make the totem invisible when put in your offhand but it wouldn't work, so I tried making it invisible in your main hand and it was invisible in both. Haven't checked if it is in other versions.