So I noticed at my wither skeleton farm, that I saw a lot more wither skeleton skulls dropping than what ended up in the chest, I figured it was a visual glitch and ignored it. Then today when I was grinding I heard an item pickup noise. That's when it clicked in my mind. The dead wither skeletons have been dropping skulls, and the still living ones have been picking them back up, and since there is only a chance to get the stone swords that they are holding, that means that there is also only a chance to get the skull they are wearing, which means that I was losing many skulls when grinding.