For me, it's very rare. A taiga village was generated and it is large but the villagers, three for be exactly and the same job, they are working on a invisible block.

I killed them and generate new villagers but no, they take de job from the invisible block (o a specific coordinate) because I tried destroying the block with tnt too or in creative, and the villager always goes to this coordinate to work or take the work...

My screenshots show the villager when interact with the job site block put his face down and it sounds... well I take ones at perfect moment.