If there is a block above a villager's workstation, such as a solid block, or even a fence or lantern or other block (notably not trap doors), they will very rarely actually work at their workstation. Putting the workstation next to a wall or in a wall reduces the amount it is worked at even further, a workstation surrounded on three of four sides with a lantern or other block on top does not seem to ever get worked at.

Also, villagers with blocks on their workstation tend to try and pathfind above the workstation, and will spend time on floors above the workstation if they can, villagers with no block above their workstation will tend to stay near their workstation (during working hours). This may be the cause (villagers aren't near their workstations), or could be a separate or related issue.

Interestingly, I also noticed villagers trying to jump onto a block while in only a two high area during testing.

This was noticed on a survival server, and then reproduced in creative.