I've tested with and without other villagers around as well as the waterlogged spruce stair (as seen in picture) and no matter what, they will eventually just stare at a crop they want to harvest all day until they go to bed.