It appears that when an illager raid party spawns when inside or near a village, the majority of villagers despawn. This has happened in two separate worlds, after an illager raid party spawned in close proximity or inside the village, all but one villager was left spawned in. They were not killed by a mob, as the night was skipped every day with a bed, and there were no nearby caves for them to come from.