At night, once villagers are in bed. If anything touches the bed, villagers fly away. In the morning, they warp back. I made a video of some of the events.  

https://youtu.be/8QixN0_0gHw

I also noticed that sometimes they will freeze in a position. Sometimes you can not touch them and other times you can. Same with iron golems.