Baby villagers seem to be able to walk through blocks. For example, you can wall off a house will cobblestone and some how the baby village will enter or leave said house. Same goes for doors of any kind and gates.

This makes keeping baby villagers inside the village area or houses extremely difficult.