this seems to happen when you have a lot of them in the same fenced of area. this applies to both fences and stone walls

To reproduce:
1. Create fenced off area
2. Fill it with farm animals, then wait a few minutes and they will be outside. best way to see this is dye sheep pink and watch around the pen.
3. farm animals should float outside fence, then teleport back in when you get close.