This was tested in a 22w13a multiplayer snapshot server with some friends to actually see what would happen to confirm if this is true.

Players can take the item of an Allay that doesn't belong to them, meaning you can effectively steal Allays from other players without their consent. This would be akin to being able to tame other player's already tamed wolves, and is inconsistent with how an Allay is only immune to damage from its owner. I imagine this was simply an oversight.

For reference in the attached image, the Allay named "Azalea" didn't belong to me.