I gave the allay a diamond block, then took it out of its hands, then hit it, then gave it back and it was off-center

To reproduce:

/summon minecraft:allay ~ ~ ~ {LeftHanded:1b,NoAI:1b,HandItems:[{id:"minecraft:magenta_concrete",Count:1b}]}