When retrieving an item from a flower pot while holding something else in your main hand, and not holding anything in your offhand, the retrieved item will end up in your off hand.

The expected behavior would be for the item to end up in the first available slot in the inventory, or stack with existing items in the inventory of the same type.