Reproduction steps (example)

1. Create a villager with custom trades via this command:

   /summon villager ~1 ~ ~ {NoAI:1b,Offers:{Recipes:[{buy:{id:"paper",Count:1b},maxUses:9999999,sell:{id:"stone",Count:64b},rewardExp:false}]}}
   

   (The important part is the large stack size of the 'sell' item. But any sell item with count > 1 has this problem potentially.)
   Note: The villager has NoAI:1b to prevent MC-164353 from happening.

2. Give yourself 1x paper and 63x stone.

   /give @s paper
   /give @s stone 63
   

3. Insert the 1x paper into the buy slot of the trading inventory.
   → It unlocks the trade (it puts 64x stone into the result slot).
4. Double click the 63x stone stack inside your player inventory.
   → Minecraft will try to collect stone items to the cursor. Regardless of the cursor having only space for at most 1 more item of stone on the cursor, it picks up and merges with the 64 stone from the trading result slot, resulting in a single stack of 64x stone on the cursor and the 1x paper gone due to the trade. So the player has now effectively traded his 1x paper worth of 64x stone for only 1x stone.

Possible solutions

• Don't allow collecting from the trading result slot to the cursor at all via double click. (Side fact: This behaves rather weird for the trading inventory view anyways in some situations: Due to first checking and merging with non-full stacks and then merging with full stacks afterwards, depending on the stack size of the result item and the player's other items inside the inventory, this sometimes results in the trade being triggered once in some situations, and twice in other situations.)
• Only allow picking from the result slot, if the cursor has enough space left to hold all the traded items.
• Handle the excess items in some other way (ex. dropping them).