I've observed that in 1.16.3, even in a completely vanilla environment on a singleplayer world, when you use the scroll wheel to move your cursor to an item in the hotbar which has an attack cooldown, the cooldown bar will "hitch" slightly, causing the actual wait before being able to use the item to be extended.

This is exacerbated on servers with less-ideal ping, which is how I noticed the issue, with my sword sometimes taking 1/2 a second longer to be readied since it would get half way through the initial cooldown, then restart, with the sword getting lowered then raised again as if I had switched from it to another item and back, when I hadn't. This doesn't seem to occur when using number keys to directly select the slot, oddly.

Attached is a video I obtained in a singleplayer world, showing the difference between scrolling to an item that has no cooldown, and one that does, showing the "hitch" effect at its weakest, though as stated it does get signficantly worse on servers that have higher ping.