Background

On macOS, each function key has a "system" functionality (usually indicated by the symbol on the key) and a "standard" functionality. By default, pressing a function key will perform the system functionality, and the user must hold down the Fn/Globe key while pressing the function key to access the standard functionality; there is a system setting to reverse this behavior. Additionally, system functions can be bound to a "standard" function key in System Settings > Keyboard > Keyboard Shortcuts, meaning apps like Minecraft cannot make use of these keys unless they are rebound. By default, F3 (Mission Control) and F11 (Show Desktop) are affected by this.

The issue

At some point in Minecraft's history, it was decided that when both the "system" and "standard" functionality for a function key are being used by macOS, the player should be able to hold down the control key to access any Minecraft-specific functionality for that key, in effect bypassing the keyboard shortcut. This seems to be hardcoded, and the Key Binds screen does not give any indication that it works this way, much less offer an option to disable or rebind it. For F11 especially, this is counterintuitive when using macOS fullscreen mode since macOS' Show Desktop functionality does not do anything if triggered from a full screen app; the key seemingly does not work for players who have not gone into System Settings and disabled or rebound Show Desktop.

TL;DR: The control key acts as a keyboard shortcut bypass, but this is not mentioned anywhere in the Key Binds screen and cannot be disabled or rebound.

Is a keyboard shortcut bypass even needed?

In terms of impact, this bug is similar to MC-147718 in that it's about a lack of communication and customization. However, when considering ways to fix this, it seems valid to ask whether a keyboard shortcut bypass is something users should actually be expected to use. In the case of the F3+X combos, a Mac user with default system settings and default Minecraft controls must press fn+ctrl+F3, then release fn+ctrl, then press X, then release F3+X. I'm sure there are people who do this, but frankly it's too cumbersome for the average player. If Mojang were simply to remove the current functionality whereby control is the KS bypass and instead prod users to set up their function keys correctly, that would arguably be the best solution for this issue, as well as the simplest.

Ok, so what might such a solution actually look like?

As previously stated, the hardcoded KS bypass functionality would be removed. Instead, in the Key Binds screen, Mac users would see a little exclamation point next to any control which is bound to a function key which is also a keyboard shortcut. Clicking this would open a new page that looks something like this:

Key Command Blocked by macOS Keyboard Shortcut

This key will not be able to perform its associated action because it is bound to a keyboard shortcut in macOS System Settings. To fix this, please choose a different key or go to System Settings > Keyboard > Keyboard Shortcuts and rebind the corresponding shortcut. [Take me there.]

(Back button)

This would be sufficient on its own, but would still leave some macOS users feeling like they're worse off than their Windows counterparts because they need to press the fn key. The way to remedy this it to put a different-colored exclamation point next to any control which is bound to a function key, regardless of whether there's a corresponding keyboard shortcut. Clicking this would open a different page:

Using Function Keys in Minecraft: Java Edition on macOS

On macOS, in order to use the function keys with third-party apps like Minecraft, you must either hold down the fn (Globe) key while pressing the key in question or configure the function keys to perform "standard" actions when pressed. To enable "standard" function keys system-wide, go to System Settings > Keyboard > Keyboard Shortcuts > Function Keys. [Take me there.] To enable "standard" function keys for just Minecraft, follow [these instructons] for how to install Fluor and set up "standard" function keys for Minecraft.

(Back button)

Ideally the second set of exclamation points wouldn't appear if standard function keys are already enabled for Minecraft, but I don't think Minecraft can reasonably be expected to know whether this is the case.

Are there any alternative solutions which preserve KS bypasses?

There are. The one I like the most, and the only one that I can think of without some inherent downside, would involve removing the hardcoded bypass and changing the Key Binds page to accept key combinations instead of only individual keystrokes, probably as part of a fix for MC-147718. This would allow for F3 and F11 to work when bound to ctrl+F3 and ctrl+F11 respectively (when the function keys are in standard mode). As part of this solution, the game would have to bypass all function keys when one or more modifier keys is held down when rebinding a control; I would think this would be technically feasible, given how the current system works.

This is a fine solution, but when compared to the one I described previously it would probably leave more users pressing an extra key when playing the game, thinking that there's no other solution if they want their screen brightness/system volume/etc broadly accessible. It would also be a bit trickier to implement, so I suspect Mojang may prefer the first solution.