I have two Macs running MC 1.11.2, macOS 10.12.3, and the latest Java. In a single player game on one computer, I selected "Open to LAN," but the game didn't show up on the other. I tried refresh and restarting the game. One is at IP address 192.168.1.104, and the other is 192.168.1.111, on the same network, connected to the same wireless access point. On both machines, the firewall is OFF.

I've written UDP broadcast-based discovery services before, in both C and Java. It's not even slightly difficult. Nevertheless, complaints about Minecraft "Open to LAN" not working go back years, talking to people on IRC, they tell me that it simply doesn't work for them on Mac. Seriously, if you want me to write you some discovery service code in Java to do this, I'd be happy to do it for you. (Opening the firewall port in Windows would have to be handled separately.)