I know that custom Dimensions were marked as experimental in 1.16, so I am not sure if this a "bug" or "experimental feature that was never intended to work in 1.17." However, we can't be the only server to play with that new feature of 1.16. I figured I should post the bug/fix if anyone else has the same issue.

We had a custom dimension added from a datapack for a server event. Once we updated to 1.17 our Nether and End were no longer accessible. 

We eventually found the problem. In Level.dat under WorldGenSettings all of the dimensions were gone except the Overworld. It pruned not just our custom dimension, but the End and Nether as well.

We restored from backup, deleted the custom dimension from Level.dat, and the update went smoothly. Simple solution. Took a long time to solve. Could not find another entry on the bug tracker. Hope this helps someone.