Whenever I try and join my friend on Nintendo switch and i'm on Xbox it won't let me at times. Sometimes it takes a long time on the loading screen "generating world" and "locating sever" and it does that for a long time in other words like more than 5 mins.

At other times it just says connecting to multiplayer game for like 2 mins and then it says "connection timed out" after that at.

there is a random unpredictable chance that it will let me like after 3 - 6 tries i join but then the game is unstable and chunks won't load in.

The funny thing is is that it's not either one of our connections as I can join other player fine and servers and same with my friend on Nintendo switch can join other players on severs.

We have double checked that we are playing on the same version (v1.16.1) and our internet connections are good. This also happens vice versa where my friend cannot join me.
Also both of our accounts do not have parental controls.