Can't join friends worlds on Xbox One

When you try to join a friends game if its public or invite only, it will load for a minute then will say "Connection to the server was lost". I restarted my console, tryed it on different accouts, and had my friends do it and for all of them it said that exact same thing.