I've had a realm for almost a year now and constantly have to re-whitelist my friends because the game constantly removes them. I am unsure if this is a realms server issue or an internal game issue, but it has been happening for a few weeks now.