On the Profile settings tab there is a button named "Manage who can send you invites" that links to a forum post on xbox.uservoice.com that was used to tally votes for a suggestion to allow blocking invites from non-friends. This post was closed in July 2016 when the suggestion was implemented. The button should be changed to take you to the page account.xbox.com/en-US/Settings instead.