If you set render distance to 4 chunks,no hostile mobs will spawn in overworld,if you set it to 6 chunks,hostile mobs will rarely spawn,if you set it to 8 chunks there will be over 10-20 hostile mobs in 100*90*100 area,and if you make it 10 chunks,hostile mobs will spawn too much.

In the nether and the end,hostile mobs spawn even if render distance set to 4 chunks because I found 61 zombie pigmen when I used kill command and render distance was 6 chunks.
same thing happens with endermen.their spawning cap does not count on render distance.