Pillagers spawn too often. In one of the screenshoots I took I already had 2 ominous banners on my slot bar because I already had killed another patrol of them nearby, only to find more next to the water. I took the pictures without fullscreen so that you can see the time on the bottom right of my screen. As far as I know they only spawn every 100 minutes meaning every 5 in-game days.