When creating a new world, Minecraft's VRAM usage starts at about 1GB. When exploring chunks, the usage continues to increase until your GPU runs out of VRAM, which can happen really quickly with higher render distances. When VRAM runs out, it starts to spill over into system RAM which makes the game stutter a lot and even crash when the system RAM also runs out. With a 4GB GPU the VRAM can max out already after 15 minutes, and so this is very impacting for a huge amount of people.

I haven't tested this on versions other than 1.19.2 and 1.20.1, but it already means this is a slightly older bug, which for some reason still isn't in the bug tracker even though it's really gameplay impacting.

I already talked about this with OptiFine's author, they explain the problem a little more in detail here in a Github thread.