Edit: After further testing, this bug seems to happen when powered redsrone dust is in front of the observer face. But not with other components like droppers.
Sometimes observers activate when unloaded chunks are reloaded. However, it is random if the player is still in the world and just left the chunks.

How to Reproduce:

-place any redstone component in front of an observer

(Components that visually show activation are recommended; pistons, lamps ,etc.)

-exit the world or go far away  from the observer

-re-enter that world

You'll notice that the redstone component will activate. This should not happen as it makes redstone contraptions with observers unreliable.