This shouldn't even happen; I can't imagine this helping performance if it is an intended "optimization", it's literally just a few lerps to do a properly smooth transition. 

Sky colors just "pop" back or from normal when the weather changes