If I'm in Survival Mode, holding a Spectral Arrow in my offhand slot, and drawing an arrow using a bow in my main hand, then I will see a single Spectral Arrow in my offhand slot but another arrow being drawn in a bow in my main hand. This doesn't make much sense - I'm holding two arrows at once! Plus, if I'd look in my inventory, then I wouldn't see another arrow - only the Spectral Arrow. How could I be firing two at once? Only when I'd shoot the Spectral Arrow, the Spectral Arrow in the offhand slot would disappear.

What should happen instead is that the Spectral Arrow should disappear when I'm drawing the bow, and reappear if I stop drawing the bow. If I shoot the Spectral Arrow in Survival/Adventure mode (since you have infinite arrows in Creative Mode and you can't see what's in your main hand or offhand in Spectator Mode), then it should remove 1 Spectral Arrow from my offhand slot. This should apply to every kind of arrow - default, tipped, and spectral.