I strongly suspect that this is a duplicate, however I have not found this bug reported through any searches.

Ender chests are the only way to access a player's "ender chest inventory" stored in their player data. As the player can access all other inventories, it makes sense that the player should be able to access ender chests.

Either
a) Add functionality to ender chests for spectators
b) Add a separate way to access the ender chest inventory (perhaps through the regular inventory screen)

Option b would have the advantage of allowing for spectators to view any player's ender chest inventory.