Steps to reproduce:

• Place jukebox
• Place record in jukebox
• Close world
• Reopen world

Result: The music will keep playing, no matter if the record is removed, the block is broken, the game is paused, the world is closed again, etc.

Going away does actually make the sound quieter/makes it stop.