After removing a resource pack, some of its textures will persist in the texture atlas. Furthermore if the atlas is resized the game will duplicate vanilla textures once the resourcepack is removed to prevent the atlas from shrinking back down.

You can observe this by following the steps below

• Save the atlas by pressing F3 + S
• Load this texture pack VanillaTweaks_Bug.zip
• Unload this pack (go back to vanilla textures)
• Save the atlas again and view it

You will notice the following:

• Non vanilla textures are still present in the atlas
• Various vanilla textures have been duplicated

Before:

After:

Why this matters:
I think it's a reasonable expectation that when you remove a texture pack the game will go back to its default state. Because of this bug I've had a much harder time testing for an unrelated issue caused by a resized block atlas because I didn't expect the atlas to remain modified after a texture pack was removed.