What I expected to happen was that the Redstone torch will break as soon as the block next to it was broken

What actually happened was that the Redstone Torch still was in the air.

Steps to Reproduce:
1. Place a block next to where ever you want your floating redstone torch
2. Place a block beneath it
3. Place a block right below where you want your torch, so now with the three blocks, it looks like an L.
4. You don't need the corner, so you can lop it off if you want, like I did in the screenshot
5. Place the Torch on top of the block where you want it, then remove the block beneath it.
6. SUCESS!! (do not remove the block next to it or else the redstone torch will drop like I did)