When a Nether portal is not in the standard shape in the overworld, it will not properly link to the portal located in the Nether.

Entering the custom portal will send the player to the correct portal in the Nether; however, going through the portal in the Nether will spawn a new portal at the player's spawn much more often than returning the player through the custom portal.