Edit: Replaced my crummy explanation with a better one, courtesy of Torabi.

This is a complicated issue, caused by a collision of multiple trading mechanics. The code doesn't really distinguish between the enchantments placed on the books – other than to set the price based on the level of the enchantment. So an offer of a book with a particular enchantment can be replaced with a book with any other enchantment, because all it checks when generating an offer and determining whether to add a new one or replace an existing one is the item ID.

The first component is MC-349: all potential offers for Enchanted Books are considered the same item, regardless of the enchantment or its level.
When generating a new offer, if the item matches an existing offer, it only replaces it if the price is better. If it's not better, nothing changes.
Enchanted books are priced differently than other items, based on enchantment level: (2 + level * 3 + random(0, level * 10 + 5)).

The result of these three components is that low-level books can easily replace higher-level books, but high-level books can never be priced low enough to ever replace some low-level offers. For example, the minimum cost for a level 5 enchantment is 17 emeralds. Once a librarian generates an offer for an enchanted book priced lower than that, they will never again offer a level 5 enchanted book. (Ranges for all levels, from the wiki: level 1: 5 – 19, level 2: 8 – 32, level 3: 11 – 45, level 4: 14 – 58, level 5: 17 – 71). Because of the random nature of generating offers, it's guaranteed that the more you trade with a librarian, the worse their offers will get. This is the exact opposite of how trading works for every other type of trade, including the librarian's offers for other types of items, in which it's guaranteed that offers will never get worse, and, if you trade with a villager enough, you will eventually unlock all their potential trades, at the lowest possible prices.

The enchanting offers provided by priests work entirely differently than enchanted books. The cost only ranges from 2-4, and is not related to the enchantment or its level. Offers can be replaced with a different enchantment, and that enchantment may be better or worse, but the full range (all enchantments that would cost 5-19 exp levels if you used a enchanting table) is always available for the offer generator to pick from, and nothing about the existing enchantment offer will ever exclude part of that range.

It is almost certain that the behavior of enchanted book offers is an oversight, not an intentional deviation.