in the modifier you need to set 2 backslashes like \ \ n(without the spaces, it does a line break here without), or else it won't work, but when using it, then the game adds a backslash in the books data for every backslash, making \ \ n into \\\\n in the nbt

book made with the following command:

/give @p written_book[written_book_content={title:"",author:"",pages:['[{"text":"hello\\n"},"no"]']}] 1

book made with /item modify command:

/give @p written_book
/item modify entity @s weapon.mainhand {"function":"minecraft:set_written_book_pages","pages":[{"text":"hello\\n","extra":[{"text":"no"}]}],"mode":"append"}