Damage taken can reset/slow velocity in some cases.
I did a quick test where I swam up lava, and with fire resistance, it took about 4 seconds. Without fire resistance, where I took damage, it took about 5.5 seconds to reach the top.
It is also slightly noticable when jumping with wither/poison/fire, your jump can inconsistently be shorter than normal.