In item NBT data, CustomModelData is an int. Even if commands are used to set it to a different type, it will be read as a 32-bit integer if possible, and ignored if not. This is correct and expected.

32-bit signed integers have a range from -2,147,483,648 to 2,147,483,647. We would expect all of these values to be available to us for model overriding. However, this is not the case. When checking the CustomModelData against an item model's "custom_model_data" predicate, the 32-bit integer is converted to a single-precision float. Single precision floats can only exactly represent integers between -16777216 and 16777216; values beyond that range are rounded to the nearest multiple of two, then the nearest multiple of four, and so on as they grow.

 More info on the precision limits of 32-bit floats.

Long story short, only 4% of the possible CustomModelData values can actually be displayed. To fix this, "custom_model_data" predicates would have to be handled differently than other predicates, so that no floating point conversion was performed. Alternatively, all predicate calculations could be switched over to using double-precision floats, which can represent all possible 32-bit integer values.