CSV doesn't contain numbers at all; it contains strings. How those should be parsed into numbers is not part of the format, unless by CSV you additionally mean "to be imported by Excel". Which is certainly a common use case but not always the case.
The distinction between file format and payload format is perfectly normal and not at all useless.
All file formats have limits for how much they define types for the payload. Most have very limited types. JSON for example knows objects, arrays, strings, numbers, booleans and null. It's not that big a difference to have a format that has only strings.
