refactor: explicit charset validation functions for JSON and URL-encoded payloads

[Phillip9587]

This PR improves the handling and validation of character encodings for the JSON and URL-encoded parsers, ensuring compliance with relevant standards and providing more robust, maintainable code.

The main change is extracting the previously inlined charset validation functions into utility functions. I also included tests for these functions.

isValidJsonCharset

• Previously only checked if the charset starts with utf-
• Now checks if the charset is utf-8 , utf-16 or utf-32 case-insensitive according to RFC 7159 Section 8.1 (Stricter, Maybe semver major)
• RFC 7159 was superseded by RFC 8259 which is even stricter and defines:

JSON text exchanged between systems that are not part of a closed
ecosystem MUST be encoded using UTF-8 [RFC3629].
Previous specifications of JSON have not required the use of UTF-8
when transmitting JSON text. However, the vast majority of JSON-
based software implementations have chosen to use the UTF-8 encoding,
to the extent that it is the only encoding that achieves
interoperability.

isValidUrlencodedCharset

• Previously checked case-senstive for utf-8 and iso-8859-1
• Now checks case-insensitive the charset is already lower-cased in the getCharset() helper function

Edit 21.01.2025: I changed the PR so that the JSON Parser now is RFC 8259 compatible and ONLY allows utf-8.

[bjohansebas]

This is definitely a breaking change. For example, if I currently provide charset=utf-16be for JSONs, it would be decoded correctly and no error would be thrown, with your current PR this would be considered invalid, which makes me wonder whether the specification takes UTF-16BE or UTF-16LE encoding into account.

Additionally, how does this behave with iconv-lite? Given that iconv-lite correctly supports UTF-16BE and UTF-16LE, it should be able to decode the JSONs correctly.

My initial comments, i’ll continue reviewing these changes.

[ctcpip]

why wouldn't we just limit to UTF-8 at this point?

[Phillip9587]

@ctcpip I orgininally intented to keep this backward compatible but if we target a new major we should definitly limit it to utf-8.

[ctcpip]

it's already not backward-compatible, so my suggestion is to just adhere to the latest JSON RFC and only allow utf-8