C: Use hb_string_T for token_T.value

[timkaechele]

This pull request changes the token_T.value to use hb_string_T and adapts the call sites. Token values become non-owning slices into the source buffer, eliminating per-token strdup calls during lexing and parsing. As a result, token_copy becomes a shallow struct copy and token_free no longer frees the value.

Two constants, HB_STRING_EMPTY and HB_STRING_NULL, are introduced to distinguish a valid empty string from the absence of a value. Call sites that previously checked token->value == NULL now use hb_string_is_null, while sites that checked for empty content use hb_string_is_empty.

The public API is simplified by removing herb_lex_file and herb_lex_to_buffer.

herb_lex_file had an inconsistent lifetime contract, it read a file, lexed it, then freed the source, leaving tokens with dangling pointers. Callers should now read the file themselves and pass the source to herb_lex. herb_lex_to_buffer was only used by the C-CLI and C tests, so it moves to an internal lex_helpers.h header.

Since token values are non-owning, callers must keep the source string alive for as long as tokens or AST nodes are in use. All existing bindings already satisfy this naturally, they hold a reference to the source (Ruby string, JNI string, std::string, etc.) for the duration of the operation and convert to native objects before releasing it.

Comparison

make bench_allocs

Before (current main 2990a34cc77681bcd45bb21b4c4320065c5ea129)

=== Allocation Benchmark ===

[small] (35 bytes input)
  lex  small       allocs: 32      deallocs: 0       bytes_alloc: 691       tokens: 16
  parse small      allocs: 106     deallocs: 68      bytes_alloc: 2597

[medium] (650 bytes input)
  lex  medium      allocs: 420     deallocs: 0       bytes_alloc: 9260      tokens: 210
  parse medium     allocs: 1355    deallocs: 897     bytes_alloc: 33354

[large] (2878 bytes input)
  lex  large       allocs: 1384    deallocs: 0       bytes_alloc: 31250     tokens: 692
  parse large      allocs: 4394    deallocs: 2963    bytes_alloc: 110160

After (this pull request)

=== Allocation Benchmark ===

[small] (35 bytes input)
  lex  small       allocs: 16      deallocs: 0       bytes_alloc: 768       tokens: 16
  parse small      allocs: 59      deallocs: 34      bytes_alloc: 2844

[medium] (650 bytes input)
  lex  medium      allocs: 210     deallocs: 0       bytes_alloc: 10080     tokens: 210
  parse medium     allocs: 764     deallocs: 451     bytes_alloc: 36038

[large] (2878 bytes input)
  lex  large       allocs: 692     deallocs: 0       bytes_alloc: 33216     tokens: 692
  parse large      allocs: 2465    deallocs: 1491    bytes_alloc: 117003

Conclusion

This pull request does 50% less allocations while lexing and 40-50% less allocations while parsing, though overall, it has to slightly allocate more total bytes.