[pull] master from BYVoid:master#159
Merged
Merged
Conversation
* Add mmap-backed zip OpenCC resources
## Resource archive
- Add //data:opencc_resources_zip.
- Produce bazel-bin/data/opencc-resources.zip.
- Include standard config files and cleaned text dictionaries.
- Keep entries flat so the archive can replace share/opencc.
## Packaging
- Derive text-backed configs at build time.
- Rewrite dictionary type ocd2 to text.
- Rewrite .ocd2 file references to .txt.
- Reuse //data/dictionary:text_dictionaries.
- Include generated dictionaries such as TSCharactersExt.
- Strip blank lines and comment lines from text dictionaries.
- Write ZIP_STORED entries for dependency-free runtime reads.
- Stamp zip entries with the build-time local timestamp.
## Runtime loading
- Extend ResourceProvider with GetResource().
- Keep Resolve() for existing path-based loading.
- Add ZipResourceProvider for stored zip archives.
- mmap the zip once on POSIX platforms.
- Index the central directory and return entry memory views.
- Avoid extracting temporary files.
- On Windows, read the zip through UTF-8 to wide paths.
## Dictionary parsing
- Add Lexicon buffer parsing.
- Let TextDict load directly from ResourceProvider data.
- Preserve existing FILE*-based loaders for path resources.
- Preserve binary dictionary loading through existing paths.
## Entry points
- Add hidden native CLI flag --resource-zip <file>.
- Add Node binding support for resource zip loading.
- Add OpenCC({ resourceZip }) support.
- Add hidden npm CLI parsing for --resource-zip.
- Update TypeScript declarations.
## Tests
- Cover ZipResourceProvider in ConfigTest.
- Add a Bazel CLI test using opencc-resources.zip.
- The CLI test invokes command_line without resource --path args.
## Verification
- bazel test //src:config_test //src:text_dict_test
- bazel test //test:command_line_converter_test
- bazel build //src:opencc_lib
- bazel build //src/tools:command_line //data:opencc_resources_zip
- bazel build //node:opencc_node
- Native CLI smoke test for hidden --resource-zip
- Native and npm help output checks
- node --check node/cli.js node/opencc.js
- Checked opencc-resources.zip uses stored entries only
* Address resource zip review feedback
- Replace the zip provider raw internal pointer with a proper PIMPL.
- Disable accidental copies of zip-backed resource state.
- Cache text resources after building the final Marisa dictionary.
- Keep config file fallback behavior explicit when zip lookup misses.
- Allow the resource zip script to write to the current directory.
* Fix Windows resource provider build
- Include <cstdint> before using fixed-width integer types.
- Keep the fix isolated to the zip resource implementation.
* Fix resource provider edge cases
- Preserve provider-loaded config directories for relative plugin resources.
- Reject absolute resource names before zip basename fallback.
- Include file freshness in filesystem and zip resource cache keys.
- Add regression tests for absolute config paths and cache keys.
* Fix Windows config test path comparison
- Normalize path separators before comparing resource names.
- Build the legacy cache key check from the actual resource name.
* Fix Windows Unicode resource loading
- Read filesystem provider resources through wide-path file IO on Windows.
- Keep the POSIX provider path on the existing ifstream implementation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )