Skip to content

Commit acb947d

Browse files
diliopmeta-codesync[bot]
authored andcommitted
remove ALL tokenizers multiversions & upgrade the single remaining version to 0.22.2
Summary: Remove all versions of multi-versioned tokenizer (`[tokenizer-20, tokenizer-21, tokenizer-22]`) and upgrade default `tokenizer` to `0.22.2` Reviewed By: dtolnay Differential Revision: D89590447 fbshipit-source-id: 9905594de5a3dd7203380453104b6f9a5df41464
1 parent 16d8633 commit acb947d

File tree

2 files changed

+0
-2
lines changed

2 files changed

+0
-2
lines changed

eden/fs/Cargo.toml

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,6 @@ riscv-pac = { git = "https://github.com/jsgf/riscv.git", rev = "90cfc5f2404e39e0
3434
riscv-rt = { git = "https://github.com/jsgf/riscv.git", rev = "90cfc5f2404e39e085e12845944d5bbbc869c290" }
3535
sqlx = { git = "https://github.com/launchbadge/sqlx", rev = "24317d5eab40fbc33caf1142946e2f39caad73ea" }
3636
stabby = { git = "https://github.com/nitinsingh-meta/stabby", rev = "304c895c34e502fc635b85df6d4f22f746747ed4" }
37-
tokenizers-21 = { package = "tokenizers", git = "https://github.com/huggingface/tokenizers", rev = "133db48d4f272070076fbc7a649c9e8aa8cd0646" }
3837
vm-allocator = { git = "https://github.com/rust-vmm/vm-allocator", rev = "5fb545cc6e41fdc9f27d48d97ed26274a35bc6c6" }
3938
warp = { git = "https://github.com/jdthomas/warp.git", rev = "c794232d182abaa3add8184e7d661e084cc516c5" }
4039

eden/mononoke/Cargo.toml

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,6 @@ riscv-pac = { git = "https://github.com/jsgf/riscv.git", rev = "90cfc5f2404e39e0
3434
riscv-rt = { git = "https://github.com/jsgf/riscv.git", rev = "90cfc5f2404e39e085e12845944d5bbbc869c290" }
3535
sqlx = { git = "https://github.com/launchbadge/sqlx", rev = "24317d5eab40fbc33caf1142946e2f39caad73ea" }
3636
stabby = { git = "https://github.com/nitinsingh-meta/stabby", rev = "304c895c34e502fc635b85df6d4f22f746747ed4" }
37-
tokenizers-21 = { package = "tokenizers", git = "https://github.com/huggingface/tokenizers", rev = "133db48d4f272070076fbc7a649c9e8aa8cd0646" }
3837
vm-allocator = { git = "https://github.com/rust-vmm/vm-allocator", rev = "5fb545cc6e41fdc9f27d48d97ed26274a35bc6c6" }
3938
warp = { git = "https://github.com/jdthomas/warp.git", rev = "c794232d182abaa3add8184e7d661e084cc516c5" }
4039

0 commit comments

Comments
 (0)