Search results

Support for python 3.13 and 3.14

Status: Open.

#344 In OpenNMT/Tokenizer;

· racinmat opened on Mar 26, 2026

ICU_INCLUDE_DIRS for cli projects

Status: Open.

#331 In OpenNMT/Tokenizer;

· emabiz opened on Jul 15, 2024

How to return token ranges when tokenizing text?

Status: Open.

#328 In OpenNMT/Tokenizer;

· anderleich opened on Apr 16, 2024

tokenize by script boundaries - only

Status: Open.

#327 In OpenNMT/Tokenizer;

· mediabuff opened on Mar 8, 2024

SentencePiece 0.1.97 changed API to take std::vector<string_view> "everywhere", breaking SetVocabulary here

Status: Open.

#323 In OpenNMT/Tokenizer;

· Micket opened on Mar 16, 2023

space/none mode potentiel issue with case_markup

enhancement

Status: Open.

#176 In OpenNMT/Tokenizer;

· Zenglinxiao opened on Oct 9, 2020