cpp.md

SentencePiece C++ API and CMake Integration

Integrating with CMake (version >= 0.2.3)

SentencePiece provides official CMake targets (sentencepiece::sentencepiece and sentencepiece::sentencepiece_train). Header search paths, dependencies, and required C++ standards are automatically propagated to your targets.

1. Using find_package (Installed Package)

When SentencePiece is installed on your system or in a custom prefix:

cmake_minimum_required(VERSION 3.15)
project(my_project CXX)

# Find installed sentencepiece package
find_package(sentencepiece CONFIG REQUIRED)

add_executable(my_app main.cc)

# Link inference library
target_link_libraries(my_app PRIVATE sentencepiece::sentencepiece)

# Or link trainer library (if using SentencePieceTrainer)
# target_link_libraries(my_app PRIVATE sentencepiece::sentencepiece_train)

If SentencePiece is installed in a custom directory, pass -DCMAKE_PREFIX_PATH:

cmake -B build -DCMAKE_PREFIX_PATH=/path/to/sentencepiece_install

2. Using FetchContent (Direct from Git)

To build SentencePiece automatically as part of your project without manual installation:

cmake_minimum_required(VERSION 3.15)
project(my_project CXX)

include(FetchContent)
FetchContent_Declare(
  sentencepiece
  GIT_REPOSITORY https://github.com/google/sentencepiece.git
  GIT_TAG        v0.2.3
)
FetchContent_MakeAvailable(sentencepiece)

add_executable(my_app main.cc)
target_link_libraries(my_app PRIVATE sentencepiece::sentencepiece)

3. Using add_subdirectory

If you include SentencePiece as a git submodule or in-tree vendored directory (e.g. third_party/sentencepiece):

add_subdirectory(third_party/sentencepiece)

add_executable(my_app main.cc)
target_link_libraries(my_app PRIVATE sentencepiece::sentencepiece)

---

Load SentencePiece model

To start working with the SentencePiece model, include the sentencepiece_processor.h header file. Instantiate the sentencepiece::SentencePieceProcessor class and call the Load method to load the model using a file path or std::istream.

#include <sentencepiece_processor.h>

sentencepiece::SentencePieceProcessor processor;
const auto status = processor.Load("//path/to/model.model");
if (!status.ok()) {
   std::cerr << status.ToString() << std::endl;
   // error
}

// You can also load a serialized model from std::string.
// const std::string str = // Load blob contents from a file.
// auto status = processor.LoadFromSerializedProto(str);

Tokenize text (preprocessing)

Call the SentencePieceProcessor::Encode method to tokenize text.

std::vector<std::string> pieces;
processor.Encode("This is a test.", &pieces);
for (const std::string &token : pieces) {
  std::cout << token << std::endl;
}

You will obtain the sequence of vocabulary IDs as follows:

std::vector<int> ids;
processor.Encode("This is a test.", &ids);
for (const int id : ids) {
  std::cout << id << std::endl;
}

Detokenize text (postprocessing)

Call the SentencePieceProcessor::Decode method to detokenize a sequence of pieces or IDs into text. In general, it is guaranteed that the detokenization is an inverse operation of Encode, i.e., Decode(Encode(Normalize(input))) == Normalize(input).

std::vector<std::string> pieces = { "▁This", "▁is", "▁a", "▁", "te", "st", "." };   // sequence of pieces
std::string text;
processor.Decode(pieces, &text);
std::cout << text << std::endl;

std::vector<int> ids = { 451, 26, 20, 3, 158, 128, 12  };   // sequence of ids
processor.Decode(ids, &text);
std::cout << text << std::endl;

Sampling (subword regularization)

Call the SentencePieceProcessor::SampleEncode method to sample one segmentation.

std::vector<std::string> pieces;
processor.SampleEncode("This is a test.", &pieces, -1, 0.2);

std::vector<int> ids;
processor.SampleEncode("This is a test.", &ids, -1, 0.2);

SampleEncode has two sampling parameters, nbest_size and alpha, which correspond to l and alpha in the original paper. When nbest_size is -1, one segmentation is sampled from all hypotheses with forward-filtering and backward sampling algorithm.

Training

Call the SentencePieceTrainer::Train function to train a SentencePiece model.

You can pass training parameters as a single command-line-like string:

#include <sentencepiece_trainer.h>

sentencepiece::SentencePieceTrainer::Train("--input=test/botchan.txt --model_prefix=m --vocab_size=1000");

Alternatively, you can pass parameters as a std::unordered_map<std::string, std::string>:

#include <sentencepiece_trainer.h>

sentencepiece::SentencePieceTrainer::Train({
  {"input", "test/botchan.txt"},
  {"model_prefix", "m"},
  {"vocab_size", "1000"}
});

Vocabulary management

Use the following methods to convert between IDs and pieces.

processor.GetPieceSize();   // returns the vocabulary size.
processor.PieceToId("foo");  // returns the vocab id of "foo"
processor.IdToPiece(10);     // returns the string representation of id 10.
processor.IsUnknown(0);      // returns true if the given id is an unknown token. e.g., <unk>
processor.IsControl(10);     // returns true if the given id is a control token. e.g., <s>, </s>