Langformers is a flexible and user-friendly library that unifies NLP pipelines for both Large Language Models (LLMs) and Masked Language Models (MLMs) into one simple API.

What makes Langformers special? Whether you're generating text, training classifiers, labelling data, embedding sentences, reranking sentences, or building a semantic search index... the API stays consistent:

from langformers import tasks

component = tasks.create_<something>(...)
component.<do_something>()

No need to juggle different frameworks — Langformers brings Hugging Face Transformers, Ollama, FAISS, ChromaDB, Pinecone, and more under one unified interface.

Use the same pattern everywhere:

tasks.create_generator(...)   # Chatting with LLMs
tasks.create_labeller(...)    # Data labelling using LLMs
tasks.create_embedder(...)    # Embeding Sentences
tasks.create_reranker(...)    # Reranking Sentences
tasks.create_classifier(...)  # Training a Text Classifier
tasks.create_tokenizer()      # Training a Custom Tokenizer
tasks.create_mlm(...)         # Pretraining an MLM
tasks.create_searcher(...)    # Vector Database search
tasks.create_mimicker(...)    # Knowledge Distillation
tasks.create_chunker(...)     # Chunking for LLMs

Langformers can be installed using pip.

pip install -U langformers

This installs the latest version with core dependencies.

Langformers includes optional integrations you can install depending on your use case:

• For FAISS support: pip install -U langformers[faiss]

• For ChromaDB support: pip install -U langformers[chromadb]

• For Pinecone support: pip install -U langformers[pinecone]

• To install all optional features: pip install -U langformers[all]

Below are the pre-built NLP tasks available in Langformers. Each link points to an example in the documentation to help you get started quickly.

• Seamless Chat with LLMs
• LLM Inference via API
• Data Labelling with LLMs
• Chunking

• Train Text Classifiers
• Pretrain MLMs from scratch
• Continue Pretraining MLMs on custom data

• Embed Sentences
• Semantic Search
• Rerank Sentences
• Mimic a Pretrained Model (Knowledge Distillation)

Complete documentation and advanced usage examples are available at: https://langformers.com.

Langformers is released under the Apache License 2.0.

We welcome contributions! Please see our contribution guidelines for details.

Built with ❤️ for the future of language AI.