pg_tokenizer

pg_tokenizer is a standalone multilingual text analyzer and vocabulary tokenizer extension for PostgreSQL, designed to power full-text BM25 ranking with vchord_bm25 and hybrid retrieval with vchord.

Website | Documentation | vchord | vchord_bm25 | Blog | PGEXT | Releases | Issues | Upstream

---

Vchord Suite

pg_tokenizer is part of the Vchord extension family maintained by PGSTY:

• vchord (Docs · PGEXT · Pigsty): Scalable, high-performance, and disk-friendly vector similarity search (vchordrq & vchordg).
• vchord_bm25 (Docs · PGEXT · Pigsty): Native BM25 full-text ranking index (Block-WeakAnd).
• pg_tokenizer (Docs · PGEXT · Pigsty): Standalone multilingual text analyzer and vocabulary tokenizer.

---

Features

• Modular Text Analysis Pipeline: Chain character filters (to_lowercase, unicode_normalization), pre-tokenizers (unicode_segmentation, regex, Chinese jieba), and token filters (stopwords, Snowball stemmer, synonym, pg_dict, ngram) via declarative TOML configurations.
• Built-in & Hugging Face Vocabularies: Ship with built-in pre-trained models (bert_base_uncased, wiki_tocken, gemma2b, llmlingua2) or import any Hugging Face tokenizer.json directly into PostgreSQL.
• Corpus-Trained Custom Models: Automatically build domain-specific vocabularies from your own table columns and maintain token embeddings via database triggers (create_custom_model_tokenizer_and_trigger).
• Multilingual Support: Native support for English, European languages (30 Snowball stemmers), Chinese (jieba), Japanese/Korean (lindera), and multilingual LLM vocabularies.

---

Version 0.2.0

This release adds transaction-end invalidation for backend-local analyzer, tokenizer, model, stopword, and synonym caches, including dependent tokenizer objects and rollback cleanup. It also adds native PostgreSQL tsvector output and assigns positions consecutively after empty tokens are removed. Existing tokenize() calls still return INT[]. See custom models, native full-text search, and upgrade instructions.

Quick Start

For full setup instructions (Debian/ZIP packages and building from source), see the Installation Guide or docs/01-installation.md.

1. Install via Native Linux Packages (pig)

On EL 8/9/10, Debian 12/13, and Ubuntu 22.04/24.04/26.04 (PostgreSQL 14–18), install the native package built for your distribution and architecture from the Pigsty repository (see the package matrix on pgext.cloud):

curl -fsSL https://repo.pigsty.io/pig | bash
pig repo add pgsql -u
pig ext add pg_tokenizer

Then add pg_tokenizer.so to shared_preload_libraries and restart PostgreSQL:

psql -U postgres -c "ALTER SYSTEM SET shared_preload_libraries = 'pg_tokenizer.so';"
sudo systemctl restart postgresql

2. Run with Docker

You can also start a ready-to-use container with the tensorchord/vchord-suite image, which bundles PostgreSQL, vchord, vchord_bm25, and pg_tokenizer:

docker run \
  --name vchord-suite \
  -e POSTGRES_PASSWORD=postgres \
  -p 5432:5432 \
  -d tensorchord/vchord-suite:pg18-latest
  # Or from GHCR: ghcr.io/tensorchord/vchord-suite:pg18-latest

Connect to the database using psql:

psql -h localhost -p 5432 -U postgres

3. Basic SQL Usage

Enable the extension and add tokenizer_catalog to search_path:

CREATE EXTENSION IF NOT EXISTS pg_tokenizer;
SET search_path TO "$user", public, tokenizer_catalog;

Create a tokenizer backed by the built-in llmlingua2 model and tokenize text into an integer array of token IDs:

SELECT create_tokenizer('tokenizer1', $$
model = "llmlingua2"
$$);

SELECT tokenize(
    'PostgreSQL is a powerful, open-source object-relational database system. It has over 15 years of active development.',
    'tokenizer1'
);

More SQL examples (custom corpus models, Chinese jieba, Japanese lindera) can be found in docs/03-examples.md.

When upgrading from Lindera 0.42 to 6.2, regenerate stored Lindera token arrays and rebuild their search indexes: the embedded IPADIC dictionary now assigns different token IDs. Existing model configurations remain accepted; see the Lindera upgrade notes.

---

Documentation

Full online documentation is available at vchord.pgsty.com/docs/tokenizer, alongside the local docs in docs/:

• Installation Guide (docs/01-installation.md)
• Tokenizer Overview & Usage (docs/04-usage.md)
• Text Analyzer Pipeline (docs/05-text-analyzer.md)
• Models (Built-in, Custom, Hugging Face, Lindera) (docs/06-model.md)
• Multilingual Segmentation & Examples (docs/03-examples.md)
• SQL & Configuration Reference (docs/00-reference.md)
• Development · Limitations

---

License

This software is licensed under the Apache License 2.0 (Apache-2.0).

• Copyright (c) 2025-2026 TensorChord Inc.
• Copyright (c) 2026 PGSTY PTE. LTD.
• Maintained by Ruohang Feng @ PGSTY