Can I add custom model pricing or tokenizers?

Yes. You can configure custom model rates and tokenizers directly via request headers or the developer dashboard.

Universal BPE Intelligence

Multi-model tokenizer encoding intelligence

Native token profiling optimized for OpenAI o200k, Anthropic Claude BPE, DeepSeek 128k, and Universal 256k vocabularies.

Supported Tokenizers

13+ Frontier LLMs

OpenAI, Anthropic, DeepSeek, Moonshot, Qwen, GLM, Meta.

Sub-token Fragmentation

-45%

Drastically fewer orphan sub-tokens on code and punctuation.

Pre-warmed Memory

0ms Cold Start

Hash tables loaded at boot time for instant encoding.

The production engineering challenge

Different LLM providers use fundamentally different tokenization algorithms. A compression strategy optimized for OpenAI o200k can trigger severe sub-token fragmentation when processed by Anthropic Claude BPE or DeepSeek Byte BPE.

Technical architecture & mechanics

Vocabulary-aware tokenization profiles

Maintains native in-memory tokenizer models for every major provider to preview exact sub-token boundaries.

Cross-tokenizer normalization

Normalizes multi-byte unicode characters, code indentation, and delimiter blocks to prevent sub-token splits.

Dynamic model routing detection

Automatically detects the target model specified in the request payload and applies the optimal tokenization profile.

Tokenizer Profile Inspection

import { SUPPORTED_TOKENIZERS, calculateTokens } from '@/lib/tokenizer'; // Profile token counts across different vocabularies const claudeTokens = calculateTokens(prompt, 'claude-bpe'); const o200kTokens = calculateTokens(prompt, 'openai-o200k'); const deepseekTokens = calculateTokens(prompt, 'deepseek-llama'); console.log({ claudeTokens, o200kTokens, deepseekTokens });

Architecture comparison

Evaluation metric	Curtly.dev gateway	Traditional / naive approach
Tokenizer Awareness	Native profile matched to target LLM architecture	Generic regex replacement ignoring BPE boundaries
Claude BPE Indentation	Optimized space normalization avoiding 2-token splits	Severe token bloat on indented code lines
Multilingual Token Efficiency	Byte-level support for CJK and non-Latin scripts	High character-to-token explosion on Asian languages

Frequently asked questions

Yes. The gateway inspects the model field in each payload (e.g. gpt-6-astra vs claude-fable-5-1) and applies the corresponding tokenizer engine instantly.

Explore related platform features

Financial Simulator

Ready to optimize your AI infrastructure?

Start optimizing in 60 seconds with a 1-line baseURL change. Free tier available with zero credit card required.

Start optimizing free Simulate token ROI