---
title: 'Count Tokens in TypeScript: Fit the Context Window (2026)'
description: Count OpenAI tokens in TypeScript with gpt-tokenizer - no WASM, no API key. Count chat messages accurately and trim history to fit the model context window.
url: https://nerdleveltech.com/count-tokens-typescript-context-window
language: en
date: '2026-06-25'
category: llm-integration
tags:
  - gpt-tokenizer
  - typescript
  - tokens
  - context window
  - openai
  - llms
  - ai-ml
markdown_url: https://nerdleveltech.com/count-tokens-typescript-context-window.md
attribution: 'Source: NerdLevelTech — https://nerdleveltech.com/count-tokens-typescript-context-window'
terms: https://nerdleveltech.com/llms.txt
---

# Count Tokens in TypeScript: Fit the Context Window (2026)

To count OpenAI tokens in TypeScript without calling the API, install `gpt-tokenizer` and call `countTokens(text)` for strings or `encodeChat(messages, model).length` for a chat array. It is pure JavaScript - no WASM, no API key - and it powers a `trimToFit()` that fits a conversation to the model's context window.

## TL;DR

You will build a small TypeScript module that counts tokens locally with `gpt-tokenizer` 3.4.0, measures chat-message overhead correctly (not the stale "~4 tokens per message" guess), and trims a growing conversation so it fits inside a model's context window while reserving room for the reply. The stack is Node 22 LTS + TypeScript 6 + `tsx`, every snippet is copy-paste-runnable, and the whole thing runs offline. It takes about 15 minutes.

## What you'll learn

- Why you'd count tokens locally instead of calling the API - and when local counting is *not* enough
- How to count tokens in a string with `encode`, `decode`, and `countTokens`
- How to check a budget cheaply with `isWithinTokenLimit` (and its `number | false` return value)
- How to count a chat-messages array accurately with `encodeChat`, including the per-message framing overhead
- How to pick the right encoding (`o200k_base` vs `cl100k_base`) for your model
- How to write a `trimToFit()` that keeps the system prompt and reserves output tokens
- How to split a long document into token-bounded chunks for embedding or summarizing
- How `gpt-tokenizer` compares to `tiktoken` and `js-tiktoken` on the edge

## Prerequisites

- **Node 20.19+ or 22+** (this guide was verified on Node 22 LTS)
- **TypeScript 6.0.3** and **tsx 4.22.4** to run `.ts` files directly
- **gpt-tokenizer 3.4.0** - the pure-JavaScript BPE tokenizer for OpenAI models[^gpt-tokenizer]
- Basic familiarity with the OpenAI chat-messages shape (`{ role, content }`)

A token is the unit a model actually reads: roughly a word-piece. OpenAI's GPT models use a Byte-Pair Encoding (BPE) tokenizer, and `gpt-tokenizer` is a faithful pure-JS port of OpenAI's `tiktoken` that ships the BPE rank tables inline, so it works in Node, the browser, and edge runtimes without a WASM binary or a runtime download.[^github]

## Why count tokens locally (and when not to)

Counting locally is instant, free, and private - you don't ship your prompt to a third party just to size it. That's what you want for trimming history, splitting documents, sizing a batch, or showing a live token counter in a UI.

But be honest about the boundary: a local BPE tokenizer counts **text only**. It does not see the tokens added by tool/function schemas, images, or files, and providers occasionally adjust how requests are framed. OpenAI's own documentation says character-based estimates are inaccurate and that tools and images need a server-side count.[^openai-count] So use local counting for fast, approximate sizing of text, and fall back to the provider's official token-counting endpoint when you need an exact, billable number that includes non-text inputs. We'll wire up both.

## Step 1 - Set up the project

Create a project and install the runtime and dev dependencies with pinned versions:

```bash
mkdir token-budget && cd token-budget
npm init -y
npm install gpt-tokenizer@3.4.0
npm install -D typescript@6.0.3 tsx@4.22.4 @types/node@22.20.0
```

Mark the package as an ES module and add a strict `tsconfig.json`. Because we run `.ts` files with `tsx` (no build step), we enable `allowImportingTsExtensions` and `noEmit`:

```bash
npm pkg set type=module
```

```json
{
  "compilerOptions": {
    "target": "ES2023",
    "module": "nodenext",
    "moduleResolution": "nodenext",
    "strict": true,
    "noUncheckedIndexedAccess": true,
    "verbatimModuleSyntax": true,
    "allowImportingTsExtensions": true,
    "noEmit": true,
    "skipLibCheck": true,
    "types": ["node"]
  },
  "include": ["src"]
}
```

That's the whole setup. No model download, no API key, no `.env`.

## Step 2 - Count tokens in a string

This is the core of OpenAI token counting: turn text into token IDs and back. Create `src/count.ts`:

```ts
import { encode, decode, countTokens, isWithinTokenLimit } from 'gpt-tokenizer'

const text = 'The quick brown fox jumps over the lazy dog.'

const tokens: number[] = encode(text)
console.log('token ids:', tokens)
console.log('token count:', countTokens(text)) // same as encode(text).length
console.log('round-trips:', decode(tokens) === text)
```

Run it:

```bash
npx tsx src/count.ts
```

Expected output:

```text
token ids: [
    976,  4853, 19705,
  68347, 65613,  1072,
    290, 29082,  6446,
     13
]
token count: 10
round-trips: true
```

Three things worth noticing. First, `countTokens(text)` is the number you want for budgeting - it returns the same value as `encode(text).length` but counts natively without building the token array. Second, `decode(encode(text))` round-trips exactly, which is how you know the tokenizer is lossless. Third, the default import uses the `o200k_base` encoding, the one used by every modern OpenAI model (gpt-5, gpt-4o, gpt-4.1, and the o-series).[^github] We'll handle older models in Step 5.

## Step 3 - Check a limit cheaply with isWithinTokenLimit

Often you don't need the exact count - you only need to know "does this fit under N tokens?" Encoding a huge string just to compare its length is wasteful. `isWithinTokenLimit` stops early once the limit is exceeded, and its return type is the small gotcha worth internalizing: it returns the **token count** (a number) when the text fits, or **`false`** when it doesn't. Append this to `src/count.ts`:

```ts
// isWithinTokenLimit returns the token COUNT if within the limit, or false if over
const within: number | false = isWithinTokenLimit(text, 20)
console.log('within 20?', within) // 10

const over: number | false = isWithinTokenLimit(text, 5)
console.log('within 5?', over) // false
```

Output:

```text
within 20? 10
within 5? false
```

Because `false` and `0` are both falsy, always check the type explicitly. Write `if (within === false)` rather than `if (!within)` - otherwise an empty string (legitimately `0` tokens) would be misread as "over the limit." Under TypeScript's `strict` mode the `number | false` union forces you to handle both arms, which is exactly the safety you want around a budget check.

## Step 4 - Count chat messages accurately with encodeChat

Here's where most token-counting blog posts go wrong. The OpenAI Chat Completions API doesn't send the model your raw `content` strings - it wraps each message in special boundary tokens that mark the role and the start/end of every message. If you only count `content`, you **undercount**, and undercounting is how you blow past the context window in production.

`encodeChat` does the real thing. Create `src/count-chat.ts`:

```ts
import { encodeChat, countTokens } from 'gpt-tokenizer'

type ChatMessage = { role: 'system' | 'user' | 'assistant'; content: string }

const messages: ChatMessage[] = [
  { role: 'system', content: 'You are a helpful assistant.' },
  { role: 'user', content: 'What is the capital of France?' },
]

// encodeChat needs the model so it can apply that model's chat framing
const chatTokens = encodeChat(messages, 'gpt-4o')
console.log('chat token count:', chatTokens.length)

// Naive raw-text counting UNDERCOUNTS because it ignores per-message framing:
const naive = messages.reduce((sum, m) => sum + countTokens(m.content), 0)
console.log('naive content-only count:', naive)
console.log('framing overhead:', chatTokens.length - naive)
```

Run it:

```bash
npx tsx src/count-chat.ts
```

Output:

```text
chat token count: 24
naive content-only count: 13
framing overhead: 11
```

The two messages are 24 tokens, not 13. The 11-token difference is the ChatML framing - `encodeChat` emits the `<|im_start|>` / `<|im_sep|>` / `<|im_end|>` boundary tokens that wrap each message's role and content in OpenAI's chat format, which naive character- or content-based counting misses entirely. Pass the **model name** so `encodeChat` applies that model's framing; an empty chat still costs a few special tokens because the conversation itself has a start and end.

If you call `encodeChat` a lot, import a model-bound version once instead of repeating the model string:

```ts
import { encodeChat, countTokens } from 'gpt-tokenizer/model/gpt-4o'

// encodeChat(messages) now uses gpt-4o framing without a second argument
```

That subpath import binds the encoding at import time, so `encodeChat(messages)` and `countTokens(text)` both use gpt-4o's `o200k_base` settings.

## Step 5 - Pick the right encoding for your model

A tokenizer is only correct if it uses the encoding your model actually uses. OpenAI models map to two encodings you'll meet in practice:

| Encoding | Models | gpt-tokenizer import |
|----------|--------|----------------------|
| `o200k_base` | gpt-5, gpt-4o, gpt-4.1, o-series | default import (or `gpt-tokenizer/model/gpt-4o`) |
| `cl100k_base` | gpt-3.5-turbo, gpt-4 | `gpt-tokenizer/encoding/cl100k_base` |

The default import is `o200k_base`, so if you're on a current model you're already correct. Using the wrong encoding doesn't crash - it silently returns a different number, which is the dangerous kind of wrong. The gap is biggest for non-English text, where `o200k_base` is markedly more efficient. Create `src/encoding.ts`:

```ts
import { countTokens as o200k } from 'gpt-tokenizer/encoding/o200k_base'
import { countTokens as cl100k } from 'gpt-tokenizer/encoding/cl100k_base'

const text = '日本語のトークン数を数える'
console.log('o200k:', o200k(text)) // 10
console.log('cl100k:', cl100k(text)) // 14
```

Output:

```text
o200k: 10
cl100k: 14
```

Same string, 10 tokens versus 14 - a 40% difference driven purely by encoding choice. If you hardcoded `cl100k_base` out of habit while serving gpt-4o, every budget calculation would be inflated. Match the encoding to the model and the numbers line up. For deeper strategy on what to put *in* that window, see our companion piece on [context window optimization for LLMs](https://nerdleveltech.com/mastering-context-window-optimization-for-llms).

## Step 6 - Trim chat history to fit the context window

Now the payoff. A chat app accumulates history until it overflows the model's context window. The fix is to trim the oldest turns - while always keeping the system prompt, which steers the model - and to **reserve** tokens for the reply, because the context window is shared by input *and* output. (gpt-4o, for example, has a 128,000-token context window but caps output at 16,384 tokens, so you must leave room.[^context])

Create `src/trim-to-fit.ts`:

```ts
import { encodeChat } from 'gpt-tokenizer'

export type ChatMessage = { role: 'system' | 'user' | 'assistant'; content: string }
export type GptModel = 'gpt-4o' | 'gpt-4o-mini' | 'gpt-4.1' | 'gpt-5'

export interface TrimOptions {
  model: GptModel
  /** Total context window of the model, in tokens. */
  maxContextTokens: number
  /** Tokens to leave free for the model's reply. */
  reserveForOutput: number
}

export function countChatTokens(messages: ChatMessage[], model: GptModel): number {
  return encodeChat(messages, model).length
}

/**
 * Drop the oldest non-system messages until the chat fits inside
 * maxContextTokens - reserveForOutput. The system message is always kept.
 */
export function trimToFit(messages: ChatMessage[], opts: TrimOptions): ChatMessage[] {
  const budget = opts.maxContextTokens - opts.reserveForOutput
  if (budget <= 0) throw new RangeError('reserveForOutput must be smaller than the context window')

  const system = messages.filter((m) => m.role === 'system')
  const turns = messages.filter((m) => m.role !== 'system')

  // Keep removing the oldest turn until we fit (or only the system prompt remains).
  const kept = [...turns]
  while (kept.length > 0 && countChatTokens([...system, ...kept], opts.model) > budget) {
    kept.shift()
  }
  return [...system, ...kept]
}
```

Now exercise it with a synthetic conversation in `src/demo.ts`. We use a deliberately tiny 400-token window so the trimming is visible; in production you'd pass the model's real window:

```ts
import { trimToFit, countChatTokens, type ChatMessage } from './trim-to-fit.ts'

const history: ChatMessage[] = [
  { role: 'system', content: 'You are a terse assistant.' },
]
for (let i = 1; i <= 40; i++) {
  history.push({ role: 'user', content: `Question number ${i}: tell me a fact about the number ${i}.` })
  history.push({ role: 'assistant', content: `Fact ${i}: ${i} is a number with interesting properties worth a sentence or two of explanation here.` })
}

const model = 'gpt-4o' as const
console.log('full history tokens:', countChatTokens(history, model), 'messages:', history.length)

const trimmed = trimToFit(history, { model, maxContextTokens: 400, reserveForOutput: 120 })
console.log('trimmed tokens:', countChatTokens(trimmed, model), 'messages:', trimmed.length)
console.log('budget was:', 400 - 120)
console.log('system kept?', trimmed[0]?.role === 'system')
console.log('fits budget?', countChatTokens(trimmed, model) <= 400 - 120)
```

Run it:

```bash
npx tsx src/demo.ts
```

Output:

```text
full history tokens: 1773 messages: 81
trimmed tokens: 277 messages: 13
budget was: 280
system kept? true
fits budget? true
```

The 81-message, 1,773-token history is trimmed to the 13 most recent messages (277 tokens), which fits the 280-token budget (400 minus the 120 reserved for output), and the system prompt survives at index 0. Swap `maxContextTokens: 400` for `128_000` and the same function manages a real gpt-4o conversation.

Two production refinements once the shape is clear. First, dropping a lone `shift()` can leave a dangling assistant reply with no matching question; if message coherence matters, remove turns in user+assistant pairs. Second, if you'd rather summarize old turns than delete them, replace the dropped messages with a single "summary so far" system message - count that summary with `countTokens` so it, too, stays inside the budget.

## Step 7 - Split a long document by token budget

Trimming handles a conversation that's too long; **chunking** handles a single document that's too long - the everyday prep step before embedding text for search or feeding it to a summarizer one piece at a time. The pattern is the same primitive (`countTokens`) used greedily: pack words until the next one would push the chunk over the limit, then start a fresh chunk. Create `src/chunk.ts`:

```ts
import { countTokens } from 'gpt-tokenizer'

/**
 * Greedily pack whitespace-separated words into chunks that each stay
 * at or under maxTokens. Useful for splitting a document before embedding
 * or summarizing it.
 */
export function chunkByTokens(text: string, maxTokens: number): string[] {
  const words = text.split(/\s+/).filter(Boolean)
  const chunks: string[] = []
  let current: string[] = []

  for (const word of words) {
    const candidate = [...current, word].join(' ')
    if (countTokens(candidate) > maxTokens && current.length > 0) {
      chunks.push(current.join(' '))
      current = [word]
    } else {
      current.push(word)
    }
  }
  if (current.length > 0) chunks.push(current.join(' '))
  return chunks
}

const doc = Array.from({ length: 120 }, (_, i) => `sentence ${i + 1} has a few words in it.`).join(' ')
const chunks = chunkByTokens(doc, 64)
console.log('document tokens:', countTokens(doc))
console.log('chunks:', chunks.length)
console.log('max chunk tokens:', Math.max(...chunks.map((c) => countTokens(c))))
console.log('all within 64?', chunks.every((c) => countTokens(c) <= 64))
```

Run it:

```bash
npx tsx src/chunk.ts
```

Output:

```text
document tokens: 1200
chunks: 19
max chunk tokens: 64
all within 64? true
```

The 1,200-token document is split into 19 chunks, none exceeding the 64-token budget. Two notes for real documents: the `current.length > 0` guard prevents an infinite loop when a single "word" is itself longer than `maxTokens` (it gets its own oversized chunk rather than wedging the loop) - if that's a real risk for your input, split such tokens further. And for retrieval quality you usually want a small overlap between chunks; seed each new `current` with the last sentence of the previous chunk so context isn't sheared at the boundary.

## Verification

You've already seen each script print its expected output, but here's a single end-to-end check. Type-check the whole project, then confirm the trim demo reports `fits budget? true`:

```bash
npx tsc --noEmit && npx tsx src/demo.ts | tail -1
```

Expected:

```text
fits budget? true
```

A clean `tsc --noEmit` (it prints nothing and exits 0) plus `fits budget? true` means the types are sound and the trimming actually respects the budget you set.

## gpt-tokenizer vs tiktoken vs js-tiktoken

There are three pure-or-near-pure JavaScript options, and the right choice depends on where the code runs:

| Library | Version | Mechanism | Edge / offline notes |
|---------|---------|-----------|----------------------|
| `gpt-tokenizer` | 3.4.0 | Pure JS, BPE ranks bundled inline | No WASM, no network fetch - works on any runtime[^gpt-tokenizer] |
| `tiktoken` | 1.0.22 | WASM binding to the Rust tokenizer | High throughput on large inputs, but needs WASM support[^tiktoken] |
| `js-tiktoken` | 1.0.21 | Pure JS | Main build bundles ranks; the `js-tiktoken/lite` build expects you to supply rank data, commonly fetched from a CDN[^jstiktoken] |

For Cloudflare Workers, Vercel Edge, or any cold-start-sensitive path, `gpt-tokenizer` is the low-friction default: it has no WASM dependency and no runtime download, so there's nothing to fetch and nothing to fail. `tiktoken` (the WASM package) is built around a compiled WASM core aimed at raw throughput on very large inputs, at the cost of needing WASM support. `js-tiktoken` sits in between - its full build also bundles ranks, but the `lite` variant's bring-your-own-ranks pattern is the one that bites people on the edge when the CDN fetch is slow or blocked.

## Troubleshooting

**`Error: Model 'gpt-4o-banana' does not support chat.`** - You passed an unknown model name to `encodeChat`. The model string must be one `gpt-tokenizer` recognizes (e.g. `gpt-4o`, `gpt-4.1`, `gpt-5`). Fix the typo or import the model-bound version, `gpt-tokenizer/model/gpt-4o`, and call `encodeChat(messages)` without the second argument.

**`Error: Disallowed special token found: <|endoftext|>`** - Your input text literally contains a special token like `<|endoftext|>`, and `encode` refuses to silently merge it for safety. If you trust the input and want it tokenized as that special token, opt in explicitly:

```ts
import { encode } from 'gpt-tokenizer'

const ids = encode('hello <|endoftext|> world', {
  allowedSpecial: new Set(['<|endoftext|>']),
})
console.log(ids.length) // 9
```

**`ERR_MODULE_NOT_FOUND: Cannot find package 'gpt-tokenizer'`** - You ran a script from outside the project directory, so Node couldn't resolve `node_modules`. Run scripts from the project root (where `package.json` and `node_modules` live).

**`tsc` errors about CommonJS under `verbatimModuleSyntax`** - You forgot `"type": "module"` in `package.json`, so TypeScript treats `.ts` files as CommonJS and rejects the ESM `import` syntax. Run `npm pkg set type=module`. If `tsc` then complains about importing `./trim-to-fit.ts`, confirm `allowImportingTsExtensions` and `noEmit` are set as shown in Step 1.

**My local count doesn't match the API's billed tokens.** - Expected, within a small margin. Local counting ignores tool/function schemas, images, and files, and providers add framing of their own. For an exact number, use the provider's count endpoint (next section).

## When local counting isn't enough

Two cases need the network. First, **exact, billable counts that include non-text inputs.** OpenAI exposes an input-token-count endpoint that accepts the same payload as the Responses API and returns the precise `input_tokens`, covering text, messages, images, files, and tools - the things a local BPE tokenizer can't see.[^openai-count] Reach for it when accuracy matters more than latency, e.g. right before a costly call or for usage metering.

Second, **other model families.** `gpt-tokenizer` and `tiktoken` are OpenAI-specific. Anthropic's Claude uses a *different* tokenizer, so counting Claude prompts with a GPT tokenizer is simply wrong; use Anthropic's `POST /v1/messages/count_tokens` endpoint (or `client.messages.countTokens()`) instead.[^anthropic] Google's Gemini has its own `countTokens` method as well. The rule of thumb: tokenizers are per-vendor, sometimes per-generation, so match the counter to the model you're actually calling.

For the broader cost picture once you can count, see our guides on [saving tokens and optimizing prompts](https://nerdleveltech.com/saving-tokens-and-optimizing-prompts-the-art-of-efficient-ai-conversations) and [prompt caching with Claude in TypeScript](https://nerdleveltech.com/claude-prompt-caching-typescript-tutorial), which both build on the budgeting muscle you just developed.

## Next steps

- Add a **summarization fallback** to `trimToFit`: when you'd drop a turn, fold it into a running summary system message instead of deleting it.
- Wire the local count into a **pre-flight guard** that rejects requests over budget before they hit the API.
- For exact metering, call the provider's count endpoint and reconcile it against your local estimate to learn your typical overhead.

You now have a fast, offline, type-safe way to count OpenAI tokens in TypeScript and keep any conversation inside its context window - without a WASM binary, a CDN fetch, or an API key.

[^gpt-tokenizer]: `gpt-tokenizer` on npm - 3.4.0 is the `latest` dist-tag as of 25 June 2026. Pure-JavaScript BPE tokenizer for OpenAI models. https://www.npmjs.com/package/gpt-tokenizer
[^github]: niieani/gpt-tokenizer (GitHub) - the default export uses the `o200k_base` encoding, used by gpt-5, gpt-4o, gpt-4.1, and the o-series. https://github.com/niieani/gpt-tokenizer
[^openai-count]: OpenAI - Counting tokens. The input-token-count endpoint accepts the same input as the Responses API and returns the exact `input_tokens`, including tools and images that local tokenizers can't measure. https://developers.openai.com/api/docs/guides/token-counting
[^context]: OpenAI gpt-4o has a 128,000-token context window and, since the 2024-11-20 update, a 16,384-token maximum output; input and output share the window. https://platform.openai.com/docs/models/gpt-4o
[^tiktoken]: `tiktoken` on npm (1.0.22) - the WASM binding to OpenAI's Rust tokenizer. https://www.npmjs.com/package/tiktoken
[^jstiktoken]: `js-tiktoken` on npm (1.0.21) - pure-JS port; the `js-tiktoken/lite` build expects caller-supplied rank data. https://www.npmjs.com/package/js-tiktoken
[^anthropic]: Anthropic - Token counting. Claude uses a different tokenizer from OpenAI; count via `POST /v1/messages/count_tokens` or `client.messages.countTokens()`. https://platform.claude.com/docs/en/build-with-claude/token-counting