LWON

Benchmarks

How many tokens the same data costs as JSON, TOON and LWON, and whether Claude still answers questions about it correctly. Every LWON result is decoded again and checked against the original.

Claude tokens per dataset, relative to minified JSON
  • JSON (minified)
  • TOON
  • LWON

users Flat table, 100 rows

JSON (minified)
4,295
TOON
2,510 · −42%
LWON
2,503 · −42%

orders Nested objects and arrays, 50 orders

JSON (minified)
8,259
TOON
7,435 · −10%
LWON
4,808 · −42%

issues GitHub-style issues with nullable objects, 30 rows

JSON (minified)
4,378
TOON
4,414 · +1%
LWON
2,760 · −37%

menu Recursive tree, 3 levels

JSON (minified)
562
TOON
578 · +3%
LWON
362 · −36%

logs Log events with optional fields, 80 rows

JSON (minified)
4,393
TOON
4,472 · +2%
LWON
3,452 · −21%

search Search tool result: object with a result list

JSON (minified)
2,930
TOON
2,200 · −25%
LWON
2,298 · −22%

Counted with the Claude count_tokens API (Haiku, Sonnet and Opus 5.5 share a tokenizer). Data only. TOON: @toon-format/toon 4.4.0. Lower is better.

Claude tokens

Counted with the Claude count_tokens API; Haiku, Sonnet and Opus 5.5 share a tokenizer. The last column adds the LWON primer, about 328 tokens, which a model needs once per conversation.

Dataset JSON minified TOON LWON LWON vs JSON LWON + primer
usersFlat table, 100 rows 4,295 2,510 2,503 −42% 2,831 (−34%)
ordersNested objects and arrays, 50 orders 8,259 7,435 4,808 −42% 5,136 (−38%)
issuesGitHub-style issues with nullable objects, 30 rows 4,378 4,414 2,760 −37% 3,088 (−29%)
menuRecursive tree, 3 levels 562 578 362 −36% 690 (+23%)
logsLog events with optional fields, 80 rows 4,393 4,472 3,452 −21% 3,780 (−14%)
searchSearch tool result: object with a result list 2,930 2,200 2,298 −22% 2,626 (−10%)

LWON is smaller than minified JSON on every dataset and smaller than TOON on all but the search result, where long free-text snippets dominate and TOON is 4% smaller. On the 20-node menu the primer costs more than LWON saves, so small payloads are better sent as JSON.

Accuracy

Each model answered 44 questions per format about the same datasets: finding a record, following nested fields, counting and summing, walking the tree, and telling an absent key from a null one. Answers were checked against values computed from the data.

Model JSONTOONLWON
Claude Haiku 5.5 44/44 44/44 44/44
Claude Haiku 5.5 (effort low) 44/44 44/44 44/44
Claude Sonnet 5.5 44/44 44/44 44/44
Claude Opus 5.5 44/44 44/44 44/44

Every model answered every question correctly in every format, including Haiku with thinking turned down to low effort. LWON 0.x, which wrote every key by position, missed one count at low effort; the current encoding did not.

Harder test: sparse data

150 log events where most fields are optional and some are null: 10,922 Claude tokens as JSON, 11,433 as TOON and 8,835 as LWON (−19%). The questions count keys across rows, add up values, list the keys of one row, and ask whether a key is absent or null. Each was asked 3 times to Haiku and 2 times to Sonnet.

ModelJSONTOONLWON
Claude Haiku 5.5 61/6665/6663/66
Claude Haiku 5.5 (effort low) 63/6664/6660/66
Claude Sonnet 5.5 43/4443/4442/44

Over all runs, models got 11 of 176 answers wrong with LWON, 9 with JSON and 4 with TOON. Every wrong answer, in every format, was a count over 150 rows that was off by one to three, or a sum; every question about a single row, including absent versus null, was answered correctly. With this many questions, LWON and JSON are within noise of each other. TOON counted best: on irregular data it writes one key: value line per field and states the row count, which makes keys easy to tally, at +29% more tokens than LWON.

Cost per question

Fewer input tokens is only half the bill. Models also spend output tokens, including thinking, to read the data, and output costs several times more than input. Sent once prices the data as normal input; cached prices it as a cache read, when the same data answers many questions.

LWON 0.x wrote every key by position, so on sparse data models had to count positions through rows full of ~: Haiku thought about four times as long as with JSON. LWON now writes keys that most rows lack by name, in a * slot, which removed most of that extra thinking. The tables compare both.

Regular data (six datasets): change in cost against JSON
Model Output tokens JSON / LWON 0.x / LWON LWON sent once LWON cached LWON 0.x cached TOON sent once
Claude Haiku 5.5 167 / 235 / 186 −21% −2% +17% −10%
Claude Haiku 5.5 (effort low) 146 / 210 / 154 −23% −7% +16% −9%
Claude Sonnet 5.5 86 / 108 / 83 −25% −15% −2% −12%
Claude Opus 5.5 120 / 130 / 108 −25% −15% −1% −15%
Sparse data: change in cost against JSON
Model Output tokens JSON / LWON 0.x / LWON LWON sent once LWON cached LWON 0.x cached TOON sent once
Claude Haiku 5.5 629 / 2,499 / 721 −9% +7% +212% +4%
Claude Haiku 5.5 (effort low) 521 / 2,175 / 604 −10% +6% +214% +4%
Claude Sonnet 5.5 273 / 545 / 353 −11% +9% +41% +4%

On regular data LWON is now the cheapest of the three formats for every model when the data is sent once, and cheaper than JSON when it is cached and reused. On sparse data with counting it is cheaper than JSON sent once and slightly more expensive cached, where LWON 0.x cost up to three times as much.

Another tokenizer

The same data counted with o200k_base (gpt-tokenizer 2.8.1), used by other model families.

Dataset JSON indented JSON minified TOON LWON LWON vs JSON
users 5,042 2,942 1,744 1,649 −44%
orders 9,226 5,400 5,753 3,624 −33%
issues 4,607 2,971 3,526 2,018 −32%
menu 696 395 530 282 −29%
logs 4,522 3,126 3,619 2,636 −16%
search 2,722 1,985 1,608 1,623 −18%

Method

  • Datasets are generated from a fixed seed so every run is identical. You can load each one in the playground.
  • The baseline is minified JSON. Indented JSON is shown only for reference; comparing against it would overstate the savings.
  • TOON output uses @toon-format/toon 4.4.0 with default options; LWON uses the reference encoder with default options.
  • In the accuracy test, JSON and TOON get a one-line label and LWON gets its primer. Each question is asked once per model and format, with adaptive thinking at medium effort unless noted.

Limits

  • The datasets are synthetic, shaped after common API and tool payloads, and up to about 8,000 tokens.
  • The regular-data test sits at the ceiling: it shows LWON does not hurt accuracy there, not small differences. The sparse test separates the formats, but with 22 questions and 2–3 trials, a difference of a few answers is within noise.
  • LWON gets a primer of about 328 tokens; JSON and TOON get a one-line label.

Reproduce

git clone https://github.com/ahmetzencirli/lwon
cd lwon
npm install
npm run bench                 # o200k token counts
npm run eval:tokens           # Claude token counts (needs ANTHROPIC_API_KEY)
npm run eval -- --budget 2    # accuracy test, about $1.30 for three models
npm run eval -- --suite sparse --trials 3 --budget 2
npm run eval:report
npm run eval:report -- --suite sparse