chunk-standalone

"As shown above" reaches the model alone

A retriever returns chunks one at a time. A chunk that opens with "As shown above", promises "the following settings" that sit in the next chunk, or calls the product "it" reaches the answering model without the text it depends on, and the answer built on it comes out vague or wrong. A regular expression can find "above" and "this", but it cannot tell whether the chunk explains them itself. chunk-standalone checks each chunk and suggests a fix for the ones that lean on their neighbours.

Each chunk judged next to its neighbours

The chunks of each file go out in reading order. One request holds up to eight consecutive chunks (--batch), plus the last 1,500 characters of the chunk before them and the first 1,500 characters of the chunk after them, so every chunk is judged next to both neighbours. Jev, TypeSafe AI's System One model, answers three questions for every chunk, pointing at it by its position:

• Can a reader understand it without the pieces before it?
• Does it refer to something that is not inside it, such as "this", "above", "the following" or "as mentioned"?
• What should be done with it so it reads well on its own: keep it, merge it with the previous chunk, merge it with the next, or split it? A merge is offered only when that neighbour exists.

At the default batch size that is 24 questions per request, and code decides which answers matter, with one threshold (0.8 by default). A chunk is ok when it confidently stands alone, points nowhere else and should be kept; it needs a fix when a merge or a split is confident and a problem is confirmed; everything else goes to review. A split stands on its own, because a chunk can make sense alone and still mix two topics. The report prints the probabilities next to every verdict.

Split the way your chunker does

Files can be split the three common ways, so the report shows the problems your own chunker is likely to make: at every heading outside code blocks, at every blank line, or into pieces of about N tokens cut between words. If your pipeline already made its chunks, a JSONL file of them works too, including LangChain-style exports with the text in page_content and the source in metadata.source. If your pipeline adds overlap, titles or heading paths to its chunks, export the chunks and check those, so the check sees what your retriever sees.

What is sent

Only when you run it with a key and without --dry-run: the text of up to eight consecutive chunks of one file per request, the neighbour snippets, the fixed question text (which names chunks only by position), and the model name. File names, folder names, chunk ids, line numbers and other JSONL fields are not sent, and the tool writes nothing to disk. The exit code is 1 when a chunk needs a fix, so it can fail a CI job before indexing.

Limits

• Jev sees the chunk text and its neighbours, not what your pipeline adds at query time.
• The fix is one of four answers. For a chunk that only lacks its section name, adding the heading is often the better fix, and the report does not suggest it. A merge names one neighbour, even when a chunk needs text from both sides.
• The splitting modes approximate common chunkers: no overlap, only # headings, and token counts estimated at four characters per token.
• Chunks estimated above 14,000 tokens are listed as skipped, not checked.
• English is where Jev is most accurate; other scripts use more tokens per character, so lower --batch if a request is rejected as too large.
• Text written to steer a model, such as an instruction hidden in a document, can move Jev's answers. The tool reports and never edits files.
• The example in the repository is two short pages about a made-up backup feature, with hand-written probabilities: it shows the report's format, not a live run.

Install and run

It needs Node.js 20 or later.

npm install -g github:hamzaahmadaslam/chunk-standalone
export TYPESAFE_API_KEY=<your-key>
chunk-standalone docs/ --dry-run
chunk-standalone chunks.jsonl --threshold 0.9