Which dataset shapes are supported?

OpenAI chat format with a messages array, prompt/completion pairs and arbitrary JSON objects, which are measured as serialised text.

Why do token percentiles matter?

p90 and p99 reveal the long tail that blows your context window or your bill, which averages alone completely hide.

Is the cost figure the fine-tuning price?

No — it is the input cost of sending the dataset once at the selected model's rate. Training prices differ per provider; use it as a relative size signal.

How are duplicates counted?

Records are compared on their lower-cased, trimmed text. For removal and fuzzy matching, use the AI Dataset Deduplicator.

Is my data uploaded to a server?

No. This tool runs entirely in your browser using standard web APIs, so your input never leaves your device and nothing is logged or stored on our servers.

AI Dataset Statistics Analyzer

Profile a JSONL training set: token stats, roles, duplicates and cost.

Support

Works Offline

Privacy First

No Login

No API

Encoding

Model (context & pricing)

Tokenizer

Estimating…

Dataset (JSONL)

Drop a file here

or choose a file — it never leaves your device

Report

Records

Invalid lines

Total tokens

Average tokens

10.3

Min / Max

6 / 14

p50 / p90 / p99

11 / 14 / 14

Messages

Empty records

Duplicates

Over context

Characters

124

One pass input cost

$0.000005

Token length distribution

0–20
2–40
4–60
6–81
8–100
10–121
12–140
14–161
16–180
18–200

Role distribution

user2
assistant2
system1
prompt1
completion1

About the AI Dataset Statistics Analyzer

Analyse a fine-tuning or evaluation dataset in one pass: record count, total and average tokens, min, max and p50/p90/p99 percentiles, token length histogram, role distribution, empty and duplicate rows, records that exceed the model context and the input cost of a single pass.

Examples

Chat dataset

3 JSONL records

Output

avg 24 tokens, p90 38, 0 duplicates

Keyboard shortcuts

Copy the main outputCtrl / ⌘ + Shift + C
Download the resultCtrl / ⌘ + S
Share this toolCtrl / ⌘ + Shift + S
Reset the inputsAlt + R
Open the tool search paletteCtrl / ⌘ + K

Related tools

AI Dataset Deduplicator

AI & LLM

Remove exact and near-duplicate rows from fine-tuning datasets.

Chat Dataset Converter

AI & LLM

Convert between OpenAI chat, ShareGPT, Alpaca and prompt/completion.

AI PII & Secret Redactor

AI & LLM

Strip emails, keys, tokens and personal data before prompting an LLM.

AI Token Counter

AI & LLM

Count exact LLM tokens for GPT, Claude and Gemini prompts.

AI Tool Schema Builder

AI & LLM

Build function-calling schemas for OpenAI, Claude, Gemini, MCP and the AI SDK.

AI Tool Schema Converter

AI & LLM

Convert function-calling schemas between OpenAI, Anthropic, Gemini and MCP.

Frequently asked questions

Version 1.0.0 · Updated 2026-08-15 · Runs entirely in your browser