Why do duplicates hurt fine-tuning?

Repeated examples are over-weighted during training, which encourages memorisation, wastes tokens you pay for and skews evaluation scores.

How does near-duplicate matching work?

Each record is turned into a set of three-word shingles and compared with the Jaccard index; anything at or above your threshold is treated as a duplicate.

What threshold should I use?

0.85 is a good default. Lower it towards 0.7 to catch heavier paraphrasing, raise it towards 0.95 if you only want near-identical rows removed.

Can I dedupe on just the prompt?

Yes — enter the field name to compare, for example prompt or instruction, and leave it blank to compare the whole record.

Is my data uploaded to a server?

No. This tool runs entirely in your browser using standard web APIs, so your input never leaves your device and nothing is logged or stored on our servers.

AI Dataset Deduplicator

Remove exact and near-duplicate rows from fine-tuning datasets.

Support

Works Offline

Privacy First

No Login

No API

Match mode

Compare field (blank = whole record)

Keep first occurrence

Records in

Unique kept

Duplicates removed

Invalid lines

Input JSONL

Drop a file here

or choose a file — it never leaves your device

Deduplicated JSONL

JSON

{"prompt":"What is RAG?","completion":"Retrieval augmented generation."}
{"prompt":"Explain embeddings","completion":"Vectors that capture meaning."}

Removed records — 2

Line 2 — Duplicate after normalisation: {"prompt":"what is rag?","completion":"Retrieval augmented generation."}
Line 3 — Duplicate after normalisation: {"prompt":"What is RAG?","completion":"Retrieval augmented generation."}

About the AI Dataset Deduplicator

Load a JSONL dataset and strip duplicates by exact text, case and whitespace insensitive match, or fuzzy near-duplicate detection using word shingles with an adjustable similarity threshold. Compare on any field or the whole record, and export the cleaned file.

Examples

Dedupe prompts

4 rows, 2 duplicate prompts

Output

3 unique rows kept

Keyboard shortcuts

Copy the main outputCtrl / ⌘ + Shift + C
Download the resultCtrl / ⌘ + S
Share this toolCtrl / ⌘ + Shift + S
Reset the inputsAlt + R
Open the tool search paletteCtrl / ⌘ + K

Related tools

AI Dataset Statistics Analyzer

AI & LLM

Profile a JSONL training set: token stats, roles, duplicates and cost.

Chat Dataset Converter

AI & LLM

Convert between OpenAI chat, ShareGPT, Alpaca and prompt/completion.

AI PII & Secret Redactor

AI & LLM

Strip emails, keys, tokens and personal data before prompting an LLM.

AI Token Counter

AI & LLM

Count exact LLM tokens for GPT, Claude and Gemini prompts.

AI Tool Schema Builder

AI & LLM

Build function-calling schemas for OpenAI, Claude, Gemini, MCP and the AI SDK.

AI Tool Schema Converter

AI & LLM

Convert function-calling schemas between OpenAI, Anthropic, Gemini and MCP.

Frequently asked questions

Version 1.0.0 · Updated 2026-08-15 · Runs entirely in your browser