Open your Downloads folder and count the files called something like 'report (1).pdf' or 'invoice - Copy.docx'. Now multiply that by every device, chat app and shared folder you use. Duplicate files are one of the biggest hidden sources of wasted storage, and unlike large videos, they don't feel like a problem until you're already out of space.
Where duplicates come from
- Downloading the same attachment or file more than once.
- Messaging apps saving a fresh copy of every image or document you receive or forward.
- Importing photos from a phone or camera twice, or from two overlapping backups.
- Making a 'safety copy' before editing and forgetting to delete it.
- Sending copies of a file to several people instead of sharing one link, which then come back edited.
- Backup tools that store the same file under different folders or dates.
What duplicates actually cost
The obvious cost is space. But the bigger cost is confusion. When three copies of a proposal exist, which is the latest? Someone edits the wrong one, work is lost, and time goes on comparing them. Duplicates also clutter search results and slow down backups, which have to copy every redundant file.
How duplicates are found: names versus content
There are two ways to decide that two files are the same. Comparing names and sizes is quick but unreliable. Two different files can share a name, and identical files often have different names. The reliable method is to compare content.
That's where hashing comes in. A hash function reads a file and produces a short, fixed-length fingerprint. SHA-256 is one such function. The same file always produces the same fingerprint, and changing even a single byte produces a completely different one. The chance of two different files producing the same SHA-256 fingerprint is so small it is treated as impossible in practice.
| Method | Reliable? | Notes |
|---|---|---|
| Same file name | No | Different files often share names, and copies are renamed |
| Same name and size | Somewhat | Better, but still misses renamed copies and can wrongly match different files |
| Same content hash (such as SHA-256) | Yes | Identical bytes always match, regardless of file name or location |
What hashing can't do
A content hash finds exact copies. If you export the same photo twice with different settings, edit it slightly, or re-save it in another format, the bytes differ and so does the hash. Finding near-duplicates, such as similar burst photos, requires other techniques that compare how images look. Hashing removes the perfectly identical files, and human judgment handles the rest.
Deduplication in storage systems
Some storage services use hashes to avoid keeping the same content twice: when you upload a file whose fingerprint already exists, the system keeps one physical copy and simply links to it. NGDS identifies uploaded files by their SHA-256 hash for this purpose, so identical files aren't stored again and again.
Habits that prevent duplicates
- Keep one source of truth for every document, and make everything else a link.
- Share links rather than sending copies, so edits happen in one place.
- Import photos once, then check the destination before importing again.
- Rely on version history for drafts instead of saving 'final_v2_copy' files.
- Clear your Downloads folder monthly. It is where most duplicates start.
- Tell messaging apps not to save every received image to your gallery automatically.
A quick cleanup routine
- Sort a folder by size and look at the largest files first, where the biggest savings usually are.
- Look for pairs with the same size and similar names, and open them to confirm they match.
- Keep the copy in the best location, delete the others, and empty the recycle bin only after checking that nothing is missing.
- Repeat every few months, or use a tool that finds duplicates by content.
A little prevention beats regular cleanups. If everyone in the household or team works from a single shared copy, and links replace attachments, most duplicates simply never appear.