De-Duplication
Jump to navigation
Jump to search
Tools for this function
| Tool | Purpose |
|---|---|
| AllDup | A brief description |
| Autopsy Digital Forensics | Open source, free digital forensics tool |
| CloneSpy | Finds duplicates, also precisely, deletes with rules. |
| Czkawka | Czkawka is a simple, fast and free app to remove unnecessary files from your computer. |
| Demystify | Format Identification Analysis and Reporting |
| Double Commander | Open source file manager with two panels side by side |
| DupeGuru | A brief description |
| Emailchemy | Converts proprietary emails to standard portable formats |
| FileVerifier++ | Windows utility for verifying file contents |
| FolderMatch | Compares two directory trees and flags up duplicates |
| FreeCommander | Split-screen file manager with desirable extras |
| Fslint | Set of utilities to find and clean various forms of lint on a filesystem, such as duplicate files, empty directories, and bad file names. |
| GNU Diffutils | GNU Diffutils is a package of several programs related to finding differences between files. |
| Java library implementing Pairtree | The PAIRTREE LIBRARY is a software library that supports the mapping between identifiers and filepaths according to the Pairtree Specification. |
| Matchbox Tool | Matchbox: Duplicate detection tool for digital document collections. |
| NT (New Tool) | This is a presrvation tool |
| SSDeep | Recursive piecewise hashing tool |
| Sumfolder1 | sumfolder1 is a utility for use within the archival and digital preservation community to generate checksums for file system directories, and to generate an overall "collection" checksum for a given set of files. The utility may be used in support of de-duplication at a directory/folder level. |
| The DeDuplicator (Heritrix add-on module) | The DeDuplicator is an add-on module for Heritrix to reduce the amount of duplicate data collected in a series of snapshot crawls. |
| WinMerge | A visual tool for differencing and merging of file collections, images and texts. |
| XcorrSound | The xcorrSound package compares sound waves using cross correlation. |
For some guidance on approaches to de-duplication see: