Matchbox Tool - Revision history

BertrandCaron: /* Description */ Encoding issues corrected

2021-12-06T14:09:50Z

Description: Encoding issues corrected

Ania Molenda at 15:28, 22 April 2021

2021-04-22T15:28:39Z

COPTR Bot: Import from spreadsheet via script.

2013-11-13T21:23:58Z

Import from spreadsheet via script.

COPTR Bot: Trial import from script.

2013-11-12T17:39:37Z

Trial import from script.

COPTR Bot: Trial import from script.

2013-11-12T13:47:43Z

Trial import from script.

COPTR Bot: Trial import from script.

2013-11-10T22:08:53Z

Trial import from script.

New page

{{Infobox_tool
|purpose=Matchbox: Duplicate detection tool for digital document collections.
|image=
|homepage=https://github.com/openplanets/scape/tree/master/pc-qa-matchbox
|license=Open source
|platforms=
}}


[[Category:Quality Assurance]]
[[Category:De-Duplication]]
[[Category:Image]]

= Description =
The Matchbox tool is responsible for finding duplicatre pairs in a collection of digital documents based on SIFT features and SSIM methods. Consequently the tool takes a collection path with associated parameters as input. Currently three scenarios are implemented. These are: ===

* Duplicate search in one turn (parameter â€˜allâ€™) ===

* Professional duplicate search (experienced user can execute particular step in â€˜FindDuplicatesâ€™ workflow) ===

* Quick check if two documents are duplicates (based on previous BoW dictionary). ===

Further parameters that influence and adjust duplicate analysis are currently investigated.

Image processing method:

The image processing algorithm can be described in 4 steps: ===

1. Document feature extraction

* Interest point detection (applying Scale Invariant Feature Transform (SIFT) keypoint extraction)
* Derivation of local feature descriptors (invariant to geometrical or radiometrical distortions)

2. Learning visual dictionary

* Clustering method applied to all SIFT descriptors of all images using k-means algorithm
* Run over collection and collect local descriptors in a visual dictionary using Bag-Of-Words (BoW) algorithm

3. Create visual histogram for each image document ===

4. Detect similar images based on visual histogram and local descriptors. Evaluate similarity score â€” pair-wise comparison of corresponding keyword frequency histograms for all documents. Conduct structural similarity analysis applying Sturctural SIMilarity (SSIM) approach (1 means identical and 0 means very different)

* Rotate
* Scale
* Mask
* Overlaying

Usage:

FindDuplicates script can be invoked from command line. For standard usage two parameters are required: path to the collection documents and â€˜allâ€™. ===

scape/pc-qa-matchbox/Python# python2.7 FindDuplicates.py h ===

usage: FindDuplicates.py [-h] [-threads THREADS|â€”threads THREADS] [-sdk SDK|â€”sdk SDK] [-precluster PRECLUSTER|â€”precluster PRECLUSTER] [-clahe CLAHE|â€”clahe CLAHE] [-config CONFIG|â€”config CONFIG] [-featdir FEATDIR|â€”featdir FEATDIR] [-bowsize BOWSIZE|â€”bowsize BOWSIZE] [-csv|â€”csv] [-v] dir ''all,extract,compare,train,bowhist,clean''

= User Experiences =
currently installed at Austrian National Library

= Development Activity =
=== PRONOM updates ===

=== Release Feed ===

=== Activity Feed ===

← Older revision		Revision as of 17:39, 12 November 2013
Line 59:		Line 59:

	= Development Activity =		= Development Activity =
		+
		+	{{Infobox_tool_details
		+	\|ohloh_id=Matchbox Tool
		+	}}

← Older revision		Revision as of 13:47, 12 November 2013
Line 59:		Line 59:

	= Development Activity =		= Development Activity =
−	~~=== PRONOM updates ===~~
−
−	~~=== Release Feed ===~~
−
−	~~=== Activity Feed ===~~

@@ Line 12: / Line 12: @@
 The Matchbox tool is responsible for finding duplicatre pairs in a collection of digital documents based on SIFT features and SSIM methods. Consequently the tool takes a collection path with associated parameters as input. Currently three scenarios are implemented. These are:
-* Duplicate search in one turn (parameter â€˜allâ€™)
+* Duplicate search in one turn (parameter 'all')
-* Professional duplicate search (experienced user can execute particular step in â€˜FindDuplicatesâ€™ workflow)
+* Professional duplicate search (experienced user can execute particular step in 'FindDuplicates' workflow)
 * Quick check if two documents are duplicates (based on previous BoW dictionary).
@@ Line 36: / Line 36: @@
 . Create visual histogram for each image document
-. Detect similar images based on visual histogram and local descriptors. Evaluate similarity score â€” pair-wise comparison of corresponding keyword frequency histograms for all documents. Conduct structural similarity analysis applying Sturctural SIMilarity (SSIM) approach (1 means identical and 0 means very different)
+. Detect similar images based on visual histogram and local descriptors. Evaluate similarity score – pair-wise comparison of corresponding keyword frequency histograms for all documents. Conduct structural similarity analysis applying Sturctural SIMilarity (SSIM) approach (1 means identical and 0 means very different)
 * Rotate
@@ Line 45: / Line 45: @@
 Usage:
-FindDuplicates script can be invoked from command line. For standard usage two parameters are required: path to the collection documents and â€˜allâ€™.
+FindDuplicates script can be invoked from command line. For standard usage two parameters are required: path to the collection documents and 'all'.
 scape/pc-qa-matchbox/Python# python2.7 FindDuplicates.py h
-usage: FindDuplicates.py [-h] [-threads THREADS|â€”threads THREADS] [-sdk SDK|â€”sdk SDK] [-precluster PRECLUSTER|â€”precluster PRECLUSTER] [-clahe CLAHE|â€”clahe CLAHE] [-config CONFIG|â€”config CONFIG] [-featdir FEATDIR|â€”featdir FEATDIR] [-bowsize BOWSIZE|â€”bowsize BOWSIZE] [-csv|â€”csv] [-v] dir ''all,extract,compare,train,bowhist,clean''
+usage: FindDuplicates.py [-h] [\--threads THREADS] [\--sdk SDK] [\--precluster PRECLUSTER] [\--clahe CLAHE] [\--config CONFIG] [\--featdir FEATDIR] [\--bowsize BOWSIZE] [\--csv] [-v] dir all,extract,compare,train,bowhist,clean
 = User Experiences =

@@ Line 1: / Line 1: @@
-{{Infobox_tool
+{{Infobox tool
 |purpose=Matchbox: Duplicate detection tool for digital document collections.
 |homepage=https://github.com/openplanets/scape/tree/master/pc-qa-matchbox
 |license=Open source
-|platforms=
+|function=Quality Assurance, De-Duplication
 }}
 = Description =
 The Matchbox tool is responsible for finding duplicatre pairs in a collection of digital documents based on SIFT features and SSIM methods. Consequently the tool takes a collection path with associated parameters as input. Currently three scenarios are implemented. These are:
@@ Line 59: / Line 55: @@
 = Development Activity =

@@ Line 14: / Line 14: @@
 = Description =
-The Matchbox tool is responsible for finding duplicatre pairs in a collection of digital documents based on SIFT features and SSIM methods. Consequently the tool takes a collection path with associated parameters as input. Currently three scenarios are implemented. These are: ===
+The Matchbox tool is responsible for finding duplicatre pairs in a collection of digital documents based on SIFT features and SSIM methods. Consequently the tool takes a collection path with associated parameters as input. Currently three scenarios are implemented. These are:
-* Duplicate search in one turn (parameter â€˜allâ€™) ===
+* Duplicate search in one turn (parameter â€˜allâ€™)
-* Professional duplicate search (experienced user can execute particular step in â€˜FindDuplicatesâ€™ workflow) ===
+* Professional duplicate search (experienced user can execute particular step in â€˜FindDuplicatesâ€™ workflow)
-* Quick check if two documents are duplicates (based on previous BoW dictionary). ===
+* Quick check if two documents are duplicates (based on previous BoW dictionary).
 Further parameters that influence and adjust duplicate analysis are currently investigated.
@@ Line 26: / Line 26: @@
 Image processing method:
-The image processing algorithm can be described in 4 steps: ===
+The image processing algorithm can be described in 4 steps:
 . Document feature extraction
@@ Line 38: / Line 38: @@
 * Run over collection and collect local descriptors in a visual dictionary using Bag-Of-Words (BoW) algorithm
-. Create visual histogram for each image document ===
+. Create visual histogram for each image document
 . Detect similar images based on visual histogram and local descriptors. Evaluate similarity score â€” pair-wise comparison of corresponding keyword frequency histograms for all documents. Conduct structural similarity analysis applying Sturctural SIMilarity (SSIM) approach (1 means identical and 0 means very different)
@@ Line 49: / Line 49: @@
 Usage:
-FindDuplicates script can be invoked from command line. For standard usage two parameters are required: path to the collection documents and â€˜allâ€™. ===
+FindDuplicates script can be invoked from command line. For standard usage two parameters are required: path to the collection documents and â€˜allâ€™.
-scape/pc-qa-matchbox/Python# python2.7 FindDuplicates.py h ===
+scape/pc-qa-matchbox/Python# python2.7 FindDuplicates.py h
 usage: FindDuplicates.py [-h] [-threads THREADS|â€”threads THREADS] [-sdk SDK|â€”sdk SDK] [-precluster PRECLUSTER|â€”precluster PRECLUSTER] [-clahe CLAHE|â€”clahe CLAHE] [-config CONFIG|â€”config CONFIG] [-featdir FEATDIR|â€”featdir FEATDIR] [-bowsize BOWSIZE|â€”bowsize BOWSIZE] [-csv|â€”csv] [-v] dir ''all,extract,compare,train,bowhist,clean''