adam/Piwigo
1
0 Fork 0
mirror of https://github.com/Piwigo/Piwigo.git synced 2026-11-03 07:22:22 +01:00
Code Issues 657 Packages Projects Releases Wiki Activity

[other] [Batch Manager] Find duplicates based on md5sum #210

New Issue
Closed
opened 2026-10-04 23:47:23 +02:00 by adam · 7 comments
adam commented 2026-10-04 23:47:23 +02:00
Owner
Copy link
Copy source

Reported by sakanaou on 11 Dec 2012 14:12

Version: 2.4.5

I recently noticed that the Batch Manager finds its duplicates based on the original file name. This is very confusing as normally duplicated files get uploaded because they happen to have different filenames. It is much safer to determine duplicates based on the file's md5sum, which is calculated after upload and stored in the database anyway.

I have created a small patch that searches for duplicates based on the md5sum instead of filename and will attach it here.

Piwigo Bugtracker #2801

**Reported by sakanaou on 11 Dec 2012 14:12** **Version:** 2.4.5 I recently noticed that the Batch Manager finds its duplicates based on the original file name. This is very confusing as normally duplicated files get uploaded because they happen to have different filenames. It is much safer to determine duplicates based on the file&#039;s md5sum, which is calculated after upload and stored in the database anyway. I have created a small patch that searches for duplicates based on the md5sum instead of filename and will attach it here. [Piwigo Bugtracker #2801](http://piwigo.org/bugs/view.php?id=2801) <!-- github-sync-id:97226498 -->
adam closed this issue 2026-10-04 23:47:23 +02:00
adam commented 2026-10-04 23:47:24 +02:00
Author
Owner
Copy link
Copy source

Comment posted by thimo on 20 Jan 2013 16:16

As mentioned in the forum the property "uniqueness_mode" should be incorporated:
http://piwigo.org/forum/viewtopic.php?id=20983

File /admin/batch_manager.php:

// perform 2 queries instead. We hope there are not too many duplicates.

  • $field = ($conf[&#039;uniqueness_mode&#039;] == &#039;md5sum&#039;) ? &#039;md5sum&#039; : &#039;file&#039;;

    $query = '
    -SELECT file
    +SELECT '.$field.'
    FROM '.IMAGES_TABLE.'

  • GROUP BY file

  • GROUP BY '.$field.'
    HAVING COUNT(*) > 1
    ;';

  • $duplicate_files = array_from_query($query, 'file');

  • $duplicate_files = array_from_query($query, $field);

$query = '
SELECT id
FROM '.IMAGES_TABLE.'

  • WHERE file IN (''.implode("','", $duplicate_files).'')

  • WHERE '.$field.' IN (''.implode("','", $duplicate_files).'')
    ;';

    array_push(

And add a hash to imports from the galleries folder (file /admin/site_update.php):

$insert = array(
  &#039;id&#039;             =&gt; $next_element_id++,
  &#039;file&#039;           =&gt; $filename,
  &#039;name&#039;           =&gt; get_name_from_file($filename),
  &#039;date_available&#039; =&gt; CURRENT_DATE,
  &#039;path&#039;           =&gt; $path,
  &#039;representative_ext&#039;  =&gt; $fs[$path][&#039;representative_ext&#039;],
  &#039;storage_category_id&#039; =&gt; $db_fulldirs[$dirname],
  &#039;added_by&#039;       =&gt; $user[&#039;id&#039;],

  •  &#039;md5sum&#039;         =&gt; md5_file($path),
    );
**Comment posted by thimo on 20 Jan 2013 16:16** As mentioned in the forum the property &quot;uniqueness_mode&quot; should be incorporated: http://piwigo.org/forum/viewtopic.php?id=20983 File /admin/batch_manager.php: ``` // perform 2 queries instead. We hope there are not too many duplicates. ``` - ``` $field = ($conf[&#039;uniqueness_mode&#039;] == &#039;md5sum&#039;) ? &#039;md5sum&#039; : &#039;file&#039;; ``` $query = &#039; -SELECT file +SELECT &#039;.$field.&#039; FROM &#039;.IMAGES_TABLE.&#039; - GROUP BY file - GROUP BY &#039;.$field.&#039; HAVING COUNT(*) &gt; 1 ;&#039;; - $duplicate_files = array_from_query($query, &#039;file&#039;); - $duplicate_files = array_from_query($query, $field); $query = &#039; SELECT id FROM &#039;.IMAGES_TABLE.&#039; - WHERE file IN (&#039;&#039;.implode(&quot;&#039;,&#039;&quot;, $duplicate_files).&#039;&#039;) - WHERE &#039;.$field.&#039; IN (&#039;&#039;.implode(&quot;&#039;,&#039;&quot;, $duplicate_files).&#039;&#039;) ;&#039;; array_push( And add a hash to imports from the galleries folder (file /admin/site_update.php): ``` $insert = array( &#039;id&#039; =&gt; $next_element_id++, &#039;file&#039; =&gt; $filename, &#039;name&#039; =&gt; get_name_from_file($filename), &#039;date_available&#039; =&gt; CURRENT_DATE, &#039;path&#039; =&gt; $path, &#039;representative_ext&#039; =&gt; $fs[$path][&#039;representative_ext&#039;], &#039;storage_category_id&#039; =&gt; $db_fulldirs[$dirname], &#039;added_by&#039; =&gt; $user[&#039;id&#039;], ``` - &#039;md5sum&#039; =&gt; md5_file($path), ); <!-- github-comment-id:124892954 -->
adam commented 2026-10-04 23:47:24 +02:00
Author
Owner
Copy link
Copy source

Comment posted by msakik on 11 Feb 2015 11:59

Hi, based on piwigo 2.7 this could add a checkmark to find by md5sum. This also could add an option to uncheck the filename option.

**Comment posted by msakik on 11 Feb 2015 11:59** Hi, based on piwigo 2.7 this could add a checkmark to find by md5sum. This also could add an option to uncheck the filename option. <!-- github-comment-id:124892964 -->
adam commented 2026-10-04 23:47:25 +02:00
Author
Owner
Copy link
Copy source

+1 to this request.

+1 to this request. <!-- github-comment-id:257342288 -->
adam commented 2026-10-04 23:47:26 +02:00
Author
Owner
Copy link
Copy source

So I have added a new checkbox "checksum" on the list of "find duplicates based on" options on the batch manager. The main problem is that photos added with synchronization have no md5sum (in the database, I mean), so this filter won't work for them.

We need a feature to compute checksums. I don't think it's a good idea to have these features in core, because it's only useful to very few people. ie people searching for duplicates based on checksum + using sync which is less and less used compared to other upload methods. The best idea I have now is to create a plugin "Compute Checksum" that would add features on Batch Manager:

  1. a filter "with no checksum"
  2. an action "compute checksum"
So I have added a new checkbox "checksum" on the list of "find duplicates based on" options on the batch manager. The main problem is that photos added with synchronization have no md5sum (in the database, I mean), so this filter won't work for them. We need a feature to compute checksums. I don't think it's a good idea to have these features in core, because it's only useful to very few people. ie people searching for duplicates based on checksum + using sync which is less and less used compared to other upload methods. The best idea I have now is to create a plugin "Compute Checksum" that would add features on Batch Manager: 1. a filter "with no checksum" 2. an action "compute checksum" <!-- github-comment-id:266300438 -->
adam commented 2026-10-04 23:47:27 +02:00
Author
Owner
Copy link
Copy source

What is the status of this? I would like to be able to search for duplicates of images already uploaded and have the option to delete the dupe. Even better if I could merge the tags, views, etc of both.

What is the status of this? I would like to be able to search for duplicates of images already uploaded and have the option to delete the dupe. Even better if I could merge the tags, views, etc of both. <!-- github-comment-id:314309424 -->
adam commented 2026-10-04 23:47:27 +02:00
Author
Owner
Copy link
Copy source

Hello!

I just recognized that most of my files have NULL in the md5sum field. Therefor I do not find duplicates based on md5sum. It would be cool to have an option to compute the md5sum for all pictures.

Any news on this issue?

Best regards

Jens
PS: Keep on going with the cool Piwigo software.

Hello! I just recognized that most of my files have NULL in the md5sum field. Therefor I do not find duplicates based on md5sum. It would be cool to have an option to compute the md5sum for all pictures. Any news on this issue? Best regards Jens PS: Keep on going with the cool Piwigo software. <!-- github-comment-id:375453124 -->
adam commented 2026-10-04 23:47:28 +02:00
Author
Owner
Copy link
Copy source

@jheinitz that's right, see #272

@jheinitz that's right, see #272 <!-- github-comment-id:375593226 -->
Sign in to join this conversation.
No Branch/Tag Specified
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
adam (Adam Wojcieszonek)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: adam/Piwigo#210