adam/Piwigo
1
0 Fork 0
mirror of https://github.com/Piwigo/Piwigo.git synced 2026-11-03 07:22:22 +01:00
Code Issues 657 Packages Projects Releases Wiki Activity

[database] md5sum is not saved if photos are uploaded with FTP #272

New Issue
Closed
opened 2026-10-04 23:50:10 +02:00 by adam · 6 comments
adam commented 2026-10-04 23:50:10 +02:00
Owner
Copy link
Copy source

Reported by flextron on 9 Sep 2014 09:57

I was trying to use the pwg.images.exist method of the API and it didn't work for some files. The problem is that the md5sum is not saved if the photos are uploaded with FTP. The md5sum column in the database is null.

For the photos uploaded through the web, the md5sum is correct and is saved in the BD.

Steps to reproduce: 1. Upload a photo with FTP
2. Use pwg.images.exists with the md5sum of the photo previously uploaded.
3. It won't work.

Expected behaviour:

  1. Upload a photo with FTP
  2. Use pwg.images.exists with the md5sum of the photo previously uploaded.
  3. It works.

Piwigo Bugtracker #3137

**Reported by flextron on 9 Sep 2014 09:57** I was trying to use the pwg.images.exist method of the API and it didn&#039;t work for some files. The problem is that the md5sum is not saved if the photos are uploaded with FTP. The md5sum column in the database is null. For the photos uploaded through the web, the md5sum is correct and is saved in the BD. **Steps to reproduce:** 1. Upload a photo with FTP 2. Use pwg.images.exists with the md5sum of the photo previously uploaded. 3. It won&#039;t work. Expected behaviour: 1. Upload a photo with FTP 2. Use pwg.images.exists with the md5sum of the photo previously uploaded. 3. It works. [Piwigo Bugtracker #3137](http://piwigo.org/bugs/view.php?id=3137) <!-- github-sync-id:97226875 -->
adam closed this issue 2026-10-04 23:50:10 +02:00
adam commented 2026-10-04 23:50:11 +02:00
Author
Owner
Copy link
Copy source

Maybe we should have such a Batch Manager action as "compute checksum" and a prefilter "with no checksum". It won't completely solve the problem, because we would still have missing checksum as long as we don't launch the batch manager action.

Maybe we should have such a Batch Manager action as "compute checksum" and a prefilter "with no checksum". It won't completely solve the problem, because we would still have missing checksum as long as we don't launch the batch manager action. <!-- github-comment-id:375593478 -->
adam commented 2026-10-04 23:50:12 +02:00
Author
Owner
Copy link
Copy source

And we can't, for a performance reason, compute the checksum during sync.

Maybe we could have an automatic task on any open of index.php "compute checksum for X photos where checksum is missing", it would act as a background task (but would slightly slow down operations...)

And we can't, for a performance reason, compute the checksum during sync. Maybe we could have an automatic task on any open of index.php "compute checksum for X photos where checksum is missing", it would act as a background task (but would slightly slow down operations...) <!-- github-comment-id:375593971 -->
adam commented 2026-10-04 23:50:13 +02:00
Author
Owner
Copy link
Copy source

It turned our to be easier then I thought, atleast to get done what I needed.
I added it to the sync_metadata action.
Would you be interested to have the filter integrated her aswell?

I obviously had to raise the max execution time in php a bit to handle 18k+ foto's nin one go ... but it was done in under 5 minutes.

Just for reference, I'm also importing keywords from my previous photo solution Shotwell. Here's the code for that ... doesn't seem worthy of a pull request or separate issue, but hopefully someone looking for a similar thing will find it here.

diff --git a/admin/include/functions_metadata.php b/admin/include/functions_metadata.php
index 750953e..d758988 100755
--- a/admin/include/functions_metadata.php
+++ b/admin/include/functions_metadata.php
@@ -206,6 +206,27 @@ function get_sync_metadata($infos)
 
   $infos['md5sum'] = md5_file($file);
 
+  // Open database.
+  $db = new SQLite3('/tmp/local-share-shotwell-data/photo.db', SQLITE3_OPEN_READONLY);
+
+  // Select the photo ID.
+  $results = $db->query("select id from PhotoTable where md5='" . $infos['md5sum'] . "'");
+  $row = $results->fetchArray(SQLITE3_ASSOC);
+  $showtwell_id = $row['id'];
+
+  // Lookup tags on both ID and thumbnail name.
+  $thumb_name = sprintf("thumb%016x", $showtwell_id);
+  $results = $db->query("select `name` from TagTable where ',' || photo_id_list LIKE '%,$showtwell_id,%' OR ',' || photo_id_list LIKE '%,$thumb_name,%'" );
+
+  $tags = array();
+  while ($row = $results->fetchArray(SQLITE3_ASSOC)) {
+    $tags[] = $row['name'];
+  }
+  if (!empty($tags)) {
+    $infos['tags'] = implode(',', $tags);
+  }
+  $db->close();
+
   if ($is_tiff)
   {
     // back to original file
It turned our to be easier then I thought, atleast to get done what I needed. I added it to the sync_metadata action. Would you be interested to have the filter integrated her aswell? I obviously had to raise the max execution time in php a bit to handle 18k+ foto's nin one go ... but it was done in under 5 minutes. Just for reference, I'm also importing keywords from my previous photo solution Shotwell. Here's the code for that ... doesn't seem worthy of a pull request or separate issue, but hopefully someone looking for a similar thing will find it here. ```patch diff --git a/admin/include/functions_metadata.php b/admin/include/functions_metadata.php index 750953e..d758988 100755 --- a/admin/include/functions_metadata.php +++ b/admin/include/functions_metadata.php @@ -206,6 +206,27 @@ function get_sync_metadata($infos) $infos['md5sum'] = md5_file($file); + // Open database. + $db = new SQLite3('/tmp/local-share-shotwell-data/photo.db', SQLITE3_OPEN_READONLY); + + // Select the photo ID. + $results = $db->query("select id from PhotoTable where md5='" . $infos['md5sum'] . "'"); + $row = $results->fetchArray(SQLITE3_ASSOC); + $showtwell_id = $row['id']; + + // Lookup tags on both ID and thumbnail name. + $thumb_name = sprintf("thumb%016x", $showtwell_id); + $results = $db->query("select `name` from TagTable where ',' || photo_id_list LIKE '%,$showtwell_id,%' OR ',' || photo_id_list LIKE '%,$thumb_name,%'" ); + + $tags = array(); + while ($row = $results->fetchArray(SQLITE3_ASSOC)) { + $tags[] = $row['name']; + } + if (!empty($tags)) { + $infos['tags'] = implode(',', $tags); + } + $db->close(); + if ($is_tiff) { // back to original file ``` <!-- github-comment-id:388473831 -->
adam commented 2026-10-04 23:50:13 +02:00
Author
Owner
Copy link
Copy source

@helmo you have implemented the solution to compute the md5sum at sync time. My intuition is that it costs too much time. Did you run some test. How much does it slow down syncing?

@helmo you have implemented the solution to compute the md5sum at sync time. My intuition is that it costs too much time. Did you run some test. How much does it slow down syncing? <!-- github-comment-id:453572564 -->
adam commented 2026-10-04 23:50:14 +02:00
Author
Owner
Copy link
Copy source

It probably depends on multiple factors but mostly on disk IO.
I did not do extensive benchmarks but my 18.000 + images in under 5 minutes lets me believe that a few hundred would not be a problem in the default 30 seconds php timeout.

What would you optimize for? should 1000 images be synced within a few seconds? or 10.000? or 100.000?

It probably depends on multiple factors but mostly on disk IO. I did not do extensive benchmarks but my 18.000 + images in under 5 minutes lets me believe that a few hundred would not be a problem in the default 30 seconds php timeout. What would you optimize for? should 1000 images be synced within a few seconds? or 10.000? or 100.000? <!-- github-comment-id:453623354 -->
adam commented 2026-10-04 23:50:15 +02:00
Author
Owner
Copy link
Copy source

I tried to sync 50k photos (average of 2MB per photo) without md5sum computation and it takes 13s, then with md5sum for more than 25min.

So now, i'm going to implement a solution like we already have for deleting orphans photos but to compute md5sum this time: one click action, progress indicator...

I tried to sync 50k photos (average of 2MB per photo) without md5sum computation and it takes 13s, then with md5sum for more than 25min. So now, i'm going to implement a solution like we already have for deleting orphans photos but to compute md5sum this time: one click action, progress indicator... <!-- github-comment-id:455119283 -->
Sign in to join this conversation.
No Branch/Tag Specified
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
adam (Adam Wojcieszonek)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: adam/Piwigo#272