Errors importing edited images / false duplicates

[rwjack]

Okay, debugging time...

From what I notice, my photoslibrary somehow contains files that have the same base name, but have a different extension. E.g

IMG_1234.JPG and IMG_1234.heic - two completely different images, the JPG taken with an older iphone, and the heic taken with a newer one. Something might have gotten jambled in the migration process when updating phones, and these just error out during the import process (for good reason I'd say).

What would be the best way to go about this?

I'm thinking I try the import again - this time tee the output to a file, grep for all errors so I can have a list of failures (or maybe I can grep for errors in the db itself?)

• Then go through the list manually
• copy the duplicate files to a temp directory (for easier filtering when importing afterwards)
• if the error list is
  • IMG_1234.jpg
  • IMG_1234.heic
  • IMG_2456.jpg
  • IMG_2456.heic
• I make copies of all these without extensions - like shown (just so we can capture the xmp files as well):
  • IMG_1234*
  • IMG_2456*
• I then add some prefix to each duplicate image, so they can be differentiated, e.g. the older one
• Finally, I import all files from this temporary folder - only problem I see is the export db would be wrong, and we could only rely on the xmp metadata in that case

---

Other than that, I'm completely bumboozled, some images just straight up failed to import. I see no correlation other than them all being edited versions - but I also see some others that successfully imported which are also edited - so it doesn't directly seem to be a problem with images being edited themselves.

[rwjack]

I figure it might be best to keep it one issue at a time - as you originally said Rhet. So the focus would be the top bit, and then we'll figure out afterwards why certain edited images aren't importing.

[RhetTbull]

IMG_1234.JPG and IMG_1234.heic - two completely different images, the JPG taken with an older iphone, and the heic taken with a newer one. Something might have gotten jambled in the migration process when updating phones, and these just error out during the import process (for good reason I'd say).

I'm trying to understand this assertion. When you have photos with identical stem (e.g. IMG_1234), osxphotos export automatically changes the name of the conflicting file(s) on export to avoid name collision. For example, renaming files to IMG_1234 (1).jpeg, etc. Here's example output of a test I just ran:

Exporting IMG_0102.HEIC (4FE579D2-888D-49A0-89E5-178600088399.heic) (1/3)
Exported IMG_0102.HEIC to /Users/rhet/Desktop/export/IMG_0102.HEIC
Exported /Users/rhet/Desktop/export/IMG_0102.HEIC
Exporting IMG_0102.JPEG (76F5188C-E304-49EF-A8A3-34B56380E76B.jpeg) (2/3)
Exported IMG_0102.JPEG to /Users/rhet/Desktop/export/IMG_0102 (1).JPEG
Exported /Users/rhet/Desktop/export/IMG_0102 (1).JPEG

Notice that IMG_0102 had both a .HEIC and a .jpeg version. The .jpeg version was name IMG_0102 (1).JPEG upon export.

Now this is only an issue if the duplicate names are exported to the same directory. If you're exporting by date and the duplicates are from different phones as you said, then it's highly unlikely to have a duplicate in the same directory. This can happen when you have duplicated the photo inside of Photos using the "Duplicate 1 Photo" right-click feature. But even then, osxphotos de-duplicates the name on export. Unless your workflow involves a lot of intentional duplication these should be fairly limited.

Now if there's an edit, the same thing happens with the AAE files (but I noticed a bug with these on import...more later)

Notice below that the HEIC this time was deemed the duplicate and received the IMG_0102 (1).HEIC name. (The order in which OSXPhotos processes photos is currently non-deterministic so the HEIC was processed 2nd whereas above it was processed first).

Exporting IMG_0102.JPEG (76F5188C-E304-49EF-A8A3-34B56380E76B.jpeg) (1/3)
Exported IMG_0102.JPEG to /Users/rhet/Desktop/export/IMG_0102.JPEG
Exported adjustments for IMG_0102.JPEG to /Users/rhet/Desktop/export//IMG_0102.JPEG
Exported /Users/rhet/Desktop/export/IMG_0102.JPEG
Exported /Users/rhet/Desktop/export/IMG_0102.AAE
Exporting edited version of IMG_0102.JPEG (76F5188C-E304-49EF-A8A3-34B56380E76B.jpeg)
Exported IMG_0102.JPEG to /Users/rhet/Desktop/export/IMG_0102_edited.jpeg
Exported /Users/rhet/Desktop/export/IMG_0102_edited.jpeg
Exporting IMG_0102.HEIC (4FE579D2-888D-49A0-89E5-178600088399.heic) (2/3)
Exported IMG_0102.HEIC to /Users/rhet/Desktop/export/IMG_0102 (1).HEIC
Exported adjustments for IMG_0102.HEIC to /Users/rhet/Desktop/export/IMG_0102 (1).AAE
Exported /Users/rhet/Desktop/export/IMG_0102 (1).HEIC
Exported /Users/rhet/Desktop/export/IMG_0102 (1).AAE
Exporting edited version of IMG_0102.HEIC (4FE579D2-888D-49A0-89E5-178600088399.heic)
Exported IMG_0102.HEIC to /Users/rhet/Desktop/export/IMG_0102_edited (1).heic
Exported /Users/rhet/Desktop/export/IMG_0102_edited (1).heic

So for edited images we end up with:

IMG_0102.JPEG
IMG_0102.JPEG
IMG_0102_edited.jpeg

and

IMG_0102 (1).HEIC
IMG_0102 (1).AAE
IMG_0102_edited (1).heic

Unfortunately `osxphotos import doesn't properly handle the edited version in the second case:

Processing import group with .AAE file with edited version: IMG_0102.JPEG, IMG_0102.AAE, IMG_0102_edited.jpeg
Importing /Users/rhet/Desktop/export/IMG_0102.JPEG, /Users/rhet/Desktop/export/IMG_0102.AAE, /Users/rhet/Desktop/export/IMG_0102_edited.jpeg (1/3)
Imported IMG_0102.JPEG with UUID E2C05F9F-A9CC-457D-A354-841423A77693
Processing import group with .AAE file: IMG_0102 (1).HEIC, IMG_0102 (1).AAE
Importing /Users/rhet/Desktop/export/IMG_0102 (1).HEIC, /Users/rhet/Desktop/export/IMG_0102 (1).AAE (2/3)
Imported IMG_0102 (1).HEIC with UUID AC79EEFF-9CF2-4139-9F7C-B2E3C424B8F1
Processing import group: IMG_0102_edited (1).heic
Importing /Users/rhet/Desktop/export/IMG_0102_edited (1).heic (3/3)
Imported IMG_0102_edited (1).heic with UUID 5DFD51F7-2527-49B3-8F41-6D19604E2309

Note how IMG_0102_edited (1).heic was not detected as the edited image for MG_0102 (1).HEIC and was imported separately. I'll open a new issue for this. I thought this might be a regression in v0.75.0 which changed import logic to handle a special case for portrait images (where the original can also have an AAE file) but I just tested with 0.74.0 and it is broken there too.

To help you solve this I need to narrow down one specific issue at a time.

1. Can you confirm that the duplicate file names are ending up in the same directory upon export?
2. If so are you seeing the (1), (2) naming scheme or are you really seeing IMG_1234.JPG, IMG_1234.HEIC in the same export directory?
3. When you run import with --verbose, what is the error output you're seeing?
4. Are the photos importing at all or just importing without the edited adjustments?
5. Does this happen only on photos with AAE files or are you getting errors associated with duplicate images that haven't been edited?

[rwjack]

Can you confirm that the duplicate file names are ending up in the same directory upon export?

1. The duplicates (by exact name) don't seem to be in the same directory, they end up in their own date directory

---

If so are you seeing the (1), (2) naming scheme or are you really seeing IMG_1234.JPG, IMG_1234.HEIC in the same export directory?

2. I'm not seeing a whole lot of (1)'s (maybe 15~) - likely because 1. is false - but where I do see them, these are exact copies so IMG_1234.MOV and a copy probably renamed as IMG_1234 (1).MOV during the export (again, unsure how these duplicates came to be, I for sure know I don't have 2x of any these in my old library, nor were they edited). Also I notice that these (1)'s are always a bit smaller in size than the original, maybe 80-90% of the original size.

---

When you run import with --verbose, what is the error output you're seeing?

3. No sidecar file found for X - 18 out of 23 failures - These were .mov and there truly isn't a sidecar for these - also to note, all these come in the form of jpg (and xmp) + heic (and xmp) + mov (no xmp) - I'm thinking something definitely got funked up with live photos, as 100% of these errored out ones date back to my iphone 7 library - but again, inconclusive because there's many other live photos from the iphone 7 era that imported without issue.

---

Are the photos importing at all or just importing without the edited adjustments?

4. Out of the 23 ones that show an error, I found that 21 were imported as their final edited version

---

Does this happen only on photos with AAE files or are you getting errors associated with duplicate images that haven't been edited?

5. Inconclusive

[RhetTbull]

No sidecar file found for X - 18 out of 23 failures - These were .mov and there truly isn't a sidecar for these - also to note, all these come in the form of jpg (and xmp) + heic (and xmp) + mov (no xmp) - I'm thinking something definitely got funked up with live photos, as 100% of these errored out ones date back to my iphone 7 library - but again, inconclusive because there's many other live photos from the iphone 7 era that imported without issue.

To clarify, you have exported photos that look like this in the same directory?

IMG_1234.jpg
IMG_1234.jpg.xmp
IMG_1234.heic
IMG_1234.heic.xmp
IMG_1234.mov

If so, this is a situation that OSXPhotos definitely does not handle. This shouldn't be possible as OSXPhotos would deconflict the filenames unless the live image really had both JPG and HEIC images and I'm not sure how osxphotos export would handle this.

[rwjack]

Exactly, it's very confusing, I don't get at all why I'm getting both the jpg and the heic during the export, when it's just a single live photo originally - not to mention the fact that the import did actually manage to successfully stitch these Frankenstein live photos back together.

[rwjack]

Now if there's an edit, the same thing happens with the AAE files (but I noticed a bug with these on import...more later)

I'd be willing to do the export / import procedure one more time, before I'm finally done with the Tahoe partition - Just so we can try to keep the edits and the originals, otherwise, we could just call it a day. Both of you have been of great help in any case!