The PR adds parallel_cosine_similarity to find_duplicates which is set to True by default so that it won't break any existing code. I have a really huge dataset and not enough RAM to use multiprocessing so introducing the ability to disable parallel computation of cosine similarity was the only way to use the package.

Not sure if this is a recent issue, or related to my using an M1 Mac. Nevertheless, the tail of a very long traceback is below:

ERROR: Cannot install imagededup==0.0.1, imagededup==0.0.2, imagededup==0.0.3, imagededup==0.0.4 and imagededup==0.1.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    imagededup 0.1.0 depends on tensorflow==2.0.0
    imagededup 0.0.4 depends on tensorflow==2.0.0
    imagededup 0.0.3 depends on tensorflow==1.13.1
    imagededup 0.0.2 depends on tensorflow==1.13.1
    imagededup 0.0.1 depends on tensorflow==1.13.1

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

The code below always returns empty duplicates when two files are compared - regardless PNG files are same or not. (Python 3.8, OSX Catalina)

import sys
import os

from imagededup.methods import PHash

if __name__ == '__main__':
    hasher = PHash()
    image_map = {}

    for i in range(1,3):
        if not os.path.exists(sys.argv[i]):
            sys.exit(sys.argv[i] + " not found")
        image_map[sys.argv[i]] = hasher.encode_image(image_file=sys.argv[i])
    
    
    duplicates = hasher.find_duplicates(
        encoding_map=image_map,
        max_distance_threshold=0,
        scores=True)

    print(duplicates)

I can see why you may think you need to lock these down to exact versions, but in general for a library like this it's better to keep them more liberal for several reasons:

1. Locking gives a false sense of security - it doesn't actually lock the whole tree, so a release of any of the dependencies of these packages could cause a break. You need to use a tool like Pipenv, poetry or pip-tools to lock the entire dependency tree.

2. This is a library, you need to ensure that it fits in nicely with larger applications that may well have similar dependencies. By locking them to exact versions you are going to cause dependency issues as their specifiers may conflict (i.e Pillow >6.0.0)

3. It's conceptually wrong - you're saying that this library only works with Pillow ==6.0.0. Not 6.0.1, which could be a bugfix release that fixes a bunch of issues.

Testing dependencies, like pytest and others, should also not be locked. They are not critical to the application as a whole, and it's fairly obvious if they break. Which they won't.

I forked the repository and I worked on the "dev" branch and while running the command:

python setup.py install

I got this error message:

Processing dependencies for imagededup==0.1.0
Searching for tensorflow~=2.0.0
Reading https://pypi.org/simple/tensorflow/
No local packages or working download links found for tensorflow~=2.0.0

I have a LOT of images (roughly 1/3 of my entire personal library) that register in PIL as MPO. Your code barfs on all of them, but changing image_utils.py line 15 to IMG_FORMATS = ['JPEG', 'PNG', 'BMP', 'MPO'] fixes this.

I didn't want to submit a PR in case you already know this and it causes a headache somewhere else. If not, please change this.

I have several errors during the execution of this project, the main error is that after launched "pip install imagededup" command, this error comes out:

I already looked at all other issues but no one can solve this problem

Running python setup.py install on the dev branch fails.

Tensorflow and numpy aren't playing well with each other. Leave tf>1.0 in setup.py with no mention of numpy (i.e., relying on tf to get numpy) leads to an error since tf 2.4.1 gets installed along with numpy 1.20.1 (due to pip resolver algo), but tf 2.4.1 needs numpy=~1.9.2. Explicitly mentioning numpy <1.20.0, makes the installation work. Additionally, newer versions of numpy (1.20.1), scipy (1.6 onwards) and matplotlib do not support python 3.6 anymore. The changes proposed in this PR will also work from Python 3.7 onwards.

Getting this error, cannot identify image file 'filename.png' 2021-01-17 20:12:33,709: WARNING Invalid image file filename.png: in v0.2.4 for .png file

Hi there, Thanks for your repo. To use encode_image(image_array=img_array) method, must the input numpy image array format be as BGR or RGB (because for example OpenCV default format is BGR, but Pillow is RGB)? Best

Currently we have failing Linux tests because we rely on the order of how images are loaded which varies across OSs. We don't care in which order images are loaded so we shouldn't test for it, that's why I removed the lines from tests/test_hashing.py

We also have a failing test for macOS Python 3.6 on Azure pipelines which was not reproducible on my MacBook but was fixed by initialising a new CNN object for the failing test.

Hey!

I believe most duplicates are created through compression. For example, if I upload an image to a different service, I re-download it. It's usually compressed, and its metadata may have changed. I have many duplicates from various platforms like Google, Facebook, etc.

I haven't seen anything in the documentation about how this repository handles compression. Will it be able to recognize duplicate images with various levels of compression?

Thanks!

WHAT/WHY

Introduces new optional multiprocessing parameters to several methods:

• num_enc_workers - Change number of processes to generate encodings (Addresses #156)
• num_sim_workers/num_dist_workers - Change number of processes to compute similarity/distances (Addresses #95, #113)

HOW

APIs impacted

For both CNN as well as Hashing functions, the following user-facing api calls get new parameters-

• encode_images
• find_duplicates
• find_duplicates_to_remove

Choice of default values

• num_enc_workers: For CNN methods, this is set to 0 by default (0=disabled). Furthermore, parallelization of CNN encoding generation is only supported on linux platform. This is because pytorch requires the call to Dataloader to be wrapped within an if ___name__ == '__main__' construct to enable multiprocessing on non-linux platforms (due to the difference between fork() vs spawn() call used for multiprocessing on different systems). Such a construct does not fit well with the current code structure. For Hashing methods, this parameter is set to the cpu count of the system. The default values preserve backward compatibility.
• num_dist_workers/num_sim_workers: Set to cpu count of the system. The default values preserve backward compatibility.

Hello @tanujjain / @clennan / @datitran, and https://github.com/idealo ,

Kudos to you for bringing up imagededup. I worked on developing a simple streamlit based webapp on the same and I think it will be fruitful to have it as a part of README here as the motivation behind developing this came from your work 😄! The entire webapp codebase has also been added as a separate folder while the main README.md of the project has been updated with the same.

You can find the entire webapp source-code in the stream_app directory of the repo.

Happy opensourcing!

Cheers, Prateek

What I would like to achieve I about the following:

EXISTING_HASHES: set = set()
def is_duplicate(img_bytes: bytes):
    if get_hash(img_bytes) in EXISTING_HASHES:
        return True
    return False

def main():
    image_bytes = get_new_image()
    if is_duplicate(image_bytes)
        return

    with open(file) as f:
       f.write(image_bytes)
        

Hello,

I got a problem when I tried to find duplicating image in a dataset of 40000 images.

I have already tried this solution but it didn't work https://github.com/idealo/imagededup/issues/95

Do you know how to fix it ? Thank you in advance.

Best regard