Find duplicated files (of the same md5) within a folder using awk.

find_dup.awk

#!/usr/bin/env -S awk -f

# Example usage:
# `fd -tf -x md5 -r | find_dup | jq -s`
# (the example assume you have `fd` and `jq`, which is not required for this script.

{
    md5=$1;                         # md5 is expected as 1st argument
    $1="";                          # shift
    gsub(/^[ \t]+|[ \t]+$/, "");    # trim
    name=$0;                        # filename is expected as 2nd argument

    files[md5] = md5 in files \
        ? files[md5] "\",\"" name \
        : name;
    count[md5]++;
} END {
    for ( md5 in files )
        if ( count[md5] > 1 )       # wrapped duplicates in json format
            printf "{\"MD5\": \"%s\", \"files\":[\"%s\"]}\n", md5, files[md5]
}

~README.md

This gist helps to find all the files of the same md5 in a folder you can save the awk above as a executable on your path such as ~/.local/bin/find_dup

chmod +x ~/.local/bin/find_dup

then you can run the script below:

find . -type f -exec md5 -r {} \; | find_dup | jq -s

This assumes you are on mac with md5 instead of md5sum, and have jq installed.

Also recommmend fd, which is much faster and respect your .gitignore.

fd -tf -x md5 -r | find_dup | jq -s