Cleaning up years of duplicate photos turned into my first real Rust project, and some of the implementation ended up being fun enough to share.
The core is a custom 127-bit dHash that tracks horizontal and vertical luminance gradients, compared by Hamming distance. The whole hash fits in a u128 so comparison is just XOR + count_ones, basically free. Threshold of 10/127 catches WhatsApp recompression and resizes without grouping photos that are merely similar.
That approach can't catch retakes though — same scene, camera moved an inch, completely different pixel layout. For those I added an optional mode that runs a local CLIP model and groups by cosine similarity of the embeddings. Calibrating the cutoff was interesting: true retakes score 0.90+, unrelated photos stay under 0.65, so there's a comfortable gap.
For Synology libraries it skips downloading originals entirely and hashes ~10KB server-generated thumbnails from the DSM API, with rayon fanning the fetch+hash out across cores.
The review UI is tiny_http on localhost with per-run CSRF tokens and Host-header validation, because a tool that deletes files really shouldn't take instructions from random web pages (DNS rebinding is sneaky).
submitted by /u/Musician_Useful to r/rust
[link] [comments]
Source: r/rust · by /u/Musician_Useful