Someone asked what the video deduplication and “pseudo-original” content space looks like right now. I went through the whole chain from the beginning—how platforms detect duplicates, what techniques commercial deduplication services use, whether open-source repos can reproduce them, and whether this can be turned into a product. Below is what I found, organized by layer and experimental data. I’ll leave the trade-offs to readers to judge for themselves.

Platform duplicate detection is not one layer, but six

Mature platforms use a six-layer cascade for duplicate detection. If any layer hits, the content is flagged as duplicate:

1. File layer: Full-file MD5/SHA1 hashes, plus container metadata and encoder fingerprints. Re-encoding changes this.
2. Visual layer: Extract keyframes, either at scene cuts or fixed intervals, then calculate pHash/dHash and compare Hamming distance against a threshold. This is what most people think “duplicate detection” means.
3. Temporal layer: Frame-sequence fingerprints, motion trajectories, and scene-cut rhythms.
4. Audio layer: Spectral peak fingerprints, similar to Shazam-style Chromaprint, plus ASR to convert speech into text for comparison.
5. Deep learning layer: CNN features such as ResNet, CLIP embeddings, and cosine similarity.
6. Cross-modal layer: Visual + audio + temporal voting and fusion for an overall judgment.

At the moment, Douyin, Bilibili, and Video Channel have already reached the fifth layer. Bilibili’s duplicate-collision system uses self-supervised ResNet50 + FAISS vector search; Video Channel uses the CVPR 2023 dual-track winning solution + Tencent Cloud Milvus; YouTube Content ID has evolved from its early Waveprint approach, based on wavelets + MinHash + LSH, to what it is today. In other words, platform duplicate detection is no longer just hash comparison—semantic models are already in use.

Techniques used by commercial deduplication services

Common techniques used by “deduplication services” on the market include: re-encoding, horizontal mirroring, RGB channel shifting, slight jitter and speed changes, mixed frame dropping and frame insertion, picture-in-picture mask overlays with low opacity, edge cropping and scaling, color filters, audio pitch shifting, metadata cleaning, and remixing/reordering clips.

These techniques mainly operate at the pixel layer. Their effectiveness varies across different duplicate-detection layers: visual hashes such as pHash are sensitive to pixel-level changes, so these techniques can alter hash values; CNN/CLIP, however, extract semantic features and are less sensitive to transformations that only change pixels without changing the content. The experiment below measures this difference.

A controlled experiment

I took the same test video, applied a horizontal mirror transformation, and then compared “original vs mirrored” using two different types of fingerprints:

• pHash (visual layer): Original hash f6c3f6965c00f600, mirrored hash f63ef6b6a300f600, Hamming distance 16/64—pHash judged them to be different.
• Semantic layer (ResNet/CLIP): Cosine similarity 0.94—the semantic layer judged them to be near-duplicates.

Note: The actual CLIP weights could not be loaded on this machine because HuggingFace was unreachable, so the script automatically fell back to ResNet50. It is still a semantic-layer feature extractor, and the conclusion is directionally the same. In a networked environment, the script will automatically use real CLIP.

The same mirroring transformation produced different judgments from the two fingerprints: pHash considered them different, while the semantic layer treated them as near-duplicates. If we expand the scope to techniques such as re-encoding, RGB shifting, color grading, cropping, masks, and frame mixing, they all change pixels without changing semantics. As a result, their effects also differ between the semantic layer and the pHash layer.

How duplicate-detection mechanisms are evolving

The evolution path from 2023 to 2026 looks like this: pHash → CNN → CLIP multimodal → visual + audio + semantic multi-channel detection in 2025. Possible future additions include: audio ASR comparison, since pitch shifting can change the spectrum but not the semantics of the transcribed text; cross-account graph-based association, which looks not only at individual videos but also at reposting account networks; and AIGC/synthetic-content watermark detection, targeting AI-generated content at scale. The overall direction is that all layers are gradually tightening. As for which technique works at which point in time, that depends on each platform’s iteration pace.

Several possible directions

If this technology is turned into a product, there are roughly three directions, each with its own technical and legal profile:

1. Reposting / pseudo-original tools: Technically, pixel-layer methods are effective against pHash but have limited effect against semantic layers. Legally, they involve reproduction rights, the right of communication through information networks, and adaptation rights. Platform rules also explicitly restrict this behavior, creating both infringement and compliance risks. This direction is constrained both technically and legally.

2. Self-deduplication / asset-library management: This manages one’s own assets and is legal. Technically, pHash can be used for initial recall, CLIP for reranking, and FAISS/Milvus for billion-scale vector search; the whole pipeline can be run locally. Demand comes from content agencies, MCNs, and independent media operators that need to deduplicate their own materials, and willingness to pay is relatively clear.

3. Theft detection / copyright enforcement: This uses the same technology as direction 2, but in reverse, helping copyright owners discover stolen content. It is legal and has social value. The challenge is customer acquisition: copyright owners are fragmented, willingness to pay is relatively low, and the product needs to connect with platform complaint channels.

The same fingerprinting technology sits in very different technical and legal positions depending on whether it is used in the “forward” direction, such as self-deduplication and theft detection, or in the “reverse” direction, helping reposting. Which direction to take depends on each party’s own trade-offs.

Full report

This article is the condensed version. The full research report includes 10 runnable ffmpeg scripts, Python fingerprinting code, a list of 15 GitHub repositories verified through APIs, a layered architecture diagram of platform duplicate detection, a self-built pipeline architecture diagram, and literature sources. It is stored in a private repo; we can discuss it further if you need the details.