Hash-Based Deduplication Replaces Filename Checks
n8n has released a workflow template that automatically detects and manages duplicate files in Google Drive using content-based hashing rather than filename comparison. The workflow monitors a designated folder, generates MD5 hashes of uploaded files, and cross-references them against a Supabase database to identify duplicates.
When a duplicate is detected, the workflow automatically moves the file to a dedicated Duplicates folder and sends a notification via Slack. All events including unique uploads, duplicates, race conditions, and errors are logged in an audit table.
The template addresses a common pain point for teams managing high file volumes. Traditional duplicate detection relies on filename matching, which fails when users rename files or when multiple versions of the same content exist under different names. By comparing file content through MD5 hashes, the workflow identifies true duplicates regardless of naming conventions.
Technical Implementation and Setup Requirements
The workflow integrates several services to deliver automated deduplication:
- Google Drive monitoring: Watches a specified folder for new file uploads
- Binary content processing: Downloads and normalizes file metadata, then generates MD5 hashes from the binary content
- Supabase database: Stores file hashes in a
file_hashestable and maintains an audit log indedup_audit_log - Conditional logic: Routes files based on hash lookup results
- Automated file management: Moves duplicates to a designated folder
- Slack notifications: Alerts team members when duplicates are found
Setup requires existing accounts for Google Drive, Supabase, and Slack. Users must create two tables in Supabase (file_hashes and dedup_audit_log) and configure the Google Drive folder ID for duplicate storage. According to the workflow template page, setup typically takes 10 to 15 minutes with a prepared Supabase project.
The workflow includes error handling and logging for edge cases including race conditions, where multiple files might be processed simultaneously.
Workflow Automation for Storage Management
The template joins n8n's growing library of pre-built workflows designed for technical teams. Content-based deduplication becomes particularly valuable as organizations accumulate large file repositories across distributed teams. Unlike manual periodic cleanup, the workflow operates continuously, preventing duplicate accumulation rather than addressing it retroactively.
The MD5 hashing approach means that even files with identical content but different names, formats, or metadata will be correctly identified as duplicates. This provides more reliable storage optimization than simple filename comparison, which often misses duplicates or requires strict naming conventions.
Teams can access the workflow template through n8n's workflow library. The open-source nature of n8n, as we previously covered, allows organizations to self-host the automation platform and maintain control over file processing and metadata.
The workflow template is available at n8n.io/workflows and requires active Google Drive, Supabase, and Slack integrations to function.
Sources
1 checkedHow we cover tool news: Create With's tool desk drafts these reports with AI from the sources listed above and checks them against those sources before publishing.







