How to Dedupe an Upload by Content Hash

Introduction

The hardest part of skip duplicate uploads is not writing the code — it is deciding what the code is allowed to reject. A content hash setup that accepts everything is easy to ship and expensive to operate, because every malformed input becomes a production surprise. The discipline is to enumerate the shapes the system actually receives, encode them in content hash, and let the library reject the rest loudly. This keeps the happy path fast and the failure path visible.

A content hash guard for skip duplicate uploads should be proportional to the risk. A ten-field form does not need a hundred-line schema; it needs the five checks that catch the five mistakes the users actually make. Ship the five, measure the rejections, and add the sixth only when the traffic proves it exists. Speculative generality is the bug you write on purpose, and content hash makes it easy to write, which is why discipline matters.

The sections below walk through the concrete decisions: where to put the content hash check, how to keep it fast under load, what to do when the contract changes, and how to test the failure paths so they degrade on purpose instead of by accident.

Compute the hash

Compute the hash is the first decision because it determines what the rest of the content hash pipeline can assume. A wrong choice here turns into skip duplicate uploads under load, and the invariant you thought content hash guaranteed no longer holds. Measure the impact before committing, because the cost of a bad choice is paid in incidents, not in a code review.

The test for Compute the hash is the one most teams skip: drive content hash to the failing condition and assert the system degrades gracefully. Write it with a fake clock for time-based logic, a stub for the failure path, and an assertion on the user-visible outcome. A test that passes by timing out is a hope, not a test.

async function hash(file) {
  const buf = await file.arrayBuffer();
  const h = await crypto.subtle.digest('SHA-256', buf);
  return [...new Uint8Array(h)].map(b => b.toString(16).padStart(2,'0')).join('');
}

The javascript above is the shape to ship for skip duplicate uploads. It is small enough to audit and specific enough that the failure behavior is obvious. Adapt the constants to your traffic and latency budget, and resist the urge to handle cases the system has not seen yet.

Check the store

The mechanism behind Check the store is straightforward in content hash, but the consequences ripple. A wrong choice here turns into skip duplicate uploads under load, and the invariant you thought content hash guaranteed no longer holds. Measure the impact before committing, because the cost of a bad choice is paid in incidents, not in a code review.

The test for Check the store is the one most teams skip: drive content hash to the failing condition and assert the system degrades gracefully. Write it with a fake clock for time-based logic, a stub for the failure path, and an assertion on the user-visible outcome. A test that passes by timing out is a hope, not a test.

Skip if exists

Skip if exists is where most content hash implementations leak performance. A wrong choice here turns into skip duplicate uploads under load, and the invariant you thought content hash guaranteed no longer holds. Measure the impact before committing, because the cost of a bad choice is paid in incidents, not in a code review.

The test for Skip if exists is the one most teams skip: drive content hash to the failing condition and assert the system degrades gracefully. Write it with a fake clock for time-based logic, a stub for the failure path, and an assertion on the user-visible outcome. A test that passes by timing out is a hope, not a test.

test('same content same hash', async () => {
  expect(await hash(f1)).toBe(await hash(f2));
});

Pin the behavior with this test. A content hash setup that passes this test degrades on purpose instead of by accident, which is the difference between a managed failure and an incident.

Store the mapping

The hard part of Store the mapping is not the happy path — it is what happens when content hash fails halfway. A wrong choice here turns into skip duplicate uploads under load, and the invariant you thought content hash guaranteed no longer holds. Measure the impact before committing, because the cost of a bad choice is paid in incidents, not in a code review.

The test for Store the mapping is the one most teams skip: drive content hash to the failing condition and assert the system degrades gracefully. Write it with a fake clock for time-based logic, a stub for the failure path, and an assertion on the user-visible outcome. A test that passes by timing out is a hope, not a test.

Handle hash collisions

Handle hash collisions should be observable without a debugger. A wrong choice here turns into skip duplicate uploads under load, and the invariant you thought content hash guaranteed no longer holds. Measure the impact before committing, because the cost of a bad choice is paid in incidents, not in a code review.

The test for Handle hash collisions is the one most teams skip: drive content hash to the failing condition and assert the system degrades gracefully. Write it with a fake clock for time-based logic, a stub for the failure path, and an assertion on the user-visible outcome. A test that passes by timing out is a hope, not a test.

Test dedup

Test dedup is a trade, not a winner. A wrong choice here turns into skip duplicate uploads under load, and the invariant you thought content hash guaranteed no longer holds. Measure the impact before committing, because the cost of a bad choice is paid in incidents, not in a code review.

The test for Test dedup is the one most teams skip: drive content hash to the failing condition and assert the system degrades gracefully. Write it with a fake clock for time-based logic, a stub for the failure path, and an assertion on the user-visible outcome. A test that passes by timing out is a hope, not a test.

Common pitfalls

The common pitfalls in skip duplicate uploads all reduce to one mistake: treating content hash as a formality instead of a contract. The contract is the thing the rest of the system relies on, and a weak contract propagates weakness everywhere.

• Skipping the content hash boundary so skip duplicate uploads enters unvalidated.
• Assuming content hash is fast without measuring the tail latency under load.
• Letting skip duplicate uploads fail silently instead of failing loudly with a metric.
• Testing only the happy path so the content hash failure surfaces in production.
• Sharing mutable state across requests so one bad input poisons the next.

Conclusion

Do not conflate the conclusion with the implementation. They look similar from the outside but the failure modes differ: content hash fails fast and the caller knows, while an unchecked path fails slow and the caller guesses. Choose the one whose failure mode your system can absorb, and document the choice at the call site.

Self-check: write a test that drives content hash to the failing condition and assert the system degrades gracefully. Then skip duplicate uploads is a property you maintain, not a feature you hope for.

Sources: SubtleCrypto, Content-addressable storage

Report a problem with this articleOpens a pre-filled issue on GitHub.