阅读中文版本

Optimize Array Deduplication with a Set

Replace O(n squared) deduplication loops with a hash set so large arrays dedupe in linear time.

function dedupe(items) {
  const seen = new Set();
  const out = [];
  for (const item of items) {
    if (seen.has(item)) continue;
    seen.add(item);
    out.push(item);
  }
  return out;
}

console.time('set'); dedupe(bigArray); console.timeEnd('set');

What this recipe does

Convert a quadratic dedup loop into a set-based linear pass.

Deduplicating a large array with nested loops becomes the slowest part of a data import.

Prerequisites

• JavaScript, Python, or a comparable runtime
• An array with many items
• A benchmark or profiler

Procedure

1. Replace the nested scan with a set membership check.

function dedupe(items) {
  const seen = new Set();
  const out = [];
  for (const item of items) {
    if (seen.has(item)) continue;
    seen.add(item);
    out.push(item);
  }
  return out;
}

Expected result: Deduplication runs in O(n) average time instead of O(n squared).

2. Benchmark against the original implementation.

console.time('set'); dedupe(bigArray); console.timeEnd('set');

Expected result: The set version completes dramatically faster on large inputs.

How to confirm it worked

Run the commands above and confirm every expected result in order. If a step does not produce the stated output, use the troubleshooting notes before continuing. A passing run shows the deduplicated output is identical to the slow version and the benchmark shows a large speedup.

Troubleshooting

Object identity differs — For objects, dedupe by a stable key string rather than reference equality.

Order changes — The set-based pass preserves first-occurrence order; verify it matches expectations.

Limitations

• Assumes hashable keys.
• Memory grows with unique items.
• Does not handle fuzzy duplicates.