阅读中文版本

Scrub PII from a Dataset with pandas

Detect and pseudonymise direct identifiers in a dataframe before sharing it with analysts.

import re, pandas as pd
patterns = {'email': r'[^@\s]+@[^@\s]+', 'phone': r'\+?\d{7,15}'}
for name, pat in patterns.items():
    hits = df.astype(str).apply(lambda c: c.str.match(pat).mean())
    print(name, hits[hits > 0.5].index.tolist())

import hashlib
def pseudonym(value, salt):
    return hashlib.sha256((salt + str(value)).encode()).hexdigest()[:16]
df['email'] = df['email'].map(lambda v: pseudonym(v, salt))

What this recipe does

Scan a dataframe for PII-like columns and replace direct identifiers with stable pseudonyms.

An analyst requests a dataset containing emails and phone numbers, which must not leave the trusted zone raw.

Prerequisites

• pandas and optionally a PII library such as presidio
• A dataset to de-identify
• A mapping policy for pseudonyms

Procedure

1. Detect likely PII columns by pattern.

import re, pandas as pd
patterns = {'email': r'[^@\s]+@[^@\s]+', 'phone': r'\+?\d{7,15}'}
for name, pat in patterns.items():
    hits = df.astype(str).apply(lambda c: c.str.match(pat).mean())
    print(name, hits[hits > 0.5].index.tolist())

Expected result: Columns dominated by PII patterns are listed for treatment.

2. Replace direct identifiers with stable pseudonyms.

import hashlib
def pseudonym(value, salt):
    return hashlib.sha256((salt + str(value)).encode()).hexdigest()[:16]
df['email'] = df['email'].map(lambda v: pseudonym(v, salt))

Expected result: Emails become stable but irreversible pseudonyms that still join across tables.

How to confirm it worked

Run the commands above and confirm every expected result in order. If a step does not produce the stated output, use the troubleshooting notes before continuing. A passing run shows a scan of the output finds no raw emails or phones, and pseudonyms remain stable across runs with the same salt.

Troubleshooting

Pseudonyms reversible via join — Use a secret salt and avoid publishing the mapping; consider tokenisation.

Missed free-text PII — Scan free-text columns with a PII detector, not just exact column names.

Limitations

• Pattern scanning misses unusual formats.
• Pseudonymisation is not anonymisation.
• Free-text columns need dedicated detection.