阅读中文版本

Build an Environment Variable Manager CLI

A CLI that parses .env files with full quoting rules, merges them with process environment, validates against a schema, and detects leaked secrets before they reach git.

envm/
├── bin/envm.js
├── src/
│   ├── parser.js
│   ├── merge.js
│   ├── schema.js
│   └── scan.js
├── test/
│   ├── parser.test.js
│   └── scan.test.js
├── env.schema.json
└── package.json

function parseLine(line) {
  let i = 0;
  while (i < line.length && /\s/.test(line[i])) i++;
  if (i >= line.length || line[i] === '#') return null;
  if (line.startsWith('export ', i)) i += 7;
  const key = /^([A-Za-z_][A-Za-z0-9_]*)/.exec(line.slice(i));
  if (!key) throw new Error('invalid key at: ' + line);
  i += key[1].length;
  while (i < line.length && /\s/.test(line[i])) i++;
  if (line[i] !== '=') throw new Error('expected = after key ' + key[1]);
  return { key: key[1], value: parseValue(line, i + 1) };
}

function merge(layers) {
  const out = { values: {}, origins: {} };
  for (const { name, values } of layers) {
    for (const [k, v] of Object.entries(values)) {
      out.values[k] = v;
      out.origins[k] = name;
    }
  }
  return out;
}

for (const rule of schema.required) {
  const v = merged.values[rule.key];
  if (v === undefined) {
    errors.push({ key: rule.key, error: 'missing required variable' });
    continue;
  }
  if (rule.type === 'number' && Number.isNaN(Number(v))) {
    errors.push({ key: rule.key, error: 'expected number, got ' + JSON.stringify(v) });
  }
}

const PATTERNS = [
  { name: 'aws-access-key', re: /AKIA[0-9A-Z]{16}/ },
  { name: 'pem-private-key', re: /-----BEGIN (RSA |EC |OPENSSH )?PRIVATE KEY-----/ },
  { name: 'hex-token', re: /[a-f0-9]{40,}/i },
  { name: 'github-token', re: /gh[pousr]_[A-Za-z0-9]{36,}/ },
];

Project overview

Environment files are deceptively simple until quoting, escapes, and multi-line certificates arrive, and until the one variable everyone forgot to set reaches production. This project builds the tool that makes both failure classes boring: a parser that handles the real grammar, and a check command that fails in CI instead of at 2am.

The second half is about the mistake that makes headlines: committing secrets. A scanner over staged files catches the AWS key before it lands in history, where rotation is the only real remedy afterwards.

The finished tool delivers five capabilities:

• Parse .env files with single quotes, double quotes, escape sequences, comments, export prefixes, and multi-line quoted values.
• Merge layers — file, process environment, CLI flags — with documented precedence and a per-key origin report.
• Validate against a schema declaring required keys, types, and allowed ranges; check exits nonzero with precise errors.
• Scan for secrets — staged files containing key-shaped values like AKIA tokens, PEM blocks, and long hex strings are flagged with file, line, and a masked preview.
• Print in two formats — print --format json|shell emits values safe to eval or consume programmatically.

The architecture is four small modules over one core: a tokenizer handles quoting and escapes character by character, a parser turns tokens into an ordered key-value map preserving later-wins semantics, a merger folds layers with origin tracking, and command modules expose parse, check, print, and scan over the same core. The secrets scanner deliberately works on file bytes rather than parsed values, because its job is catching what would leak, not what is already loaded.

Setup

Verify the prerequisites before starting: Node.js 20 LTS or later, familiarity with dotenv conventions and shell quoting, and basic regex comfort for the secret-pattern detector.

The dependency footprint is one runtime package plus the test runner:

Create the project and install:

mkdir envm && cd envm
npm init -y
npm install commander@^12.0.0
npm install -D vitest@^2.1.0

Configure the environment in package.json — the bin entry, the test script, and nothing else; the tool reads .env and env.schema.json from the working directory by convention:

{
  "name": "envm",
  "bin": { "envm": "bin/envm.js" },
  "scripts": { "test": "vitest run" }
}

The directory structure mirrors the core: parser.js and merge.js are pure functions, schema.js and scan.js own their own I/O, and the CLI composes them.

Phase 1: Env file tokenizer and parser

The parser works line by line with a character-level state machine handling quotes, escapes, inline comments, and multi-line quoted values.

// src/parser.js
export function parseLine(line) {
  let i = 0;
  while (i < line.length && /\s/.test(line[i])) i++;
  if (i >= line.length || line[i] === '#') return null;
  if (line.startsWith('export ', i)) i += 7;
  const key = /^([A-Za-z_][A-Za-z0-9_]*)/.exec(line.slice(i));
  if (!key) throw new Error('invalid key at: ' + line);
  i += key[1].length;
  while (i < line.length && /\s/.test(line[i])) i++;
  if (line[i] !== '=') throw new Error('expected = after key ' + key[1]);
  return { key: key[1], value: parseValue(line, i + 1) };
}

function parseValue(line, start) {
  const out = [];
  let i = start;
  while (i < line.length) {
    const c = line[i];
    if (c === "'") {            // single quotes: verbatim until close
      const close = line.indexOf("'", i + 1);
      out.push(line.slice(i + 1, close));
      return { value: out.join(''), rest: close + 1 };
    }
    if (c === '"') return parseDoubleQuoted(line, i);   // may span lines
    if (c === '#' && out.length === 0) return { value: '', rest: i }; // trailing comment
    if (c === '\\' && line[i + 1] !== undefined) { out.push(line[i + 1]); i += 2; continue; }
    out.push(c);
    i++;
  }
  return { value: out.join(''), rest: i };
}

The parser is a state machine rather than a regex because the grammar has states: inside single quotes nothing escapes, inside double quotes backslash escapes work and interpolation does not, and a # only starts a comment where a value has not begun. Multi-line double-quoted values are consumed across physical lines, joining with real newline characters — which is exactly how PEM keys and JSON web tokens actually appear in env files, and the feature most hand-rolled parsers silently break.

When parsing fails, the error carries the offending line and a reason — “expected = after key DATABASE_URL” — because a parser that says “invalid file” is a parser people stop using and rerun by hand.

Acceptance for this phase: parser tests pass for every quoting form, escaped characters, trailing comments outside quotes, and multi-line double-quoted values joined with real newlines.

Phase 2: Layered merge with origins

The merger folds parsed files, the process environment, and CLI key=value flags into one map plus per-key origin tracking.

// src/merge.js
export function merge(layers) {
  const out = { values: {}, origins: {} };
  for (const { name, values } of layers) {
    for (const [k, v] of Object.entries(values)) {
      out.values[k] = v;
      out.origins[k] = name;
    }
  }
  return out;
}

The fold order is the contract: file first, then process environment, then explicit CLI flags — later layers win, so a --set DATABASE_URL=... override beats the environment, which beats the file. That ordering matches how twelve-factor deployment actually composes: base file for defaults, environment for deployment differences, flags for one-off runs.

Origins make the merge debuggable: the question “why is the database URL wrong” has an answer command instead of an investigation, because the origin report names the winning layer per key and a key present nowhere resolves to undefined with origin null. This is the same pattern that makes the config-center project debuggable, applied here at the process level — and the implementation is deliberately thirty lines rather than a dependency, because merge-with-origins is exactly as complicated as it looks.

Acceptance for this phase: later layers win, the origin report names the winning layer per key, and absent keys resolve with null origin.

Phase 3: Schema check command

The check command validates the merged map against env.schema.json, which declares required keys, string or number types, and optional enums and ranges.

// src/schema.js
export function checkAgainst(merged, schema) {
  const errors = [];
  for (const rule of schema.required) {
    const v = merged.values[rule.key];
    if (v === undefined) {
      errors.push({ key: rule.key, error: 'missing required variable' });
      continue;
    }
    if (rule.type === 'number' && Number.isNaN(Number(v))) {
      errors.push({ key: rule.key, error: 'expected number, got ' + JSON.stringify(v) });
    }
    if (rule.enum && !rule.enum.includes(v)) {
      errors.push({ key: rule.key, error: 'expected one of ' + rule.enum.join(', ') });
    }
    if (rule.min !== undefined && Number(v) < rule.min) {
      errors.push({ key: rule.key, error: 'below minimum ' + rule.min });
    }
  }
  return errors;
}

The load-bearing decision is all-errors-reported semantics: check collects every violation in one run and exits 1 with the full list, instead of failing one key at a time across three CI attempts. A deploy that is missing two variables and has a mistyped third fails once, with a machine-readable list that CI can annotate onto the exact schema lines — and the next run is green or has genuinely new information.

The schema file is deliberately JSON rather than a custom format: it diffs in review, it loads with one JSON.parse, and it can be generated from infrastructure tooling later without a parser dependency.

Acceptance for this phase: check exits 0 for a conforming environment and exits 1 listing every violated key with a reason, not just the first.

Phase 4: Secrets scanner

The scanner reads staged or given files byte-wise, matches secret-shaped patterns, and reports file, line, and a masked preview.

// src/scan.js
const PATTERNS = [
  { name: 'aws-access-key', re: /\bAKIA[0-9A-Z]{16}\b/ },
  { name: 'pem-private-key', re: /-----BEGIN (RSA |EC |OPENSSH )?PRIVATE KEY-----/ },
  { name: 'hex-token', re: /\b[a-f0-9]{40,}\b/i },
  { name: 'github-token', re: /\bgh[pousr]_[A-Za-z0-9]{36,}\b/ },
];

export function scanFiles(files, { sizeCap = 5 * 1024 * 1024 } = {}) {
  const findings = [];
  for (const file of files) {
    const stat = statSync(file);
    if (stat.size > sizeCap) continue;         // skip huge bundles
    const lines = readFileSync(file, 'utf8').split('\n');
    lines.forEach((line, i) => {
      for (const p of PATTERNS) {
        const m = line.match(p.re);
        if (m) findings.push({
          file, line: i + 1, pattern: p.name,
          preview: m[0].slice(0, 4) + '…' + m[0].slice(-4),
        });
      }
    });
  }
  return findings;
}

The scanner trades recall for precision, deliberately: word boundaries anchor every pattern, a size cap skips files where the patterns would false-positive, and the tool never attempts entropy scoring — a threshold that flags high-entropy blobs also flags every legitimate SHA hash in a lockfile, and false positives destroy trust in a secrets scanner faster than anything else. The masked preview shows enough to recognize the secret and not enough to copy it: first four characters, ellipsis, last four.

The git integration is one child invocation — git diff --cached --name-only for the file list — so scan runs as a pre-commit hook over exactly what is about to land, which is the moment interception is cheapest.

Acceptance for this phase: a fixture directory with planted secrets reports each with file, line number, pattern name, and a masked preview, while clean files pass silently.

Design trade-offs

Character-level state machine over regex parsing. A regex per line handles the eighty percent case and breaks on multi-line values, escaped quotes inside quotes, and comments after values — the exact cases that carry real secrets and real certificates. The state machine costs a hundred lines and survives the grammar.

Precision-first scanning over recall-first. An entropy-based detector catches more secret shapes; it also flags every hash in every lockfile. Anchored patterns for the four shapes that constitute the majority of real leaks keep the signal trustworthy, and the limitation is documented rather than papered over.

All-errors check over first-error check. Collecting violations costs one loop; reporting them all saves a CI round-trip per bug. The exit contract stays binary for scripting.

No interpolation by design. dotenv-style variable expansion is convenient and ambiguous — does $VAR expand inside single quotes, and what about escaping the dollar? Refusing interpolation keeps expansion semantics unambiguous, and shell users already have envsubst for the cases that need it.

Testing strategy

Run npx vitest. Parser tests are table-driven over every quoting and escaping form — single, double, escaped quotes, export prefixes, trailing comments, multi-line values, Unicode payloads — and each case pins the exact parsed value, so a regression is a one-case diff rather than a mystery. Merge tests assert precedence and origin reporting across the three-layer fold. Schema tests cover missing keys, type mismatches, enum violations, and range violations with all-errors-reported semantics.

Scanner tests use a fixture tree with planted secrets of each pattern and assert exact line numbers and masked previews, plus — the case that matters most — zero false positives on benign files: minified JavaScript, a lockfile full of hashes, a changelog. The false-positive fixtures are the tests that keep the tool trustworthy enough to stay installed.

The check command’s tests deserve one extra discipline beyond coverage: assert the exit code and the machine-readable error list shape, not just that validation failed. CI pipelines consume both — the exit code drives the gate, and the error list drives the annotation — and a refactor that keeps the exit code while mangling the list shape would break the integration silently. A two-line assertion of the full JSON structure keeps the contract whole.

Troubleshooting playbook

Values contain literal quotes after parse. PASSWORD="hunter\"2" parses with a stray backslash or an unbalanced quote. The cause is escape handling applied uniformly — but by convention single-quoted values are verbatim, so escapes only run inside double quotes. The fix is scoping the escape branch to the double-quote path, and the parser table includes both quoting forms with the same input to pin the difference.

Scanner flags minified bundles. A single-line 2MB JavaScript bundle matches the hex-token pattern in its sourcemap comment. The cause is unanchored matching over enormous lines. The fix is the word boundaries plus the size cap from the scan phase — and if a repository genuinely needs to scan bundles, an ignore pattern list is the documented escape valve.

Multi-line values lose newlines. A PEM key in a .env file arrives flattened or with literal \n sequences. The cause is line-by-line parsing that never looks past the physical newline. The fix is the unterminated-quote continuation: while a double-quoted value is open, the parser keeps consuming physical lines and joins them with real newline characters.

Extending the project

The schema is the natural growth point: generating .env.example from env.schema.json is one read-and-render command, and it keeps the example file from drifting behind the schema. Schema-driven shell completion — emitting valid key names for an editor integration — reuses the same source. A watch mode is a file watcher over the parse layer that already exists. Interpolation remains out of scope by design, and the honest answer for teams that need it is composition with envsubst rather than growing the parser’s ambiguity surface.

For deployment, install per-repository as a devDependency: wire check into CI before the build step, and scan as a pre-commit hook over staged files. Ship env.schema.json in the repository root so every environment — local or deployed — validates against the same declaration, and generate .env.example from it so the file everyone copies starts life correct. The print command deserves a paragraph on format safety, because it is the command most likely to run in a pipeline. --format shell emits KEY='value' lines with values single-quoted and internal quotes escaped, which is safe to eval in POSIX shells and safe to source; --format json emits a single JSON object, which is safe to pipe into jq or read from any language. The two formats exist because the safe quoting rules differ: a value containing $(...) must never reach a double-quoted shell context unescaped, and a value containing newlines cannot be a shell assignment at all. Pinning both formats in tests — with hostile values like x'; rm -rf / and multi-line PEM payloads — is what keeps the print path a convenience instead of a footgun.

The other deployment pattern worth naming is the dependency-audit scenario: teams in locked-down environments often cannot add a dotenv validation package without a review cycle, and this tool is small enough to vendor — the parser, merger, and schema check are three files with one dependency between them. Vendoring a schema checker you have read is a different trust proposition than vendoring one you have not, and the all-errors check contract makes the vendored copy verifiable against the same test suite.