阅读中文版本

Build a URL Metadata Scraper Service

A complete URL metadata scraper: SSRF guards, robots-aware fetching, OpenGraph and Twitter card parsing, caching, and hostile-input tests.

url-metadata-scraper/
├── src/fetch.ts
├── src/guard.ts
├── src/parse.ts
├── src/cache.ts
├── tests/scrape.test.ts
└── package.json

if (isPrivate(ip)) throw new Error('blocked address');

for (let hop = 0; hop < 5; hop++) { url = check(await fetchStep(url)); }

const title = $('meta[property="og:title"]').attr('content') ?? $('title').text();

const image = new URL(raw, baseUrl).href;

await redis.set(key, JSON.stringify(meta), 'EX', 3600);

Project overview

A service that fetches an arbitrary URL is a request-forgery tool unless it is carefully constrained. This build resolves and checks the host before fetching, caps the response, and only then parses the metadata a link preview needs.

You will validate the scheme, resolve the host and reject private and loopback addresses, stream the response with a size cap, parse the head tags for OpenGraph and fallbacks, and cache the result by URL hash with a TTL.

The SSRF guard is the security boundary and it must run on the resolved address rather than the hostname, because a hostname can resolve to an internal address and a redirect can change the target after the first check.

Streaming with a size cap is what stops a hostile server from returning an unbounded response, which would otherwise be a memory problem triggered by a single client-supplied URL.

Each phase adds one property: the guard, the safe fetch, the parsing, the image resolution, and the cache, so the service is safe and useful at every stage.

It is for teams that show link previews and have discovered that a preview fetch is an outbound request driven entirely by user input. The service makes that request as narrow as possible so the preview feature does not become a network probe.

It is also a reminder that a preview is a display surface, so the extracted text and image are attacker-controlled and must be escaped and constrained before they reach a user interface.

What you will build

The service provides five capabilities, each with a test that fetches a local page or asserts a rejection:

• SSRF guard
• Safe fetch
• OpenGraph parsing
• Image URL resolution
• Cache with TTL

The SSRF guard blocks internal targets; a safe fetch bounds the response; parsing extracts OpenGraph and Twitter tags with fallbacks; image resolution makes relative URLs absolute; and a URL-keyed cache bounds repeated work.

The design assumes the target is hostile, so it treats the fetched content as untrusted text and never follows a URL it has not checked.

Architecture and data model

The service validates the scheme, resolves the host, and rejects private or loopback addresses before fetching. It streams the response with a size cap, parses head tags, and caches the result by URL hash with a TTL.

A request validates the scheme, resolves the host, and rejects any address in a private or loopback range before the HTTP client is invoked. The check runs on the resolved address, since the hostname alone can be misleading.

The fetch streams the response with a size and time cap, so a slow or enormous response is bounded rather than allowed to consume the worker.

The parser reads only the head of the document and extracts OpenGraph tags, then Twitter card tags, then fallbacks such as the title, so a page with no metadata still produces a usable preview.

Results are cached by a hash of the URL with a TTL, which keeps a popular link cheap and avoids re-fetching a page that never changes.

Parsing only the head is both a performance and a security decision: the head is small, so the response cap can be tight, and the rest of the document is never needed to produce a preview.

Caching by URL hash rather than by raw URL keeps the key bounded and avoids storing a URL that may itself contain sensitive query parameters.

Prerequisites and dependencies

You need an HTTP client that follows redirects under control, a cache, and an egress policy that restricts where the service may connect.

• Node.js 20 LTS or later
• A reachable Redis 7+ instance
• Outbound network access

The dependencies cover the HTTP surface, the fetch, the parsing, and the cache:

• undici (^6.19.0) — Fetching with redirect and timeout control
• cheerio (^1.0.0) — HTML head parsing
• ioredis (^5.4.1) — Metadata cache

pnpm init
pnpm add undici cheerio ioredis
pnpm add -D typescript tsx vitest

Configure an egress restriction or an allowlist so the service cannot reach internal services even if the guard is bypassed, because defense in depth is the point of a fetcher that takes URLs from users.

Directory structure

Separate the SSRF guard, the safe fetch, the metadata parsing, image URL resolution, and the cache so each can be tested with a fixed page:

url-metadata-scraper/
├── src/fetch.ts
├── src/guard.ts
├── src/parse.ts
├── src/cache.ts
├── tests/scrape.test.ts
└── package.json

Phase 1: SSRF guard

Reject loopback, link-local, and private ranges before fetching.

if (isPrivate(ip)) throw new Error('blocked address');

Without a guard, a crafted URL can scan internal services.

Resolve the host and reject a private or loopback address before fetching, because the guard is the only thing between a user-supplied URL and your internal network.

Phase 2: Safe fetch

Follow a bounded number of redirects and enforce a size and time cap.

for (let hop = 0; hop < 5; hop++) { url = check(await fetchStep(url)); }

Redirects can lead back to blocked hosts, so re-check each hop.

Fetch with a timeout and a size cap, and re-check the address after any redirect, since a redirect is the standard way to defeat a one-time URL check.

Phase 3: OpenGraph parsing

Read og:title, og:description, og:image with sensible fallbacks.

const title = $('meta[property="og:title"]').attr('content') ?? $('title').text();

Structured tags give better previews than the page title alone.

Parse the head for OpenGraph tags, then Twitter card tags, then fallbacks, so a preview degrades gracefully rather than disappearing when a site omits metadata.

Phase 4: Image URL resolution

Resolve relative image URLs against the base.

const image = new URL(raw, baseUrl).href;

Relative og:image values break naive previews.

Resolve a relative image URL against the document’s base, and re-apply the guard to the image host if the image is fetched or embedded, because a relative URL can point inward.

Phase 5: Cache with TTL

Cache parsed metadata by URL hash.

await redis.set(key, JSON.stringify(meta), 'EX', 3600);

Repeated previews must not hammer the target site.

Cache the result by a URL hash with a TTL, so a repeated lookup is cheap and a page that changes is refreshed after the window rather than on every request.

Testing

Serve a local page with OpenGraph tags and assert extraction. Point the fetcher at 127.0.0.1 and a metadata service address and assert both are blocked.

Serve a local page with OpenGraph tags and assert the extraction, which exercises parsing without depending on a third-party site.

Point the fetcher at a loopback address and at a well-known metadata address and assert both are blocked, because these are the two cases the guard exists for.

Assert that an oversized response is truncated or rejected rather than buffered, since this is the property that bounds the memory a hostile target can consume.

Assert that a page with no metadata still yields a title fallback, because a preview that is empty for half the web is a broken feature.

Serve the fixture page from an in-process server on a loopback address and reach it through the resolver path, so the guard is exercised together with the fetch.

Assert too that a redirect to a blocked address is rejected, because a guard that only checks the initial URL passes a server that fails this exact case.

Assert that extracted fields are bounded in length, because a page can declare an enormous title and a preview should carry a truncated version rather than the whole thing.

pnpm vitest run

Deployment and operations

Run fetches from an egress-restricted network so even a guard bypass cannot reach internal services. Cache aggressively and rate-limit per caller.

Run fetches from an egress-restricted network, so even a guard bypass cannot reach internal services, and treat the network policy as part of the design rather than as an extra.

Cache aggressively and rate-limit per caller, because the service is a proxy for other people’s pages and a popular link should not be fetched once per viewer.

Monitor fetch latency and failure rate by host, since a single slow host can dominate the tail without any rise in traffic.

Alert on guard rejections, because a sustained rate is either a misconfigured client or an attempt to reach internal resources.

Treat the egress policy as a first-class part of the deployment rather than an optional hardening step, because it is the only control that still holds if the guard is bypassed.

Security considerations

This is an SSRF-sensitive service. Validate the resolved IP, not the hostname text, re-check on redirects, and never reflect untrusted HTML back to the browser unescaped.

Check the resolved address, not only the hostname, and repeat the check after redirects, because a resolver and a redirect are both ways to reach an internal target.

Cap the response size and the redirect count, since unbounded responses and redirect loops are both denial-of-service vectors controlled by the target.

Do not fetch arbitrary schemes or ports, because a fetcher that speaks more than HTTP and HTTPS is a probe for the rest of the network.

Treat the extracted text as untrusted and escape it wherever it is rendered, because preview metadata is attacker-controlled content shown in your UI.

Bound the number of redirects and re-check the address at each hop, because a redirect is the most common technique for turning an allowed URL into an internal request.

Do not forward the fetcher’s own credentials or cookies to the target, because a preview request should look like an anonymous client rather than like your infrastructure.

Do not include the fetched page’s raw HTML in any error returned to the caller, because that exposes server-side markup and, potentially, secrets embedded in the page.

Observability

Track fetch latency, blocked-address counts, and cache hit ratio. Alert on a spike in blocked addresses, which indicates probing.

Track cache hit ratio and fetch rate, because the ratio is the main determinant of both latency and how much of the internet you actually contact.

Chart guard rejections by reason, since a cluster of one reason usually points at a misconfigured caller rather than at an attack.

Measure response size and fetch duration, because a rising size with a stable duration suggests the cap is about to bind.

Alert on redirect chains approaching the limit, as a chain that always hits the cap is a target trying to exhaust the service.

Track fetches by host, because a single host dominating the traffic is either a misbehaving client or an attempt to use the service as a proxy.

Track the response size distribution too, because a cluster of sizes just under the cap suggests the limit is being probed rather than merely occasionally exceeded.

Troubleshooting

Preview shows an internal service. The guard checked the hostname string instead of the resolved IP; resolve first.

Fetch hangs on a slow site. No timeout was set; always bound the fetch and the body read.

If a legitimate site is blocked, check the resolved address first, because a site behind a private CDN can legitimately resolve to a range the guard rejects.

If a preview is empty, inspect the raw head tags, because a page may declare metadata in a form the parser does not yet recognize rather than having none.

If the fetch hangs, check the timeout and the size cap together, because a target that streams slowly can hold a connection until the timeout even when it never sends much.

If the same URL returns different metadata on each call, check the cache key and the TTL, because an unstable key turns the cache into a growing file of duplicates.

Limitations and next steps

• Only head metadata is parsed; no article extraction.
• No JavaScript-rendered pages.
• Cache invalidation is TTL-only.

The service extracts metadata from the head only and does not run a full page render, so a client-side-rendered preview is returned empty.

There is no per-site customization for tags beyond the standard vocabularies, which limits coverage for bespoke markup.

The cache is keyed by URL and does not invalidate on content change, so an update is visible only after the TTL expires.

The fetcher does not execute JavaScript, which is a deliberate security choice rather than a missing feature.

The service returns metadata only and does not store the page content, so a downstream feature that needs the document would have to fetch it again under the same constraints.

There is no structured extraction for schema.org JSON-LD, so a site that marks up only in JSON-LD falls back to the generic tags rather than a rich result.