Decode Chunked Transfer Encoding Errors

Parse malformed chunked bodies frame by frame to pinpoint a broken chunk size or missing terminator in a streaming response.

~30 minhttpdebuggingstreaming

Task

Decode a chunked HTTP body manually to find the malformed chunk boundary causing a client parse error.

When to use this

A streaming exporter works in most environments but one proxy corrupts the trailing chunk, breaking clients with obscure parse errors.

Prerequisites

A raw response saved to a file
Python 3 for manual framing
Knowledge of the target endpoint

Steps

1. Capture the raw chunked body without decoding.

curl --raw -sS https://stream.example.com/export -o raw.http
head -c 200 raw.http | xxd | head

Expected output: The raw file shows chunk size lines in hexadecimal followed by data and CRLF.

2. Decode chunks and assert each size line is valid hex.

import re
body = open('raw.http','rb').read().split(b'\r\n\r\n',1)[1]
for line in body.split(b'\r\n'):
    if re.fullmatch(rb'[0-9a-fA-F]+', line.strip()):
        size = int(line, 16)
        if size < 0 or size > 1048576:
            print('suspicious chunk size', line)

Expected output: Any invalid or absurd chunk size is flagged with its raw value.

Troubleshooting

Problem

Proxy re-chunks the body

Solution

Capture at the origin to confirm whether the proxy or the app emits the bad frame.

Problem

Terminator missing

Solution

A final 0-length chunk followed by CRLF is required; absence makes clients hang.

Limitations

Manual framing is for debugging, not for production parsing.
Does not validate the semantic content of chunks.
Assumes HTTP/1.1 chunked encoding rather than HTTP/2 frames.

What this recipe does

Decode a chunked HTTP body manually to find the malformed chunk boundary causing a client parse error.

A streaming exporter works in most environments but one proxy corrupts the trailing chunk, breaking clients with obscure parse errors.

Prerequisites

A raw response saved to a file
Python 3 for manual framing
Knowledge of the target endpoint

Procedure

Capture the raw chunked body without decoding.

curl --raw -sS https://stream.example.com/export -o raw.http
head -c 200 raw.http | xxd | head

Expected result: The raw file shows chunk size lines in hexadecimal followed by data and CRLF.

Decode chunks and assert each size line is valid hex.

import re
body = open('raw.http','rb').read().split(b'\r\n\r\n',1)[1]
for line in body.split(b'\r\n'):
    if re.fullmatch(rb'[0-9a-fA-F]+', line.strip()):
        size = int(line, 16)
        if size < 0 or size > 1048576:
            print('suspicious chunk size', line)

Expected result: Any invalid or absurd chunk size is flagged with its raw value.

How to confirm it worked

Run the commands above and confirm every expected result in order. If a step does not produce the stated output, use the troubleshooting notes before continuing. A passing run shows the decoder identifies the exact malformed frame, and capturing after the proxy fix shows a valid terminating chunk.

Troubleshooting

Proxy re-chunks the body — Capture at the origin to confirm whether the proxy or the app emits the bad frame.

Terminator missing — A final 0-length chunk followed by CRLF is required; absence makes clients hang.

Limitations

Manual framing is for debugging, not for production parsing.
Does not validate the semantic content of chunks.
Assumes HTTP/1.1 chunked encoding rather than HTTP/2 frames.