Protocol Buffers
Protocol Buffers is a language-neutral schema and binary serialisation format, giving compact, forward-compatible structured messages.
Definition
Protocol Buffers (protobuf) is a schema-first serialisation system: messages are defined in a .proto file with numbered fields, and a compiler generates code in many languages. On the wire it is a compact binary format keyed by field number rather than name, so payloads are small and parsing is fast. Field numbers, not names, give compatibility: unknown fields are preserved and new optional fields can be added without breaking older readers.
Context and scope
The standard message format for gRPC and for high-throughput internal services and streaming pipelines where JSON size and parsing cost matter.
Common misconceptions
- Protobuf is human-readable — it is binary; a schema is needed to decode it, which is also why it is compact
- Changing a field name breaks compatibility — numbers identify fields; renaming is safe, renumbering or reusing numbers is not
- It is only for gRPC — it is a general serialisation format usable for storage, queues, and any structured data exchange
Anti-patterns
- Deleting a field and reusing its number, corrupting readers that still expect the old type
- Making every field required, so schema evolution becomes a breaking change
- Storing protobuf without a schema registry, leaving old data undecodable when schemas drift
Related concepts
Related examples
Authoritative source
What it is
Protobuf begins with a schema: messages are declared with fields, each tagged with a unique integer. A compiler turns that schema into strongly typed classes in the target languages, handling serialisation and deserialisation. The wire format encodes each field as a small integer tag plus its value, so the field name never travels and the payload is far smaller than equivalent JSON. Because decoding reads tags, unknown fields can be skipped and preserved.
Why it matters
JSON is readable but verbose and slow to parse at scale, and it carries no schema. Protobuf gives smaller messages, faster parsing, and generated types that catch mistakes at compile time. Its field-number scheme provides compatibility by design: a producer can add fields that old consumers ignore, and old data remains readable as the schema grows. This makes it a good fit for evolving, high-throughput service and streaming interfaces.
How to use it well
Treat field numbers as permanent: never reuse a retired number, and reserve numbers instead of deleting them, because a reused tag reinterprets old data as the new type. Default fields to optional, add rather than remove, and evolve schemas in backward-compatible steps. Pair protobuf with a schema registry or versioned contracts so every stored or streamed message can still be decoded, and remember a schema is required to interpret the bytes — store it alongside the data.