کدگذاری و تکاملEncoding and evolution

This chapter has two distinct halves:

• First half — encoding formats: from language-native formats (like java.io.Serializable) through JSON/XML/CSV to schema-driven binary formats like Protocol Buffers and Avro. Each has different schema-evolution rules.
• Second half — dataflow modes: how does data move between processes? Via a database (writing = messaging your future self), via services (REST/RPC), via workflow engines (durable execution), or via a message broker and actors (event-driven architecture). Each mode needs a different mix of backward/forward compatibility.

Programs hold data in two shapes:

• In memory: objects, structs, lists, arrays, hash tables, trees — structures optimized for fast CPU access (usually with pointers).
• On disk / network: must be a self-contained byte sequence (like one JSON document); a pointer means nothing to another process.

Translating from the in-memory representation to bytes is encoding (common synonyms: serialization, marshaling); the reverse is decoding (parsing, deserialization, unmarshaling).

• Sometimes no encode/decode is needed: e.g. an engine working directly on compressed data read from disk (the Query Execution part of Chapter 4), or zero-copy formats like Cap’n Proto and FlatBuffers where the same bytes are both the execution and the storage view, with no separate conversion.
• But most systems must convert between in-memory objects and byte sequences — and the format choice isn’t just about size and speed; it’s about evolvability.

Changes in program features usually mean changes in the data it stores: a fresh field, a fresh record type, or a fresh rendering of old data. But code never changes in one instant:

• Server side: rolling upgrades (or staged rollouts) — the new version deploys node by node, gets monitored, and moves on. No downtime, with small frequent releases.
• Client side: users may not update the app for a long time; that timing is out of your hands.

So old and new versions of the code and old and new versions of the data formats must live side by side:

The famous danger of the book’s Figure 5-1: old code reads a record the new code wrote with a fresh field, updates it, writes it back — and if the decoder doesn’t deliberately keep unknown fields, that field is gone forever.

Now that we know which compatibilities are needed, what are the options? This part surveys three families: language-native formats, textual formats (JSON/XML/CSV), and schema-driven binary formats (Protocol Buffers and Avro).

• Most languages ship a built-in encoding library: java.io.Serializable in Java, pickle in Python, Marshal in Ruby, plus libraries like Kryo. They’re comfortable: store and restore objects with minimal code.
• But they carry four fundamental problems:
• Language lock-in: encoding is tied to one language; reading the data in another language is hard, and integrating with other organizations becomes impossible.
• Security: to rebuild objects the decoder must instantiate arbitrary classes — if an attacker sends arbitrary bytes, arbitrary code execution (RCE) usually follows.
• Versioning as an afterthought: these libraries are built for “fast and easy” and mostly ignore forward/backward compatibility.
• Performance as an afterthought: Java’s built-in serialization is famously slow and bloated.
• Verdict: never use language-internal encoding except for very transient purposes.

Standard multi-language formats: JSON and XML (plus CSV, which holds only tabular data and doesn’t nest). They’re textual and fairly human-readable, but their problems are real:

• XML is usually criticized as verbose and over-complex.
• Number ambiguity: in XML and CSV you can’t tell a number from a numeric string at all (without an external schema). JSON separates strings from numbers but not integers from floats, and pins no precision. Numbers above 2^53 don’t survive IEEE 754 exactly; e.g. X returns each post’s 64-bit id both as a JSON number and as a text string so JavaScript won’t corrupt it.
• No binary strings: JSON/XML support Unicode (human-readable text), not raw byte sequences. The common workaround is Base64, which roughly works and inflates size by about a third.
• XML Schema and JSON Schema are powerful and hence complex; programs that skip them must hardcode the encode/decode logic.
• CSV has no schema at all; the program must define each row and column’s meaning, and escaping commas and newlines is murky.
• Still, for cross-organization exchange these are “good enough”: as long as both sides agree on the format, beauty and efficiency matter less.

Protocol Buffers (protobuf) is Google’s binary encoding library; Apache Thrift (from Facebook) is very similar. Every datum needs a schema — in the IDL language:

• The codegen tool builds classes in many languages from this schema; application code calls into that generated code. The schema language is far simpler than JSON Schema: just fields and types, no complex validation rules.
• Encoding the same example record takes only 33 bytes. Unlike MessagePack, field names aren’t in the data; instead come tag numbers: the 1, 2, 3 you saw in the schema — a compact alias for each field.
• A field’s type and tag fit in one byte; numbers are variable-length (varint): 1337 fits in two bytes (each byte’s top bit means “more follows”). Numbers −64..63 fit in one byte, −8192..8191 in two.
• There is no explicit list; repeated marks a multi-valued field, and in the bytes, repeating the same tag represents the items.

Apache Avro (since 2009, a Hadoop subproject) differs from protobuf in interesting ways. It has two schema languages: Avro IDL for humans and a JSON rendering for machines. Example:

• No tag numbers at all. Encoding the same record takes only 32 bytes — the most compact of all. The bytes hold neither field names nor data types; just concatenated values (a string = length + UTF-8 bytes). Correct decoding is possible only with exactly the schema used at write time.
• So at read time the program works with two schemas: the writer’s schema (the one at write time) and the reader’s schema (what current code expects). Avro translates the data from the writer’s view to the reader’s view by comparing the two.

• Fields pair by name; field order doesn’t matter.
• A field present in the writer but not the reader is ignored.
• A field the reader expects but the writer never wrote is filled with the reader’s default.
• For compatibility you may only add or remove a field with a default. Adding a default-less field breaks backward; removing one breaks forward.
• In Avro, null isn’t everything’s default: for nullability you must write union { null, ... }, and you may default to null only when null is the union’s first branch.
• Renaming a field is possible with an alias in the reader’s schema → backward yes, but not forward. Adding a union branch is backward-only too.

Keeping a “schema-versions database” is useful anyway: it is documentation, and every change’s compatibility can be checked before deploy. Version numbers can be increasing integers or the schema’s own hash.

• Dynamic schemas are easier with Avro: with no tags in the schema you can generate an Avro schema straight from a relational schema (each table → one record, each column → one field) and write a full database dump. If the database schema changes, re-run the same converter; since fields are known by name, the old reader’s schema still pairs up. In protobuf you must allocate tags by hand and watch none get reused.
• Schema-driven binary formats carry big wins: denser than “JSON binaries” (field names drop out); the schema itself is living documentation (unlike hand docs, always current since decoding needs it); with a registry you can test a change’s compatibility before deploy; and they generate code for statically typed languages (compile-time type checks).
• These ideas aren’t new: ASN.1 has had tag-based schema evolution since 1984 (still used in SSL/X.509 certificates with DER encoding) but is complex and badly documented. Most relational databases also have their own proprietary binary protocol that the driver (JDBC/ODBC) decodes.
• Practical rule: keep the number of simultaneous formats in the system small; each extra format means more work keeping compatibility.

So far we’ve seen how writer’s and reader’s schemas get along via resolution rules: the writer’s extra field is ignored, the writer’s missing field is filled with the reader’s default. But these rules only save us if schema change arrives step by step: in a live system, during a rolling upgrade old and new code versions run side by side for hours or days, and every message must stay readable to both. So “adding one field” isn’t one deploy; it’s a multi-stage cycle, each step guaranteeing one compatibility.

This cycle’s golden rule is readers first: first deploy code that reads the fresh field and tolerates its absence, then code that writes it. The cheapest changes add an optional field with a default — in Avro a union { null, ... } with a proper default — since old readers ignore it and new readers fill it with their own default. But renaming or retyping a field equals “delete + add”: done in one step, one of the two backward/forward compatibilities breaks.

And why is deletion always the last step? Because data outlives code: old messages stuck in queues, backups and archive files keep their old schema forever. So first stop writing the field (deprecated) while readers must still read records containing it, and only drop it from the schema once no reader needs it. Diagram 6 draws this cycle as a state machine: five main field states, two compatibility gates for the hard calls, and one dead-end state for wrong ordering.

Compatibility is a relation between one process that encodes and another that decodes. But data travels between processes in assorted ways. This part looks at four common modes.

• In a database the writer process encodes and the reader process decodes. With a single process the reader is the same program in the future: storing means messaging your future self — so backward compatibility is definitely needed.
• But usually several processes touch the database at once (different services, or several instances of one service), and mid-rolling-upgrade some are new and some old; so forward compatibility is needed too.
• Data outlives code: one value may be 5 milliseconds old, another 5 years. Code changes in minutes; data keeps its old format until explicitly rewritten.
• Rewriting (migrating) data to a new schema is possible but expensive on big data; so engines do it deferred and best-effort: LSM engines rewrite in the newest format during compaction; relational databases add nullable columns without rewrites (reading an old row fills null). Result: the whole database looks “as if” encoded with one schema, though the underlying storage mixes versions.
• Trickier changes (single- → multi-valued, moving to a separate table) still need application-level rewrites, and keeping compatibility through such migrations is still research territory.
• Archival: a snapshot or dump is usually written uniformly with the newest schema (since the actual data is being copied). Since a dump is written at once and frozen, Avro object container files are ideal — and a good moment for columnar encodings like Parquet (Chapter 11).

• The common pattern: client and server. The server puts an API on the network and the client sends it requests. The web works this way (GET for HTML/CSS/JS, POST for sending data) with a standard protocol set (HTTP, URL, TLS, HTML).
• Clients aren’t just browsers: mobile and desktop apps and browser JavaScript apps also reach servers over HTTP, usually taking JSON.
• Services resemble databases (they take and give data) but instead of a general query language they expose a program-specific API with predefined inputs/outputs — which brings encapsulation.
• The microservices goal is for each service to deploy and evolve independently (usually one team, one service, frequent releases). So old and new server and client versions coexist, and teams are free to change internals only while APIs stay compatible.
• A web service is a service over HTTP; used in three settings: user-device clients (the public internet), service-to-service inside one organization (private network), and org-to-org (public APIs like payment processing or OAuth).
• REST is the most popular design philosophy: simple formats, URLs for resource identity, and HTTP facilities for caching, auth and content negotiation. But the client must still know which endpoint and format; hence IDLs: OpenAPI (Swagger) for JSON services, Protocol Buffers for gRPC. IDL tools build docs, check schema-change compatibility, and give testing GUIs.

The book’s example: a payment that must charge the card and deposit money to the bank account — separate services for fraud detection, card connectivity and bank connectivity. Running one payment = several service calls; this sequence is a workflow, each step a task (Temporal calls them activities; some say durable functions). Workflows are usually graphs of tasks, defined in a general language, a DSL, or BPEL.

• A workflow engine decides when and on which machine each task runs, what to do on failure, and how much to parallelize; it is an orchestrator (scheduling) plus one or more executors (running). Runs start with a trigger: a schedule, a web service, or a human.
• Families: data tools like Airflow / Dagster / Prefect (ETL orchestration), graphical tools like Camunda / Orkes with BPMN for non-engineers, and durable execution frameworks like Temporal / Restate.

• Outside services (like third-party payment gateways) must still be idempotent; you must block repeat runs with unique IDs.
• The framework records calls in order and expects the next run to make exactly the same calls in the same order; so swapping two code lines can blur the replay behavior of in-flight invocations. The safe fix: deploy workflow-code changes as a separate version so old invocations replay with old logic.
• Replay must be deterministic: same input → same output. Random numbers and system-clock reads are banned; frameworks give deterministic versions of these functions, and tools like Workflow Check catch violations. (Back to the determinism debate in Chapter 9.)

• In this architecture a request is an event or message and the sender usually doesn’t wait for the receiver’s processing; the message passes through a message broker (event broker, message queue, MOM) holding it temporarily.
• Broker advantages over direct RPC: buffering when the receiver is down or overloaded; automatic redelivery after a crash; no service discovery needed; sending one message to several receivers; and logical sender/receiver decoupling (the sender only publishes).
• Two distribution patterns: queue (one of several consumers takes the message) and topic / pub-sub (every subscriber gets its own copy).
• Brokers impose no data model: a message = bytes + metadata. Common: Protobuf/Avro/JSON with a schema registry; AsyncAPI as the message-world OpenAPI. Examples: RabbitMQ, ActiveMQ, NATS, Redpanda, Kafka (open source) and Kinesis, Azure Service Bus, Cloud Pub/Sub (cloud) — compared in Chapter 12.
• Message durability: many write to disk to survive crashes; unlike databases, many auto-delete after consumption. For event sourcing you need a broker keeping messages forever (Chapter 3).
• If a consumer republishes a message to another topic it must preserve unknown fields — otherwise Figure 5-1’s curse strikes mid-pipeline.
• Actor model: instead of threads, locks and deadlocks, logic is encapsulated in actors; each actor owns its local state, talks over async messages, processes one message at a time, and delivery isn’t guaranteed. In distributed frameworks like Akka, Orleans and Erlang/OTP the same mechanism works locally or remotely (messages transparently encode here, decode there). Location transparency works better here than in RPC because the model accepts “messages may be lost” from the start. Rolling upgrades still need compatible encoding.

• Swapping backward and forward; the rule: always ask “who is reading what?”
• Thinking “JSON has no schema, so no evolution thought needed” — JSON data lives for years too, and an added field may be wiped by old code.
• Changing a protobuf tag or reusing a deleted field’s tag — destroying existing data.
• Adding a default-less field in Avro, then wondering why backward compatibility broke.
• Assuming “unknown-field preservation” in every library; the behavior must be checked and tested explicitly.
• Equating “binary = compact”; MessagePack is only 66 vs 81 bytes — real density comes from dropping field names with a schema.
• Treating RPC like a local function and forgetting timeout, retry and idempotence.
• Reordering activities in a durable workflow and expecting healthy replay (or forgetting determinism).
• Forgetting that third-party APIs must be idempotent in event-driven and workflow designs.

1. Backward/forward definitions + their request/response translation in APIs.
2. Tell Figure 5-1’s story in 3 sentences: why the unknown field vanishes and how to stop it.
3. The three format families and the final verdict: language-native (banned), textual (human-readable exchange), schema-driven binary (compact and evolvable).
4. Protobuf evolution rules (frozen tags, free renames, reserving) and Avro’s (mandatory defaults, unions for null, backward-only aliases).
5. Avro’s three writer’s-schema sources + resolution (ignore / default).
6. “Data outlives code” and compatibility duty in the four dataflow modes (database, RPC, workflow, broker) — recite section 4’s table.
7. Five RPC-vs-local differences + three durable-execution limits (idempotence, order, determinism).