Networking

Serialization: Protobuf & Base64

Two ideas that get confused constantly because they're often used together: serialization turns a structured, in-memory object into a flat sequence of bytes (and deserialization turns it back); Base64 has nothing to do with that — it just disguises arbitrary binary bytes as plain text so they survive text-only channels. Protobuf is one specific, very common serialization format; Base64 is an encoding, not a format for structuring data at all.

What serialization and deserialization actually mean

A running program keeps its data as objects, structs, or classes in memory — a layout full of things that are meaningless outside that one process: compiler-chosen padding between fields, pointers that are just addresses valid only in this process's address space, and a byte order (endianness) tied to the CPU it's running on. None of that survives being sent to another machine, written to a file, or even read by a different program on the same machine.

Serialization is the process of converting that in-memory structure into a flat, well-defined sequence of bytes that means the same thing to anyone who reads it, regardless of language, compiler, or CPU. Deserialization is the reverse: take that byte sequence and rebuild an equivalent in-memory structure on the receiving end.

Text vs. binary formats, at a glance

Base64 is deliberately left out of this table on purpose — it's covered later, but it is not a serialization format. It's an encoding that can be layered on top of any of these (or any binary data at all) when the transport only accepts text.

Protocol Buffers (Protobuf)

Protobuf is a binary serialization format built around a schema you write yourself in a small interface-definition language, saved as a .proto file:

syntax = "proto3";

message SensorReading {
  int32  sensor_id  = 1;
  float  temperature = 2;
  string location    = 3;
}

A compiler (protoc) reads that .proto file and generates real code in whatever language you need — a class in Python or Java, a struct in C++, or (the option most relevant on a microcontroller) tiny C code via nanopb, a Protobuf implementation built specifically for embedded targets: no dynamic memory allocation, a tiny code footprint, and no C++ required. Whatever language it's generated in, the generated code gives you two functions that matter: one to serialize an instance into bytes, and one to parse bytes back into an instance.

# Python, using code generated from the .proto file above
reading = SensorReading(sensor_id=300, temperature=21.5, location="A1")
data = reading.SerializeToString()   # -> bytes, ready to send/store

# ... elsewhere, maybe on a different machine entirely ...
received = SensorReading()
received.ParseFromString(data)       # -> received.sensor_id == 300

How Protobuf actually encodes bytes

Unlike JSON, a Protobuf message on the wire never contains field names — only field numbers. Every field is written as a small tag byte (which field, and how its value is encoded) followed by the value itself:

tag = (field_number << 3) | wire_type

The wire type tells the parser how many bytes to read next: 0 for a variable-length integer ("varint" — used for int32, bool, enums), 5 for a fixed 4 bytes (float, fixed32), and 2 for "length-delimited" (a varint length, then that many raw bytes — used for string, bytes, and nested messages). Encoding SensorReading{ sensor_id: 300, temperature: 21.5, location: "A1" } from the schema above looks like this on the wire:

The varint for 300 (AC 02) is worth unpacking once: a varint spends 7 bits of each byte on the actual number and reserves the top bit to mean "another byte follows." 300 in binary is 1 00101100; split into 7-bit chunks from the low end that's 0101100 and 10. The first byte becomes 1_0101100 (continuation bit set) = 0xAC; the second becomes 0_0000010 (no more bytes) = 0x02. Small numbers (the overwhelmingly common case) fit in a single byte this way instead of always spending 4 bytes like a fixed-width int.

Schema evolution: why field numbers matter

In an IoT fleet, the firmware on a device and the schema the cloud expects are rarely upgraded in lockstep — some devices in the field may run firmware from years ago (see OTA at fleet scale). Protobuf is designed around that reality:

• Adding a new field is always safe — give it a new, never-used field number; old code that doesn't know about it simply skips those bytes, new code sees it normally.
• Removing a field means retiring its number — mark it reserved in the .proto file so it can never accidentally be reused for something unrelated later, which would make old encoded data misinterpreted as the new field.
• Never change a field's number once shipped — the number, not the name, is what's on the wire; renaming a field in the .proto file is free, renumbering it silently breaks every message already encoded with the old number.
• Never change a field's wire type (e.g. int32 to string) without a new field number — old and new wire types aren't generally interchangeable at the byte level.

Base64 encoding and decoding

Plenty of systems are only willing to carry text — a JSON string value, a URL, an email body, an XML document. Raw binary bytes can contain values that those formats treat as control characters, delimiters, or just invalid — so they can't be dropped in directly. Base64 solves this by re-representing any sequence of bytes using only 64 characters that are always safe in text: A–Z, a–z, 0–9, +, and /.

The trick is a mismatch of sizes: a byte is 8 bits, but Base64 maps 6 bits at a time to one character (2⁶ = 64 possible values, exactly the alphabet size). So 3 bytes (24 bits) divide evenly into 4 groups of 6 bits — 3 bytes in, 4 text characters out, which is why Base64-encoded data is always about 4/3 (~33%) larger than the original.

When the input length isn't a multiple of 3 bytes, the last group is padded with zero bits and the output string is padded with = characters to keep every encoded block a multiple of 4 characters — e.g. Base64("Hi") (2 bytes) comes out as "SGk=", one = short of a full 4-character group. Decoding just runs the whole diagram in reverse: characters back to 6-bit values, regrouped into 8-bit bytes, padding discarded.

Real-world applications

These two ideas show up together constantly in IoT systems — a device that measures data, serializes it compactly, then has to squeeze that binary payload through a transport that only speaks text (a REST/JSON API is extremely common even when the payload inside it isn't JSON-native):

• gRPC APIs — gRPC uses Protobuf as its wire format by default, over HTTP/2, and is the standard choice for high-throughput service-to-service calls.
• Device telemetry — a compact Protobuf payload matters far more over a constrained link (cellular, LoRa, low-bandwidth MQTT) than it does on a typical web API.
• HTTP Basic Authentication — the Authorization: Basic <credentials> header is just username:password Base64-encoded — not encrypted at all (see Gotchas).
• Data URIs and small embedded assets — a small image embedded directly in HTML/CSS as data:image/png;base64,... avoids a separate network request.
• JWTs — each of a JSON Web Token's three dot-separated parts is Base64url-encoded JSON — readable by anyone with a decoder, which is exactly the point (the security comes from a separate signature, not from hiding the content).

Gotchas