truewire
Source on GitHub

ADR 0016: Protobuf WebSocket frames are decoded at run time from spec/proto sources, as ProtoJSON

  • Status: accepted
  • Date: 2026-09-15

Context

mexc's spot WebSocket pushes are Protocol Buffers: every frame is a PushDataV3ApiWrapper whose body oneof holds the one message a channel carries. The spec already records the facts a client needs:

  • <id>.messages.protobuf.json: the binary frames, base64, which truewire mock sends in place of <id>.messages.json and truewire check validates as base64.
  • <id>.messages.json: a decoded, human-readable sidecar.
  • meta.proto_field on each stream endpoint, declared by the project's [cores.<name>].meta schema: which body member the channel narrows to.

What the spec did not hold was the schema itself. Python's mexc core carries betterproto2-generated classes in the package and narrows with getattr(wrapper, meta['proto_field']); the nine leaves stay surface: handwritten. The TypeScript and Go runtimes had no way to decode a binary frame at all (G10), and Rust will need the same thing next.

The options were:

  1. A protoc/buf step per language (protoc-gen-es, protoc-gen-go, prost-build). Typed classes, but every generating machine needs protoc and a plugin per language, the generated names differ per plugin, and Python's betterproto2 output already differs from all three.
  2. Compile the .proto sources at generation time into a FileDescriptorSet and embed the bytes. Needs protoc (or grpc_tools) in the generator.
  3. Embed the .proto sources verbatim in the generated package and compile them at run time with a pure-library compiler, reading the result as ProtoJSON.

Decision

Option 3.

Spec. A project keeps its .proto files under spec/proto/ (any depth; imports resolve by path relative to that directory). They are spec, like schemas.json: copied from upstream, never generated. gRPC (G9) reads the same directory.

Generator. Each non-Python backend writes the sources into the package verbatim, keyed by their path under spec/proto/, and nothing else:

Language File Symbol
TypeScript <package>/proto.ts PROTO_SOURCES
Go <package>/protos/protos.go (protos is a reserved root directory) protos.Sources
Rust <package>/protos.rs SOURCES

A project with no spec/proto/ gets no such file. No protoc runs anywhere.

Runtime. A core compiles the sources once and decodes each binary frame against its envelope message:

Language Module Library
TypeScript @truewire/core/protobuf: ProtoFrames.compile(sources, message), frames.decode(data) protobufjs (optional peer dependency)
Go truewire.dev/core/protoframes (its own module, so truewire.dev/core has no protobuf dependency): Compile/MustCompile, (*Frames).Decode github.com/bufbuild/protocompile, google.golang.org/protobuf (dynamicpb, protojson)
Rust truewire_core::proto (feature proto): Protos::from_sources(SOURCES), decode, narrow protox (compile) + prost-reflect (decode, serde ProtoJSON)

A decoded Frame has the same small surface in every language: has(field), field(field) (the value as ProtoJSON, absent when unset), string(field), oneofCase(oneof) and json() (the whole frame). Fields are named by their .proto name, the name meta.proto_field carries.

Value rendering is canonical ProtoJSON: lowerCamelCase json_name keys, 64-bit integers as strings, bytes as base64, enums by name, implicit-presence fields at their default left out. It is the one rendering every protobuf library implements the same way, so one frame decodes to one JSON value in every language. packages/testing-ts/test/fixture-protobuf/frames.golden.json holds every recorded mexc frame with its decode by betterproto2 (Python); the TypeScript and Go tests assert byte-for-byte agreement with it, and the Rust tests will read the same file.

Where narrowing lives. Decoding the envelope and narrowing it by meta.proto_field is a core's job, as in Python: the core's parseMsg/Parse decodes binary frames and routes them by their channel field, and the endpoint core filters on has(proto_field) and maps to field(proto_field). The runtime gives the mechanism; the venue dialect (JSON subscribe/ack beside binary pushes) stays in the hand-written core.

Testing. truewire mock already replays binary frames. @truewire/testing's WsExample.frames and Go twtest.Example.Frames now carry the decoded sidecar bytes, so a project test can decode every recorded frame through its own core. Both runtimes test a protobuf-framed streams core against truewire mock over the fixture project.

Consequences

  • mexc's spot stream leaves become implementable in TypeScript and Go with no build step: a core holds ProtoFrames.compile(PROTO_SOURCES, 'PushDataV3ApiWrapper'). The spec still needs spec/proto/*.proto (upstream mexcdevelop/websocket-proto); a scratch copy reconstructed from the Python classes decodes all nine recorded frames to their declared proto_field in both languages.
  • The pushed value is untyped ProtoJSON (unknown / json.RawMessage), not a generated class. A generated method's message codec can still validate it, but only when the endpoint's payload schema mirrors ProtoJSON names. mexc's docs-derived schemas do not (publicdeals, dealsList, sendtime beside sendTime), which is why its leaves stay hand-written. Aligning those schemas to ProtoJSON would let the generated stream methods serve them. That is spec work, left open.
  • Run-time compilation costs a few milliseconds once per process, and protobufjs, protocompile and protox weigh more than generated code would. The weight only reaches a project that imports the protobuf module: it is an optional peer dependency in TypeScript and a separate package in Go.
  • Well-known types (google/protobuf/*.proto) resolve in Go (protocompile's standard imports) but must be vendored under spec/proto/ for TypeScript. No mexc message uses them.
  • Python is unchanged: its cores keep betterproto2 classes. A Python truewire_core equivalent can come later if a second protobuf venue appears.