Wire protocols

The protobuf schema (proto/algolang/v1/algolang.proto) is the field-number reference. make proto regenerates gen/ using buf generate. Preserve wire compatibility: add fields, never renumber or repurpose existing fields. OrderbookAction values use the ORDERBOOK_ACTION_ prefix to avoid proto3 enum-name collisions. DBN payloads use Envelope.DbnRecordBatch (field 60) inside the protobuf control framing.

Strategy CSV carries one response per bar, cannot represent attached-order trees, and has narrower identifier/field support than protobuf. Use protobuf for multi-order decisions, brackets and richer data declarations. The exact grammars and validation live in internal/wire/csv_strategy.go and internal/wire/csv_data.go; changing them also requires checking the raw clients in strategies/demo. The stdio-csv-v2 grammar (its codec is internal/wire/csv2_strategy.go) answers each decision with a batch of cancels and orders and still refuses attached-order trees; the engine speaks it on a run under --lifetime v2 (engine/stdiohost_csv2.go), and the Go SDK runs a strategy over it (sdk/strategy/run_csv2.go) (The stdio-csv-v2 grammar).

The current adjusted strategy CSV records are abar and afill, with roll for notices. Earlier implementation briefs called them cbar and cfill; those names did not ship. Protobuf bars carry raw OHLC, additive offset, multiplicative ratio, is_adjusted, and optional session metadata. Fills carry raw_price and adjusted_price, not a price_adjustment field. For additive adjustment, adjusted = raw + cumulative_adjustment. For ratio adjustment, adjusted = raw * cumulative_price_ratio.

InitAck.data_requirements (8) carries a declared bar series’ own construction (GLE-390). DataRequirement.bar_spec (4) carries every BarSpec field the declaration sets, and DataRequirement.session (5) carries its session; a declaration that sets neither marshals byte for byte as before. The CSV declaring record keeps its fixed eight fields and refuses a session or a construction field beyond kind, interval and threshold. sdk/barspec holds the shared Construction type. Its zero value is the run’s construction; its Validate, Normalise, Key, BarSpec and FromBarSpec are what the SDK, the run config and the engine all apply.

InitAck.aligned_series (field 12, message AlignedSeries: series_id 1, source 2, grid 3) carries the strategy’s aligned-series declarations (GLE-353). The engine reads them to validate each view and fetch its reach-back; the samples themselves never cross a wire. An InitAck without them marshals byte for byte as before. The CSV strategy wire has no grammar for them: the SDK’s CSV encoder refuses an InitAck that carries one (a raw CSV replay drops them, having no engine to read them). Bar carries no new wire field: SourceOpenTime, SourceCloseTime and Updates exist only on the SDK’s Bar.

Bar.tick_count (11), vwap (12), notional (13), buy_volume (14), sell_volume (15) and close_threshold (16) are the construction statistics of a trade-built bar. Since GLE-386 they cross the strategy wire as well as the data wire: internal/pbconv and the SDK’s mirrored converter map them to and from Bar.Stats, a block that exists exactly when tick_count > 0. A bar without the block marshals byte for byte as before. The CSV bar and abar lines have no slots for them, so a CSV strategy sees Stats == nil; the equity adjuster adjusts vwap with the bar’s prices.

Statistic.price (4) and Statistic.quantity (5) are proto3 optional since GLE-396, so a settlement carries a price and no quantity and an open interest a quantity and no price; Statistic.symbol names the dated contract. InstrumentDef.symbol (17) is the canonical dated symbol and maturity_year (18), maturity_month (19) and maturity_day (20) the calendar maturity the venue symbol encodes (0 where the publisher gives no such granularity). GetDataRequest.stat_types (16) names the statistic kinds a STATISTICS request wants; empty asks for every kind. The data SDK serves both schemas through data.StatisticSource and data.DefinitionSource on the pb data wire only and refuses them on DBN and CSV; the engine reads them with DataClient.FetchStatistics and FetchDefinitions (Statistics and instrument definitions).

BarSpec.lead_in (13, a google.protobuf.Duration) asks an event-bar request for a constructor lead-in (GLE-397). SupportedBarKind.serves_lead_in (6) advertises one per kind, and the data SDK refuses a lead-in the adapter did not advertise, a negative one, and one on time bars. GetDataResponse.lead_in (5, message LeadInReport: start 1, clamped 2, warmup_bars_discarded 3) rides the request’s final envelope, and DataClient.FetchBars returns it as FetchStats.LeadIn. Both travel on the pb data wire only, and the engine refuses a lead-in on the CSV wire. The bar cache key carries the lead-in as lead_in (a Go duration string, omitted when there is none, so earlier entries keep their hashes). GetDataRequest.session may name a custom:HHMM-HHMM window when the adapter’s SupportedSessions lists custom (Custom session windows).

Envelope.market_events (field 62, message MarketEvents: repeated MarketEvent, each a series_id (1) and a oneof event holding a Trade (2), a Statistic (3) or an InstrumentDef (4)) carries declared trades (GLE-389), statistics and definitions (GLE-406) to the strategy between bars. The strategy SDK sets InitAck.needs_market_events (15) when it declares a TRADES, STATISTICS or INSTRUMENT_DEFINITION series or implements MarketTradeHandler, StatisticHandler or DefinitionHandler. The push is pb-only and gets no reply; a declaring strategy speaks pb already. A trade’s RecordMeta.flags carries the Databento flags byte (last, top_of_book, snapshot, aggregated for F_MBP, bad_ts_recv, maybe_bad_book; the two publisher-specific low bits are not carried), so MarketTrade.BadTsRecv reaches strategy code.

GetDataRequest.grid_revision (field 11) carries the calendar revision of an intraday grid request; the CSV get line carries it as an optional trailing grid_revision=HEX token, emitted only when set.

Error.request_scoped (3) and Error.request_id (4) carry a request-scoped refusal on the data wire (GLE-324 W10). Without the flag, which is every producer that predates the fields, an Error is fatal in either direction and ends the run. With it, the error answers the one request request_id names: the adapter keeps serving, and the engine fails that request alone. The data client returns it as an engine.RequestRefusal carrying the feed’s code and text; test for it with errors.As. The data SDK’s rule is that every error on a request’s path is request-scoped: a handler’s returned error (source_error), the SDK’s own request validation (bad_request, unsupported_schema) and a failed coverage lookup (coverage_failed). The fatal errors are the ones no request owns: a failed DataInit (init_failed), a protocol violation (protocol_error: a request before DataInit, an envelope or line the data wire has no meaning for), an engine-sent Error, and a broken stream, on which the SDK exits because nothing can be answered. On the CSV data wire the request-scoped form is its own line, refused,REQUEST_ID,CODE,MESSAGE; the error,CODE,MESSAGE line stays fatal. A refusal is the request’s own only on an exact, non-empty request_id: requests are served one at a time, so a refusal naming another request, or none, is a protocol error on the engine side, never a RequestRefusal, and the data client then refuses every later request on that connection before writing it, naming the earlier cause — as it does after an unscoped Error, a failed read or write, or a response its decoder rejected mid-stream (whose remaining envelopes are still on the stream). Every request kind — data, coverage, instrument info — sent before DataInit is a fatal protocol_error that never reaches the adapter’s handler. Compatibility in both directions: an engine that predates the fields reads a request-scoped Error as fatal and fails the fetch as it always has (the adapter is still up, so a later fetch on the same connection succeeds), and an engine with them reads an older adapter’s unscoped Error as fatal, exactly as before. Benchmark sleeves are the first consumer (see Report benchmark findings); a series fetch the feed refuses still fails the run, with the feed’s text rather than a dead connection’s.

ROLL_SCHEDULE is data schema 14. Its protobuf records contain the root, recipe hash, generation, election time, half-open held spans, paired seam prices and typed degradation flags. Three fields added for the calendar schedule (GLE-270, populated by marketfeed’s schedule v2, GLE-274) say what the pair and the chain are:

• RollSchedule.seam_policy (12) names the measurement contract every seam’s pair conforms to; schedule v2 sends r1-settlement@trade-date-boundary/v1 (the R−1 settlement pair at the trade-date-boundary cut). Empty means a producer that predates the field, whose pair is the feed’s 1 m contemporaneous measurement at the cut.
• RollSeam.degraded_reason (10) names the fallback rung that supplied the pair when degraded is true. Schedule v2 uses settle_window (the 1 m pair in the settlement window), last_common (the last common 1 m pair before the cut) and temporal.
• RollSchedule.held_rank (7, optional) is the contract rank the rule holds: 0 is the front, 1 to 3 are B1 to B3. A rule may hold a deferred month on purpose, and the engine follows it (GLE-387). With no presence the rank is undisclosed, which does not mean the front; the engine refuses that.

The feed-owned continuous product’s wire additions landed in GLE-314 P0 (additive; no adapter or engine behaviour changed with them). On RollSchedule, the GLE-214 E0 lineage fields took their reserved numbers: superseded_by (8; 0 when the served generation is current, else the current generation number), derived_at (9; when the deriver last examined the lineage), last_bar_at (10; the end of the newest stored bar of the open front) and chain_checksum (11; sha256 hex over the canonical chain), plus snapshot_token (13), the identity of the served state. RollScheduleSpan gained optional cumulative_adjustment (6) and cumulative_price_ratio (7), the span’s constants at the request’s adjustment anchor (absent means undisclosed). GetDataRequest gained the pins snapshot_token (12), generation (13), as_of (14) and adjust_anchor (15), which apply to composite BARS and ROLL_SCHEDULE requests, and adjust_anchor also to the BARS request of an adjusted equity (eq:V:T:adj=split|splitdiv; GLE-328 sets it to the run end on the warm-up and main fetches alike, so they share one basis), and which an adapter that predates them ignores; GetDataResponse.snapshot_token (4) echoes the served token on the final envelope only, and the data SDK sets it from DataContext.SetSnapshotToken. Bar gained seam_prior_contract (28) and seam_partial (29), the straddled-seam facts of a composite bar built across a roll cut, and CorporateActionType gained CA_SPINOFF (11) and CA_SUSPENSION (12) (CSV tokens spinoff and suspension). None of these travel on the CSV or DBN wires.

EXCHANGE_CALENDAR is data schema 15 (GLE-380, increment I11 unit A1 of the order-lifetime programme): versioned exchange-calendar facts, one record per exchange trade date of a futures root under a root-level header. The messages, the record rule, the adapter contract and the validator are in Exchange calendar facts; this paragraph is the wire. It is a GetData schema only: GetDataResponse.exchange_calendar (22, message ExchangeCalendarBatch: header 1, days 2) is its one response member, and StreamData and AppendDataRequest carry no counterpart, because a calendar is neither streamed nor appended. The framing is header first: the first envelope of a request carries the header and the first days, every later envelope carries days only, and total_count counts days (not envelopes, and not the header). The data SDK batches a calendar as it batches every schema: full envelopes of batch_size days go out with complete = false, and the final envelope carries the remaining days, possibly none, with complete = true and the total; so five days at batch_size 2 travel as three envelopes of 2, 2 and 1 days, four days as 2, 2 and 0, and no days as one envelope (the header, no days, complete, total 0). Of the request, symbol (a bare root, a dated contract or a :cont composite; the three share one calendar and the adapter resolves the root), start, end and batch_size (0 = 4096) reach the adapter; session, bar_spec, grid_revision and the composite pins (snapshot_token, generation, as_of, adjust_anchor) are ignored on this schema. GetDataResponse.snapshot_token rides the final envelope when the adapter set one, as on every schema; the SDK does not fill it from the calendar’s revision. pb wire only: DBN has no record type for it and the CSV grammar no encoding, and both refuse it (the refusal table in Exchange calendar facts).

Two existing messages gained a field each. InstrumentInfo.exchange_timezone (8) is the IANA name of the venue’s exchange-local clock (America/Chicago); empty from an adapter that predates the field or does not know it, and absent from the CSV init line, which is byte-identical with it set. The SDK mirrors it as InstrumentInfo.ExchangeTimezone (omitempty, so every artefact that embeds an InstrumentInfo is unchanged when it is empty), and it reaches a strategy through InitConfig.Instruments and ctx.Instrument(sym). InitAck.needs_calendar (14) mirrors needs_rolls: the strategy SDK sets it when the strategy consumes calendar facts (unit A3; Strategy SDK lookups), gating the engine’s push, and leaves it false otherwise, so the InitAck of a strategy that consumes none is byte-identical; the CSV init_ack line does not carry it. The contract’s draft numbered it 13, which I2’s capabilities had taken by the time the unit froze, so 14 is its number; the draft’s 15, 22 and 8 stand. The push is Envelope.exchange_calendar (61, message ExchangeCalendarBatch): one batch per root with the header and every served day, engine to strategy, after InstrumentsResolved and before the first warm-up bar, only to a strategy that set needs_calendar. 61 is the lowest tag above every allocated and reserved one (the strategy-side tags 1–29 are exhausted, 30–60 belong to the data-adapter, timer and DBN blocks, 53–54 are reserved), and the engine sends it since unit A2a (Engine side). Old readers see identical bytes when the new fields are unset; a reader built from the proto with every I11 addition stripped decodes every earlier field of an Init, an InitAck, a GetDataResponse and an Envelope that carry the new ones, keeps the new bytes as unknown fields, and a current reader recovers them intact.

The order-lifetime v2 wire (GLE-367, increment I2 of the order-lifetime programme) is additive and in effect only on a run that negotiated the lifetime-v2 capability (next subsection). Every field and message below is lifetime-v2: a v1 run sets none of them, so each is zero or absent there, and a message with none of them set marshals byte for byte as before. The strategy-side Envelope gains order_update (27, message OrderUpdate: one attempt’s status transition) and cancel_response (28, message CancelResponse: the classified answer to one cancel request), and since GLE-377 lifecycle_turn (29, message LifecycleTurn: one lifecycle decision turn), all engine to strategy; a v1 run receives none of them. Init.capabilities (6) carries the tokens the engine puts in effect and InitAck.capabilities (13) the tokens the strategy accepts. On the order messages:

• Order.attempt (13): the attempt number of this physical submission under client_id, 1-based and increasing per label within one strategy instance. client_id stays the strategy’s label; (client_id, attempt) names one attempt, and every engine-to-strategy event about the attempt echoes both. 0 on a v1 submission.
• Fill 11–14: attempt; exec_id, the venue’s execution identifier and the de-duplication key of a partial execution; order_ref, the engine’s durable canonical reference for the attempt; remaining_quantity, the quantity still working after this execution (0 on a full fill).
• OrderCancel 5–8: attempt, order_ref, request_id (echoes the OrderCancelRequest the cancel answers, when the reason is CANCEL_REASON_EXPLICIT) and remaining_quantity, the quantity still working when the order was cancelled.
• OrderCancelRequest 2–3: attempt, the attempt to cancel (0 = the label’s current attempt, the v1 meaning), and a strategy-chosen request_id that the CancelResponse and the resulting OrderCancel echo.
• AdoptedState.owner_id (3): the deployment identity the state was recovered under; every working order’s owner_id equals it. AdoptedOrder 10–19, the complete order frame: trail_amount, oco_group, attached_orders (each child a complete AdoptedOrder; a child of an unfilled parent carries status INACTIVE), attempt, order_ref, status, filled_quantity, remaining_quantity, submitted_at and owner_id. A v1 snapshot carries the nine v1 fields alone.
• OrderDenied 4–5 and OrderRejected 5–6: attempt and order_ref of the attempt denied or rejected.
• OrderUpdate, fields 1–13: client_id, attempt, order_ref, symbol, status, quantity, filled_quantity, remaining_quantity (the attempt’s original, cumulative filled and still-working quantities), timestamp, reason and broker_code (the venue’s or the engine’s text for a terminal or held transition), parent_client_id and owner_id.
• CancelResponse, fields 1–9: client_id, attempt, request_id, outcome, reason, status and remaining_quantity (the attempt’s state as evidence), timestamp and order_ref.
• LifecycleTurn, fields 1–3: at, the instant; turn, 1-based within the instant; budget, the turns the run grants per instant. The strategy answers it exactly as it answers a Bar: zero or more OrderCancelRequest envelopes, then one OrderResponse whose for_bar echoes at. No bar travels with it, so a turn advances no series (Lifecycle turns).
• Enums: OrderStatus 0–12 (UNSPECIFIED, SUBMITTING, SUBMISSION_UNKNOWN, WORKING, PARTIALLY_FILLED, PENDING_CANCEL, HELD, INACTIVE, then the terminal FILLED, CANCELLED, REJECTED, DENIED, EXPIRED); CancelOutcome 0–4 (UNSPECIFIED, CONFIRMED, PENDING, REFUSED, UNKNOWN); and CancelReason gains GTC_EXPIRED (7, a GTC order cancelled at the venue’s quarter-end deadline) and VENUE (8, the venue cancelled and the cause was not determined). The SDK mirrors each value for value (OrderStatus, CancelOutcome, CancelReasonGTCExpired, CancelReasonVenue); its CancelReasonParentCancelled (6) was added at the same time, the proto having carried the value since June.

The three new instants, OrderUpdate.timestamp, CancelResponse.timestamp and AdoptedOrder.submitted_at, are emitted only when set and decode as the zero time when absent; the v1 instants (Fill.timestamp, OrderCancel.timestamp, AdoptedPosition.opened_at) keep their v1 encoding, always emitted and the Unix epoch when absent, so the v1 bytes are unchanged. CancelReject is unchanged: it stays the v1 answer to a cancel the engine could not honour, and a run that negotiated lifetime-v2 answers every cancel request with a CancelResponse instead and sends no CancelReject. I2 defined the protocol and the conversions on both sides of the wire (internal/pbconv for the engine, sdk/strategy/run_pb.go and lifetime.go for the SDK; ToPBOrder also carries oco_group now, a correction with no production caller). The behaviour behind each field is its own increment: the attempt-indexed working view (I3, landed under GLE-372: on a run that negotiated lifetime-v2 the SDK stamps Order.attempt on its submissions, sends attempt and request_id on its cancel requests and keeps the view the events move; see The working view under lifetime v2); the lifecycle turns and the cancel classification on the simulated path (I4 unit 4a, landed under GLE-377: a backtest under --lifetime v2 sends OrderUpdate, CancelResponse and LifecycleTurn, stamps attempt on every Fill and OrderCancel, echoes request_id and carries the simulator’s sim-N reference as order_ref; see Lifecycle turns); the live path’s turns and cancel classes (unit 4b); admission limits (I6); persisted ownership (I7: the live order_ref, exec_id and the extended adoption frame); recovery (I8) and CSV v2 (I10, landed: the grammar and its codec, the engine host and the Go SDK’s runner; The stdio-csv-v2 grammar).

Capability negotiation

A capability is a named token the engine puts in effect for a run; the vocabulary has one, lifetime-v2 (wire.CapabilityLifetimeV2, mirrored as algolang.CapabilityLifetimeV2). The handshake rides the Init / InitAck pair and both sides check it (internal/wire/capability.go):

1. The offer. The engine puts the tokens it selects in Init.capabilities (InitConfig.Capabilities in the SDK). An engine that predates the field offers none, and so does every run under the default profile: algo run --lifetime v2 is the one switch that selects lifetime-v2, on a plain backtest or an emulated-live run (Lifecycle turns); its config key is data.lifetime, and the engine’s own tests set InitConfig.Capabilities and drive the run through StdioHost.Init.
2. The acceptance. A strategy declares what it requires by implementing CapabilityRequirer (Reacting to fills and cancels). After the input values are applied and before OnInit, the SDK applies the strategy-side law, wire.AcceptCapabilities(offered, required), walking the required tokens in order. A token this build does not define is an initialisation error, Error code init_failed with the text wire: the strategy requires unknown capability "lifetime-v1" (this build knows: lifetime-v2). A known token the engine did not offer refuses the run with Error code capability_unsupported and the text wire: capability not offered by the engine: the strategy requires "lifetime-v2" but the engine offered none; run it under an engine and execution profile that offer lifetime-v2 (GLE-200 I2) (none becomes the offered list, as [other], when the engine offered something else; the error wraps the sentinel wire.ErrCapabilityUnsupported, and algolang.Run returns it). Otherwise the required tokens go out in InitAck.capabilities, duplicates collapsed. On a refusal OnInit is not called. A strategy that requires nothing accepts nothing, and its InitAck is byte-identical whatever the engine offered.
3. The check. The engine’s strategy host applies the engine-side law, wire.CheckCapabilities(offered, accepted), to the InitAck and refuses the run, prefixing the text with engine: <strategy name>:, when a token was accepted twice (wire: the strategy accepted capability "lifetime-v2" twice), when a token was accepted that was not offered (wire: the strategy accepted capability "lifetime-v2" that was not offered (offered: none), the list again rendered [a b] when non-empty), or when lifetime-v2 was offered and not accepted (wire: the run selected lifetime-v2 but the strategy did not accept it; a strategy that does not implement the v2 order lifetime must run under the legacy execution profile (GLE-200 I2)), so no strategy runs under a lifetime it does not implement. Only lifetime-v2 is mandatory when offered: another offered token the strategy leaves unaccepted is tolerated, as is a token offered twice.

The outcomes, with O = the engine offers lifetime-v2 and R = the strategy requires it:

stdio-csv-v1 has no grammar for capabilities; stdio-csv-v2 (increment I10) carries them on its init and init_ack lines (The stdio-csv-v2 grammar), and a run under --lifetime v2 offers it ahead of v1. The v1 CSV formatters refuse an Init that offers any and an InitAck that accepts any with one text, wire: capabilities [lifetime-v2] are not expressible in stdio-csv-v1; run the strategy on stdio-pb-v1 (GLE-200 I2), and the CSV parsers yield none. So the host cannot offer lifetime-v2 over CSV v1 (its Init fails before any line is written, with that text behind the engine: <strategy name>: prefix), and a strategy that requires it on the v1 CSV wire, where no offer reaches it, is refused by the SDK with the same capability_unsupported code, as the line error,capability_unsupported,<message, commas replaced by ';'>. The code is not request-scoped on either wire; init_failed covers an unknown token and every other initialisation failure.

Compatibility runs in both directions. A reader built from the pre-I2 schema decodes every v1 field of a v2 message, keeps the v2 fields as unknown bytes and re-encodes them unchanged; CancelReason values 7 and 8 reach it as their numbers (proto3 enums are open); a v2-only envelope decodes on it with no message set, which the v1 SDK reports as the unexpected <nil> on the strategy wire protocol error — the reason a host sends OrderUpdate and CancelResponse only to a strategy that accepted lifetime-v2. In the other direction a v1 run is byte-identical on both wires: an Order with attempt 0, a Fill and an OrderCancel with their four new fields zero, an AdoptedState with no owner and v1 frames, an Init and an InitAck with no capabilities each marshal as the v1 message built from the v1 fields alone. That identity reaches the run’s artefacts too: sim.FillReport embeds algolang.Fill, so the new fields on the existing SDK structs (Order, Fill, OrderCancel, InitConfig, InitAck, AdoptedState, AdoptedOrder) carry json:",omitempty" and a v1 run’s --fills-json blotter is unchanged; the three KBD masters and the engine/sim FillReport JSON golden pin it.