Architecture
Package map
| Package or directory | Responsibility |
|---|---|
cmd/algo | CLI, run planning and offline report rendering |
engine/runconfig | Commented JSON parsing, input sweeps, symbol templates and capital allocation |
engine | Process hosts, data client, validation, plain/continuous/live execution; the equity consumer (equity_consumer.go, equity_frame.go: the eq symbol grammar and wire gates, the corporate-action event builder, the raw projection and the ratio frame) |
engine/sim | Deterministic venue, order lifecycle, fills and cash accounting: cash in several currencies converted at the rate in force (currency.go), the margin models (margin.go), combo orders (combo.go), GTC expiry, corporate-action cancels and contract expiry |
engine/venueibkr, internal/ibkrclient | IBKR venue boundary and HTTP client |
engine/ledger, engine/recon, engine/reconcile | Durable order identity, startup reconciliation and adjustment verification |
engine/report | Recording, FIFO trade pairing, statistics, benchmarks, artefacts and HTML |
engine/algorun | The .algorun export format: its schema, chunk codec and validator (The export file) |
engine/adjust, engine/equity | Shared adjustment math; equity corporate-action accounting (the event manifest, the security master) and the engine’s forward-adjust schedule, which serves live mode only since GLE-328 |
engine/continuous, engine/resolver | The served composite consumer and the continuous run’s definition types (continuous.SymbolDef, PriceSpec, AdjustmentSpec), and contract translation by the served constants |
internal/wire, internal/pbconv, proto, gen | Framing, codecs, conversions, schema and generated bindings |
sdk/strategy, sdk/data, sdk/dbnx | Strategy SDK, adapter SDK and DBN encoding |
sdk/data/barconstruct | Adapter-side bar construction and resampling |
internal/seriescache | Cross-process disk caching |
indicators | Thin bridge to nseries; indicator semantics come from that library |
strategies/kit | Shared strategy helpers: sizing, stops, aligned auxiliary reads, cooldowns, the desired-order controller and the exit-once gate (Exiting); every helper, its order-lifetime assumption and the profile it serves are in The strategy kit |
strategies/conform | The shared conformance suite, imported by test files alone: market-only parity between the two lifetime profiles through engine.Run, the host’s refusal of an undeclared strategy, the port harness (Replay, Decisions) and the scripted session harness (Session), a strategy run in-process under a scripted engine on either wire |
internal/goldentest | Legacy blotter comparison and reporting replay |
Process boundaries
The engine launches one adapter and one strategy process for a run. The simulator and venue adapters live inside the engine. Each run owns its strategy state and account; a multi-symbol run remains one strategy instance.
run config -> engine -> data client <-> adapter SDK -> archive/files
|
+-> strategy host <-> strategy SDK -> OnBar
|
+-> venue (simulator or IBKR) -> fills/account state
|
+-> recorder -> result.json, CSVs, report.htmlData protocol preference is DBN, protobuf, then CSV. Strategy preference is protobuf, then CSV; a run under lifetime v2 offers CSV v2 ahead of CSV v1. Session/path requirements can demote DBN during automatic negotiation, as can an activity kind, a non-native interval and a composite among the planned symbols; a continuous run asks for protobuf outright, an adjusted equity forces it, and a raw equity, a constructor lead-in, an exchange-calendar option and a lifetime-v2 run holding a dated futures contract move it to the front of the list with the fallback kept (the contract’s final settlement travels on the statistics stream, which protobuf alone carries; see Contract expiry (lifetime v2)). A forced --data-protocol is honoured as given. JSONL and Unix-domain-socket protocol names are reserved but have no implementation. A message’s presence in the schema does not imply that the engine drives it.
The handshake
Every child process speaks the same two-line ASCII handshake, implemented in internal/wire/wire.go:
- Child writes
ALGO/1 protocols=stdio-pb-v1,stdio-csv-v2,stdio-csv-v1 sdk=go-0.1.0(wire.FormatHello) and flushes. - Engine reads it (
wire.ParseHello), intersects the child’s offer with its own preference list (wire.Negotiate– the engine’s order wins), and repliesALGO/1 use=stdio-pb-v1(wire.FormatUse). - Both sides switch to the negotiated protocol on the same pipes.
Two one-shot adapter modes skip the handshake: the intraday grid capability probe (Intraday grids) and the discovery probe, --algolang-probe (Finding the adapter).
The spawn-and-handshake mechanics live in engine/negotiator.go::spawn. Two details there are load-bearing:
- One
bufio.Readerper child, for its whole lifetime. The handshake line and every subsequent frame or CSV line come off the same buffered reader (child.r). Constructing a second reader after the handshake would silently discard whatever the first one had already buffered. The SDKs obey the same rule on their side (sdk/strategy/run.go::runWith,sdk/data/run.go::run). - stderr is an explicit
os.Pipe, notcmd.StderrPipe(), socmd.Waitdoes not race the stderr-scanner goroutine that re-prints child diagnostics prefixed[name]. A reaped child’s scanner gets a bounded grace period (stderrDrainTimeout, 1s) because a grandchild that inherited the write end can hold the pipe open forever. - A strategy’s hello line has a deadline (
spawnWithin, GLE-494): 30 s from its start (strategyHandshakeTimeout), and the run’s context. The read runs on a goroutine of its own; when the deadline passes or the context ends first, the child is killed and reaped and its stdout closed, which ends the read, andNewStdioHostreturns the deadline’s error or a*HostInterrupt. An adapter’s handshake has no deadline (spawn), as its fetches have none.
Framing
internal/wire/frame.go implements the binary framing shared by stdio-pb-v1 and the stdio-dbn-v1 control plane: a 4-byte big-endian length prefix followed by a serialized Envelope protobuf, capped at wire.MaxFrame (16 MiB). wire.EnvelopeWriter/wire.EnvelopeReader reuse their buffers across calls; neither is safe for concurrent use.
stdio-csv-v1 is newline-delimited comma-separated text with no quoting: fields can never contain commas. Free prose (names, log messages) is sanitized into lookalike characters (internal/wire/csv_strategy.go::SanitizeCSVText); semantic values that cannot ride the grammar are rejected, never rewritten (validateCSVValue), because silently rewriting an identifier would corrupt its meaning. stdio-csv-v2 percent-escapes every field of its command and event lines instead, so they carry any text losslessly (Escaping and fixed arity).
Flush-per-frame is an invariant, not an optimization choice. Both wires are strict request/response at every step, apart from a data subscription’s pushes, so a buffered-but-unflushed message deadlocks both processes. Every write path in the codebase flushes immediately: engine/negotiator.go::conn.writeEnvelope and conn.writeLine, the strategy SDK’s send/writeLine closures in sdk/strategy/run_pb.go, run_csv.go and run_csv2.go, and the data SDK’s sdk/data/run.go::wireConn (which additionally holds a mutex so source goroutines can emit Log lines while a record batch is streaming, and subscriptions can push while the loop answers a request). The one deliberate exception is the engine’s stdio-csv-v2 warm-up block, buffered and flushed once as a whole, which a strategy answers only after its last bar (Warm-up blocks). The engine’s writes to a strategy stay on the caller’s goroutine through that one writer, so the drain below changes no byte of any wire.
Cancellation and strategy output. Since I4 unit 4c of the order-lifetime programme (GLE-416; engine/hostio.go) the engine reads a strategy’s output concurrently: one drain goroutine per connection, started by the first host call and the connection’s only reader from then on. A Log envelope or a log line is printed on stderr as it arrives, in wire order, whether or not a host call is waiting for a reply; every other frame is queued for the host in wire order, at most 256 of them beyond the one in hand, and a strategy that writes more unsolicited frames than that is held by back-pressure: nothing is dropped and nothing grows while the connection is open. So a strategy whose handlers log while the engine writes it a run of events (a bar with many fills: OrderUpdate, Fill, CancelResponse, one after another with no reply) cannot deadlock the engine, and a reply’s log lines are printed before the reply is handed over. When a run’s context ends while a host call is blocked in a write (a strategy that stopped reading) or a read (one that stopped answering), the connection’s raw streams are closed, which fails the blocked operation where it is and gives the strategy end of input, and the call returns within 2 s with the error engine: <strategy name>: strategy I/O interrupted while writing: context canceled (or while reading; the value is an *engine.HostInterrupt). Its cause is the context’s (context.Cause): context canceled or context deadline exceeded for a plain context, and for a run’s the cause the run ended with, interrupt signal received for a Ctrl-C or the stall timeout’s text (Stopping a run); either way it unwraps to the context’s error. The first cause sticks: every later call on the connection returns the same interrupt, an interrupted Shutdown returns it at once, and the engine’s deferred Close kills and reaps the strategy process, then waits up to 2 s for the drain to stop and reports a leaked drain goroutine otherwise. A host call entered with a context that has already ended returns the context’s cause before anything is written, without closing the connection. The wire bytes are unchanged on every wire, pb and CSV alike, and the KBD masters are byte-identical; the one difference an operator may notice is that a strategy’s log line can now appear on stderr before the engine’s own later lines, where it used to wait until the engine next read a reply. At a run’s end the strategy’s last lines are printed too (since GLE-495, landed 2026-10-11; engine/childout.go): the child’s stdout is a pipe of the engine’s own, which reaping the child does not close, so the log lines a strategy writes on receiving shutdown are printed, in order, before Shutdown returns, and the drain ends at the pipe’s end rather than on a pipe closed under it. The bound is the reap as before (the stdin closed, the exit awaited, the kill after 5 s for a strategy that ignores end of input), then up to 2 s for the drain to reach the end of the pipe: a strategy that exited and closed its stdout is drained in milliseconds, while a helper process the strategy left holding its stdout costs the whole grace, about 3 s of Shutdown with the second the stderr scanner is given first, after which the read end is closed and the helper’s later writes fail. Nothing extra is written to stderr when the grace expires: the strategy exited 0, and what holds its stdout is not the strategy. What ends a run’s context, the signals and the strategy stall timeout, is under Stopping a run; the limits that remain (the data wire; the kill path, which loses what is in the pipe at the kill) are listed under The engine host.