Data adapters

The three bundled adapters

make build puts them in bin/ with an -adapter suffix; make install copies them into the adapter directory without it (Installing).

bin/file-csv-adapter serves pre-aggregated time bars from a CSV file in the Algolang bar-file format – header then one bar per line:

symbol,ts_open,ts_close,open,high,low,close,volume
MSFT,2022-06-10T12:30:00Z,2022-06-10T12:31:00Z,265.76,265.76,261.01,261.06,3006

Timestamps are RFC 3339; prices are plain decimals. Inputs (passed via --adapter-inputs): Path (required) and Interval, the file’s native bar interval (default 1m). It serves time bars only – at the native interval, or any whole multiple of it by deterministic resampling. A release-stamped symbol (cot:, macro:, fx:RBA:) is served row for row at its own cadence, whatever the requested interval, so one file can hold hourly prices and a FRED series with its optional ref column. An optional CorporateActions input names a ledger CSV; a request on a qualified equity symbol (eq:V:T:adj=raw) is answered with the rows filed under the base symbol eq:V:T as well, as marketfeed’s adapter does. Its bars are raw only: it serves no adjusted view, so an eq:…:adj=split or adj=splitdiv run against it is refused by the engine’s served-bar check (GLE-328). An optional Observations input names a vintages CSV, from which it serves the OBSERVATIONS schema: the revision history of a release-stamped series (Revisions: the OBSERVATIONS schema). An optional Listing input names a futures inventory CSV, from which it serves the CONTRACT_LISTING schema: the contracts of a root listed at an instant (Contract listings). Beside TickSize and PointValue, the optional MarginInitial, MarginMaintenance (defaulting to the initial margin) and MarginCurrency (default USD) inputs serve a margin for every symbol, as an undated registry seed (Instrument metadata). An optional Currencies input (GLE-391) serves each named symbol’s quote currency in its instrument metadata, as SYMBOL:CCY pairs separated by semicolons (Currencies=fut:XCME:NIY:H26:JPY;MSFT:USD; the currency is the text after a pair’s last colon, upper-cased). A symbol it does not name is served no currency, and the engine takes it in the base currency (Currencies). An optional Descriptions input (GLE-1358) serves each named symbol’s description, its name for people, the same way: SYMBOL:NAME pairs separated by semicolons (Descriptions=MSFT:Microsoft;fut:XCME:ES:E-Mini S&P 500; the name is the text after a pair’s last colon). A symbol it does not name is served none.

Optional IBKR inputs (GLE-1464) serve the IBKR contract terms a live run needs to find each symbol’s contract at the IBKR venue, the emulator’s included. IBKRSecType (FUT or STK) turns them on and applies to every symbol. Each symbol is searched by its own name unless IBKRSymbols names another, as SYMBOL:SEARCH pairs separated by semicolons (IBKRSymbols=fut:XCEC:SIL:Z26:SI). IBKRExchange is the exchange (a stock’s listing exchange, for example NASDAQ). IBKRTradingClass and IBKRCurrency are optional; the currency defaults to the symbol’s Currencies entry, else USD. IBKRContractMonth (yyyymm) is a dated future’s contract month, and Expiration (RFC 3339) is served as every symbol’s expiration, which a future’s terms need. IBKRReason serves a reason in place of the terms. A symbol is served the same terms as every other, so one run holds one kind of instrument. A live run on the emulator’s AAPL fixture adds:

--adapter-inputs Path=AAPL.csv,IBKRSecType=STK,IBKRExchange=NASDAQ

bin/databento-adapter serves a local Databento trades file, CSV or DBN (auto-detected by content); it reads the file only and never contacts Databento. Inputs: Path (required) and Dataset (default XNAS.ITCH). Because it owns raw trades it can serve the TRADES schema (which also powers tick-resolution fills) and construct bars of every one of the fourteen kinds, at any interval.

bin/ibkr-adapter is a data adapter, despite sharing a provider with the IBKR venue adapter. It reads native 1-minute historical bars from a Client Portal gateway or bin/ibkr-emulator. It serves those 1-minute bars only, so a run on it names --bar-interval 1m: the default, 1d, is refused. Its required input is EmulatorURL (the gateway or emulator base URL); its optional inputs are AccountID, a static Symbols symbol-to-conid table, and DatabentoFixture, below. It is separate from order execution: selecting it with --adapter does not select live brokerage, and selecting --mode live does not automatically select it as the data source.

Corporate actions come from marketfeed’s adapter, not this one. A run that needs them takes them from marketfeed’s adapter, which serves them from the archive (the lake; Equities and live reconciliation). The IBKR adapter never asks Databento for them: only marketfeed talks to Databento (the owner’s rule of 20 Nov 2026, GLE-1790). Its HTTP backend is removed (GLE-1795), and with it the DatabentoURL and DatabentoKey inputs; it does not read DATABENTO_API_KEY, so a shell that exports the key changes nothing, and the only host it calls is the gateway at EmulatorURL. A run that still passes one of the two removed inputs gets the SDK’s unknown input ... ignored warning.

One input remains, for tests: DatabentoFixture, the path of a recorded file of Databento’s corporate-action records (a JSON array). With it the adapter serves the file’s records; without it the adapter serves none. It advertises the corporate-actions schema either way, so the engine does not refuse an adjusted equity symbol on it: without the fixture such a run has no corporate actions, and the adapter’s start-up log line says so (CA backend none: DatabentoFixture is not set ...).

All three match a requested symbol in any case (GLE-514): a file that spells it MSFT serves a request for msft, and the bars, trades, observations and listings they serve carry the request’s spelling. That matters because the engine checks the symbol on every observation and listing record and matches orders to bars by symbol. An adapter written for Algolang should do the same. The engine sends every symbol in its canonical spelling (Symbols are case-insensitive), but an adapter run by hand, or a data file written by hand, may use another.

Finding the adapter

--adapter is optional. Without it, algo run asks every data adapter in the adapter directory whether it can serve the run and runs the one that says yes. The adapters decide: the engine keeps no table of file formats or symbol families, and an adapter’s input names stay its own.

The adapter directory is $ALGOLANG_ADAPTERS when that is set, and otherwise /usr/local/algolang/adapters, or wherever PREFIX put it at build time (Installing). Each executable file in it is an adapter; files whose names start with ., subdirectories and files without an execute bit are skipped. A third-party adapter is installed by copying it there, and its name needs no suffix. The bundled adapters say yes in these cases:

Installed asSays yes when
databentoPath names a file it reads: DBN, or a CSV with Databento’s trades header. It reads the first bytes only.
file-csvPath names an Algolang bar file, told from its header line.
ibkrEmulatorURL is set. The gateway is not contacted.
marketfeedThe archive (Dir, then $MARKETFEED_DIR, then /usr/local/marketfeed) has a coverage manifest for every requested symbol. It stats one file per symbol and says nothing about the range or interval.

Discovery runs in four steps:

  1. The engine starts every adapter in the directory at once, each with the one argument --algolang-probe. It sends each the run’s request: the symbols, the schema, the bar kind, the interval (time bars only), the session, the range and the --adapter-inputs.

  2. Each adapter answers yes or no, with a reason for a no. An adapter that does not answer within 10 seconds, exits non-zero or prints anything else counts as a no.

  3. When exactly one says yes, the run uses it. The report’s adapter line says where it came from:

      adapter:    databento (stdio-dbn-v1), inferred from /usr/local/algolang/adapters

    The adapter’s path, not the fact that it was discovered, is what the series-cache key, the BENCH line and result.json record, exactly as for an adapter named with --adapter.

  4. Otherwise the run stops before it starts. With no yes, the error gives each adapter’s reason:

    algo: no adapter in /usr/local/algolang/adapters can serve MSFT@1m BARS; pass --adapter
      databento: required input Path was not supplied
      file-csv: required input Path was not supplied
      ibkr: required input EmulatorURL was not supplied

    With several, it names them and asks you to choose one with --adapter.

The inputs help choose. An adapter answers no when an input name is not one of its own, so --adapter-inputs Path=... rules out every adapter that takes no Path. The file’s content then decides between databento and file-csv:

bin/algo run \
  --adapter-inputs Path=testdata/xnas-sample/xnas-itch-20220610.trades.csv \
  --strategy bin/hello \
  --symbols MSFT --bar-interval 1m

--adapter overrides discovery and probes nothing. A path (--adapter bin/databento-adapter) runs as given. A bare name (--adapter databento) is looked up in the adapter directory first and, when it is not there, on $PATH as before.

A config file still names its adapter: data.adapter is required, and a bare name there resolves as --adapter does. A run matrix is a record of what ran, so it does not depend on what happens to be installed.

The algodata subcommands that read through an adapter (info, listing, stats, defs and obs) and cmd/tools/rollsched resolve -adapter the same way. Without it, they ask the adapters about -symbol (or -root) and the schema they read, and name the adapter they chose on stderr.

Making an adapter discoverable

An adapter built on the Go data SDK answers the probe by implementing data.Prober. It returns nil for yes and an error for no; the error’s text is the reason the user sees.

func (a *MyAdapter) Probe(req data.ProbeRequest) error {
    for _, sym := range req.Symbols {
        if !a.covers(sym) { // a manifest lookup, not a data read
            return fmt.Errorf("the archive has no %s", sym)
        }
    }
    return nil
}

The SDK applies the request’s inputs to the adapter’s tagged fields, with their defaults, before it calls Probe; it does not call OnInit. It answers no itself, without calling Probe, in three cases: an input name is not one of the adapter’s, a required input is missing or invalid, or no capability serves the request’s schema. Probe must be cheap, because the engine probes every installed adapter before every run that names none. It must make no network request, and should read only enough of its data to recognise it (a header, a manifest).

An adapter that does not implement data.Prober answers no: the adapter does not implement the discovery probe. One built before the probe existed refuses the flag and exits non-zero. Discovery never chooses either; --adapter still runs them.

An adapter written without the Go SDK implements the protocol itself. Started with the single argument --algolang-probe, it reads one JSON object and a newline from stdin:

{"version":1,"symbols":["fut:XCME:ES"],"schema":"BARS","bar_kind":"BARS_TIME","interval":"1d","start":"1970-01-01T00:00:00Z","end":"2100-01-01T00:00:00Z","inputs":{}}

bar_kind, interval and session are absent when they do not apply, and inputs is always an object. The adapter writes one line on stdout, yes, no or no: <reason>, and exits 0. No handshake follows.

The algodata tool

bin/algodata converts between the formats (this is exactly what make testdata runs):

$ bin/algodata csv2dbn -in testdata/xnas-sample/xnas-itch-20220610.trades.csv -out testdata/msft-trades.dbn
csv2dbn: wrote testdata/msft-trades.dbn and verified 1000 records (count, first/last price and ts_recv match)

$ bin/algodata csv2bars -in testdata/xnas-sample/xnas-itch-20220610.trades.csv -out testdata/msft-1m.csv -bar-interval 1m
csv2bars: wrote 23 1m bars to testdata/msft-1m.csv

$ bin/algodata replicate -in testdata/xnas-sample/xnas-itch-20220610.trades.csv -out testdata/msft-trades-x1000.csv -n 1000
replicate: wrote 1000 copies (1000000 rows, shift 23m0s) to testdata/msft-trades-x1000.csv

csv2dbn builds a real binary DBN file from a Databento trades CSV (and re-decodes it to verify); csv2bars aggregates trades into the bar-file format the file-csv adapter reads, at -bar-interval (default 1d, as for algo run; -interval is its older name); replicate clones a trade file N times with time shifts, for benchmarking. (stats and defs ask an adapter for venue statistics and instrument definitions; see Statistics and instrument definitions. obs asks for the revisions of a release-stamped series; see Revisions: the OBSERVATIONS schema. listing asks which contracts of a futures root were listed at an instant; see Contract listings. info prints one symbol’s instrument metadata, its margin included; see Instrument metadata.)

Statistics and instrument definitions

Besides bars and trades, the data wire carries two event schemas a futures strategy needs (GLE-396):

  • STATISTICS: the venue’s published statistics for a dated contract: the settlement price, open interest, cleared volume and the other kinds of the StatType enum. A record carries its kind and either a price (price kinds) or a quantity (count kinds), never both, so an open interest of 0 is a real zero rather than “does not apply”. It also carries the instant it applies to (ts_ref, a session’s trade date for a settlement), the venue’s sending instant (ts_event), when it was published (ts_recv), and whether it adds or deletes a value. A point-in-time reader gates on ts_recv: a settlement for Friday’s session is not known at Friday’s close.
  • INSTRUMENT_DEFINITION: what the venue said a contract was on a given day: its venue symbol (ESM6), security type, activation and expiration, tick size, multiplier, currency and maturity year, month and day, under its canonical dated symbol (fut:XCME:ES:M26). ts_event is when the definition was current.

A request names a dated contract or a bare root, and every record names its dated contract. A STATISTICS request can name the kinds it wants (GetDataRequest.stat_types); naming none asks for every kind the adapter serves. Both schemas travel on the protobuf data wire only: the DBN and CSV data wires have no records for them, and the data SDK refuses a request for them there.

Querying an adapter. algodata stats and algodata defs ask any adapter that advertises the schema and print one record per line, through the same data client a run uses:

$ bin/algodata stats -adapter bin/marketfeed-adapter -symbol fut:XCME:ES:M26 \
    -start 2026-03-09 -end 2026-03-10 -types settlement_price,open_interest
symbol,type,ts_ref,ts_recv,price,quantity,action
fut:XCME:ES:M26,open_interest,2026-03-06T00:00:00Z,2026-03-09T13:51:03.41604285Z,,61208,added
fut:XCME:ES:M26,settlement_price,2026-03-09T00:00:00Z,2026-03-09T20:00:32.680045343Z,6852,,added
fut:XCME:ES:M26,settlement_price,2026-03-09T00:00:00Z,2026-03-09T21:38:51.842204856Z,6852,,added
fut:XCME:ES:M26,settlement_price,2026-03-09T00:00:00Z,2026-03-09T23:49:41.154481064Z,6852,,added

$ bin/algodata defs -adapter bin/marketfeed-adapter -symbol fut:XCME:ES \
    -start 2026-03-02 -end 2026-03-03
symbol,raw_symbol,maturity,activation,expiration,tick_size,ts_event
fut:XCME:ES:H26,ESH6,2026-03,2023-08-18T21:30:00Z,2026-03-20T13:30:00Z,0.25,2026-03-01T13:02:33.442208215Z
fut:XCME:ES:H27,ESH7,2027-03,2023-08-18T21:30:00Z,2027-03-19T13:30:00Z,0.25,2026-03-01T13:02:33.442208215Z
...

marketfeed windows statistics on publication: a window holds the records whose ts_recv falls in it, so Friday’s open interest, published on Monday, belongs to Monday’s window above. Every published record is served — the settlement for 9 March arrives three times, at 20:00, 21:38 and 23:49 UTC — and a point-in-time reader keeps the latest one published by its decision instant. Records come in the adapter’s order (ts_ref, contract, kind), not in ts_recv order. A definition request serves the daily snapshots of the day files the window selects; a snapshot’s ts_event is when the venue last sent it, usually the day before. The adapter’s side is documented in marketfeed’s “Serving statistics and definitions to the engine”.

-types takes the StatType names in lower case without the STAT_ prefix (settlement_price, open_interest, cleared_volume, …). -adapter-inputs K=V,... passes adapter inputs, as algo run does.

Serving them from an adapter. An adapter implements data.StatisticSource or data.DefinitionSource (or both) and advertises the schema in Capabilities(). The SDK refuses to start when an advertised schema has no source or a source is not advertised:

func (a *MyAdapter) OnGetStatistics(ctx *data.DataContext, req data.StatisticRequest, emit chan<- []data.Statistic) error {
    // req.Types is nil when the request named no kinds: serve every kind.
    for _, s := range a.read(req.Symbol, req.Start, req.End, req.Types) {
        emit <- []data.Statistic{{
            Symbol: s.Contract, Type: algolangpb.StatType_STAT_OPEN_INTEREST,
            Quantity: &s.OpenInterest, TsRef: s.TradeDate, TsRecv: s.Published,
            Action: algolangpb.StatUpdateAction_STAT_ADDED,
        }}
    }
    return nil
}

OnGetDefinitions has the same emit contract with data.DefinitionRequest and data.Definition. A source pushes slices of any size and must not close emit; the SDK re-batches to the request’s batch size.

Reading them in the engine. DataClient.ServesStatistics() and ServesDefinitions() report the adapter’s capabilities; FetchStatistics(symbol, types, start, end, batchSize) and FetchDefinitions(symbol, start, end, batchSize) return the records over [start, end) in the adapter’s order, check that each names its contract and that the final count matches, and need the pb data wire. A strategy receives either schema by declaring it, as market events between bars; see Statistics and definitions as events.

Contract listings

A store’s contract inventory lists whatever is on disk today: ask it on 16 March 2026 for the contracts of ES and it also names contracts first listed months later. A backtest that chooses among contracts, a calendar spread or a curve, needs the answer as it stood at each decision instant. The CONTRACT_LISTING schema (data schema 17, GLE-399) gives it: a request names a futures root and an instant, as_of, and the response is every contract of the root listed then, one ContractListing record each.

The rule. A contract is listed at instant t when all of these hold:

  • the exchange had activated it and not yet expired it: activation <= t < expiration;
  • the store holds a bar of it that started by t: first_bar <= t;
  • the store had its definition by t: known_at <= t, or the store does not know when it first held the definition.

data.Listed(c, t) is the one statement of this rule in the SDK. A contract the exchange has defined but not yet activated is not listed; nor is an activated contract that has not traded, because a strategy could not have seen a price for it.

The record. Each record carries the canonical dated symbol (fut:XCME:ES:M26) and the contract code (M26), activation and expiration, the maturity year, month and day (0 where the exchange gives no such granularity), tick_size, the store’s coverage first_bar and last_bar, and known_at. Two fields are point-in-time on purpose:

  • last_bar is the end of the contract’s newest stored bar, clamped to the request instant: nothing stored after as_of is reported. It is an upper bound on what had been stored by then, not the end of the last bar before it.
  • known_at is when the listing fact became true and known: the latest of the definition’s first appearance, activation and first_bar. It is never after as_of, and it is the record’s meta.ts_recv.

Records ascend by expiration, then symbol, one per contract.

Querying an adapter. algodata listing asks any adapter that advertises the schema and prints the records in the column order of the file-csv adapter’s listing file, so a printed listing can be served again. With the real ES inventory in testdata/gle399/es-listing.csv (activation, expiration and maturity from marketfeed’s GLBX definitions, coverage from its hourly bars):

$ bin/algodata listing -adapter bin/file-csv-adapter \
    -adapter-inputs Path=testdata/gle357/FRED_GDP.csv,Listing=testdata/gle399/es-listing.csv \
    -symbol fut:XCME:ES -at 2026-03-16T14:00:00Z
symbol,contract,activation,expiration,maturity,tick_size,first_bar,last_bar,known_at
fut:XCME:ES:H26,H26,2023-08-18T21:30:00Z,2026-03-20T13:30:00Z,2026-03,0.25,2025-01-31T15:00:00Z,2026-03-16T14:00:00Z,2025-01-31T15:00:00Z
fut:XCME:ES:M26,M26,2023-08-18T21:30:00Z,2026-06-18T13:30:00Z,2026-06,0.25,2025-04-23T17:00:00Z,2026-03-16T14:00:00Z,2025-04-23T17:00:00Z
fut:XCME:ES:U26,U26,2023-08-18T21:30:00Z,2026-09-18T13:30:00Z,2026-09,0.25,2025-07-28T15:00:00Z,2026-03-16T14:00:00Z,2025-07-28T15:00:00Z
fut:XCME:ES:Z26,Z26,2021-09-17T13:30:00Z,2026-12-18T14:30:00Z,2026-12,0.25,2025-09-10T17:00:00Z,2026-03-16T14:00:00Z,2025-09-10T17:00:00Z
fut:XCME:ES:H27,H27,2023-08-18T21:30:00Z,2027-03-19T13:30:00Z,2027-03,0.25,2025-10-03T11:00:00Z,2026-03-16T14:00:00Z,2025-10-03T11:00:00Z

The inventory also holds M31 (June 2031). CME sent its definition on 8 March 2026, but it activates on 20 March, so it is not listed on the 16th. Asked again at -at 2026-03-23, after the March expiry, H26 has gone and M26 to H27 remain; M31 is activated by then but has no bar, so it is still not listed. -at takes a date (UTC midnight) or an RFC 3339 instant; -adapter-inputs passes adapter inputs, as algo run does.

The file-csv adapter’s Listing input is a CSV with the fixed header symbol,contract,activation,expiration,maturity,tick_size,first_bar,last_bar,known_at, one row per contract, in any order. Times are RFC 3339; maturity is YYYY, YYYY-MM or YYYY-MM-DD; first_bar and last_bar are empty together when the store holds no bars of the contract; known_at is empty when the first sight of the definition is unknown. A row is the store’s inventory, not a listing: the adapter applies the rule.

Serving it from an adapter. An adapter implements data.ListingSource and advertises CONTRACT_LISTING. data.SelectListing turns a root’s whole inventory into exactly the records a request gets: the rule, the two point-in-time adjustments, the order, and one record per symbol. Use it rather than restating the rule:

func (a *MyAdapter) OnGetListing(ctx *data.DataContext, req data.ListingRequest, emit chan<- []data.ContractListing) error {
    // req.Symbol is the root; req.AsOf the instant, never zero (the SDK
    // refuses a request without one).
    inventory, err := a.contracts(req.Symbol) // every contract the store holds, with its coverage
    if err != nil {
        return err
    }
    if out := data.SelectListing(inventory, req); len(out) > 0 {
        emit <- out
    }
    return nil
}

The schema travels on the protobuf data wire only; the SDK refuses it on the DBN and CSV wires, and refuses a request without as_of as bad_request. The file-csv adapter reads a root’s rows on the root’s first request and keeps them for the run, so a run may ask for the listing at every decision instant.

Reading it in the engine. DataClient.ServesListing() reports the capability, and FetchListing(root, at, snapshotToken, batchSize) returns the listing at at. It refuses a zero instant before any I/O, and checks the whole response before returning it:

  • every record is a contract of the root (<root>:<contract>);
  • the rule holds at at;
  • known_at is set and equals meta.ts_recv;
  • known_at is not before activation or first_bar, since the listing cannot have been known before it was true;
  • last_bar lies in [first_bar, at];
  • the tick size is finite and not negative;
  • the maturity month and day are in range;
  • records ascend strictly by (expiration, symbol);
  • no contract appears twice, even under a later expiration.

SelectListing’s output always passes. A refused response returns no records.

A strategy reads the listing at its decision instant with ctx.Curve(root) (see The listed curve), which the engine serves from these requests.

Contract curves

A root’s contracts trade along a curve: the front, the contract after it, and so on out. A strategy reads the curve through deferred composites (GLE-410): the root held one, two or three contracts behind the front, named by the ofs= qualifier on the :cont symbol, fut:XCME:ES:cont:ofs=B1. Each is a continuous series of its own. It is stitched from the lineage the feed stores for that rank, rolls at the root’s served cuts (the front and B1 step together), is adjusted under its own adj= method and is delivered under its own symbol. Which contracts the curve holds at a decision instant is ctx.Curve(root), the root’s point-in-time listing (see The listed curve below).

Where the data comes from. marketfeed serves a deferred composite only at a rank it stores: the rank the root’s own schedule holds, or an extra rank the root declares in contract.extra_ranks, which its nightly derives (GLE-404). ES declares B1. The feed refuses any other rank, naming the ranks it stores, and the run fails before its first bar. The engine checks that the schedule it is served holds the rank it asked for, and refuses one held at another (HELD_RANK_MISMATCH).

Naming one. ofs=B1, B2 or B3 (the value case-insensitive) may appear once, before or after adj=. ofs=F is refused: the plain composite is the root’s own product, which need not hold the front (ZQ holds B1). A run names a deferred composite as a series (--symbol fut:XCME:ES:cont:ofs=B1), or a strategy declares it in DataRequirements. A run takes each rank of a root once.

Several ranks of a root can take orders (GLE-484). The front and B1 hold the same contracts at different times: ESM26 is B1’s until the March cut and the front’s after it. Both may trade. Configure both (--symbols fut:XCME:ES,fut:XCME:ES:cont:ofs=B1), or declare the deferred one and name it in TradeableSymbols (under universe-expansion=full). Every fill and cancel belongs to the series whose order produced it, not to the chain that holds its contract, so each series’ account is its own:

  • Positions. ctx.Position("fut:XCME:ES:cont:ofs=B1") is B1’s lots, whatever the front holds in the same contract. The venue holds the sum, per contract.
  • Rolls. At the root’s cut each series rolls its own position with legs of its own. The legs are not netted, so a contract one rank leaves as the other enters it is traded twice at the cut, paying commission and slippage on both legs. Ranks roll innermost first: a rank rolls once no other still holds the contract it enters, so at one cut B2 rolls before B1 and B1 before the front. A roll touches only its own series’ working orders: a GTC order B1 rests on ESM26 is cancelled by B1’s roll at the cut (reason roll), not carried into the front.
  • Order lifetime v2. The race guard books an order on a shared contract on the series that places it, the one whose active contract it is, and keeps it there for the attempt’s life. A contract the ranks hold when it expires is settled once per holder, each settlement fill and contract_expiry record on its own series, even when their lots net to nothing at the venue. Any remainder no series holds settles through the plain delivery, on the contract’s own symbol, and no series counts it. The summary’s contract_expiries still counts contracts. Such an expiry refuses only the held attempts pinned to the expiring contract, with the contract named in the reason; a holder’s attempt on its own logical symbol, bound for its active contract, stays held (GLE-491; Contract expiry (lifetime v2)). An order for a contract that the guard holds across a cut is refused at the roll of the rank leaving the contract, hold_rolled, stamped at the cut and delivered before the roll’s own cancels, and its exposure is released from the placing rank’s book (GLE-491; Order lifetime v2 across a roll). Before that it was released after the cut and routed to the rank whose active contract the contract had become, while its exposure stayed booked on the rank that placed it, so the guard’s book and the fill’s attribution parted ways; now they cannot.
  • No cross-rank cap. Each series’ exposure is guarded on its own. Nothing limits the root’s net or gross position across its ranks.

Two series of one rank are still refused, and so are two roots sharing a contract. Live, the opposite legs two ranks can send on one contract at one instant (B1 selling ESM26 as the front buys it) are orders the venue may treat as a self-trade. Whether IBKR’s self-trade prevention cancels one of them is not verified, as GLE-471’s assumptions about native OCA groups are not (live runs of them wait for GLE-490).

A rank declared without TradeableSymbols is reference-only: - its bars arrive; - its own orders are refused (… is declared for reference only: the run takes no orders on it); - it claims no fill and no position; - its rolls touch nothing on the venue.

A reference-only rank’s bars match no order in the simulator, so an order resting on a contract a trading rank will hold later fills on that rank’s prices, not on the reference’s. Contract facts (tick, point value, margin) are the root’s, the same on every rank. The engine fixtures are in engine/gle484_two_ranks_test.go and engine/gle484_lifetime_v2_test.go.

A run whose own series is the deferred composite trades its active contract: an order on fut:XCME:ES:cont:ofs=B1 fills on ESM26 before the March cut, and the position rolls to ESU26 there. The run report names the rank the symbol asked for: continuous: fut:XCME:ES:cont:ofs=B1 holds B1 as its symbol names (the contract after the front). The engine fixtures are in engine/gle410_deferred_e2e_test.go.

The example. strategies/demo/go/termcarry trades fut:XCME:ES and declares fut:XCME:ES:cont:ofs=B1:adj=none for reference. On each trade date with both bars it reads the raw close of the contract each series holds, and computes the annualised roll yield from them and the months between their deliveries (carry.go): ln(front/next) × 12 / months. A positive yield is backwardation, which a long in the front earns as it rolls up the curve; a negative one is contango, which it pays. It holds one lot while the yield’s 5-day average is above its 60-day average, and is flat otherwise:

front, next, ok := s.pair.Add(leg, date, Quote{Close: bar.RawClose, Contract: bar.ActiveContract})
if !ok {
    return nil, nil // the trade date's other series has not arrived yet
}
y, ok := RollYield(front, next) // e.g. ESZ24/ESH25: -0.044
make run-termcarry   # 2015-2024; TERMCARRY_START/TERMCARRY_END move the window

Over 2015–2024 on the archive it makes 171 fills and ends at $196,312.50 on $100,000, in the market 19% of the time. Since the order-lifetime programme’s I21 (GLE-467) it declares optional lifetime-v2 support, and golden_test.go beside it runs that window and the spread’s under both profiles through the marketfeed adapter (make test-golden), the fills equal under each, the spread’s roll legs among them (Conformance under both lifetimes). The yield it reads tracks the financing cost of holding the index against its dividends: about +1.5% in 2015, when rates were near zero, and about −4.4% in December 2024. Across the December 2024 cut both series step together, the front from ESZ24 to ESH25 and B1 from ESH25 to ESM25. make test-golden pins the run.

With Spread=true it holds the calendar spread instead: both ranks are the run’s series, and while the yield’s fast average is above its slow one it is long one front lot and short one B1 lot (GLE-484):

make run-termcarry-spread TERMCARRY_START=2023-01-01T00:00:00Z   # both ranks as series, --inputs Spread=true

Over 2023–2024 it places 38 orders, and its spread is open at two cuts. At each one, each rank rolls its own lot: at the December 2023 cut B1 buys back ESH24 and sells ESM24, then the front sells ESZ23 and buys ESH24. That is 46 fills, and the run ends at $99,675.00 on $100,000, still holding the spread (ESH25 long, ESM25 short). Each series’ trades.csv rows are its own: ten each, two of them rolled once. make test-golden pins it.

The listed curve

A strategy asks which contracts of a root are listed at its decision instant with ctx.Curve(root) (GLE-410). It declares the root’s CONTRACT_LISTING (see Contract listings), and the engine delivers the listing to it as market events:

func (s *CurveWatch) DataRequirements() []algolang.DataRequirement {
    return []algolang.DataRequirement{{SeriesID: "curve", Symbol: "fut:XCME:ES",
        Schema: algolang.DataSchemaSpec{Kind: algolang.ContractListing}}}
}

func (s *CurveWatch) OnBar(ctx *algolang.Context, bar algolang.Bar) ([]algolang.Order, error) {
    for _, l := range ctx.Curve("fut:XCME:ES") { // front first
        fmt.Println(l.Contract, l.Expiration, l.FirstBar, l.LastBar)
    }
    return nil, nil
}

ctx.Curve(root) returns the contracts listed at ctx.Now(), front first (ascending expiration, then symbol), as algolang.MarketListing records: the fields of a ContractListing, the declared SeriesID, and TsRecv, the instant the record was delivered. It is data.SelectListing, the rule every adapter applies, over the latest record delivered for each contract, so a contract joins the curve once it is activated, has traded and is known, and leaves it at its expiration. A contract listed after the bar is never in it, although the store holds it today. The slice is the caller’s. It is nil before anything of the root has been delivered.

What the engine delivers. The declaration names a bare futures root (fut:<venue>:<root>) and takes no Lookback: the curve keeps the latest record of each contract. The engine takes the root’s listing at the start of the event window, at the start (00:00 UTC) of each later day on which a bar of the run closes, and at the run’s last close, one adapter request each. From those listings:

  • a contract’s first record is delivered at its known_at, with last_bar clamped there; its other fields are those of the listing that first holds it. A contract that first traded at 15:00 is in the curve at that day’s close, not at the next day’s listing, even when that close is the run’s last. If the previous listing had already passed the contract’s known_at without holding it (the store learned of it late), it is delivered at the listing that first holds it instead;
  • a record is delivered again at a daily listing when it has changed. Its last_bar grows each day the contract trades, so the curve’s coverage is at most a day behind;
  • each delivered record’s meta.ts_recv is its delivery instant, and visibility follows the market-event rule: a record delivered strictly before a bar’s close is seen at that bar.

The optional ListingHandler interface, OnListing(ctx, l), is called for every delivered record, after it has joined the curve and with ctx.Now() at its TsRecv. Like the other market-event handlers it is observe-only.

On a continuous run. A listing is the one market event a continuous run serves: the curve belongs to the root, not to a contract. The declaration may keep its default id, the root, although the root’s continuous series has the same id, because ctx.Curve reads by root. Trades, statistics, definitions and observations stay refused on a continuous run.

Limits.

  • Declared listings are backtest-only; a live run refuses them.
  • The adapter must serve CONTRACT_LISTING (file-csv and marketfeed do), and the strategy must speak the protobuf wire.
  • A listing costs one adapter request per trade date. On marketfeed that is about 35 ms, so a year of daily bars adds about 10 seconds.
  • Each contract in the curve is delivered again on every day it trades, and the engine holds a run’s listings in memory, so the volume grows with the curve’s length. marketfeed lists 77 NG contracts, which is about 19,000 events a year; ES’s five or six are about 1,500.
  • The engine assumes the store does not change during the run. If it did, a contract that a later listing drops before its expiration would stay in the curve. When the first listing carries a snapshot token, the later ones pin it.

The example. strategies/demo/go/curvewatch runs the continuous ES daily series, declares ES’s listing, and at each bar prints the contracts that joined or left the curve since the bar before, with the contract the series holds and its days to expiry. It places no orders; since the order-lifetime programme’s I21 (GLE-467) it declares optional lifetime-v2 support, and cf21_conform_test.go beside it runs it under both profiles over a plain symbol’s bars with the ES inventory served as its listing by the file-csv adapter, zero submissions in both traces.

$ make run-curvewatch   # 2024; CURVEWATCH_START/CURVEWATCH_END move the window
[curvewatch] 2024-01-02 +H24 (first traded 2022-12-26) +M24 (first traded 2023-03-22) +U24 (first traded 2023-06-20) +Z24 (first traded 2023-09-27) +H25 (first traded 2023-12-13); curve H24 M24 U24 Z24 H25; holding ESH24, 72 days to expiry
[curvewatch] 2024-01-18 +Z25 (first traded 2024-01-18); curve H24 M24 U24 Z24 H25 Z25; holding ESH24, 56 days to expiry
[curvewatch] 2024-02-02 +M25 (first traded 2024-02-02); curve H24 M24 U24 Z24 H25 M25 Z25; holding ESH24, 41 days to expiry
[curvewatch] 2024-03-15 -H24 (expired 2024-03-15); curve M24 U24 Z24 H25 M25 Z25; holding ESM24, 97 days to expiry
[curvewatch] 2024-06-21 -M24 (expired 2024-06-21); curve U24 Z24 H25 M25 Z25; holding ESU24, 90 days to expiry
[curvewatch] 2024-07-23 +U25 (first traded 2024-07-23); curve U24 Z24 H25 M25 U25 Z25; holding ESU24, 58 days to expiry
[curvewatch] 2024-09-20 -U24 (expired 2024-09-20); curve Z24 H25 M25 U25 Z25; holding ESZ24, 90 days to expiry
[curvewatch] 2024-12-20 -Z24 (expired 2024-12-20); curve H25 M25 U25 Z25; holding ESH25, 90 days to expiry

Each contract joins on the trade date of its first bar in the archive. The December contracts trade earlier than the others, so Z25 joins in January, before M25. A contract leaves on its expiration date: that day’s bar closes after the 13:30 UTC expiry. The archive also holds Z26 and later contracts, but none of them traded in 2024, so none is in the year’s curve. make test-golden pins the run. The engine fixtures are in engine/gle410_curve_e2e_test.go: a dated run on the file-csv inventory above, and a continuous run.

Instrument metadata: margins, tick sources and as-of

Every symbol’s tick size, point value and currency come from the adapter as InstrumentInfo. Since GLE-400 it can also carry three facts a futures backtest needs.

  • A margin per contract (InstrumentInfo.margin): initial and maintenance, in a currency, with its provenance. The source is a registry seed (a curated static value, undated) or an IBKR what-if capture (dated, and possibly dependent on the capturing account’s positions). The contract it was captured on is named when known.
  • Where the tick came from (tick_size_source): the adapter’s static per-root registry, or the exchange definitions in force at the answer’s instant. They differ for contracts whose tick changes over their life. In the GLBX definitions of 16 March 2026, SR3’s five nearest contracts (through SR3J6) tick 0.0025 and every later one 0.005, while ZQ ticks 0.0025 only in the nearest contract (ZQH6) and 0.005 beyond. The registry holds one tick per root, 0.0025.
  • An instant. A request may carry as_of and ask for the instrument as it was then: the definitions in force then, and a margin captured at or before then. Without it the answer is the latest known.

Querying an adapter. algodata info prints one symbol’s metadata through the engine’s data client, so the answer has passed the check a run applies:

$ bin/algodata info -adapter bin/file-csv-adapter \
    -adapter-inputs Path=testdata/gle357/FRED_GDP.csv,TickSize=0.25,PointValue=50,MarginInitial=46080,MarginMaintenance=33225 \
    -symbol fut:XCME:ES -at 2026-03-16
symbol: fut:XCME:ES
tick_size: 0.25
point_value: 50
margin: initial 46080, maintenance 33225 USD
margin_source: registry seed
ibkr: not served

These are marketfeed’s registry seed for ES (margin: in ES.yaml). The answer carries no as_of line because the file-csv adapter’s facts are static: it does not answer as of an instant. An adapter that serves no margin prints margin: not served. The last line is the IBKR contract terms (GLE-1464), which file-csv doesn’t serve. An adapter that serves them prints, for ES December:

ibkr: FUT ES on CME in USD
ibkr_trading_class: ES
ibkr_contract_month: 202612
ibkr_multiplier: 50
ibkr_price_magnifier: 1

One that gives the reason there are none prints ibkr: no terms: <reason>.

Serving them from an adapter. Set Margin and TickSizeSource on the InstrumentInfo that OnInstrumentInfo returns. An adapter whose facts change over time also implements data.InstrumentInfoAtSource:

func (a *MyAdapter) OnInstrumentInfoAt(ctx *data.DataContext, symbol string, asOf time.Time) (*algolangpb.InstrumentInfo, error) {
    def := a.definitionInForce(symbol, asOf) // the newest definition at or before asOf
    return &algolangpb.InstrumentInfo{Symbol: symbol, TickSize: def.MinPriceIncrement, PointValue: def.Multiplier,
        TickSizeSource: algolangpb.TickSizeSource_TICK_SIZE_SOURCE_DEFINITIONS,
        Margin:         a.marginCapturedAtOrBefore(symbol, asOf)}, nil
}

When a request carries as_of:

  • If the adapter implements OnInstrumentInfoAt, the SDK calls it and stamps the reply’s as_of with the instant. The stamp is how the engine knows the answer is point-in-time.
  • If the adapter does not, the SDK calls OnInstrumentInfo and sends the reply unstamped.
  • The SDK owns the stamp: it replaces whatever a source set. A failed lookup still degrades to bare metadata, unstamped.

Reading them in the engine. DataClient.InstrumentInfoAt(symbol, asOf) sends the instant, and a zero instant sends the request as before. InstrumentInfo(symbol) asks for the latest. Every reply is checked, and a reply that fails is an error, not metadata:

  • the tick size and point value are finite and not negative;
  • the tick source is a known value, and a tick from the definitions is not 0;
  • an as_of stamp appears only when an instant was asked, and equals it;
  • a tick from the definitions, asked as of an instant, carries the stamp. An answer that does not say which definitions it read could be hindsight.
  • A margin’s amounts are finite and not negative, maintenance is not above initial, and the currency is set.
  • A margin’s source is one of the two defined values, and an IBKR capture carries its capture time.
  • A margin captured after the instant asked about is refused: that would be hindsight.

A reply from an adapter that predates GLE-400 passes unchanged. The strategy sees the margin and the tick source on ctx.Instrument(sym). The simulator locks the margin and enforces buying power with it (Margin, GLE-411). On a deferred continuous run each held contract fills and slips on its own tick, which the strategy reads with ctx.Instrument(ctx.ActiveContract(sym)) (GLE-412, Held rank).

IBKR contract terms (InstrumentInfo.ibkr, GLE-1464). An adapter may also say how IBKR knows the instrument, or why it can’t. The terms are what IBKR’s contract search is asked for and what picks the contract from its answer: the search symbol, the trading class, the security type (FUT or STK), the exchange (a stock’s primary listing exchange), the currency, the multiplier, the price magnifier and, for a dated future, the contract month. IBKR’s names differ from the engine’s in ways a copy of the mapping would miss. Micro Silver (fut:XCEC:SIL) is searched as SI, full-size silver’s symbol, and told apart by its trading class SIL; SR3 is searched as SOFR3. The contract month is the month IBKR lists the contract under, not the month it matures: CL’s December contract matures in November. Marketfeed serves the terms (GLE-1466), and the IBKR venue finds each contract by them (Finding contracts at the IBKR venue).

When there are no terms, the message carries the reason instead, for example a venue whose IBKR exchange isn’t confirmed. A consumer refuses such an instrument rather than guess. No message at all means the adapter predates the field, or its lookup failed: the data SDK answers a failed OnInstrumentInfo with bare metadata.

info.Ibkr = &algolangpb.IBKRContract{Answer: &algolangpb.IBKRContract_Terms{Terms: &algolangpb.IBKRContractTerms{
    SearchSymbol: "ES", TradingClass: "ES", SecType: "FUT", Exchange: "CME", Currency: "USD",
    Multiplier: 50, PriceMagnifier: 1, ContractMonth: "202612", // fut:XCME:ES:Z26
}}}
// No route: the reason, and no terms.
info.Ibkr = &algolangpb.IBKRContract{Answer: &algolangpb.IBKRContract_Reason{
    Reason: "fut:XEUR:FDAX: venue XEUR has no confirmed IBKR exchange"}}

The terms are for the engine alone. Every InstrumentsResolved reaches a strategy through StdioHost.SendInstrumentsResolved, which sends it with each instrument’s ibkr cleared, and ctx.Instrument(sym) has no field for them. So a strategy never sees a broker’s names, and changing brokers never reaches a strategy.

Granularity: coarsening yes, refinement never

Every adapter reports coverage: what time range it has data for, and the finest granularity that data can honestly support – tick for a trades file, the native interval for a bar file. The engine enforces one rule: serving coarser bars than the data (aggregating) is always legitimate; serving finer bars (inventing detail) never is.

So a 1-minute bar file happily serves 5-minute bars by resampling:

$ bin/algo run --adapter bin/file-csv-adapter --adapter-inputs Path=testdata/msft-1m.csv \
    --strategy bin/hello --symbol MSFT --bar-interval 5m
[hello] INFO: hello: MSFT closed at 260.7
[hello] INFO: hello: MSFT closed at 261.1
[hello] INFO: hello: MSFT closed at 261.3
[hello] INFO: hello: MSFT closed at 261.87
[hello] INFO: hello: MSFT closed at 261.03
  bars:       5 main after replicate (+ 0 warmup), 5 loop, 0 orders

…but ask the same file for 1-second bars and the engine refuses, before anything runs:

$ bin/algo run --adapter bin/file-csv-adapter --adapter-inputs Path=testdata/msft-1m.csv \
    --strategy bin/hello --symbol MSFT --bar-interval 1s
algo: file-csv-adapter cannot honestly serve 1s bars over 2022-06-10T12:30:00Z..2022-06-10T12:53:00Z: finest granularity there is 1m (refusing rather than silently coarsening; see §8.4 granularity rule)

(Exit status 1. Non-whole multiples like 90s from a 1m file are refused the same way.) Every run report also prints the window’s granularity ceiling, so you always know what the data under your result could support:

  coverage:   MSFT: finest supportable granularity: 1m (limited by file-csv-adapter, 2022-06-10T12:30:00Z..2022-06-10T12:53:00Z)

The Databento adapter reports tick there, which is why it can serve any interval and tick-resolution fills.