Data adapters

The three bundled adapters

bin/file-csv-adapter serves pre-aggregated time bars from a CSV file in the Algolang bar-file format – header then one bar per line:

symbol,ts_open,ts_close,open,high,low,close,volume
MSFT,2022-06-10T12:30:00Z,2022-06-10T12:31:00Z,265.76,265.76,261.01,261.06,3006

Timestamps are RFC 3339; prices are plain decimals. Inputs (passed via --adapter-inputs): Path (required) and Interval, the file’s native bar interval (default 1m). It serves time bars only – at the native interval, or any whole multiple of it by deterministic resampling. A release-stamped symbol (cot:, macro:, fx:RBA:) is served row for row at its own cadence, whatever the requested interval, so one file can hold hourly prices and a FRED series with its optional ref column. An optional CorporateActions input names a ledger CSV; a request on a qualified equity symbol (eq:V:T:adj=raw) is answered with the rows filed under the base symbol eq:V:T as well, as marketfeed’s adapter does. Its bars are raw only: it serves no adjusted view, so an eq:…:adj=split or adj=splitdiv run against it is refused by the engine’s served-bar check (GLE-328).

bin/databento-adapter serves a local Databento trades file, CSV or DBN (auto-detected by content); it reads the file only and never contacts Databento. Inputs: Path (required) and Dataset (default XNAS.ITCH). Because it owns raw trades it can serve the TRADES schema (which also powers tick-resolution fills) and construct bars of every one of the fourteen kinds, at any interval.

bin/ibkr-adapter is a data adapter, despite sharing a provider with the IBKR venue adapter. It reads native 1-minute historical bars from a Client Portal gateway or bin/ibkr-emulator, and can pair them with Databento corporate-action records. Its required input is EmulatorURL (the gateway or emulator base URL); optional inputs include AccountID, a static Symbols symbol-to-conid table, and Databento credentials or a recorded fixture. It is separate from order execution: selecting it with --adapter does not select live brokerage, and selecting --mode live does not automatically select it as the data source.

The algodata tool

bin/algodata converts between the formats (this is exactly what make testdata runs):

$ bin/algodata csv2dbn -in testdata/xnas-sample/xnas-itch-20220610.trades.csv -out testdata/msft-trades.dbn
csv2dbn: wrote testdata/msft-trades.dbn and verified 1000 records (count, first/last price and ts_recv match)

$ bin/algodata csv2bars -in testdata/xnas-sample/xnas-itch-20220610.trades.csv -out testdata/msft-1m.csv -interval 1m
csv2bars: wrote 23 1m bars to testdata/msft-1m.csv

$ bin/algodata replicate -in testdata/xnas-sample/xnas-itch-20220610.trades.csv -out testdata/msft-trades-x1000.csv -n 1000
replicate: wrote 1000 copies (1000000 rows, shift 23m0s) to testdata/msft-trades-x1000.csv

csv2dbn builds a real binary DBN file from a Databento trades CSV (and re-decodes it to verify); csv2bars aggregates trades into the bar-file format the file-csv adapter reads; replicate clones a trade file N times with time shifts, for benchmarking. (A fourth subcommand, cost, prices prospective Databento data pulls against their metadata API; see the data pricing reference. stats and defs ask an adapter for venue statistics and instrument definitions; see Statistics and instrument definitions.)

Statistics and instrument definitions

Besides bars and trades, the data wire carries two event schemas a futures strategy needs (GLE-396):

STATISTICS: the venue’s published statistics for a dated contract: the settlement price, open interest, cleared volume and the other kinds of the StatType enum. A record carries its kind and either a price (price kinds) or a quantity (count kinds), never both, so an open interest of 0 is a real zero rather than “does not apply”. It also carries the instant it applies to (ts_ref, a session’s trade date for a settlement), the venue’s sending instant (ts_event), when it was published (ts_recv), and whether it adds or deletes a value. A point-in-time reader gates on ts_recv: a settlement for Friday’s session is not known at Friday’s close.
INSTRUMENT_DEFINITION: what the venue said a contract was on a given day: its venue symbol (ESM6), security type, activation and expiration, tick size, multiplier, currency and maturity year, month and day, under its canonical dated symbol (fut:XCME:ES:M26). ts_event is when the definition was current.

A request names a dated contract or a bare root, and every record names its dated contract. A STATISTICS request can name the kinds it wants (GetDataRequest.stat_types); naming none asks for every kind the adapter serves. Both schemas travel on the protobuf data wire only: the DBN and CSV data wires have no records for them, and the data SDK refuses a request for them there.

Querying an adapter. algodata stats and algodata defs ask any adapter that advertises the schema and print one record per line, through the same data client a run uses:

$ bin/algodata stats -adapter bin/marketfeed-adapter -symbol fut:XCME:ES:M26 \
    -start 2026-03-09 -end 2026-03-10 -types settlement_price,open_interest
symbol,type,ts_ref,ts_recv,price,quantity,action
fut:XCME:ES:M26,open_interest,2026-03-06T00:00:00Z,2026-03-09T13:51:03.41604285Z,,61208,added
fut:XCME:ES:M26,settlement_price,2026-03-09T00:00:00Z,2026-03-09T20:00:32.680045343Z,6852,,added
fut:XCME:ES:M26,settlement_price,2026-03-09T00:00:00Z,2026-03-09T21:38:51.842204856Z,6852,,added
fut:XCME:ES:M26,settlement_price,2026-03-09T00:00:00Z,2026-03-09T23:49:41.154481064Z,6852,,added

$ bin/algodata defs -adapter bin/marketfeed-adapter -symbol fut:XCME:ES \
    -start 2026-03-02 -end 2026-03-03
symbol,raw_symbol,maturity,activation,expiration,tick_size,ts_event
fut:XCME:ES:H26,ESH6,2026-03,2023-08-18T21:30:00Z,2026-03-20T13:30:00Z,0.25,2026-03-01T13:02:33.442208215Z
fut:XCME:ES:H27,ESH7,2027-03,2023-08-18T21:30:00Z,2027-03-19T13:30:00Z,0.25,2026-03-01T13:02:33.442208215Z
...

marketfeed windows statistics on publication: a window holds the records whose ts_recv falls in it, so Friday’s open interest, published on Monday, belongs to Monday’s window above. Every published record is served — the settlement for 9 March arrives three times, at 20:00, 21:38 and 23:49 UTC — and a point-in-time reader keeps the latest one published by its decision instant. Records come in the adapter’s order (ts_ref, contract, kind), not in ts_recv order. A definition request serves the daily snapshots of the day files the window selects; a snapshot’s ts_event is when the venue last sent it, usually the day before. The adapter’s side is documented in marketfeed’s “Serving statistics and definitions to the engine”.

-types takes the StatType names in lower case without the STAT_ prefix (settlement_price, open_interest, cleared_volume, …). -adapter-inputs K=V,... passes adapter inputs, as algo run does.

Serving them from an adapter. An adapter implements data.StatisticSource or data.DefinitionSource (or both) and advertises the schema in Capabilities(). The SDK refuses to start when an advertised schema has no source or a source is not advertised:

func (a *MyAdapter) OnGetStatistics(ctx *data.DataContext, req data.StatisticRequest, emit chan<- []data.Statistic) error {
    // req.Types is nil when the request named no kinds: serve every kind.
    for _, s := range a.read(req.Symbol, req.Start, req.End, req.Types) {
        emit <- []data.Statistic{{
            Symbol: s.Contract, Type: algolangpb.StatType_STAT_OPEN_INTEREST,
            Quantity: &s.OpenInterest, TsRef: s.TradeDate, TsRecv: s.Published,
            Action: algolangpb.StatUpdateAction_STAT_ADDED,
        }}
    }
    return nil
}

OnGetDefinitions has the same emit contract with data.DefinitionRequest and data.Definition. A source pushes slices of any size and must not close emit; the SDK re-batches to the request’s batch size.

Reading them in the engine. DataClient.ServesStatistics() and ServesDefinitions() report the adapter’s capabilities; FetchStatistics(symbol, types, start, end, batchSize) and FetchDefinitions(symbol, start, end, batchSize) return the records over [start, end) in the adapter’s order, check that each names its contract and that the final count matches, and need the pb data wire. A strategy receives either schema by declaring it, as market events between bars; see Statistics and definitions as events.

Granularity: coarsening yes, refinement never

Every adapter reports coverage: what time range it has data for, and the finest granularity that data can honestly support – tick for a trades file, the native interval for a bar file. The engine enforces one rule: serving coarser bars than the data (aggregating) is always legitimate; serving finer bars (inventing detail) never is.

So a 1-minute bar file happily serves 5-minute bars by resampling:

$ bin/algo run --adapter bin/file-csv-adapter --adapter-inputs Path=testdata/msft-1m.csv \
    --strategy bin/hello --symbol MSFT --interval 5m
[hello] INFO: hello: MSFT closed at 260.7
[hello] INFO: hello: MSFT closed at 261.1
[hello] INFO: hello: MSFT closed at 261.3
[hello] INFO: hello: MSFT closed at 261.87
[hello] INFO: hello: MSFT closed at 261.03
  bars:       5 main after replicate (+ 0 warmup), 5 loop, 0 orders

…but ask the same file for 1-second bars and the engine refuses, before anything runs:

$ bin/algo run --adapter bin/file-csv-adapter --adapter-inputs Path=testdata/msft-1m.csv \
    --strategy bin/hello --symbol MSFT --interval 1s
algo: file-csv-adapter cannot honestly serve 1s bars over 2022-06-10T12:30:00Z..2022-06-10T12:53:00Z: finest granularity there is 1m (refusing rather than silently coarsening; see §8.4 granularity rule)

(Exit status 1. Non-whole multiples like 90s from a 1m file are refused the same way.) Every run report also prints the window’s granularity ceiling, so you always know what the data under your result could support:

  coverage:   MSFT: finest supportable granularity: 1m (limited by file-csv-adapter, 2022-06-10T12:30:00Z..2022-06-10T12:53:00Z)

The Databento adapter reports tick there, which is why it can serve any interval and tick-resolution fills.

Market events Marketfeed