Intraday grids

Implemented (GLE-207). By default an intraday bar sits on one fixed grid: multiples of the bar size counted from 1 January 1970, UTC. A 7m bar drifts against the clock from day to day, and a 4h bar can mix two trading days. An intraday grid restarts the bars at every trading window instead, and cuts the last bar of each window short at its close.

A series asks for a grid by pairing a boundary session with an intraday time-bar interval (whole seconds, 1s up to but excluding 24h):

bin/algo run --adapter "$MF_ADAPTER" --adapter-inputs Dir=/usr/local/marketfeed \
  --strategy bin/printbars --symbols fut:XCME:ES:M24 \
  --interval 7m --session rth --data-protocol pb \
  --start 2024-05-22T00:00:00Z --end 2024-05-23T00:00:00Z

The run above serves 65 bars for 22 May 2024: 13:30–13:37Z, …, 20:51–20:58Z, then 20:58–21:00:01Z. An ETH or RTH window is closed at its close, so the closing second belongs to the last bar, which ends one second after the close. Bars carry nominal instants: ts_open is the grid position, and ts_close is the bar’s nominal end, not the last trade. The series label reads fut:XCME:ES:M24@7m-rth (SYMBOL@INTERVAL-BOUNDARY). eth, rth and session at 1d or 1w remain the trading-day containers; utc-day at 1d is refused (session "utc-day" anchors intraday intervals only; 1d and 1w already use the UTC day), because the UTC day at 1d is the plain 1d interval.

The capability probe and the revision. A grid’s windows come from the adapter’s calendars, so a run cannot be planned against one calendar and served from another. Before fetching any bar, and before the series cache is consulted, the engine runs the adapter once per adapter command line, symbol and boundary (--verify runs the same probe):

marketfeed-adapter --intraday-grid-capability fut:XCME:ES:M24 rth Dir=/usr/local/marketfeed
{"version":1,"boundary":"rth","revision":"c5071ba4…2035"}

The engine sends the adapter inputs as one JSON object on the probe’s stdin, as DataInit carries them, never on the command line where other local users could read them. NAME=VALUE arguments, as above, are for manual use and override the same names from stdin. The answer must be exactly one JSON line with version 1, the requested boundary, and a 64-character lowercase hex revision. The probe has a 5-second timeout; the child is always reaped, and its stdout is bounded at 64 KiB. Any other outcome refuses the run: - a non-zero exit (an adapter without grids exits 2; one built on an older SDK ignores the flag and exits 1); - malformed output; - a timeout.

The refusal reads engine: series fut:XCME:ES:M24@7m-rth: adapter "…" does not advertise intraday-grid-v1 for boundary "rth"; upgrade the adapter, quoting the adapter’s stderr. Every grid request then carries the revision in GetDataRequest.grid_revision. The SDK refuses a grid request whose revision is missing (intraday grid revision is missing; query the adapter capability first) or no longer current (intraday grid revision changed (requested …, current …); restart the request). The revision joins the series cache key.

Adapter side (Go SDK). An adapter serves grids by implementing data.IntradayGridProvider:

IntradayGridCapability(sym, boundary string) (data.IntradayGridCapability, error)

It returns Version: 1, the boundary and its revision. data.Run answers the probe from it: inputs are applied and OnInit runs, with no handshake. The SDK admits a grid request only when the adapter implements the interface, and it checks the request against the adapter’s current revision. A grid request does not consult SessionProvider: utc-day need not be listed. Grid bars travel on stdio-pb-v1, or on stdio-csv-v1 without a path. DBN refuses them (intraday grids cannot travel on stdio-dbn-v1; …), because its OHLCV record rebuilds ts_close as ts_open + interval. Automatic negotiation demotes DBN for any session request.

Paths. --include-path works on grids. With --path-interval d, every piece must sit on the bar’s own d grid from its open: - a piece starts at ts_open + j·d and ends at min(start + d, ts_close), so the last piece is cut with the bar; - pieces ascend, and empty pieces are omitted; - the pieces must reconstruct the bar’s OHLCV exactly.

The engine checks this in place of the epoch rule that every piece spans exactly d.

Warm-up. Grid bars are not evenly spaced in time: they restart each session, and a closed market has none. So a grid warm-up counts bars instead of assuming a span: 1. It fetches the max(lookback·T, 14 days) before the first bar. 2. While it has fewer distinct bars than the lookback, it widens the span by max(2, ⌈1.5·lookback / have⌉), down to the series’ coverage start, for at most six fetches. 3. It keeps the newest lookback bars, and warns on a shortfall as the epoch warm-up does.

Restrictions. Grid series refuse: - --replicate (intraday grid replication is not supported); - non-time bar kinds; - the TRADES schema; - the continuous path (intraday grids are not supported on the engine-stitched continuous path; drop the session for epoch bars, or run a dated contract fut:XCME:ES:<contract> (sessions and intraday grids work there); the advice names the root, for a bare root and for a :cont composite alike).

Continuous symbols on a grid. A bare root or a :cont composite in the run grammar routes to the continuous path (GLE-325), where the grid refusal above applies before any spawn: view a continuous grid in mf-view or mf-chart, and backtest a grid on a dated contract. The grammar’s own refusals come first and read the same with or without a grid. A back-month composite (ofs=B1, B2 or B3) reads:

engine: series fut:XCME:ES:cont:ofs=B1@7m-rth: fut:XCME:ES:cont:ofs=B1 is an adapter composite symbol, which the engine cannot run: back-month composites (ofs=B1..B3) are viewer-only in marketfeed; run a dated contract fut:XCME:ES:<contract> instead

Any other qualifier (ofs=F, straddle=split) reads … carries the qualifier "<q>", which a continuous run symbol cannot carry; name the root fut:XCME:ES or the composite fut:XCME:ES:cont:adj=<panama|ratio|none> (the straddle policy and the session are the run's, not the symbol's). A strategy-declared composite is refused too, when the run reads the strategy’s declarations (engine: declared series "<id>": … which a declared series cannot be … declare a dated contract fut:XCME:ES:<contract> instead, or configure the composite as the run's own series). --verify starts no strategy, so it cannot see a declared series; a configured symbol outside the grammar stops --verify before it prints the plan.

The marketfeed adapter serves composite bars and answers their coverage. Since ER3 (GLE-325) the continuous orchestration consumes them for a run’s own series; the plain bar stream still cannot, because each composite bar carries its dated contract’s symbol, not the series symbol, which is why a declared composite stays refused. Report benchmark sleeves fetch :cont directly, outside the strategy’s bar stream and without asking for coverage; they need the pb wire (see Report benchmark findings).

Implementation anchors: engine/grid.go, engine/gridwire.go, sdk/data/grid.go, their *_test.go files, and internal/seriescache/gridrevision_test.go.