Using the cache

Fetching bars from an adapter (which may be reading a large data lake and building bars from ticks) dominates a backtest’s wall clock, and it repeats identically run after run. algo run therefore caches fetch results on disk at ~/algolang/cache (override with --cache-dir or ALGO_CACHE_DIR) and serves repeat fetches from there. The cache is shared safely by every workspace and process on the machine, and it is invisible in results: off, cold, and warm runs produce byte-identical fills, reports, manifests, and audit records – only the BENCH timing/byte diagnostics differ. A corrupt or half-written cache file is treated as a miss, never an error.

The cache engages only for backtests whose --date-to does not extend past today 00:00 UTC (the window must be fully historical – note the default end is 2100-01-01, so pass a real --date-to to benefit; the run tells you when caching was skipped and why). Entries are keyed on everything that determines the bars: the adapter identity (its description, which for the marketfeed adapter carries the build stamp make marketfeed-adapter writes from scripts/build-id.sh: both checkouts as git describes them, plus a fingerprint of any uncommitted content, so no two adapter builds share entries; GLE-354) and its inputs, the negotiated wire protocol, the symbol, the full bar spec, the exact range and, for a continuous run’s composite, the served snapshot token and the adjustment anchor the fetch pinned (GLE-324). A continuous run caches its composite bars under that token; the roll schedule itself is fetched fresh, and a new schedule generation from the feed changes the token, so it misses the cache and takes effect without a refresh. The cache has one level: the Level-2 continuous build cache of earlier builds is gone (the engine builds nothing since GLE-323), and a continuous/ directory left by an older cache is removed by a pattern-less algo cache flush, its files counted as removed entries.

Two behaviors worth knowing:

  • Same-end reuse. A request that narrows only the start of an existing entry (same everything else, same end) is served by slicing that entry – running 2015-2024 after 2010-2024 reads no data from the adapter. Time bars only; activity bars (volume/tick/dollar/…) are construction-order dependent and always match exact ranges.
  • Invalidation is manual. There is no freshness check against the lake. If the lake is rebuilt or history revised, run once with --cache refresh (bypass reads, rewrite entries) or clear entries with algo cache flush. A composite entry is no exception: the token names the feed’s lineage state, not the tape, so a nightly bar re-derive under one state is served from the warm entry until the refresh or the flush.
algo cache                  # stats: directory, entry count, total size
algo cache list  [glob...]  # entries by level/symbol/range
algo cache flush [glob...]  # remove matching entries (none = all)

--cache off disables for a run; ALGO_CACHE=off disables wherever the flag was not given explicitly. Trading-day containers participate in cache identity. Intraday session-window filtering (hours/days) remains unsupported; it is a separate feature. See the cache reference.

Live bars from an adapter

A live run on a real-time feed warms up from history, then subscribes to each bar series and steps on each bar as it closes (GLE-1440). This section is the adapter’s half, which the Go data SDK carries. The engine’s half, algo run --mode live --feed stream, is in Running live on a real-time feed.

An adapter streams bars by setting SupportsStreaming on its BARS capability and implementing data.BarStreamSource:

func (a *Feed) Capabilities() []*algolangpb.SchemaCapability {
    return []*algolangpb.SchemaCapability{{
        Schema:             algolangpb.DataSchema_BARS,
        SupportedBarKinds:  []*algolangpb.SupportedBarKind{{BarKind: algolangpb.BarKind_BARS_TIME, NativeIntervals: []string{"1m"}}},
        SupportsHistorical: true,
        SupportsStreaming:  true,
    }}
}

// OnSubscribeBars runs on its own goroutine, once per subscription.
func (a *Feed) OnSubscribeBars(ctx *data.DataContext, req data.BarRequest, stream *data.BarStream) error {
    live, err := a.connect(req.Symbol, req.IntervalStr)
    if err != nil {
        return err // the subscription ends with an ERROR event
    }
    defer live.Close()
    stream.Event(algolangpb.SubscriptionState_CONNECTED, "receiving")
    // The catch-up: bars that closed after req.Start, before the feed's first.
    for _, b := range a.history(req.Symbol, req.Start, live.First()) {
        if err := stream.Send(b); err != nil {
            return err
        }
    }
    for {
        select {
        case <-stream.Done(): // the engine cancelled or shut down
            return nil
        case b := <-live.Bars():
            b.Provisional = true
            if err := stream.Send(b); err != nil {
                return err
            }
        case why := <-live.Lost():
            stream.Event(algolangpb.SubscriptionState_DISCONNECTED, why)
            // ...reconnect: RECONNECTING while trying, CONNECTED when back,
            // then send the bars that closed while the feed was down.
        }
    }
}

The request is a bar request with an open end. Start is the close of the last bar the engine holds: the end of its warm-up, or the last bar it received before a reconnect. The source sends every bar that closes after Start and has already closed, in order, then each bar as it closes. That way the stream follows the warm-up with no gap and no duplicate. A zero Start asks from the next bar to close. AdjustAnchor is the warm-up’s adjustment anchor, so live bars of a composite or adjusted series are on the warm-up’s frame. The rest of the request is parsed and checked as on a history request: the bar kind, the interval, a session the adapter lists, IncludePath. End and BatchSize are zero.

The SDK holds the stream to three rules:

  • Order. Send refuses a bar that does not close after the last bar sent (after Start, on the first Send) and sends nothing of that call, so the engine never sees a bar twice or out of order. A source that returns that error ends the subscription with it.
  • Lifecycle. Event reports CONNECTED, DISCONNECTED and RECONNECTING, with a reason in words for the operator. ERROR is the SDK’s: when the source returns an error, the SDK sends ERROR with data source: <the error>, and the subscription is over. A source that returns nil before Done is closed ends it as well, with ERROR and the source ended the stream, because a live feed has no end the engine can wait for. After a cancel, nothing more is sent under the request’s id: Send and Event return data.ErrStreamClosed, and a source that returns it has stopped cleanly.
  • Endpoints. A source that reads from a list of live endpoints (marketfeed’s gateways, The live source) reports them and their health with Endpoints (GLE-1740). While its last state is CONNECTED they go at once, in an event that repeats that state and reason; while the feed is down they wait, and ride on the next Event. A source with no list never calls it.
  • Provisional. Set Provisional on every bar the source may still restate: every bar of a live tail, and the catch-up bars that are not yet settled. It travels as Bar.provisional on the pb wire.

The serve loop goes on answering history requests while subscriptions stream. Each subscription has its own DataContext. A Shutdown, or the engine going away, ends every open subscription and waits up to 10 seconds for the sources to return before OnShutdown runs. As for every schema, data.Run refuses to start when the capability and the interface disagree: capabilities advertise streaming BARS but the adapter does not implement data.BarStreamSource, or adapter implements data.BarStreamSource but its BARS capability does not set supports_streaming.

A subscription is refused as a request (a request-scoped Error; the connection stays up), checked in this order:

CodeTextWhen
unsupported_schemaschema <S> is not streamed by this SDK version (BARS, ROLL_SCHEDULE, CORPORATE_ACTIONS, QUOTES_L1, QUOTES_L1_ON_TRADE and QUOTES_L1_SAMPLED only)a schema other than these six
unsupported_schemaadapter does not stream BARSthe adapter does not stream bars
bad_requestsubscription without request_idan empty request id
bad_requesta subscription takes no bar_spec.lead_in: the engine warms up from historya constructor warm-up asked
bad_requestthe history request’s text, e.g. BARS request without bar_specthe bar spec, session or path fails the history request’s checks; an intraday grid is refused there, since a subscription carries no grid revision
bad_requestsubscription "<id>" is already opena second subscription under an open id

A cancel of a subscription that is not open does nothing, because it races with the source’s own end. Subscriptions travel on the framed wires, stdio-pb-v1 and stdio-dbn-v1, as pb StreamData on both. The CSV data wire has no grammar for them.

Roll-schedule subscriptions

A futures root’s roll schedule streams on the same wire (GLE-1460, built for GLE-1458). A live continuous run on a real-time feed then learns of a roll as the stream crosses it, and of the next cut before it comes. An adapter streams schedules by setting SupportsStreaming on its ROLL_SCHEDULE capability and implementing data.RollScheduleStreamSource:

// OnSubscribeRollSchedule runs on its own goroutine, once per subscription.
func (a *Feed) OnSubscribeRollSchedule(ctx *data.DataContext, req data.RollScheduleRequest, stream *data.RollScheduleStream) error {
    stream.Event(algolangpb.SubscriptionState_CONNECTED, "following "+req.Symbol)
    last := a.schedule(req.Symbol) // the schedule as served now
    if err := stream.Send(last); err != nil {
        return err
    }
    for {
        select {
        case <-stream.Done():
            return nil
        case <-a.changed(req.Symbol): // a new generation, a crossed roll, a moved next roll
            if next := a.schedule(req.Symbol); !a.same(next, last) {
                if err := stream.Send(next); err != nil {
                    return err
                }
                last = next
            }
        }
    }
}

A schedule is a whole state, not a change, so each one supersedes the one before. The source first sends the schedule as it serves it now, which stands for every change since Start. Then it sends a schedule each time the served one changes: a new generation, a changed chain, a roll the live stream crossed, or a next roll that moved. It never sends the same state twice in a row.

The request carries Symbol as a history request does. Start is the instant of the engine’s own schedule read, and AdjustAnchor is as given. End, BatchSize and the snapshot pins are zero, because a subscription follows the lineage as it moves. Send refuses a schedule without a symbol. Event, the end on cancel and the ERROR on the source’s own end are as for bars. data.Run refuses to start when the capability and the interface disagree, in the same words as for bars.

A roll-schedule subscription is refused as a request:

CodeTextWhen
unsupported_schemaadapter does not stream ROLL_SCHEDULEthe adapter does not stream schedules
bad_requestsubscription without request_idan empty request id
bad_requestROLL_SCHEDULE cannot travel on stdio-dbn-v1; negotiate stdio-pb-v1the DBN wire, which carries no schedule
bad_requesta ROLL_SCHEDULE subscription takes no bar_spec, session or intervala bar field given
bad_requestROLL_SCHEDULE subscription without symbolan empty symbol
bad_requestsubscription "<id>" is already opena second subscription under an open id

On the engine side the data client opens one with SubscribeRollSchedule. It takes the schedules on the same reader as the bars, and refuses a wire other than stdio-pb-v1 and an adapter that does not stream the schema.

Corporate-action subscriptions

An equity’s corporate actions stream on the same wire (GLE-1460). A live run on a real-time feed then learns of a split, dividend, merger or delisting announced while it runs, not at its next restart. An adapter streams them by setting SupportsStreaming on its CORPORATE_ACTIONS capability and implementing data.CorporateActionStreamSource:

// OnSubscribeCorporateActions runs on its own goroutine, once per subscription.
func (a *Feed) OnSubscribeCorporateActions(ctx *data.DataContext, req data.CorporateActionRequest, stream *data.CorporateActionStream) error {
    stream.Event(algolangpb.SubscriptionState_CONNECTED, "following "+req.Symbol)
    last := a.actions(req.Symbol, req.Start) // every action from Start on, as served now
    if err := stream.Send(last); err != nil {
        return err
    }
    for {
        select {
        case <-stream.Done():
            return nil
        case <-a.ledgerMoved(): // the nightly landed, restated or withdrew an action
            if next := a.actions(req.Symbol, req.Start); !slices.Equal(next, last) {
                if err := stream.Send(next); err != nil {
                    return err
                }
                last = next
            }
        }
    }
}

A send is a whole state, as a roll schedule is: every action of the symbol from Start on, as the history read of that open window serves it now. It supersedes the send before it. So the source first sends the actions as served now. Then it sends the whole set again each time it changes: an action landed, restated or withdrawn. It never sends the same set twice in a row. An empty set is a state too: no action from Start on. Send refuses an action of no type, and sends nothing.

The request carries Symbol as a history request does, and Start as the engine gave it. End and BatchSize are zero, because a subscription follows the ledger as it grows. Event, the end on cancel and the ERROR on the source’s own end are as for bars. data.Run refuses to start when the capability and the interface disagree, in the same words as for bars.

A corporate-action subscription is refused as a request:

CodeTextWhen
unsupported_schemaadapter does not stream CORPORATE_ACTIONSthe adapter does not stream corporate actions
bad_requestsubscription without request_idan empty request id
bad_requestCORPORATE_ACTIONS cannot travel on stdio-dbn-v1; negotiate stdio-pb-v1the DBN wire, which carries no corporate action
bad_requesta CORPORATE_ACTIONS subscription takes no bar_spec, session, interval or adjust_anchora bar field or an adjustment anchor given
bad_requestCORPORATE_ACTIONS subscription without symbolan empty symbol
bad_requestsubscription "<id>" is already opena second subscription under an open id

On the engine side the data client opens one with SubscribeCorporateActions. It takes each state on the same reader as the bars, the empty state included. It refuses a wire other than stdio-pb-v1 and an adapter that does not stream the schema.

Quote subscriptions

Top-of-book quotes stream on the same wire (GLE-1620). AlgoDesk shows a live bid and ask from them, taken from Databento through marketfeed (GLE-1519). There are three quote schemas:

SchemaDatabentoWhat it sends
QUOTES_L1MBP-1the top of the book at every book update
QUOTES_L1_ON_TRADETBBOthe top of the book at each trade
QUOTES_L1_SAMPLEDBBO-1s, BBO-1mthe top of the book once per interval

An adapter streams quotes by setting SupportsStreaming on one or more quote capabilities and implementing data.QuoteStreamSource. One callback serves every quote schema the adapter advertises; req.Schema says which:

// OnSubscribeQuotes runs on its own goroutine, once per subscription.
func (a *Feed) OnSubscribeQuotes(ctx *data.DataContext, req data.QuoteRequest, stream *data.QuoteStream) error {
    live, err := a.connect(req.Symbol, req.Schema, req.Interval)
    if err != nil {
        return err // an interval or start it does not serve: ERROR
    }
    defer live.Close()
    stream.Event(algolangpb.SubscriptionState_CONNECTED, "receiving")
    for {
        select {
        case <-stream.Done():
            return nil
        case q := <-live.Quotes():
            q.TsLocal = time.Now() // when the adapter got it
            if err := stream.Send(q); err != nil {
                return err
            }
        case why := <-live.Lost():
            stream.Event(algolangpb.SubscriptionState_DISCONNECTED, why)
            // ...RECONNECTING while trying, CONNECTED when back.
        }
    }
}

The request carries Symbol, Schema and Consolidated. Interval is the sampling interval as the engine sent it, such as 1s or 1m, on QUOTES_L1_SAMPLED only. The SDK does not check its value. Start is zero for live from now. The SDK passes any other start through, and the source may refuse it. A source refuses what it does not serve by returning an error, which ends the subscription with ERROR.

Quotes are live only. A history request for a quote schema is refused, as before. A send is a batch of records, not a whole state: the source sends each quote as the feed delivers it. Send sends nothing when given no quotes, and refuses a quote without a symbol, sending nothing of that call. It has no order rule, because a quiet book may be sent again: the same quote, its TsRecv repeated.

A data.Quote keeps Databento’s fields. TsEvent and TsRecv are the provider’s: the matching engine’s and the capture’s. TsLocal is the adapter’s own clock when the quote reached it from the provider, and the desk judges a quote’s age by it. For a quiet book sent again, TsLocal is when the adapter last saw the feed live. A price is 0 when its side of the book is empty. Side is the side of the last trade or update, 'A', 'B' or 'N', as a trade’s aggressor. Flags is the DBN flags byte. data.ToPBQuote and data.FromPBQuote convert to and from the wire, with TsLocal as RecordMeta.ts_local.

Event, the end on cancel and the ERROR on the source’s own end are as for bars. data.Run refuses to start when the capabilities and the interface disagree: capabilities advertise streaming quotes but the adapter does not implement data.QuoteStreamSource, or adapter implements data.QuoteStreamSource but no QUOTES_L1, QUOTES_L1_ON_TRADE or QUOTES_L1_SAMPLED capability sets supports_streaming. It also refuses a quote capability without SupportsStreaming, since the SDK serves quotes on subscriptions only: capability <S> does not set supports_streaming, but this SDK version serves quotes on subscriptions only.

A quote subscription is refused as a request, checked in this order:

CodeTextWhen
unsupported_schemaadapter does not stream <S>the adapter does not stream that quote schema
bad_requestsubscription without request_idan empty request id
bad_request<S> cannot travel on stdio-dbn-v1; negotiate stdio-pb-v1the DBN wire; quotes travel on the pb wire only
bad_requesta <S> subscription takes no bar_spec, session or adjust_anchora bar field or an adjustment anchor given
bad_requestQUOTES_L1_SAMPLED subscription without intervala sampled subscription with no interval
bad_requesta <S> subscription takes no interval: only QUOTES_L1_SAMPLED is sampledan interval on QUOTES_L1 or QUOTES_L1_ON_TRADE
bad_request<S> subscription without symbolan empty symbol
bad_requestsubscription "<id>" is already opena second subscription under an open id

On the engine side the data client opens one with SubscribeQuotes(symbol, schema, interval), and StreamsQuotes(schema) says whether the adapter streams that schema. The quotes arrive on the same reader as the bars: each batch is a StreamItem whose Quotes() are set, never empty. The client refuses a schema that is not a quote schema, a wire other than stdio-pb-v1, an adapter that does not stream the schema, and an interval on a schema other than QUOTES_L1_SAMPLED. The engine’s own live runs take no quotes: quotes under a bar, roll-schedule or corporate-action subscription end the feed with an error.

Implementation anchors: sdk/data/stream.go, sdk/data/stream_test.go, sdk/data/gle1460_rollschedule_stream_test.go, sdk/data/gle1460_corpactions_stream_test.go, sdk/data/gle1620_quotes_stream_test.go, engine/datastream.go, engine/gle1620_quotes_stream_test.go, engine/gle1620_stream_quotes_e2e_test.go.

Running live on a real-time feed

--feed stream makes a live run take its bars from the adapter’s real-time feed (GLE-1440). The default, --feed history, steps the bars the run fetched before it started, as every live run did before. A deployment file’s feed key sets the same choice for a member run against the desk (Running a deployment against the desk).

bin/algo run \
  --adapter "$ADAPTER" \
  --strategy bin/smacross \
  --symbols AAPL \
  --bar-interval 1m \
  --mode live --feed stream \
  --venue-url http://127.0.0.1:5000 \
  --venue-emulated \
  --ledger var/orders.json \
  --date-to 2026-11-11T21:00:00Z

The adapter must stream bars (SupportsStreaming on its BARS capability, Live bars from an adapter). The marketfeed adapter streams bars, roll schedules and corporate actions (GLE-1441, GLE-1461). Its live bars come through marketfeed’s gateways, which MF_LIVE_SERVERS and MF_GATEWAY_KEY name (environment or ~/.marketfeed/env). With no server list it refuses every bar subscription, whatever DATABENTO_API_KEY holds: only marketfeed talks to Databento, and the adapter has no direct route (The live source, GLE-1790, GLE-1793).

The start. The run warms up from history as on any run. Then each series subscribes from the close of its last warm-up bar, or from just before --date-from when it has no warm-up, so the stream follows the warm-up with no gap and no duplicate. An open --date-from starts the run now. Nothing of the run itself is fetched ahead, so the report’s bars line counts no main bars: bars: 0 main after replicate (+ 9 warmup), 28 loop counts the warm-up and the bars the loop stepped.

The order of bars. Bars closing at the same instant go to the strategy in the backtest’s order: the other series first, then the primary (the first --symbols entry). When one series’ bar arrives, the run waits up to 2 seconds for the bars of the other series closing at the same instant. A series whose bar misses that wait is not waited for again until its next bar, and the run says so. A bar that arrives after the run has moved past its close is stepped when it arrives, with a note. A release-stamped series (cot:, macro:, fx:RBA:) is never waited for. A bar the adapter marks provisional is stepped like any other.

Outages and catching up. The feed’s state reaches the run as it changes:

  • Disconnected or reconnecting. The run notes it and stops openings until the feed is back. A bar from the series is the sign that it is back.
  • Ended with an error. The run stops openings and subscribes again from its last bar a second later. The adapter sends what closed in between.
  • A gap. When the first bar after an outage opens after the last bar before it closed, the run says that bars may be missing, and carries on.
  • Catching up. When the bar about to be stepped closed more than one interval of its series before the clock, the run is catching up: after an outage, or from a --date-from in the past. Openings stay stopped until the run is within one interval of the clock.

While openings are stopped, an order that would open or add to a position is refused before the ledger and the venue see it. The strategy hears a REJECTED cancel under its own label, and the operator line names the reason:

algo: order "smacross_1772462760000000000" (AAPL) from instance AAPL denied: openings are stopped: the real-time feed is catching up: the bar closed at 2026-03-02T14:46:00Z, 1h26m30s before the clock

An order that only reduces a position goes through, so a strategy can still exit and its stops still work. The run judges an order by the position its delivered fills leave, starting from the broker’s positions. Other working orders do not count. A timer’s decision between bars is refused openings while a series’ feed is down.

The notes. The feed’s notes are printed on standard error as they happen, each on an algo: line:

algo: the real-time feed of AAPL@1m is disconnected: gateway restart; openings stop until it is back
algo: the real-time feed of AAPL@1m ended with an error (data source: the replay lost its source); openings stop, and the run subscribes again from its last bar in 1s
algo: subscribing to AAPL@1m again from 2026-03-02T14:52:00Z
algo: bars of AAPL@1m may be missing: the feed was down, and the first bar after it opens at 2026-03-02T14:49:00Z, after the last bar before it closed at 2026-03-02T14:47:00Z
algo: released the bars closing 2026-03-02T14:50:00Z without MSFT@1m, whose bar did not arrive within 2s; it is not waited for again until its next bar
algo: a late bar: MSFT@1m closing 2026-03-02T14:50:00Z arrived after the run had moved past it, and is stepped now
algo: openings resume at the bar closing 2026-03-02T14:55:00Z

Timers. A timer fires when the clock reaches its instant, between bars as well (Scheduled decision points).

Continuous futures. A continuous series runs on the stream only with the roll rebase, on by default since GLE-1502 (Re-adjusting at each roll). The strategy then sees the bars, rebases and fills of its backtest (GLE-1458).

  • The start. The warm-up reads the composite’s bars before --date-from, on the basis of the contract held there. The run then subscribes to the :cont composite with that adjustment anchor, so the stream stays on the warm-up’s basis. It also subscribes to the root’s roll schedule (Roll-schedule subscriptions). It needs the stdio-pb-v1 data wire.
  • The chain grows. Each schedule the feed sends adds the rolls after the run’s newest contract. A schedule that moves a roll the run already holds, its cut or its contracts, ends the run. A seam measured again is noted, and the run keeps the one it holds until it restarts. A bar of a contract the run does not hold yet waits for its schedule.
  • When a roll fires. A roll fires at the later of its cut and the incoming contract’s ETH open in the exchange calendar. The loop wakes for it, as it wakes for a timer, so a daily series rolls at the cut, not at its next bar. The run fetches the root’s calendar itself, and refuses to start without one. A bar of the incoming contract that closes before the calendar’s ETH open shows the market is already open: the roll fires at its cut, before that bar, and a note says so.
  • A late seam (owner ruling Q11, 12 Nov 2026). The feed serves a roll’s seam once it has measured it, usually just after the cut. A roll whose fire instant comes first is held, and openings stop, until the schedule serves its seam. The hold lasts up to the roll deadline, counted from the fire instant: the deployment’s roll_deadline, or ten minutes (owner answer, GLE-1508). Marketfeed checks for a new seam only every 30 s, so ten minutes allows about twenty checks. Then the run stops, naming the roll, and a restart’s catch-up roll finishes it. A run that reaches its --date-to with a roll held stops the same way.
  • The venue’s contracts are fixed at the start: the held contract and the next one the schedule names. A roll into any other contract stops the run and asks for a restart.
  • Catching up. A run whose feed catches up across a cut rolls at each cut in order, each with its own rebase.
  • Starting between a cut and its first bar. A run started after a cut, before the incoming contract’s first bar, warms up on the outgoing contract. Its roll then fires before its first streamed bar, with its OnRoll and its rebase, and is made at its fire instant. A backtest started at the same moment warms up on the incoming contract and sends neither.

The notes say when a roll fires and when it waits. A roll found before a bar is made at the close of the bar before it, as on history:

algo: the roll ESH26->ESM26 of fut:XCME:ES@1d fires at 2026-03-15T22:00:00Z (its cut 2026-03-15T22:00:00Z)
algo: the roll ESH26->ESM26 of fut:XCME:ES@1d fires before the bar closing 2026-03-16T21:00:00Z (its cut 2026-03-15T22:00:00Z), made at 2026-03-13T21:00:00Z
algo: the roll ESH26->ESM26 of fut:XCME:ES@1d, due at 2026-03-15T22:30:00Z, waits for its seam from the feed; openings stop until it arrives, for up to 2m0s
algo: the feed served the seam of the roll ESH26->ESM26 of fut:XCME:ES@1d after 45s; the roll fires

Corporate actions (GLE-1460). An equity whose corporate actions the run read at its start follows them on the stream too, so an action announced while the run is going reaches it before its ex-date, not at its next restart.

  • The subscription. Each equity subscribes to its corporate actions from the start of its own start-up read (Corporate-action subscriptions). Each state the feed sends replaces the one before it. The loop takes the latest state at the top of the symbol’s next bar and rebuilds the symbol’s actions from it, as the start-up read built them. The backtest’s passes then fire each action at its ex-date, with its Announced and Effective notices, its ledger record and its cash. The first state usually matches the start-up read, and then changes nothing.
  • Restated or withdrawn. An action restated or withdrawn before its ex-date fires as last served, or not at all. One the run has already fired is never fired again or changed. A note says that the feed restated or withdrew it.
  • Late. An action whose ex-date the symbol’s bars had already reached when it arrived fires at the symbol’s next bar, with a note. An adjusted equity’s prices do not take it during the run, because the bars already sent show its raw step. A restart prices it in. A split that lands late while the run holds the symbol still stops the run at reconciliation, as on history: the broker’s position moved at the ex-date, and the ledger’s moves only at that next bar.
  • A state the backtest would refuse ends the run with the backtest’s refusal: for example, an action of a kind live mode does not run yet. A restart would refuse the same state.
  • Outages. While a symbol’s subscription is disconnected or reconnecting, openings stop. A state from the feed is the sign that it is back. An ERROR stops openings, and the run subscribes again a second later; the adapter’s first state on the new subscription is the whole set again.
  • An adapter that does not stream them. A run with an adjusted equity is refused before it starts, because its prices follow its actions. A run of raw equities goes on, with a note that an action landing during the run reaches the symbol at its next restart.

The marketfeed adapter streams them from its ledgers (GLE-1461): it looks for a change every 30 s, so an action the nightly lands reaches the run within about half a minute.

algo: the real-time corporate actions of eq:XNAS:AAPL changed: added split eq:XNAS:AAPL ex 2026-03-04
algo: the real-time corporate actions of eq:XNAS:AAPL changed: withdrew cash dividend eq:XNAS:AAPL ex 2026-03-06
algo: the real-time corporate actions of eq:XNAS:AAPL changed: restated split eq:XNAS:AAPL ex 2026-03-02, which the run already fired and keeps as fired
algo: the cash dividend of eq:XNAS:AAPL with ex-date 2026-03-03 reached the run after its ex-date; it takes effect at the bar closing 2026-03-04T21:00:00Z
algo: the real-time corporate actions of eq:XNAS:AAPL are disconnected: gateway restart; openings stop until they are back
algo: the real-time corporate actions of eq:XNAS:AAPL ended with an error (data source: ledger unreadable); openings stop, and the run subscribes again in 1s
algo: subscribing to eq:XNAS:AAPL's corporate actions again
algo: the real-time corporate actions of eq:XNAS:AAPL are back: the feed sent them
algo: <adapter> does not stream corporate actions: one that lands during the run reaches eq:XNAS:AAPL at its next restart

The end. The run’s last bar is the last one closing before --date-to. The run ends when every series has sent a bar closing after that, or when the clock has been past --date-to for 2 seconds with no bar arriving. A feed still catching up past --date-to is not cut off. With an open --date-to the run goes on until it is signalled (Stopping a run).

What the run refuses. Before anything is spawned or fetched:

RefusedMessage
a backtest--feed stream needs --mode live: a backtest reads its bars from history
a run that reads trades--feed stream streams bars, and this run reads TRADES
an adapter that does not stream barsadapter <name> does not stream BARS (its BARS capability has no supports_streaming), so it cannot be a live run's real-time feed; run with --feed history
a CSV data wire<name>: a real-time feed needs the stdio-pb-v1 or stdio-dbn-v1 data wire, but <wire> was negotiated
a continuous futures series with the roll rebase turned off--feed stream runs a continuous futures series (fut:XCME:ES@1d) only with roll rebase, and the run turns it off: a stream cannot restate the history it has sent (GLE-1458, GLE-1502); drop --roll-rebase=false (a run-config's "roll-rebase": false, a deployment's "roll_rebase": false), or run it with --feed history
an intraday grid series--feed stream does not run an intraday grid series (...): a subscription carries no grid revision; run it with --feed history

Once it has read the data, a run with a continuous series refuses a root the adapter serves no exchange calendar for:

engine: --feed stream fires each roll of fut:XCME:ES@1d at the later of its cut and the exchange's ETH open, which the exchange calendar places, and the adapter serves no calendar for fut:XCME:ES: <why>

A run with an adjusted equity also refuses an adapter that does not stream its corporate actions:

engine: --feed stream runs an adjusted equity series (eq:XNAS:AAPL:adj=split) only on an adapter that streams its corporate actions, and <name> does not (its CORPORATE_ACTIONS capability has no supports_streaming); run it raw (adj=raw) or with --feed history

During the run, an adapter’s refusal of a subscription ends the run (engine: the adapter refused the subscription to AAPL@1m: <reason>), and so does a failure of the connection to the adapter. A continuous series also ends the run when:

WhenMessage
a seam does not arrive within the roll deadlinethe roll ESH26->ESM26 of fut:XCME:ES@1d was due at <fire>, and the feed served no seam for it within 10m0s (the roll deadline); the run stops: restart it, and its catch-up roll takes the series into ESM26
the run reaches its end with a roll heldthe run reached its end with the roll ESH26->ESM26 of fut:XCME:ES@1d, due at <fire>, held for its seam, which the feed did not serve; the series did not roll: restart the run, and its catch-up roll takes the series into ESM26
a schedule moves a held rollthe real-time roll schedule of fut:XCME:ES@1d restates the roll ESH26->ESM26 at <cut>: it now serves <from>-><to> at <cut>
a roll leads outside the venue’s contractsfut:XCME:ES@1d reaches its roll ESM26->ESU26 at <cut>, and the run's venue holds no ESU26: the feed served that roll after the run started, and a run's venue contracts are fixed at its start; restart the run, and the series follows the roll into ESU26
a corporate-action state the backtest would refuseengine: the real-time corporate actions of eq:XNAS:AAPL cannot run: <the backtest's refusal>; the run stops, and a restart would refuse them too

Implementation anchors: engine/streamrun.go (the choice and the refusals), engine/livestream.go (the feed), engine/livegate.go (the openings gate), engine/datastream.go (the data client’s subscriptions), engine/streamcont.go (a continuous series’ growing frame), engine/livestreamcorp.go (the corporate-action subscriptions), engine/livecorpstream.go (a state rebuilt into the passes); tests engine/gle1440_*_test.go, which run smacross on a fake streaming adapter and match its backtest’s trades, and engine/gle1458_stream_e2e_test.go, which does the same across a roll, engine/gle1460_live_corp_stream_test.go, which lands, withdraws and restates actions during a run, and engine/gle1460_stream_corp_e2e_test.go, which runs smacross through a split its feed announces during the run.

Bars and rolls on a live feed, end to end

This section is an overview for a strategy author taking a strategy live on marketfeed’s feed. It says where the bars come from, what happens at a roll, and what the strategy must do itself. Each point links to the section that defines it (GLE-1760). Contract rolls in a live strategy walks through one real roll with diagrams and the demo rolltrend.

Where the bars come from. The run warms up from marketfeed’s archive (the lake), then subscribes from the close of its last warm-up bar. The adapter serves the bars up to the archive’s end from the archive. After that it builds each bar from Databento’s live 1 s bars (ohlcv-1s), which reach it through marketfeed’s gateways (The live source), folded into the run’s interval by the same code that folds the archive’s bars for a history read. Day, week and session bars are cut by the same calendars, so a holiday, an early close or a daylight-saving change cuts the stream where it cuts history. A bar goes out once it has closed, and the strategy decides on it as it would in the backtest. The details are marketfeed’s (“Live bars” in marketfeed’s docs/marketfeed.md); the engine’s half is Running live on a real-time feed, and the adapter’s Live bars from an adapter.

How a live bar can differ from history. Every live bar is marked provisional, and the next nightly’s archived bar replaces it. The engine steps a provisional bar like any other. marketfeed’s replay tests check that the next day’s history read serves the same bars, apart from that mark. Three exceptions:

  • Equity volumes. An equity’s live bars are EQUS.MINI’s, the feed the live licence covers, not the archive’s Nasdaq tape. Prices track the tape closely, but volumes are far lower, so a rule on volume reads differently live.
  • Futures settlements. A futures session bar takes its settlement from the feed’s replay, which reaches back about 23 hours, and less after Databento’s gateway restarts. With the archive more than a day behind, a day whose settlement fell outside that reach carries none.
  • Gaps after an outage. The adapter catches up from its gateway’s replay and from Databento’s historical API through the gateway’s /hist proxy. A stretch neither could cover is reported, and the run says that bars may be missing. With both gateways down there is no live data at all until one returns (GLE-1791).

What a live feed refuses. Trades, event bars, spreads and intraday grid series are not streamed, and an archive more than five days behind is refused (marketfeed’s LiveMaxLag): the nightly is not running. The run’s refusals and their messages are in Running live on a real-time feed.

A dated contract never rolls. A series on one contract, such as fut:XCME:ES:Z26, trades that contract until you move the run to the next one.

A continuous series needs the roll rebase. It is on by default (GLE-1502), and a deployment turns it off with "roll_rebase": false (Running a deployment against the desk). A run on the stream refuses a continuous series at its start when the rebase is off, and after Init when its strategy cannot take it (When the default gives way). The reason is that a stream cannot change bars it has already sent. marketfeed keeps the stream on the warm-up’s price basis, and the engine moves the strategy onto each new contract’s basis at the roll.

What happens at a roll:

  1. marketfeed places the roll from its forward roll calendar and measures the seam on the live bars, as the nightly would. Until the seam is measured, the run holds the roll and stops openings, up to the roll deadline.
  2. The roll fires at the later of its cut and the incoming contract’s ETH open.
  3. When it fires:
    • At the venue, the roll cancels the working orders on the outgoing contract, rolls the position with two DAY market orders, and places the orders on the logical symbol again on the incoming contract, at their moved levels. Under order lifetime v2 the engine first restates the rest of its book with the same arithmetic.
    • In the strategy, OnRoll comes first. Then the engine re-adjusts the series’ history onto the incoming contract’s basis, and the strategy hears OnRebase. The roll’s own events come last, such as a carried stop that fills on the incoming contract.

The strategy’s side is in Re-adjusting at each roll, and the venue’s in Continuous futures in live mode.

What the SDK moves for you. Before the strategy sees the rebase, the SDK moves onto the new basis:

  • ctx.Series(id), the whole lookback, so a Series the strategy kept stays valid;
  • ctx.CurrentAdjustment and ctx.CurrentAdjustmentRatio;
  • an open position’s AvgPrice and AvgPriceAdjusted, but not AvgPriceRaw or Realized;
  • every working order on the logical symbol: its limit and stop prices, trailing amount and attached orders;
  • ctx.TradeState’s entry price, entry high and low, peak and trough.

What you must move yourself. Anything else the strategy keeps on the old basis: a remembered price level, a band, a channel, or an indicator’s running state. Implement RebaseObserver and move it in OnRebase:

  • r.Level(x) moves a price level;
  • r.Distance(d) moves a price distance, such as an ATR or a trailing amount;
  • kit.Tracker, kit.StopSuite and kit.LotBook each have Rebase(r.Gap, r.Ratio);
  • r.Bars holds the re-adjusted history, oldest first, to rebuild an indicator such as an EMA;
  • r.RestateOrder moves an order exactly as the engine does.

A strategy that only compares indicators over the same bars, such as smacross, needs no OnRebase, since the rebase moves both sides alike. One that compares a price with a level it remembered across the roll does, or it trades differently from its backtest.

Aligned views and the portfolio view move with the strategy: each view of a continuous source is rebuilt from the rebased source, and the portfolio view’s rows are in the held contract’s prices, its average price the strategy’s own (Aligned views across a roll, The portfolio view across a roll).

The strategy wire. The roll rebase runs on stdio-pb-v1 and stdio-csv-v2, so a strategy written in another language takes part through a rebase block (Rebase blocks) (GLE-1505). stdio-csv-v1 cannot carry it, nor express a futures symbol. The run’s report stays on the served basis, so it compares with a run with the rebase off. The rebase is on by default (GLE-1502).

Choosing a bar size for live trading

A backtest’s order works on the bar’s clock, and a live order works on the broker’s. In a backtest an order decided at bar N’s close is matched against bar N+1 from its open, and a one-bar order ends at that bar’s close. Live, the order starts working only once IBKR has accepted it, and stops only once IBKR has processed its cancel. Those delays are a roughly fixed amount of time, so the shorter the bar, the larger the share of it they take, and the less the backtest says about the live run.

The broker’s delays. algodesk’s TWS probe (cmd/twsprobe, described in algodesk’s docs/tws.md) measured these on IBKR’s paper account, on AAPL, on 12 November 2026, in milliseconds:

Stepp50p95Max
Resting limit: submit to IBKR’s first status with a permanent order id58411271222
Market order: submit to the fill99315641725
Cancel to Cancelled463506618
Modification to IBKR’s answer469548736

These are paper figures. IBKR matches paper orders on its own servers, and a live account may be faster or slower. The network is a small part of them: a host in Chicago less than 15 ms from IBKR would still wait about 600 ms for an acknowledgement, so the delay is IBKR’s processing, not the distance.

What the delays do to an order. At these figures a resting order starts working about 0.6 s after its bar opens (1.3 s at p95), and a cancel sent at the bar’s close takes effect about 0.5 s after it. A market order fills about 1 s after the bar’s open, where the backtest fills it at the open. Two rough measures by bar size:

BarLate start at p95, as a share of the bar1 s of price drift against the bar’s typical range
5 s26%45%
15 s9%26%
30 s4%18%
1 min2%13%
5 minunder 0.5%6%

The last column assumes prices move as a random walk, so the drift over 1 s, against the range over a bar of B seconds, is about the square root of 1/B. It is a rule of thumb, not a measurement. It applies to every order, market orders included: a second of delay costs the same at any bar size, and is a larger share of a short bar’s move.

Which bar sizes are reasonable:

  • 1 minute and longer. The delays are a few per cent of the bar. This is the floor for a live run that is expected to stay close to its backtest.
  • 15 to 30 seconds. Workable only if the acknowledgements measured from the trading host are well under paper’s 600 ms, and the strategy’s edge survives a second of delay.
  • 5 seconds. Not realistic through IBKR. An order misses about a quarter of its bar, and a market order’s slippage is comparable to the bar’s whole range.

Other limits that grow as bars shrink:

  • Message volume. kit.OneBar cancels and resubmits its orders at every bar, unchanged ones included, so on 5-second bars it sends a steady stream of order messages that never fill. IBKR and the exchanges monitor the ratio of order messages to fills; check IBKR’s thresholds before running a fast strategy. kit.Controller keeps an unchanged order working, and a changed one is modified in place (GLE-1419), so it sends far fewer. The transport scheduler sends at most 5 requests per window by default (The transport scheduler), and on the stepped loop the window is a bar, so the rest queue.
  • IB Gateway’s daily restart. It drops every API connection. In the measured restart the connections came back after 24 to 36 seconds: 5 to 7 missed bars at 5 seconds, one or none at 1 minute.
  • Waiting for bars. A run of several series waits up to 2 seconds for the other series’ bars that close at the same instant (above), and that wait adds to every delay here. How late marketfeed delivers a live bar after its close has not been measured.
  • One-bar orders under lifetime v1. Live, a v1 Day order goes to IBKR as DAY and works until the session’s close. Nothing cancels it after one bar, as the simulator does (Order lifetime examples). A live strategy that wants one-bar orders runs under --lifetime v2 with kit.OneBar or kit.Controller. A live Order.CancelAt, which ends an order at a set instant without waiting for the next decision, is GLE-1753; retiring v1 is GLE-1754.

Measure before choosing. Run the probe’s round trips (twsprobe -probe fills) from the host that will trade, on a futures contract the strategy trades. Then confirm them with a few one-lot trades on the live account, since paper’s timings may not hold there; the probe refuses an account that is not paper. Also measure the time from a bar’s close to the strategy’s decision, which no tool reports yet.