Release-stamped series

The dates on these series need care. A COT report describes positions on a Tuesday, but the CFTC publishes it that Friday at the earliest, and during the 2025 shutdown seven weeks later. A FRED payrolls observation dated 1 August describes a month whose figure appears in September. A backtest that used each value from the date it describes would trade on numbers nobody had yet. Marketfeed and the engine prevent that between them: marketfeed stamps every value at the instant it was first public, and the engine hands it to the strategy at that instant and not before.

How a release reaches a strategy

The wire has no typed schema for these series, so the adapter serves each release as one synthetic bar. Open, high, low and close all equal the value, and volume is 0. The bar closes at the instant the value was first public and opens at the previous release. The requested interval is ignored and the series’ own cadence is served. A cot symbol needs its group qualifier (cot:CFTC:13874A:MM).

• COT rows close at 16:00 New York on the day the CFTC actually published the report (it publishes at 15:30), usually the Friday after the Tuesday as-of date. The date comes from marketfeed’s release calendar, which records holiday weeks, the 2013, 2018-19 and 2025 shutdowns and the 2023 ION incident. The E-mini S&P 500 report as of 2025-09-30 closes on 2025-11-19. Reports published together, as in a shutdown catch-up, close a second apart in as-of order.
• RBA FX rates close at 17:00 Sydney on the rate date, the RBA’s published release time.
• FRED has two kinds of series. 52 serve each observation’s first-release value, closing on ALFRED’s first vintage date for the observation at the original publisher’s release time (BLS and BEA 08:30 New York, H.15 16:15, H.4.1 16:30). September 2026 payrolls close at 08:30 New York on 2026-10-02. The other five (VIXCLS, VXVCLS, DFEDTARU, RRPONTSYD, MORTGAGE30US) are known on their observation date and never revised, and close at 17:00 New York that day. Observations released together close a second apart in observation order.

Each stored row also records the period its value describes: the COT as-of date, the FRED observation date or the RBA rate date. The wire carries it as Bar.ref_date (proto field 27, YYYY-MM-DD; the optional eleventh CSV field on both bar lines, after SESSION, which is left empty when the bar has none), and a strategy reads it as Bar.Ref, that date’s UTC midnight (GLE-261). Key periods by Ref, never by OpenTime: on a release-stamped bar OpenTime is the previous release. Ref is zero on every price bar, and a period that was never published has no bar. The marketfeed adapter serves it from GLE-252; until that adapter is installed Ref is zero on these series too. A bar file for the file-csv adapter may add a ninth ref column (header symbol,ts_open,ts_close,open,high,low,close,volume,ref), and that adapter, like marketfeed’s, serves a cot:, macro: or fx:RBA: symbol’s rows verbatim at their own cadence whatever interval is requested. A strategy binary built from an SDK older than GLE-261 rejects a ref-dated CSV bar that has no session (its parser refuses the empty SESSION placeholder): rebuild strategies with the engine. A run’s series cache stores bars as protobuf, so it keeps ref_date; entries cached before the adapter served it lack it until refreshed (--cache refresh).

Declare these series at 1w, as the strategies do. A bar’s close is its release, not its open plus the interval, and only the pb and csv wires carry that close verbatim; 1w has no DBN record type, so negotiation routes a configured release series there, and a forced dbn request is refused rather than shifting the release. Negotiation happens before the strategy declares its series, so a run that declares one needs --data-protocol pb. Without it, the engine refuses the declared series before any fetch and names the flag (GLE-311).

The engine merges every series of a run into one stream ordered by close time (Backtest event ordering), so a release joins ctx.Series(id) when the run’s clock reaches its stamp. At any decision, ctx.Series(id).Close(0) is the latest value already published. kit.AsOfClose(series, cutoff) returns the latest value released by an explicit cutoff, normally the primary bar’s close; use it when the cutoff can be earlier than the current decision. At a co-close the release goes after the primary (the co-close order, Backtest event ordering): a release stamped at exactly a primary bar’s close arrives just after that bar’s decision and is first seen at the next one, so a decision cannot act at a close price that predates the release. For whole windows, kit has the alignment helpers (GLE-263; see Lining up series): AsOfCloses(aux, prim, n) (the value known at each of the last n primary closes, carried forward), JoinCloses(a, b, n) (an inner join on exact close time, for same-interval legs with missing bars), ReleaseClock (the aux value and the last primary close before each release, for correlations against a slow series), AgeAt/StaleLegs/StaleSeries (each leg’s age at the decision) and Cadence/CompareLegs (equal interval and session end, or the implicit lag between legs). Each helper’s doc comment states its pairing class; never correlate carried-forward as-of values.

Lookback and Bar(k) count releases. Lookback: 104 on a COT series holds the last 104 reports known. That is 104 as-of weeks, but not always the last 104 calendar weeks: since 2010, 64 of 874 weeks had no COT release and 62 had two or three, in holiday weeks and shutdown catch-ups. FRED first-release series are stamped in observation order, but a period that was never published has no row: CPIAUCSL has no October 2025, so from November 2025 on Bar(12) of that monthly series is 13 months back, not 12. Key such lookups by reference period instead (GLE-264): kit.PeriodsAgo(cpi, kit.PeriodMonth, 12) is year on year by Ref and reports a missing month (ok false, with that month’s Ref) rather than returning its neighbour; kit.RefValue looks up one period, kit.ShiftRef does the calendar arithmetic, and kit.PairByRef pairs two release series by period, each pair known at the later release. For a second view of the same series, sampled at the bars of a series the strategy chooses and holding each value as known at that bar, declare an aligned series (GLE-353): a daily view of a COT series repeats each report on every daily bar until the next one, with the report’s own release instant in SourceCloseTime. Keep the release view for statistics over the reports themselves.

Warmup counts releases. Before the run, the engine warms each irregular series up to its depth — the declaration’s own Lookback when it names one, else the global Lookback() — in release ROWS (GLE-262): it fetches back from at least a year, doubling the window until it holds that many releases or reaches the series’ earliest coverage. A monthly series declared with Lookback: 24 starts the run with its last 24 releases (fewer only when the archive starts later, which the run records as a warmup_shortfall degradation). Guard reads with Len() all the same.

Don’t add a publication lag of your own: the stamps already carry it.

For positioning in one product, read that product’s own contract market (kit.CFTCCodeForSymbol: 13874A for the E-mini S&P 500, 088691 for COMEX gold), not a three-digit CFTC commodity sum or a -CMB consolidation. The sums mix contract sizes and unrelated markets, and the consolidations change unit with the 2 May 2023 report. marketfeed’s docs/marketfeed.md has the full stamping rules and data warnings.

Estimated release dates in early FRED history

ALFRED, the St. Louis Fed’s archive of past vintages, starts at a different date for each series. An observation older than a series’ ALFRED record has no usable first-release date, because ALFRED often adds such history in a vintage years later: DGS10’s 1962 values first appear in the 2005-12-20 vintage. Marketfeed therefore estimates the release as the observation date plus the series’ 90th-percentile release lag, measured over the ALFRED era in marketfeed’s 2026-09-28 audit, at the release’s time of day. Precisely, an observation is estimated when its date plus that lag falls before the series’ first ALFRED vintage.

The estimate can be early. If early releases were as slow as modern ones, about one estimated value in ten is stamped before it was really public, usually by days, so a backtest over those years can see it slightly early. Values just inside the record carry the record’s first vintage date, which can be later than their real release: late, never early. We accept this residual risk. Where a result depends on one of these series, start the backtest after its “Estimated through” date below, or check that the result survives doing so.

Most backtests start in 2010 or later; Databento’s futures history begins in June 2010. These series have estimated stamps in 2010 or later: STLFSI4 (to October 2022), SAHMREALTIME (July 2019), DTWEXBGS (January 2019), PCOPPUSDM (November 2014), BAA10Y, T10Y2Y, T10Y3M, T10YIE and T5YIE (January 2014), BOGMBASE (May 2013), RECPROUSM156N (June 2012), WALCL (June 2011), ANFCI and NFCI (May 2011), DCOILWTICO (March 2011), TEMPHELPS and USSLIND (November 2010), JTSQUL (May 2010), and DRTSCILM, M2V and USREC (January 2010). The spreads T10Y2Y and T10Y3M can be computed from DGS10, DGS2 and DGS3MO, whose records start in June 2005.

The table covers the 52 first-release series, measured from the lake on 2026-10-03. “Estimated through” is the last observation date with an estimated stamp; “Lag” is the estimate’s lag in calendar days. The five known-on-the-day series have no estimated stamps.

First-release values, not revisions

Served FRED values are first releases, and later revisions never change them. Marketfeed keeps every vintage in its revision store (vintages/FRED/<ID>.csv: observation date, vintage date, value), but the store is not served. There is no read of a series as known on a given date with revisions applied. This is a deliberate choice; the trade-off is recorded here in case it needs revisiting.

For first-release values:

• No look-ahead through revisions. Each value is one a trader could have acted on at its close.
• One value per observation, fixed for good. Results are reproducible, a later revision cannot change a past backtest, cached series stay valid, and live and backtest runs see the same history.
• It is the number the market reacted to. Prices move on the first print, so a release-day signal sees what was traded on.
• Simple delivery. One bar per observation, in release order, keeps the wire simple and Lookback meaningful.

Against:

• It is not everything known at the time. On any date a trader also knew the revised values of earlier observations, while the strategy sees only first prints. Payrolls are revised in each of the two following months and benchmarked every year, so a 24-month payroll trend built from first prints follows a path nobody saw at any single moment.
• Rebasings can leave steps. When an agency rebases a level series, consecutive first prints can come from different bases, and a change computed between them includes the rebasing. BLS applies its annual payroll benchmark with the January figure, released in February.
• Revisions are news in themselves. The revisions to earlier months that come with each release are invisible.
• Published models may not carry over. Studies, and models calibrated on today’s FRED, use latest-vintage data, so their thresholds may not suit first-release values.

If this is revisited, the options are:

• A real-time view: at each decision, the series’ whole history as of the latest vintage then. This is correct, but each release then carries a whole history, which needs a wire design (a vintage per bar or a snapshot read) and makes runs heavier.
• A latest-vintage research series, clearly labelled. It helps comparisons with published work, but it has look-ahead by construction and must never feed a strategy.

Either can be rebuilt from the revision store without refetching.

COT reports beyond the release calendar

Marketfeed’s COT release calendar (fetch/cot/releases.csv in marketfeed) lists every report week from 2006-06-13. In the marketfeed commit pinned here it ends with the report as of 2026-09-22. A report after the calendar’s last week is stamped at the usual Friday, 16:00 New York, if the nightly run sees it within 12 hours of that time, and otherwise at the time it was first seen. Such a stamp can be late but never early. It records our run, not the CFTC’s release, so a late or missed nightly run stamps a report later than the CFTC published it, and a later run keeps the stamp an earlier one chose.

Append each year’s dates to the calendar when the CFTC publishes its schedule. Extending the calendar re-stamps weeks already stored only when they are fetched again; marketfeed’s docs give the procedure.

Live trading

Today. This build’s live mode is an emulator-driven harness for a single series (The venue seam and the current live path). A strategy that declares a COT or FRED series has two symbols, so live mode refuses it at startup. Live mode also refuses every continuous futures symbol (a bare root or a :cont composite) at startup, before any spawn: marketfeed does not stream a composite, so a live run trades a dated contract (owner decision 2026-10-04, GLE-228 Decision 7). A live run also fetches all its data when it starts and nothing afterwards. No strategy that reads these series can run live yet; this subsection records the rules that apply once one can. GLE-295 tracks the work.

The rule. Live market data will come only from Databento. COT, FRED and RBA series will not be streamed at all: a live run reads them from the marketfeed archive and treats them exactly as a backtest does: each release joins the strategy’s series when the clock reaches its stamp, Lookback counts releases, and the values are first releases. Warmup at startup reads the same rows a backtest would. The stamps are stable, because first-release values never change and marketfeed keeps the stamp an earlier run chose, so a restarted strategy rebuilds the history it had.

Where live and backtest differ: arrival. A backtest assumes each value is in hand at its stamp. A live run has it only once the archive has fetched it and the engine has read it. With only marketfeed’s nightly run (11:30 Melbourne, which is 01:30 UTC under AEST and 00:30 UTC under AEDT), each source reaches the archive this long after its stamp at the earliest:

A decision between a stamp and its arrival sees the value in a backtest but not live. Take a strategy deciding at the CME equity close, 16:00 Chicago (17:00 New York). A backtest uses Friday’s COT report at Friday’s close; a live run first has it on Monday. A payrolls figure released at 08:30 is in hand by that day’s close in a backtest, and live only at the next session’s close. The same applies to the H.15 rates stamped at 16:15 New York.

Making them agree. There are two ways; live runs should use the first.

• Fetch promptly. Schedule archive fetches shortly after each release as well as the nightly run: COT on Fridays after 15:30 New York, FRED on release mornings after 08:30 New York and daily after 16:15 New York for H.15, and RBA after 17:00 Sydney. The engine must then re-read the archive during the run, which is not built (GLE-295): a live run reads it only at startup. FRED posts a release some minutes after the agency publishes it, so even a prompt fetch lands the value after its stamp. That matters only to a decision inside that window, such as an intraday bar closing just after 08:30.
• Delay the backtest. A backtest could instead move each release to when the live setup would have it, for example the next scheduled fetch. That makes backtests pessimistic and matches a nightly-only live setup exactly. It is not built either.

Rows stored before their stamp. A row can reach the archive before its stamp. On 2026-10-03 the archive already held T10Y2Y, T10Y3M, T10YIE and T5YIE for 2026-10-02, stamped 2026-10-05 16:15 New York because of the one-business-day floor. A live run must hold such a row until its clock reaches the stamp, as a backtest does, and never deliver it early.

Releases that land after their stamp. When a live run finds a release whose stamp has already passed, it should add the release to the series at once, before the next decision, and never re-run a decision already made. The protocol has a late flag for this: the SDK appends a late bar to the series without calling OnBar. The engine does not set the flag yet. The run should also record each such release’s stamp and arrival time, so live-versus-backtest reconciliation can explain the decisions that differ.

The earlier gaps, live.

• Estimated early FRED dates don’t affect live decisions: by the time a live run starts, those values are long public.
• First-release values apply unchanged, and they are what keeps live history equal to backtest history. A revision published while the strategy runs never alters what it has seen.
• COT reports beyond the calendar are the normal live case unless the calendar is kept current. A report the nightly run sees within 12 hours of the usual Friday is stamped at that Friday. A later one is stamped when first seen, which is when the live run had it. If the calendar is later extended over such a week and the week fetched again, its stamp moves back to the CFTC’s release, and later backtests see the report earlier than the live run did.