Marketfeed

The bundled file adapters are ideal for learning and small fixtures. Marketfeed is the normal bridge to the larger local historical archive. It is a companion Go project, not a package inside this repository:

  • Marketfeed owns acquisition, canonical names, archive layout, coverage manifests and faithful reads across data-source eras.
  • marketfeed-adapter exposes that archive to Algolang over the data wire.
  • Algolang decides how a strategy run is assembled, including continuous-contract stitching on the feed’s served roll schedule, price adjustment and execution.

The viewers and the engine roll on one rule. Marketfeed’s :cont composite, its viewers, and every engine continuous run follow the feed’s served roll schedule, compiled from the root’s roll: rule: ES’s C:EXP-12:VOL:1 becomes a backstop 12 calendar days before expiry. The engine elects nothing and reads no roll knob of its own; it fetches the schedule over the protobuf wire and executes it. That is decision D1 of the feed-owned continuous futures proposal, implemented by GLE-270.

Building and selecting the adapter

make marketfeed-adapter builds the adapter into bin/ from the marketfeed commit pinned in marketfeed.ref, compiled against this checkout (Pinned checkouts). make install here installs that build as marketfeed in the adapter directory, where algo run finds it without --adapter for any symbol the archive holds (Finding the adapter). An installed marketfeed (make install there) puts its own build at /usr/local/bin/marketfeed-adapter. Name any of them with --adapter:

make marketfeed-adapter
export MF_ADAPTER=$PWD/bin/marketfeed-adapter

bin/algo run \
  --adapter "$MF_ADAPTER" \
  --adapter-inputs Dir=/usr/local/marketfeed \
  --strategy bin/hello \
  --symbols fut:XCME:ES:M26 \
  --bar-interval 1h \
  --date-from 2026-01-01T00:00:00Z \
  --date-to 2026-04-01T00:00:00Z

Dir names the archive. If it is omitted, the adapter uses $MARKETFEED_DIR, then /usr/local/marketfeed. A particular installation may not contain the symbol or range in this example; empty or partial archive coverage is data state, not something the adapter invents around. The adapter’s other inputs tune spreads (MaxCarry, Synthetic spreads) and live subscriptions (LiveInstruments, LiveCache, LiveMaxLag and others; marketfeed’s manual lists them). No input names the live source: that comes from the environment (The live source).

For history the adapter reads the archive. It advertises time bars, the event-driven bar kinds when a trade tape is present, raw trades, instrument metadata, granularity-aware coverage, and optional intra-bar paths. A marketfeed built against this SDK also serves intraday grids and answers the engine’s capability probe with the calendar revision of the requested boundary; the marketfeed commit pinned in marketfeed.ref does. The engine negotiates the wire automatically. Force --data-protocol pb when a feature requires per-record symbol identity or nested path data.

The adapter is not only historical. It also streams bars, roll schedules, corporate actions and top-of-book quotes to a live run on a real-time feed (Running live on a real-time feed); its live bars and quotes come through marketfeed’s gateways (The live source).

Marketfeed also stores irregular COT, FX and macro series, served by release time; see Release-stamped series.

The live source

Only marketfeed talks to Databento. That is the owner’s rule of 20 Nov 2026 (GLE-1790): marketfeed’s gateways (mf-gateway on db1 and db2) hold the Databento live sessions and serve /hist, a proxy of Databento’s historical API, and every other process gets Databento’s data from a gateway or from the archive. No algolang process holds a Databento key.

The adapter’s live bars and quotes therefore come through the gateways. Two settings name them. Each is read from the environment, or else from ~/.marketfeed/env:

SettingWhat it is
MF_LIVE_SERVERSthe ordered server list, name=host:port entries separated by commas: db1, then db2
MF_GATEWAY_KEYthis client’s gateway key, used for every gateway in the list

Neither is an adapter input, because adapter inputs are public. The adapter streams from the first server in the list. After the current server has failed for 5 seconds it moves to the next, and every session resumes there from the last record it holds. When the last server of the list fails for 5 seconds, it starts again from the top (GLE-1787). Otherwise it goes back to the top only when it restarts or when its gateway tells it to. A gateway that drops off the network without closing its connections is noticed after 15 seconds of silence. Its catch-ups follow the current server: bars past the archive’s end come from the gateway’s replay and from its /hist proxy. With a list, the adapter polls each server every 5 seconds and reports their health on every bar and quote subscription (SubscriptionEvent.endpoints, Live bars from an adapter). The details are marketfeed’s (“Gateway clients” and “Gateway failover” in marketfeed’s docs/marketfeed.md).

Without a server list the adapter has no live source, whatever DATABENTO_API_KEY holds, and every live bar and quote subscription is refused with live bars are off: no live servers: set MF_LIVE_SERVERS and MF_GATEWAY_KEY (environment or ~/.marketfeed/env). History reads still work. A list that names a gateway while MF_GATEWAY_KEY is unset turns live data off too, with an error that names the missing key.

No direct route. The marketfeed commit pinned in marketfeed.ref has no way to Databento that bypasses the gateways (GLE-1793):

  • With no server list, live data is off. The adapter does not fall back to DATABENTO_API_KEY.
  • A databento entry in the list is refused, with an error that names the rule, and live data is off.

marketfeed keeps one bypass for its own occasional checks, MF_DATABENTO_DIRECT=1 (“The bypass” in marketfeed’s docs/marketfeed.md). A process run with it says so in its log, and its endpoint reports show the server databento. Do not set it for a live run.

Both gateways down. No live run has live data until a gateway returns, on any desk, LIVE included (the owner’s decision of 20 Nov 2026, GLE-1791). The run’s feed is down, so its openings stop until the feed is back (Running live on a real-time feed). LIVE’s server list is db1, db2 and nothing else: it has no Databento last resort. A third gateway may be added later; it is not planned.