Configuration and run matrices

Configuration and run matrices

Single-run flags stop scaling the moment you want “this strategy on these five symbols with three parameter sets”. A config file describes a whole matrix of runs; the engine executes them sequentially, each with its own adapter, strategy instance, and simulator account, then prints a portfolio rollup.

The file is JSON with // and # comments allowed. The vocabulary:

  • global – defaults every run inherits: strategy, bar-interval (1d when no level sets one), date-from/date-to (YYYY-MM-DD), initial-capital (global-only; its currency is the account’s base currency, so there is no base-currency key; see Currencies), inputs, name, commission-mode (per-trade or per-unit) + commission-amount (an amount in the account’s currency, 2.50, or one with its currency, "EUR2.50", which each fill converts into the account’s; see Currencies), slippage-ticks, margin-model (report, enforce or off; see Margin), and the data object naming the adapter: {"adapter": path, "inputs": {...}}, where the adapter is a path or the name of an installed adapter and is required (a config file does not discover its adapter; see Finding the adapter), plus optional bar-kind/threshold/include-path/accept-degraded-seams/roll-rebase/continuous-clock/protocol fields mirroring the CLI flags ("include-path": true opts the run into the intra-bar path), the fill settings simulation-resolution, bracket-ambiguity and limit-fill beside path-interval (GLE-1603; see Fill settings per run), and the universe-expansion governance policy (how far a strategy may grow the universe – see below), and the order-lifetime keys lifetime (v1 or v2; absent at every level, the default: v2 for a backtest, v1 for a live run), lifetime-turns and lifetime-hold-timeout with the admission policy’s six limits, max-working-orders to submission-window (see The admission policy), and the transport scheduler’s three, transport-requests, transport-window and transport-queue (see The transport scheduler), the IBKR venue’s venue-reply-allow, a list of message ids (see Order reply messages), and the exchange-calendar keys exchange-calendar, calendar-revisions, strict-calendar, calendar-accept-unverified and calendar-margin-quarters (see Engine side), and the strategy stall timeout strategy-stall-timeout, a duration such as "90s", off when absent or "0s" (see Stopping a run). cache (off/on/refresh) and cache-dir are global-only, like results-dir. The removed keys symbol-registry (at any level) and data.served-roll-schedules were refused for one release (GLE-325) and are, since ER5 (GLE-327), unknown fields ignored with a warning; delete them. data.adjustment-mode and data.dividend-treatment are refused for one release (GLE-328, GLE-228 Decision 5a): the equity price view rides the symbol, eq:<VENUE>:<TICKER>:adj=split or adj=splitdiv (no adj= is raw), and the refusal names that form.

  • runs – the run entries. One run is one strategy instance. Each entry may override any global setting, and chooses exactly one of three symbol forms:

    1. {"symbol": "MSFT"} – single series;
    2. {"symbols": "MSFT/FAKE"} – slash-separated: ONE instance fed both series, merged by close time (ties broken in listed order);
    3. {"series": [{"symbol": "MSFT"}, {"symbol": "MSFT", "bar-interval": "5m"}]} – the full form: one instance sees every listed series, each with its own interval (mixed timeframes), its own session, and its own bar construction (GLE-390). The construction keys are bar-kind, threshold, reversal-mult, threshold-mode, ewma-span-bars, ewma-span-signal, init-expected-size, init-signed-rate, lead-in, include-path, path-interval and close-on-boundary. An entry without bar-kind takes the run’s construction. An entry with one has its own complete construction, so {"symbol": "MSFT", "bar-kind": "volume", "threshold": 1000} beside {"symbol": "MSFT", "bar-interval": "1m"} mixes volume and time bars in one instance. Each entry’s series id names its construction (MSFT@volume:t=1000), and the rules are those of declared series.

    An entry may carry an id: the run’s stable name, which the desk keeps the run’s book under when the file is deployed (Running a deployment against the desk). It is 1 to 64 characters from lowercase letters, digits, - and _, starts with a letter or a digit, and is unique in the file. It names one run, so an entry that sweeps its inputs or fans out over a template cannot carry one. In a backtest the id is the run’s id, its results directory and report id, ahead of its name.

  • symbols (top level) – named template sources. A run containing {placeholder} in its symbol/name fields expands into one run per source row; several placeholders in one run are zipped (sources must be equal length). A source can be an inline array, a file path of one symbol per line, or "<name" for portfolios/name.txt next to the config file.

  • strategies (top level, optional) – batches: each entry has its own settings, optional scoped symbols, and its own runs, inheriting global. Use it to put two different strategy binaries in one file.

  • portfolio (top level, optional) – what the file’s runs share as one portfolio (GLE-1372). visibility (all, the default, totals or none) is what a strategy that reads the portfolio view may see; a strategies entry or a run overrides it with its own visibility, which is not a global key. See Reading the portfolio. director is the portfolio director, deprecated (GLE-1629): a director object, which a desk applies across every member only while its configuration names no director file of its own. The desk’s director now comes from that file, the one --director reads; algodesk director import core.json takes this object from the portfolio file into the desk. A backtest runs each run alone, without it, and says so in a note. A strategies entry or a run sets conviction_rank: own (the default) ranks the run’s orders under top-n-conviction against its own alone, with the rule’s slots to itself; portfolio ranks them in one pool shared with every other portfolio run, which shares the rule’s slots. Choose portfolio only for runs whose convictions are on one scale. It is not a global key, and a backtest does not read it. The object’s other fields are ignored with a warning.

  • Input sweeps – a strategy input given as a JSON array declares a sweep: the run expands into the Cartesian product of all its swept inputs, and each instance’s name gets a Name=value suffix.

  • Capital – initial-capital (e.g. "USD400000") is one pool, global-only. Each configured run takes a share proportional to its capital-units (default 1), and a run’s share then subdivides equally across all of its template and sweep expansions. No initial-capital anywhere means every run independently gets the default 100,000. The pool’s currency is every run’s base currency (Currencies): "JPY15000000" runs the accounts in yen.

The global-only report object configures the performance report and what the set writes (Saving the results lists its keys). Two of its keys govern the set as a whole: export writes the set to one .algorun file as it runs (Writing the export as the set runs), and keep-going carries the set past a failed run, which it otherwise stops at (When a run of a set fails). keep-going is on by default when the set exports:

"report": {"equity-sampling": "bar", "export": "sweep.algorun", "keep-going": true}

Settings resolve global < strategies-entry < run < explicit CLI flag. The three fill settings’ flags are the exception: --simulation-resolution, --bracket-ambiguity and --limit-fill are defaults, which a key at any level overrides (Fill settings per run). Only the config overrides listed in the CLI reference may accompany --config. Some compatibility fields, including unsupported session-hours and session-days, are accepted but ignored with explicit warnings. Trading-day session selection is implemented and is a separate setting.

A worked example

Build a two-symbol fixture by cloning the sample bars under a second name (a cheap stand-in for a real second data file):

{ cat testdata/msft-1m.csv
  tail -n +2 testdata/msft-1m.csv | sed 's/^MSFT,/FAKE,/'
} > testdata/pair-1m.csv

Then write pair.json:

{
  // A run matrix: per-symbol SMA sweeps, one two-symbol pair instance,
  // and one mixed-timeframe instance. JSON with // and # comments.
  "global": {
    "strategy": "bin/smacross",
    "bar-interval": "1m",
    "initial-capital": "USD400000",
    "data": {
      "adapter": "bin/file-csv-adapter",
      "inputs": {"Path": "testdata/pair-1m.csv"}
    }
  },

  "symbols": {"sym": ["MSFT", "FAKE"]},   # template source

  "runs": [
    // Expands per {sym} row, then per Fast value: 2 x 2 = 4 runs.
    {"symbol": "{sym}", "name": "SMA {sym}",
     "inputs": {"Fast": [2, 3], "Slow": 8}},

    // ONE instance fed both series, double capital weight.
    {"name": "SMA pair", "symbols": "MSFT/FAKE", "capital-units": 2},

    // ONE instance, same symbol at two timeframes (5m is resampled).
    {"name": "MSFT two-speed", "series": [
      {"symbol": "MSFT"},
      {"symbol": "MSFT", "bar-interval": "5m"}
    ]}
  ]
}

Use --verify to validate the configuration, inspect the expanded matrix, and run each run’s pre-flight checks against its adapter (the checks algo run makes before its first fetch) without executing a run:

$ bin/algo run --config pair.json --verify
Run plan: 6 run(s) from pair.json
   1. "SMA MSFT Fast=2"    strategy=bin/smacross series=MSFT@1m capital=USD 25000.00 sweep=Fast=2
      window: 1970-01-01T00:00:00Z < ts_close <= 2100-01-01T00:00:00Z
      MSFT: finest supportable granularity: 1m (limited by file-csv-adapter, 2022-06-10T12:30:00Z..2022-06-10T12:53:00Z)
   2. "SMA MSFT Fast=3"    strategy=bin/smacross series=MSFT@1m capital=USD 25000.00 sweep=Fast=3
      window: 1970-01-01T00:00:00Z < ts_close <= 2100-01-01T00:00:00Z
      MSFT: finest supportable granularity: 1m (limited by file-csv-adapter, 2022-06-10T12:30:00Z..2022-06-10T12:53:00Z)
   3. "SMA FAKE Fast=2"    strategy=bin/smacross series=FAKE@1m capital=USD 25000.00 sweep=Fast=2
      window: 1970-01-01T00:00:00Z < ts_close <= 2100-01-01T00:00:00Z
      FAKE: finest supportable granularity: 1m (limited by file-csv-adapter, 2022-06-10T12:30:00Z..2022-06-10T12:53:00Z)
   4. "SMA FAKE Fast=3"    strategy=bin/smacross series=FAKE@1m capital=USD 25000.00 sweep=Fast=3
      window: 1970-01-01T00:00:00Z < ts_close <= 2100-01-01T00:00:00Z
      FAKE: finest supportable granularity: 1m (limited by file-csv-adapter, 2022-06-10T12:30:00Z..2022-06-10T12:53:00Z)
   5. "SMA pair"           strategy=bin/smacross series=MSFT@1m,FAKE@1m capital=USD 200000.00
      window: 1970-01-01T00:00:00Z < ts_close <= 2100-01-01T00:00:00Z
      MSFT: finest supportable granularity: 1m (limited by file-csv-adapter, 2022-06-10T12:30:00Z..2022-06-10T12:53:00Z)
      FAKE: finest supportable granularity: 1m (limited by file-csv-adapter, 2022-06-10T12:30:00Z..2022-06-10T12:53:00Z)
   6. "MSFT two-speed"     strategy=bin/smacross series=MSFT@1m,MSFT@5m capital=USD 100000.00
      window: 1970-01-01T00:00:00Z < ts_close <= 2100-01-01T00:00:00Z
      MSFT: finest supportable granularity: 1m (limited by file-csv-adapter, 2022-06-10T12:30:00Z..2022-06-10T12:53:00Z)
OK

Each run is followed by its window and its per-symbol granularity ceiling. The pre-flight negotiates each run’s protocols exactly as the run would, and as the run does, it says nothing of the wire it chose. A series label carries its session (MSFT@5m-rth). Warnings the run would print, such as a continuous run’s roll-policy note, are listed under warnings:, each prefixed with its run. When a run would fail before its first bar, --verify prints that run’s exact error under it, still checks every other run, and ends with a count in place of OK. Adding "session": "rth" to the 5m series, which this adapter cannot serve as a grid, gives:

   6. "MSFT two-speed"     strategy=bin/smacross series=MSFT@1m,MSFT@5m-rth capital=USD 100000.00
      window: 1970-01-01T00:00:00Z < ts_close <= 2100-01-01T00:00:00Z
      FAILED: engine: series MSFT@5m-rth: adapter "file-csv-adapter" does not advertise intraday-grid-v1 for boundary "rth"; upgrade the adapter (adapter stderr: "algolang-data: intraday grids not supported")
FAILED: 1 of 6 run(s) failed pre-flight
algo: --verify: 1 of 6 run(s) failed pre-flight

and exit status 1. OK therefore means every run would start. A refusal made while the configuration is built, such as an @ES alias or a :cont series outside the run grammar, stops --verify before it prints the plan, exactly as it stops algo run. The pre-flight fetches no bars or trades, opens no series cache and starts no strategy (it checks only that the strategy binary resolves), so a failure that only the data or the strategy’s declarations reveal still surfaces at run time.

Check the capital math: three configured runs weigh 1 + 2 + 1 = 4 capital-units, so the USD 400,000 pool splits 100,000 / 200,000 / 100,000. The first run then expands into 2 symbols x 2 sweep values = 4 instances, which split their run’s 100,000 equally – 25,000 each. Sweep siblings always split equally; there is no per-value weighting.

Drop --verify to execute. Six reports stream past (each exactly like the first run’s), and because there is more than one run, a rollup follows:

$ bin/algo run --config pair.json
Algolang run sma-msft-fast-2
  name:       SMA MSFT Fast=2
  ...
              equity 24998.76 (capital 25000.00, return -0.0050%)
...
Algolang run sma-pair
  name:       SMA pair
  instance:   1 (MSFT@1m + FAKE@1m), 6 orders
  ...
Algolang run msft-two-speed
  name:       MSFT two-speed
  instance:   1 (MSFT@1m + MSFT@5m), 3 orders
  ...

PORTFOLIO 6 runs
  capital:    400000.00
  fills:      25
  realized:   -4.20
  equity:     399996.22 (return -0.0009%)

Note what runs 5 and 6 are not: they are not two runs each. “SMA pair” is one strategy instance receiving both symbols’ bars interleaved – cross-symbol orders within that universe are allowed. Spreading a strategy across symbols as separate instances is always the explicit template expansion of run 1, never engine guessing.

Universe expansion

A strategy can ask for data on – or to trade – symbols beyond the ones the run configures (see “Declaring extra data and universe expansion” for the strategy side). How far it may go is governed per run by the universe-expansion field on the data object:

  • reference-only (the default) – a strategy may pull in a brand-new symbol for data only; it may not add a new tradeable symbol. The common case: read extra reference series, trade only configured symbols.
  • off – no new symbols at all. A strategy that declares one fails to start. Use this to pin a run to exactly its configured universe.
  • full – new symbols may be both read and traded. Because each fanned-out run (template or sweep expansion) is an independent account with no cross-run netting, full is rejected by --verify for any run that fans out into more than one instance, and is rejected in live mode. A calendar spread (fut:XCME:ES:M26-U26) named tradeable is still refused: it is data only, and is traded as a combo order on its legs (Combo orders, GLE-409).

A new interval of a symbol already in the run is a refinement, not an expansion, and is always allowed regardless of the policy. The policy is a guardrail applied loudly at startup: a declaration it forbids stops the run with a clear error rather than being silently dropped or surfaced only through denied orders.

{
  "global": {
    "strategy": "bin/pairs",
    "data": { "adapter": "bin/databento-adapter", "inputs": { "Path": "..." },
              "universe-expansion": "full" }   // this strategy trades a declared NQ leg
  },
  "runs": [ { "symbol": "ES" } ]
}