The export file

Specified (GLE-664); writing and reading it are planned. A run set’s export is one self-contained SQLite file, NAME.algorun. It holds the runs, the bars every run saw, each run’s per-bar equity, trades, fills, the trade director’s log, rolls, instruments, calendars and the metrics Go computed. It is the source of truth for both algo report and kchart’s MarketDemo, which opens it in a run tab. This chapter is the specification of version 1, algolang.export/1. Both repositories check against it.

What exists today:

• the specification itself, this chapter;
• the package engine/algorun, which holds:
  • the canonical schema, schema.sql;
  • the chunk codec, EncodeBars, DecodeBars, EncodeEquity and DecodeEquity;
  • Check, which validates a file against this chapter;
• a hand-made reference file, testdata/algorun/reference.algorun, which kchart vendors.

Planned:

• algo export RESULTS_DIR -o out.algorun, which converts an existing results folder (GLE-668);
• algo run … --export out.algorun, which writes the file during the run (GLE-672);
• algo report out.algorun, which reads it back (GLE-676).

The writer uses the pure-Go modernc.org/sqlite, so the engine still builds with CGO_ENABLED=0. kchart reads the file through the system SQLite3 module.

Opening a file

Any SQLite reader opens the file: the sqlite3 shell, Python’s sqlite3 module, DuckDB. The bulky columns, bars and per-bar equity, sit in compressed chunk BLOBs (Chunks). Everything else is plain rows, and JSON text where algolang already has a JSON form:

sqlite3 -readonly testdata/algorun/reference.algorun \
  "SELECT key, value FROM meta WHERE key IN ('format', 'set_name');
   SELECT ordinal, run_id, status, name FROM runs ORDER BY ordinal;"

format|algolang.export/1
set_name|Reference set
1|sma-es-fast-5-slow-20|ok|SMA ES Fast=5 Slow=20
2|sma-es-fast-20-slow-20|failed|SMA ES Fast=20 Slow=20
3|es-nq-spread|ok|ES/NQ spread

Identity and versions

A file identifies itself three ways, and a reader checks all three before it reads anything else:

Finished files only. A writer writes the whole file under a temporary name, in one transaction, and renames it into place, as algolang writes its other artefacts. meta.complete = 1 is the last row it writes. A reader refuses a file without it as incomplete.

Evolution. Within version 1, tables, columns, indexes, meta keys, features and bar flags are only ever added, and a reader ignores what it does not know. The values of an enumerated column (runs.status, fills.kind, …) are fixed: a new value is a change of meaning. A change of meaning, whether a column’s unit, a value’s sense or a chunk’s layout, bumps the version, and a reader refuses a version it does not know. The name plots is reserved for strategy plots (GLE-677). A version 1 file has no such table.

Conventions

• Times are INTEGER nanoseconds since the Unix epoch, UTC: Go’s UnixNano, good until the year 2262. Dates (equity_daily.date) are TEXT YYYY-MM-DD in UTC.
• Booleans are INTEGER 0 or 1.
• JSON columns end in _json and hold UTF-8 JSON text. Where the value is an algolang type, its JSON is the type’s, with Go field names and RFC 3339 times, exactly as result.json writes it: header_json is a report.ResultHeader, metrics_json a report.Metrics.
• Ordinals (n, seq, ordinal) count from 1, without gaps, in the order the source gives. Chunk indexes count from 0.
• NULL means absent or unknown, never zero. A row never stores NaN. Only the chunks’ benchmark column does.
• Money is in the run’s base currency, and prices are in points of the instrument, unless a column says otherwise.

Bar open and bar close

A bar has two instants: t, its open (the wire’s ts_open), and tc, its close (ts_close).

• The close is the bar’s instant in the engine. A bar belongs to a run when start <= tc < end. The strategy decides at tc, and each equity mark is stamped with the close of the bar it follows.
• The open is the bar’s key in a chart. Bars are stored in t order, chunks are indexed on t, and a bar anchor names its bar by t. That is the time a chart draws the bar at.

The instants are those the engine delivered:

• a grid bar’s t is its grid position and its tc the nominal end (Intraday grids);
• a session bar’s t and tc are its first and last constituents (Session bars);
• a release-stamped bar opens at the previous release (Release-stamped series).

A bar delivered without an open time is stored with t = tc, flagged no_open. Within a series t never decreases. For time bars it strictly increases; activity bars may share a t.

Fill stamps and bar anchors

A fill’s ts is its stamp as the simulator wrote it:

• a fill inside a bar is stamped with that bar’s close: a market order decided at bar N fills at bar N+1’s open price, stamped with N+1’s close;
• a market-on-close order carries its decision bar’s close;
• a roll’s legs carry the cut;
• a tick fill carries its print’s time.

A bar anchor, fills.bar_ts and trades.entry_bar_ts/exit_bar_ts, is the t of the latest bar of the row’s series whose t is strictly before the stamp. For a fill stamped at a bar’s close that is the bar itself; for a roll leg, the outgoing contract’s last bar. Where activity bars share that t, the anchor is the last of them. Anchors are the chart’s own bar times, so a reader draws a trade without snapping it to a bar.

Adjusted and raw prices

Bars are stored as the strategy received them. Fills keep the venue’s price beside the strategy’s:

• The two columns on series say which case applies. adjusted says the bars are an adjusted view. trades_adjusted says the trade prices are in that view too.
• Trade prices are the report’s. trades.entry_price and exit_price are what trades.csv and result.json hold. The report records an adjusted equity at its raw projection (Equity qualifiers).
• On an adjusted equity, mark a trade at its fills. A reader that marks trades on the chart takes fills.price (through entry_fill and exit_fill) when trades_adjusted = 0 and adjusted = 1.
• Rolls are in raw prices. A continuous series’ rolls hold the outgoing and incoming contracts’ raw prices at each cut, the gap and the cumulative adjustment.
• The snapshot pins the adjustment. snapshot_json holds at, the as_of instant the adjustment was taken at (the run’s end), anchor, the contract the snapshot holds, and token, the feed’s snapshot token when it served one. A re-fetch of the series pins to it.

Warm-up

• Warm-up bars are the history before the first bar the strategy decided on. The engine delivers them in a batch, with no OnBar (see Lookback and the Series API). They carry the warmup flag.
• They come first. They are a prefix of the series, and series.warmup_count counts them.
• Nothing is recorded during warm-up. Equity, trades and fills begin with the first loop bar.
• Runs can share a series. A series row is one distinct delivery: the same descriptor, the same snapshot and the same loop bars. Runs that received the same loop bars share the row, as the members of a sweep do. Each run’s own warm-up then starts at run_series.warmup_start, and the series holds the deepest history any of them read. Runs over different windows get different rows.

Sweeps and config entries

A run records where it came from in its config:

• config_entry is the path of the entry it expanded from: runs[0], or strategies[1].runs[2] inside a batch.
• sweep_json holds the inputs the expansion swept and the values this run took, as the config gave them, so {"Fast":5,"Slow":20} keeps its numbers.
• template_json holds a template expansion’s placeholder values.

Sweep siblings share config_entry and differ in sweep_json; kchart groups them on that. name keeps its Name=value suffixes (Config files and run matrices), and capital_units is the entry’s share of the pool.

Failed runs

A run that failed is kept:

• it has status = 'failed' and its error, the message that stopped it;
• its config facts are recorded: name, strategy, inputs_json, capital, config_entry and sweep_json;
• it has no rows in the result tables: equity_chunks, equity_daily, trades, combo_trades, fills and benchmarks.

A sweep’s refused combination is such a run: kchart shows it as a “refused” cell.

Bars from the run, or fetched again

• bars_source = 'run': the writer recorded the bars as the engine delivered them during the run, so they are exactly what the strategy saw.
• bars_source = 'refetched': algo export fetched them again from the run’s adapter, pinned to the snapshot. The archive may have moved since the run, so a reader checks that the fills lie inside their bars.

Features

meta.features is a JSON array that names what the file holds beyond the minimum. A file converted from older results lacks some features, and a reader falls back where one is missing.

Tables

The schema below is engine/algorun/schema.sql. A writer runs it as given on an empty database. Every table, column, key and index in it is part of the specification. The sections after it give each table’s meaning.

-- The .algorun export, format algolang.export/1 (GLE-664).
-- PRAGMA application_id = 1095519055 (0x414C474F, "ALGO"); PRAGMA user_version = 1.
-- Times are INTEGER nanoseconds since the Unix epoch, UTC. Booleans are INTEGER 0 or 1.
-- JSON columns hold UTF-8 JSON text. Ordinals named n, seq and ordinal count from 1;
-- chunk indexes count from 0.

CREATE TABLE meta (
  key   TEXT NOT NULL PRIMARY KEY,
  value TEXT NOT NULL
);

CREATE TABLE runs (
  run_id                TEXT    NOT NULL PRIMARY KEY, -- the run's results-folder slug
  ordinal               INTEGER NOT NULL UNIQUE,      -- position in the expanded config
  name                  TEXT    NOT NULL,             -- display name, sweep suffix included
  status                TEXT    NOT NULL,             -- ok | failed
  error                 TEXT,                         -- why a failed run failed; NULL when ok
  strategy              TEXT    NOT NULL,             -- strategy binary base name
  strategy_name         TEXT,                         -- the name the strategy reports
  strategy_version      TEXT,                         -- the version the strategy reports
  config_entry          TEXT,                         -- runs[0], strategies[1].runs[2]
  sweep_json            TEXT,                         -- swept inputs: {"Fast":2,"Slow":30}
  template_json         TEXT,                         -- template placeholders: {"sym":"ES"}
  inputs_json           TEXT    NOT NULL,             -- every input as the run received it
  capital               REAL    NOT NULL,
  capital_units         REAL    NOT NULL,
  first_bar             INTEGER,                      -- first equity mark
  last_bar              INTEGER,                      -- last equity mark
  final_open_equity     REAL,
  final_closed_equity   REAL,
  header_json           TEXT,                         -- report.ResultHeader
  metrics_json          TEXT,                         -- report.Metrics
  director_config_json  TEXT,                         -- the run's director settings
  director_summary_json TEXT,                         -- report.DirectorSummary
  degradations_json     TEXT                          -- the run's degradation records
);

CREATE TABLE series (
  series_id        INTEGER NOT NULL PRIMARY KEY,
  label            TEXT    NOT NULL,                  -- fut:XCME:ES@1d-eth
  symbol           TEXT    NOT NULL,
  interval         TEXT,                              -- NULL for activity bar kinds
  bar_kind         TEXT    NOT NULL,                  -- time, volume, tick, ...
  session          TEXT,                              -- 1d and 1w trading-day container
  boundary         TEXT,                              -- intraday grid boundary
  clock            TEXT,                              -- continuous: session | epoch
  continuous       INTEGER NOT NULL,
  method           TEXT,                              -- continuous adjustment method
  held_rank        TEXT,                              -- continuous: F, F+1, ...
  snapshot_json    TEXT,                              -- continuous: {"at","anchor","token"}
  adjusted         INTEGER NOT NULL,                  -- bars are an adjusted price view
  trades_adjusted  INTEGER NOT NULL,                  -- trade prices are in the bars' view
  tz               TEXT,                              -- IANA zone of the venue
  tick_size        REAL,
  point_value      REAL,
  currency         TEXT,
  contracts_json   TEXT    NOT NULL,                  -- the contract dictionary; [""] at least
  first_ts         INTEGER,                           -- first bar's t; NULL when no bars
  last_ts          INTEGER,                           -- last bar's t
  bar_count        INTEGER NOT NULL,
  warmup_count     INTEGER NOT NULL,
  bars_source      TEXT    NOT NULL,                  -- run | refetched
  chart_query_json TEXT                               -- reserved for the feed's chart query
);

CREATE TABLE run_series (
  run_id       TEXT    NOT NULL REFERENCES runs (run_id),
  series_id    INTEGER NOT NULL REFERENCES series (series_id),
  ordinal      INTEGER NOT NULL,                      -- position in the run's series
  role         TEXT    NOT NULL,                      -- traded | reference
  declared     TEXT    NOT NULL,                      -- config | strategy
  name         TEXT,                                  -- the strategy's name for a declared series
  warmup_start INTEGER,                               -- t of the run's first bar; NULL when none
  PRIMARY KEY (run_id, series_id)
);

CREATE TABLE bar_chunks (
  series_id INTEGER NOT NULL REFERENCES series (series_id),
  chunk     INTEGER NOT NULL,
  t0        INTEGER NOT NULL,                         -- first row's t
  t1        INTEGER NOT NULL,                         -- last row's t
  n         INTEGER NOT NULL,                         -- rows, 1 to 8192
  codec     TEXT    NOT NULL,                         -- deflate
  data      BLOB    NOT NULL,
  PRIMARY KEY (series_id, chunk)
);
CREATE INDEX bar_chunks_window ON bar_chunks (series_id, t1);

CREATE TABLE equity_chunks (
  run_id TEXT    NOT NULL REFERENCES runs (run_id),
  chunk  INTEGER NOT NULL,
  t0     INTEGER NOT NULL,
  t1     INTEGER NOT NULL,
  n      INTEGER NOT NULL,
  codec  TEXT    NOT NULL,
  data   BLOB    NOT NULL,
  PRIMARY KEY (run_id, chunk)
);
CREATE INDEX equity_chunks_window ON equity_chunks (run_id, t1);

CREATE TABLE equity_daily (
  run_id    TEXT    NOT NULL REFERENCES runs (run_id),
  date      TEXT    NOT NULL,                         -- YYYY-MM-DD, UTC
  t         INTEGER NOT NULL,                         -- the day's last mark
  open      REAL    NOT NULL,
  closed    REAL    NOT NULL,
  in_market INTEGER NOT NULL,
  benchmark REAL,                                     -- NULL without a benchmark
  PRIMARY KEY (run_id, date)
);

CREATE TABLE fills (
  run_id           TEXT    NOT NULL REFERENCES runs (run_id),
  seq              INTEGER NOT NULL,                  -- blotter order
  ts               INTEGER NOT NULL,                  -- the fill's stamp
  kind             TEXT    NOT NULL,                  -- order | roll | expiry | corporate_action
  symbol           TEXT    NOT NULL,                  -- the symbol the strategy traded
  contract         TEXT    NOT NULL,                  -- the instrument the venue filled
  series_id        INTEGER REFERENCES series (series_id),
  bar_ts           INTEGER,                           -- t of the bar the fill is anchored to
  client_id        TEXT,
  attempt          INTEGER,
  name             TEXT,                              -- the order's name
  parent_client_id TEXT,
  broker_order_id  TEXT,
  exec_id          TEXT,
  order_ref        TEXT,
  order_type       TEXT,
  side             TEXT    NOT NULL,                  -- buy | sell
  qty              INTEGER NOT NULL,
  price            REAL    NOT NULL,                  -- in the bars' price view
  raw_price        REAL    NOT NULL,                  -- the venue's price
  commission       REAL    NOT NULL,
  slippage         REAL,                              -- points per unit
  combo_leg        INTEGER,
  PRIMARY KEY (run_id, seq)
);
CREATE INDEX fills_order ON fills (run_id, client_id, attempt);

CREATE TABLE trades (
  run_id          TEXT    NOT NULL REFERENCES runs (run_id),
  n               INTEGER NOT NULL,                   -- result.json order
  key             TEXT    NOT NULL,
  symbol          TEXT    NOT NULL,
  series_id       INTEGER REFERENCES series (series_id),
  dir             INTEGER NOT NULL,                   -- 1 long, -1 short
  qty             INTEGER NOT NULL,
  open            INTEGER NOT NULL,
  entry_ts        INTEGER NOT NULL,
  entry_price     REAL    NOT NULL,
  entry_bar       INTEGER NOT NULL,                   -- 1-based bar ordinal of the key
  entry_bar_ts    INTEGER,
  entry_fill      INTEGER,                            -- fills.seq
  entry_client_id TEXT,
  entry_attempt   INTEGER,
  entry_name      TEXT,
  entry_type      TEXT,
  exit_ts         INTEGER,                            -- NULL while open
  exit_price      REAL    NOT NULL,                   -- the last close while open
  exit_bar        INTEGER NOT NULL,
  exit_bar_ts     INTEGER,
  exit_fill       INTEGER,
  exit_client_id  TEXT,
  exit_attempt    INTEGER,
  exit_name       TEXT,
  exit_type       TEXT,
  exit_kind       TEXT,                               -- order | roll | expiry | corporate_action
  bars_held       INTEGER NOT NULL,
  mfe             REAL    NOT NULL,
  mae             REAL    NOT NULL,
  profit_points   REAL    NOT NULL,
  profit_currency REAL    NOT NULL,
  point_value     REAL    NOT NULL,
  commission      REAL    NOT NULL,
  slippage        REAL    NOT NULL,
  rolls           INTEGER NOT NULL,
  currency        TEXT,
  fx_rate         REAL,
  profit_local    REAL,
  PRIMARY KEY (run_id, n),
  FOREIGN KEY (run_id, entry_fill) REFERENCES fills (run_id, seq),
  FOREIGN KEY (run_id, exit_fill) REFERENCES fills (run_id, seq)
);
CREATE INDEX trades_entry ON trades (run_id, entry_ts);
CREATE INDEX trades_bar ON trades (series_id, entry_bar_ts);

CREATE TABLE combo_trades (
  run_id          TEXT    NOT NULL REFERENCES runs (run_id),
  n               INTEGER NOT NULL,
  client_id       TEXT    NOT NULL,
  combo           TEXT    NOT NULL,                   -- B1@fut:XCME:ES:M26|S1@fut:XCME:ES:H26
  units           INTEGER NOT NULL,
  open            INTEGER NOT NULL,
  entry_ts        INTEGER NOT NULL,
  exit_ts         INTEGER,
  entry_cost      REAL    NOT NULL,
  exit_cost       REAL    NOT NULL,
  profit_currency REAL    NOT NULL,
  commission      REAL    NOT NULL,
  PRIMARY KEY (run_id, n)
);

CREATE TABLE director (
  run_id           TEXT    NOT NULL REFERENCES runs (run_id),
  sequence         INTEGER NOT NULL,
  kind             TEXT    NOT NULL,
  event_time       INTEGER,
  instant_sequence INTEGER,
  phase            TEXT,
  stage            TEXT,
  client_id        TEXT,
  attempt          INTEGER,
  symbol           TEXT,
  side             TEXT,
  requested        INTEGER,
  accepted         INTEGER,
  reason_code      TEXT,
  cause_code       TEXT,
  rule             INTEGER,
  record_json      TEXT    NOT NULL,                  -- the log line exactly as written
  PRIMARY KEY (run_id, sequence)
);
CREATE INDEX director_time ON director (run_id, event_time);
CREATE INDEX director_order ON director (run_id, client_id, attempt);

CREATE TABLE rolls (
  series_id      INTEGER NOT NULL REFERENCES series (series_id),
  cut_ts         INTEGER NOT NULL,
  from_contract  TEXT    NOT NULL,
  to_contract    TEXT    NOT NULL,
  gap            REAL    NOT NULL,
  ratio          REAL    NOT NULL,
  cumulative     REAL    NOT NULL,
  cum_ratio      REAL    NOT NULL,
  outgoing_price REAL    NOT NULL,
  incoming_price REAL    NOT NULL,
  degraded       INTEGER NOT NULL,
  PRIMARY KEY (series_id, cut_ts)
);

CREATE TABLE instruments (
  run_id      TEXT NOT NULL REFERENCES runs (run_id),
  symbol      TEXT NOT NULL,
  point_value REAL,
  tick_size   REAL,
  currency    TEXT,
  margin_json TEXT,                                   -- report.MarginSpec
  PRIMARY KEY (run_id, symbol)
);

CREATE TABLE calendars (
  run_id    TEXT NOT NULL REFERENCES runs (run_id),
  root      TEXT NOT NULL,
  revision  TEXT,                                     -- NULL when none was served
  info_json TEXT NOT NULL,                            -- report.CalendarInfo
  PRIMARY KEY (run_id, root)
);

CREATE TABLE benchmarks (
  run_id      TEXT    NOT NULL REFERENCES runs (run_id),
  ordinal     INTEGER NOT NULL,                       -- report order
  name        TEXT    NOT NULL,
  is_primary  INTEGER NOT NULL,
  spec_json   TEXT    NOT NULL,                       -- report.BenchmarkSpec
  result_json TEXT,                                   -- report.Benchmark; NULL when not computable
  PRIMARY KEY (run_id, ordinal)
);

CREATE TABLE portfolio (
  runs_json           TEXT    NOT NULL,               -- the run ids combined
  capital             REAL    NOT NULL,
  first_bar           INTEGER,
  last_bar            INTEGER,
  final_open_equity   REAL    NOT NULL,
  final_closed_equity REAL    NOT NULL,
  avg_hold_seconds    REAL    NOT NULL,
  metrics_json        TEXT    NOT NULL,               -- report.Metrics
  contribution_json   TEXT    NOT NULL,               -- []report.Contribution
  correlation_json    TEXT    NOT NULL                -- report.CorrMatrix
);

meta

One row per key; every value is text.

runs

One row per run, in the order the config expanded. Its key is run_id, the run’s results-folder slug.

• status is ok or failed; error holds a failed run’s message (see Failed runs).
• name, strategy and strategy_name are the display name, the binary and the name the strategy reports. strategy_version is the version the strategy reports, when it reports one; none does today.
• inputs_json is every input as the run received it, an object of strings, as ResultHeader.Inputs.
• capital is the run’s allocated capital and capital_units its share of the pool.
• first_bar and last_bar are the curve’s first and last marks. final_open_equity and final_closed_equity are its last values.
• header_json and metrics_json are result.json’s header and metrics.
• director_config_json is the run’s director settings as configured. director_summary_json is ResultHeader.DirectorSummary.
• degradations_json is the run’s degradation.json.

series

One row per distinct series, as the warm-up section defines distinct.

• Naming. label is the engine’s series label: SYMBOL@INTERVAL[-SESSION], or the construction key for activity bars, such as MSFT@volume:t=1000.
• Bar construction. interval is NULL for activity bar kinds. bar_kind is the kind’s short name. session is a 1d or 1w trading-day container (eth, rth, session). boundary is an intraday grid’s boundary (utc-day, eth, rth, session). A series has at most one of the two. clock is a continuous run’s session or epoch clock.
• Continuous futures. continuous, method, held_rank and snapshot_json describe a continuous future.
• Prices. adjusted and trades_adjusted are described under Adjusted and raw prices.
• The instrument. tz is the venue’s IANA zone. tick_size, point_value and currency describe the instrument, when it has them.
• The contract dictionary. contracts_json is the dictionary the chunks’ contract column indexes. Entry 0 is always "", “no dated contract”, so a plain series’ dictionary is [""].
• Counts. first_ts and last_ts are the first and last bars’ t. bar_count counts every bar and warmup_count the warm-up ones.
• Provenance. bars_source is run or refetched.
• Reserved. chart_query_json is the feed’s chart query for the series (MF-1, GLE-670), NULL until then.

run_series

Which runs read which series. The key is (run_id, series_id).

• ordinal is the series’ position in the run’s series list.
• role is traded, a series the run’s orders could trade, or reference, data only.
• declared is config, a series the config named, or strategy, one the strategy declared. name is the strategy’s name for a series it declared.
• warmup_start is the t of the first bar the run received. It is NULL when the run received none.

bar_chunks and equity_chunks

A series’ bars and a run’s equity marks, in chunks of 8,192 rows: see Chunks. t0 and t1 are the chunk’s first and last t, and n its rows. The index on t1 serves a window query.

equity_daily

A run’s curve at the last mark of each UTC date, equity.csv’s daily sampling:

• t is the mark’s instant;
• open, closed and in_market are the mark’s values;
• benchmark is the primary benchmark’s equity, NULL when the run has none.

It is enough to recombine runs at daily resolution without decoding a chunk.

fills

One row per fill, in blotter order.

• What filled. symbol is the symbol the strategy traded, the logical one on a continuous run. contract is the instrument the venue filled, the dated contract.

• What kind of fill. kind is one of:

  • order: an order the strategy placed;
  • roll: a roll leg, whose client id is roll/<symbol>/<cut>/close or /open;
  • expiry: a contract’s expiry settlement;
  • corporate_action: a corporate action’s leg.

  Expiry and corporate-action fills have no client id. A converter that cannot tell them apart by other means takes expiry for a future and corporate_action for an equity.

• Where it sits. series_id and bar_ts are the fill’s anchor.

• The order. client_id and attempt join a fill to its trades and to the director’s records. name is the order’s name (GLE-666) and order_type its type. parent_client_id, broker_order_id, exec_id and order_ref are the venue’s identities.

• Prices and costs.

  • price is in the bars’ price view and raw_price is the venue’s.
  • commission is the fill’s, as the blotter records it.
  • slippage is in points per unit. It is NULL when the source does not record it.
  • combo_leg is a combo leg’s number (GLE-409).

trades

One row per trade, result.json’s Trades in order: closed trades as they closed, then open ones. Every field of report.Trade has its column.

• Prices. entry_price and exit_price are the report’s (see Adjusted and raw prices). An open trade’s exit_price is its symbol’s last close.
• Anchors. entry_bar and exit_bar are the report’s 1-based bar ordinals of the key. entry_bar_ts and exit_bar_ts are bar anchors on series_id, the series the trade’s fills anchor to.
• Orders. entry_fill and exit_fill are the fills’ seq. entry_client_id, entry_attempt, entry_name and entry_type describe the entry’s order, and the exit_ columns the exit’s.
• How it closed. exit_kind is the exit fill’s kind.
• Open trades. An open trade has open = 1, and no exit_ts, exit_fill, exit_bar_ts or exit_kind.
• Other currencies. currency, fx_rate and profit_local are set when the trade’s symbol is quoted in another currency.

Its indexes serve a run’s trades in time order, and a series’ trades by bar.

combo_trades

One row per combo trade (report.ComboTrade, GLE-409). The combo’s legs are also in trades. combo names the legs per unit, for example B1@fut:XCME:ES|S1@fut:XCME:NQ.

director

One row per record of a run’s trade director log, director-decisions.jsonl (The decision log), before its footer.

• The record itself. record_json is the line exactly as written, without its newline.
• Its fields. The other columns are copies of its fields, for querying:
  • event_time is converted to nanoseconds;
  • requested and accepted come from requested_quantity and accepted_quantity;
  • a field the record does not carry is NULL.
• The footer. It is not a row. Its record count and SHA-256 are in runs.director_summary_json, as Records and LogSHA256.
• Checking it. A reader checks the log by hashing each record_json plus a newline, in sequence order. The digest must equal LogSHA256 and the row count Records.

rolls

One row per roll of a continuous series, from its manifest’s forward adjustments:

• cut_ts is the roll’s effective instant;
• outgoing_price and incoming_price are the two contracts’ raw prices at it;
• gap (panama) and ratio (ratio methods, 0 otherwise) are the roll’s adjustment;
• cumulative and cum_ratio are the adjustment accumulated through it;
• degraded marks a seam built from a fallback price.

instruments

Each run’s instruments:

• point_value, tick_size and currency;
• margin_json, the run’s report.MarginSpec for the symbol, when it held a margin requirement.

calendars

Each run’s exchange-calendar provenance, one row per root, when the run asked for calendars:

• info_json is report.CalendarInfo;
• revision is its served revision, NULL when none was served.

benchmarks

Each run’s benchmarks, in report order:

• spec_json is the configured report.BenchmarkSpec;
• result_json is the computed report.Benchmark, NULL when it was not computable;
• exactly one benchmark of a run has is_primary = 1. Its curve is the benchmark column of the run’s equity.

portfolio

At most one row: the combination of the set’s ok runs as Go computed it, which is portfolio/result.json. It is present when the set has two or more ok runs.

• runs_json lists the runs combined.
• metrics_json, contribution_json and correlation_json are its metrics, contribution table and correlation matrix.

A reader shows it at once and checks its own recombination against it.

Chunks

A series’ bars and a run’s equity marks are stored in chunks:

• Rows. A chunk holds 8,192 rows, except the last chunk of its series or run, which holds the remaining 1 to 8,192. Chunks count from 0, so row i of a series is in chunk i / 8192.
• Layout. A chunk’s data holds its columns one after another, each a run of n little-endian values. There is no header: n and codec are the row’s own columns.
• Time columns. These are delta-encoded. The first value is the instant itself, and each later value is the difference from the value before it, in int64 arithmetic that wraps.
• Codec. The concatenated columns are compressed as one stream.

Bar chunk columns, 59 n bytes before compression:

Equity chunk columns, 33 n bytes before compression:

Marks strictly increase in t. The curve is the recorder’s at every bar, the curve the metrics are computed from, whatever the set’s equity_sampling.

Bar flags:

Bits 64 and 128 are free for later flags, which a reader ignores.

Codec. codec = 'deflate' is raw DEFLATE (RFC 1951) with no zlib or gzip framing:

Neither side needs a new dependency. lz4, the LZ4 block format (Apple’s COMPRESSION_LZ4_RAW), is reserved for a later version; version 1 writes deflate only.

A window of bars is one indexed query and a few decompressions:

SELECT chunk, n, codec, data FROM bar_chunks
 WHERE series_id = :series AND t1 >= :from AND t0 <= :to
 ORDER BY chunk;

Decoding a chunk needs nothing beyond a language’s standard library. In Python:

import sqlite3, struct, sys, zlib

db = sqlite3.connect(f"file:{sys.argv[1]}?mode=ro", uri=True)
codec, n, data = db.execute(
    "SELECT codec, n, data FROM bar_chunks WHERE series_id = 1 AND chunk = 0").fetchone()
assert codec == "deflate"
raw = zlib.decompress(data, -15)              # raw DEFLATE
assert len(raw) == 59 * n

def times(column):                            # undo the delta encoding
    t, out = 0, []
    for delta in struct.unpack_from(f"<{n}q", raw, column * 8 * n):
        t += delta
        out.append(t)
    return out

t, tc = times(0), times(1)
o, h, l, c, v = (struct.unpack_from(f"<{n}d", raw, (2 + k) * 8 * n) for k in range(5))
contract = struct.unpack_from(f"<{n}H", raw, 56 * n)
flags = raw[58 * n:]
print(n, t[0], tc[0], c[0], contract[0], flags[0])

Size and speed

These figures were measured on the year of smacross 20/60 on ES 1m in the design (2025-10-01 to 2026-09-30, the installed marketfeed adapter):

The results folder of the same run is 57 MB, report.html 31 MB of it.

• Page size. The page size is the writer’s choice and not part of the format. A large export reads its BLOBs faster with 64 KiB pages. A small one is smaller with SQLite’s default, because every table and index takes at least one page: the reference file uses 4 KiB.
• Vacuum. A writer VACUUMs before the rename.

Checking a file

algorun.Check(path) validates a file against this chapter. It returns every violation it finds, and it refuses outright:

• another application (ErrNotExport);
• an unknown version or format (ErrVersion);
• an unfinished file (ErrIncomplete).

The schema. Every specified table, column, index and foreign key is present, and each column declares the specified type, NOT NULL and key. More is allowed.

Values:

• every value’s storage class matches its column;
• JSON columns are valid JSON, and the ones that hold algolang types decode into them;
• enumerated columns hold their values;
• booleans are 0 or 1;
• every foreign key resolves (PRAGMA foreign_key_check).

Chunks:

• each chunk decodes to exactly n rows;
• chunks count from 0 without gaps;
• only the last chunk is short;
• t0 and t1 match the rows;
• bar times never decrease and marks strictly increase;
• every contract index is inside its dictionary;
• warm-up bars are a prefix, and the series’ counts and bounds match.

Results:

• equity_daily is the last mark of each date;
• a run’s bounds and final equity match its curve;
• failed runs have no results;
• ordinals count from 1 without gaps;
• open trades have no exit;
• every bar anchor is a bar of its series;
• every director row is its record under algolang.director/1, and the run’s summary digests exactly its rows;
• a run has one primary benchmark;
• the portfolio combines ok runs.

Writers check what they write before the rename, and the tests check the reference file the same way.

The reference file

testdata/algorun/reference.algorun is a small set made by hand to this chapter. kchart vendors it to test its reader. Its bars and fills are made by hand. Its trades, curves, metrics, benchmarks and portfolio are what engine/report computes from them, so every figure agrees with the rest. It holds three runs from one config (meta.config_json, which runconfig expands to these runs, names and capital), a portfolio, and four series.

Run 1, SMA ES Fast=5 Slow=20:

• continuous ES 1m across the March 2026 roll: 9,680 bars in two chunks, the first 20 warm-up and the last 60 degraded;
• six market orders and the roll’s two legs;
• five trades, two of them carried through the roll and one still open;
• per-bar equity in two chunks, and a primary benchmark.

Run 2, SMA ES Fast=20 Slow=20: its sweep sibling, which the strategy refused, so it is failed.

Run 3, ES/NQ spread, with a trade director:

• continuous ES and NQ 1d, and the weekly COT series cot:CFTC:13874A:LF@1w, which the strategy declares for reference;
• a combo entry and exit;
• a trimmed and a refused order in its director’s log;
• both legs rolled.

The portfolio is runs 1 and 3 combined.

The generator is engine/algorun/reference_test.go. To rewrite the file after changing it:

go test ./engine/algorun -run TestReference -update

make test fails while the committed file differs from what the generator writes.