Foundation models

A foundation model (FM) run generates limit order book activity by continuing from real market data rather than simulating agents. You give it a market and a duration; the platform slices the model's required context out of that market's data, runs inference, and writes the output using the same conventions as an agent-based run — so the existing results and download endpoints work unchanged.

Use it when you want generated activity that continues a specific real day, rather than a stress scenario on a calibrated one. Model discovery lives on client.fm; the run itself is the ordinary client.simulation.run() with engine="fm".

Run flow

A run is submitted, polled, and then read back through the standard simulation endpoints:

Models and their names

Each model has two names. The model_id identifies a specific build, and it is the id that sim_ids are minted from. The production_name is the stable product name, built as {model}-{market}-{version}-{size}: for example flow-hkex-1-100M or flow-lse-1-100M, where 100M is the parameter count. Anywhere a model_id is accepted (run() and available_data()) you can pass either name. The match ignores case, spaces, underscores and dashes, so "Flow HKEX 1 100M" finds the same model.

Every model declares what it accepts: the markets it can be prompted from (supported_data, empty meaning any), the context it needs to start (min_prompt_rows), and its knobs (model_args, each with type, default and range). The knobs differ by model family — a diffusion model exposes guidance and num_steps where an autoregressive one exposes temperature — and an undeclared or out-of-range key is rejected rather than ignored. Read ids and knobs off fm.models() rather than hard-coding either.

models = client.fm.models()["models"]

for m in models:
    markets = {(d["provider"], d["exchange"]) for d in m["supported_data"]}
    print(f"{m['model_id']} v{m['version']} ({m.get('production_name') or '-'})")
    print(f"  markets: {markets or 'any'}")
    print(f"  prompt rows: {m['min_prompt_rows']}")
    print(f"  args: {sorted(m['model_args'])}")

flob-adaln-rf-1s v1.0.14 (flow-hkex-1-100M)
  markets: {('bmll', 'hkex_securities')}
  prompt rows: 2048
  args: ['guidance', 'historical_conditions', 'impact_coefficient', 'impact_model', 'impact_seconds', 'method', 'num_steps']

Data to prompt with

A foundation model is prompted with real market data rather than calibrated parameters, so data.available_data(model_id=…) reads the raw vendor registry, not the calibrated catalog that calibrated_data() serves. The registry holds tens of thousands of symbol-days and is never returned whole: page with limit and offset and read total.

Submitting and polling a run

You name a market, not a prompt — the platform slices min_prompt_rows of real data for that symbol and date itself. Ask for an amount of market time with duration_minutes (preferred: an illiquid symbol simply produces fewer frames) or a raw frame count with horizon. n_runs defaults to 1 and is capped at 8, because an FM job's worst case must fit on one GPU; per-run seeds derive from seed exactly as on the ABM. An FM job takes at most one execution algorithm, and it needs a real market day.

Market scenarios are deliberately not implemented for foundation models: scenario, scenario_params or any other agent-based field with engine="fm" is a 422. For stress testing, use an agent-based run.

Polling is the ordinary get_job_status(), with the GPU scheduler's own status, message and detail merged in on top:

Poll on is_complete and read has_errors afterwards rather than matching status strings: the scheduler's status can take new values, while those two flags always mean the same thing.

import time

# Models differ by environment, so take one this key can run rather than
# naming it. Prompt it with the busiest instrument in its market, on that
# instrument's latest calibrated day: a quiet name can have too few rows to start.
model = client.fm.models()["models"][0]
market = (model["supported_data"] or [{}])[0]
inst = max(
    client.data.calibrated_data(provider=market.get("provider"), exchange=market.get("exchange"))["symbols"],
    key=lambda s: s["avg_number_of_msgs"],
)
day = {"symbol": inst["symbol"], "provider": inst["provider"], "exchange": inst["exchange"],
       "dates": [d["date"] for d in inst["available_dates"]]}

job_id = client.simulation.run(
    engine="fm",
    model_id=model["model_id"],  # a production_name works here too
    symbol=day["symbol"],
    cal_date=day["dates"][-1],
    provider=day["provider"],
    exchange=day["exchange"],
    duration_minutes=5,
    n_runs=1,
)["job_id"]

while True:
    status = client.simulation.get_job_status(job_id)
    # status / message / detail are absent until the scheduler reports
    print(f"{status.get('status', 'queued')}: {status.get('message', '')}")
    if status.get("detail"):
        print(f"  {status['detail']}")
    if status["is_complete"]:
        break
    time.sleep(30)

if status["has_errors"]:
    print(client.simulation.get_job_logs(job_id))

fm.live() takes the same model and generation fields but streams frames to a live chart instead of writing output files. Four more methods (fm.registry, fm.register, fm.activate, fm.deactivate) manage the model registry; they need an admin key and are for operators who publish models.

Compute and pricing

Foundation models need a Pro tier key (pro or demo). Every such account can run on CPU and can use every publicly available model. Specific models and GPU types are enabled per account by Simudyne: to use a model that is not public, or to run on a GPU, contact us.

Prices are approximate, billed per hour of compute, and subject to change. GET /fm/compute-profiles returns the current price of each profile your account can use.

Choosing where a run executes

Pass device to POST /simulation/run (engine="fm") with a profile name: cpu, t4, l4, a100, or gpu for the default GPU. Leave it blank to use the model's own default. Two calls tell you what your account can use:

profiles = client.session.get(f"{client.base_url}/fm/compute-profiles").json()
for p in profiles["profiles"]:
    print(p["name"], p["label"], p["price_per_hour"])

model_id = client.fm.models()["models"][0]["model_id"]
job = client.simulation.run(
    engine="fm",
    model_id=model_id,
    duration_minutes=5,
    device="t4",  # cpu, t4, l4, a100, or gpu for the default GPU
)

{
  "profiles": [
    {"name": "cpu", "gpu": 0, "label": "CPU", "price_per_hour": 0.10},
    {"name": "t4", "gpu": 1, "label": "NVIDIA T4 (16 GB)", "price_per_hour": 0.75}
  ],
  "default_gpu": "t4"
}

A request for a model or GPU type that is not enabled for your account is refused with 403, and the detail names what was refused — for example GPU profile 'a100' is not enabled for your account. The same applies to fm.live() and to data.available_data(model_id=…).

Prompt context in the output

The output file contains the model's prompt context as well as its generated frames. The leading rows of sim_data.parquet are the real market data the model was conditioned on, so the series runs continuously from observed history into generated activity with no gap at the handover.

Where the split is recorded

The row counts live in the parquet file-level metadata, so no extra column or row is added to the data and readers that do not care about the split are unaffected.

Only the row split is recorded, because it is the only thing the data cannot tell you itself. The model is warm-started on the opening rows of each trading session, so a session's start time is simply its first row's timestamp, and its handover is the timestamp at row n_historical.

One sim_id is one whole trading day. A day that spans a break — HKEX's lunch hour, say — is a single file whose sessions are stitched in time order with no rows between them, so the break reads as an absence of data rather than a flat line. Each session keeps its own warm-start block, which is why pulse_fm.segments exists: n_historical and n_generated describe the first session only, and segments is the authority for the rest. When a job returns several sim_ids, they are alternative starting points for the same day, not different parts of it.

Both keys are absent on agent-based simulation output, where every row is generated, and on foundation-model runs produced before the keys existed. Treat a missing key as zero historical rows.

Example: keep only the generated frames

get_sim_data() hands back a DataFrame, which drops the file-level metadata — so read the raw bytes when you need the split.

import io
import time

import polars as pl
import pyarrow.parquet as pq

model_id = client.fm.models()["models"][0]["model_id"]  # models differ by environment
job = client.simulation.run(engine="fm", model_id=model_id, duration_minutes=5)
while not client.simulation.get_job_status(job["job_id"])["is_complete"]:
    time.sleep(30)
sim_id = job["sim_ids"][0]

url = f"{client.base_url}/simulation/results/{sim_id}/data/sim_data.parquet"
raw = client.session.get(url).content

meta = pq.read_metadata(io.BytesIO(raw)).metadata or {}
n_hist = int(meta.get(b"pulse_fm.n_historical", 0))

df = pl.read_parquet(io.BytesIO(raw))

context = df.head(n_hist)    # real market data — the prompt
generated = df.slice(n_hist)  # model output

print(f"{len(context)} context rows, {len(generated)} generated rows")

Validation

The validation API excludes the prompt context automatically — it reads pulse_fm.n_historical and drops those rows before any metric runs, so reported distances describe the generated frames only. The number excluded per simulation is echoed back in the job's metadata.historical_rows_dropped, and metadata.sim_rows is the post-drop count.

A foundation-model run usually covers only part of a trading day, but validation compares against the whole historical day unless you ask otherwise. Pass match_window=True to cut the historical day to the span the runs cover, so a 30-minute run is compared with the same 30 minutes of real trading. The window used is reported in metadata.historical_window. See Validation API and validation.run.

Recipes

• Foundation-model rollout, then score it
• Validate against a stretch of the day (match_window)