Pulse-Check

Pulse-Check scores how closely simulated order book data matches the real trading day for the same symbol and date: distributional distances (L1 and Wasserstein), the Cont stylised facts, impact response, cross-level volume correlation, and the MIND and FID inception distances on DeepLOB embeddings. Use it to decide whether a simulation, a calibration or your own generative model is realistic enough to trust. It is reached through client.validation.

How a job runs

1. run() submits the job and returns a job_id immediately.
2. The historical day is fetched for you; the simulated side comes from your sim_ids, your uploaded files, or both (25 runs in total).
3. The job moves pending → running → completed or failed. Poll the light get_job_status() until is_complete, then call get_job() once for the result.

Missing historical data or an unknown sim_id does not raise at submission: the job ends failed with the reason in error. One area failing does not fail the job; it is named in errors, so check that before reading a missing payload as "nothing computed". metadata reports the resolved parameters, which areas ran, which were skipped and why (areas_skipped), and the tick size used (ticksize, ticksize_source). Tick size is always read from the historical day; a source of "fallback" means none was found and 1.0 was assumed.

What you get back

A pro result never contains the real session. Aggregate scalars (the distances, per-fact verdicts and inception scores) are returned in full, and the impact response is returned for your simulated runs. Anything derived from the historical series itself is withheld: the distribution histograms, the full stylised-fact series, and the historical half of the impact response and volume correlation.

What a job computes

Six areas, one flag each on run(). Every area defaults on for every tier, so a job with no flags returns everything below. Pass False to skip an expensive area.

Time window

By default every run is scored against the whole historical day. When your runs cover only part of it, that compares a half hour of simulation with a full session of open, lunch and close. match_window on run() chooses which stretch is compared:

True is the right choice for a run shorter than the day, such as a 30-minute foundation-model rollout. The span is matched on time of day, because generated frames need not carry the date.

An interval cuts both sides, with both ends inclusive. Each end is "HH:MM", "HH:MM:SS", "HH:MM:SS.fff" or the time part of an ISO timestamp, with no timezone. A malformed interval is rejected at submission with a 422; an interval that holds no historical rows, or none of a run's rows, fails the job with the reason in error.

result["metadata"]["historical_window"] reports what was applied: mode (off, simulation or interval), the start and end used, and the row counts before and after the cut. See validation.run for the parameter and the time window recipe for both modes side by side.

Populations

Pass sim_ids (or sim_files) as a mapping, {"fm": [...], "abm": [...]}, rather than a list and the job compares named populations instead of pooling every run. Results come back keyed by name, and the summary figures draw one series per population. The historical day is measured once however many there are. sim_ids and sim_files can be given together, and each source then becomes a population of its own:

LOB mode and the book grid

With lob=True the frames are treated as L2 snapshots only. Anything needing the message stream (impact and the order-placement and cancellation metrics) is skipped rather than failing, with the reason in metadata["areas_skipped"]. Frames that carry no message stream, such as foundation-model output, run in LOB mode automatically.

In LOB mode the historical day is put on the generated frames' grid before it is measured, and resample_method chooses how:

pulseflow needs the model's sessions, each [start_h, start_m, end_h, end_m] with the end exclusive, and is rejected without them. The day is resampled over its whole session before match_window trims anything.

Distributional metrics

With statistical on, Pulse-Check builds the distribution of each metric on the historical and simulated books and scores the gap with L1 and Wasserstein distances. Lower is better. The approach follows LOB-Bench (Nagy et al.). distances holds one value per run, in sim_ids order, null where a metric failed for that run.

Stylised facts

Pulse-Check runs the eleven Cont stylised facts on the historical day and on every run. A real session is expected to exhibit them; the question is whether your simulation does too.

{
  "heavy_tails": {
    "historical": true,
    "simulated": [true, true, false, true, true],
    "aggregate": {"n": 5, "passes": 4, "verdict": true, "significant": true},
    "policy": {
      "frequencies": ["50s", "100s", "150s", "200s", "..."],
      "judged": ["50s", "100s", "150s"],
      "mode": "any"
    }
  }
}

historical is the verdict on the real session and simulated one verdict per run; null means the fact could not be evaluated (usually too short a frame). aggregate folds the runs into one Monte Carlo signal with a one-sided binomial test at the 5% level, and significant says whether there were enough runs to call it. policy reports the sampling frequencies the fact is scored over, which the data supported (judged), and how they combine (any, all or slope).

Impact response

The Bouchaud response function for each simulated run: how far the mid-price moves, in ticks, in the events after a given kind of book event (Eisler, Bouchaud and Kockelkoren, 2012). Events that change a best price are split into market orders, limit orders and cancellations, each by whether the event itself moved the mid (_1) or not (_0): MO_0, MO_1, LO_0, LO_1, CA_0, CA_1. R(l) is the average signed mid move from just before the event to l events later, over a log ladder of lags from 1 to about 200. The result holds lags, events, and per run a {ys, ci_low, ci_high} block indexed [event][lag] with a bootstrapped 99% interval.

Inception distances (MIND and FID)

Both compare 96-dimensional DeepLOB embeddings of 100-row L2 windows. FID is the Frechet distance between the two embedding clouds; MIND is a sliced-Wasserstein distance over the same embeddings, sensitive to differences beyond mean and covariance. Lower is closer. They share one embedding pass and need the full 10 book levels; with fewer, the pass is skipped and named in areas_skipped.

Plots

plots=True renders every figure the enabled areas can draw; a list renders just those ids. get_job(job_id, plot_dir=...) writes them to disk. Match returned figures on id, not filename: statistical.radar writes two files under one id. A pro figure never draws the historical series.

Examples

Validate three runs of a cached baseline:

cached = client.simulation.list_cached(symbol="700.HK", scenario="normal")["simulations"][0]
base = cached["example_sim_id"].rsplit(":", 1)[0]
sim_ids = [f"{base}:{i:04d}" for i in range(min(cached["n_runs"], 3))]

job = client.validation.run(
    symbol=cached["symbol"],
    date=cached["date"],
    provider="omd",
    exchange="hkex_securities",
    sim_ids=sim_ids,
)
print(job["job_id"])

Wait for a job and read its result:

import time

job_id = client.validation.list_jobs(limit=1)["jobs"][0]["job_id"]

while not client.validation.get_job_status(job_id)["is_complete"]:
    time.sleep(5)

result = client.validation.get_job(job_id)
print(f"Status: {result['status']}")
print(f"Window: {(result.get('metadata') or {}).get('historical_window')}")
if result["status"] == "failed":
    print(f"Error: {result['error']}")
elif result.get("errors"):
    print(f"Areas that failed: {result['errors']}")

List past jobs; symbol and date sit under params:

jobs = client.validation.list_jobs(limit=10)

print(f"You have {jobs['total']} validation jobs")
for job in jobs["jobs"]:
    params = job["params"]
    print(job["job_id"], job["status"], params["symbol"], params["date"])

Recipes

• Validate simulation runs
• Score part of the day (time window)
• Compare named populations
• Validate your own parquet files
• Validate a foundation-model rollout

Over plain HTTP, send your key in the X-API-Key header; the routes and the config body are in the API reference.