Skip to content
BranchesTags

Repository files navigation

Sharded Tickerplant Blueprint

A KDB+/q reference implementation demonstrating scalable market data ingestion patterns. Generates synthetic NYSE TAQ-compatible market data at configurable throughput to show where single-TP architectures hit their ceiling — and how sharding fixes it.

Overview

Two architectures, one goal: show the sharding benefit under load.

  1. Basic Mode: Complete production kdb-tick stack (TP/RDB/HDB/Gateway)

    • Single tickerplant processing all data
    • RDB holds the whole day in memory (no intraday writedown) — its footprint climbs under load until the -maxmb cap, visible via gw(`.gw.footprint;::)
    • Falls behind when driven past its throughput ceiling
    • End-to-end transit lag visible via gw(`.gw.transitLag;::)
    • Demonstrates full kdb-architecture-course reference implementation

  2. Sharded Mode: Distributed architecture with separated write path

    • Symbols A-M (AAPL, MSFT, etc.) → Shard 0
    • Symbols N-Z (NVDA, TSLA, etc.) → Shard 1
    • Stays near real-time by splitting load across independent TP/RDB pairs
    • WDB (Write Database) handles disk I/O — RDB stays pure in-memory
    • RDB memory is offloaded via the WDB/IDB writedown path - memory stays consistent (visible via gw(`.gw.footprint;::))
    • IDB (Intraday Database) serves flushed historical intraday data
    • Shard-aware gateway queries three tiers: RDB + IDB + HDB

Architecture

Basic Mode (Complete Production Stack)

                                       ┌─ RDB (5011) ──┐
NYSE TAQ Data → feeds → TP (5010) ─────┤               ├─ Gateway (5014)
(bbo + trade)                          └─ HDB (5012) ──┘
  • TP (port 5010): Tickerplant publishes to RDB
  • RDB (port 5011): In-memory real-time database, writes to HDB at EOD
  • HDB (port 5012): Historical database (empty initially)
  • Gateway (port 5014): Queries both HDB and RDB, concatenates results

Single tickerplant processes all data. Falls behind under full load to demonstrate bottleneck.

Architecture Diagram

basic-mode architecture

Sharded Mode (Distributed Architecture with WDB/IDB)

                        ┌→ TP₀ (5010) ──publish→ RDB₀ (5011) ──────────────┐
                        │                  └───→ WDB₀ (5013) → IDB₀ (5015) ├─── Gateway (5030)
NYSE TAQ Data → feeds ──┤                                                  │
(bbo + trade)           └→ TP₁ (5020) ──publish→ RDB₁ (5021) ──────────────┤
                                           └───→ WDB₁ (5023) → IDB₁ (5025) │
                                                       ↓                   │
                                                   HDB (5012) ←────────────┘

Each shard has four processes:

  • TP: Tickerplant with transaction logging
  • RDB: Pure in-memory real-time database (never writes to disk)
  • WDB: Write Database — subscribes to TP alongside RDB, periodically flushes older data to disk as int-partitions, signals IDB to reload, merges and saves the final date partition at EOD
  • IDB: Intraday Database — loads flushed int-partitions into memory, serves historical intraday queries

Plus shared infrastructure:

  • HDB (port 5012): Segmented historical database (par.txt → segment0 + segment1)
  • Gateway (port 5030): Routes by symbol, queries RDB + IDB + HDB per shard

The WDB/IDB separation keeps the RDB pure in-memory — it never blocks on disk I/O during volatile periods. Based on the kx intraday writedown whitepaper with separated write path per production best practice.

Architecture Diagram

sharded-mode architecture

Quick Start

Prerequisites

Run

No download required. Both scripts default to synthetic mode (SOURCE=synthetic):

# Basic mode — single TP/RDB/HDB/Gateway
./up-basic.sh

# NOTE: NYSE rotates the daily TAQ sample window (older dates fall off as new
# ones are published). To peek at what's currently hosted:
#   curl -s "https://ftp.nyse.com/Historical%20Data%20Samples/DAILY%20TAQ/" \
#     | grep -oE 'SPLITS_US_ALL_BBO_A_[0-9]{8}\.gz'
#
# Default (A+N) downloads ~5GB compressed, decompresses to ~25GB, and parses
# ~250M rows. Expect 30-60 minutes depending on network and CPU speed.
# RSS samples print to stderr every 5s during the build phase.
#
# Two HDBs are produced:
#   data/taq_hdb     — canonical p#sym-sorted (queries, TradeFM, etc.)
#   data/taq_replay  — time-sorted copy for feed replay (built by resort_hdb.q)
# If KxSystems/taq PR #1 lands, kx.taq will emit the time-sorted HDB directly
# and the second step becomes unnecessary.
#
# ./download.sh --full downloads all A-Z (~2.3B quote rows, 18.5 GB sym col).
# This requires a commercial kdb+ license — p#sym on the full sym column
# exceeds the 16 GB workspace cap of kdb-x Community Edition.

# 2. Run basic mode and observe end-to-end transit lag
./up-basic.sh                  # auto-detects from data/taq_hdb/ (or pass YYYY.MM.DD)

# From another q:
#   q)gw:hopen 5014
#   q)gw (`.gw.transitLag;"n"$.z.p-0D00:01)  / last 60s, per table
# Each row: n batches, avg/p50/p95/max microseconds for feed→TP→RDB transit.
# At A+N scale the single TP keeps up: sub-millisecond median lag.
# Under full A-Z (commercial license) or time-compressed replay, basic mode's
# single TP becomes the bottleneck and the p95/max climb.

./down.sh

# Sharded mode — two TP/RDB/WDB/IDB pairs + shared HDB + gateway
./up-sharded.sh

# Same query on the sharded gateway
q)gw:hopen 5030
q)gw (`.gw.transitLag;"n"$.z.p-0D00:01)   / per-shard breakdown

./down.sh

Each transitLag row reports n, avg_us, p50_us, p95_us, max_us feed→TP→RDB transit time. Two TPs split the load — per-shard lag stays flat where single-TP would climb.

Want real NYSE TAQ data instead? See Real NYSE TAQ Data (Optional) — requires a one-time ~30–60 min download, then SOURCE=taq ./up-basic.sh.

Tuning throughput

Two knobs drive the feed, one bounds the RDB:

  • BBO / TRADES — rows per 1-second market-time bucket (volume density). Default: 100K BBO, 5K trade.
  • SPEEDUP — how fast the simulated clock runs. Default: 100× (one bucket every 10 ms wall-clock).
  • RDB_MAXMB — soft per-RDB memory cap (MB). At the cap the RDB drops new batches and stays alive/queryable instead of aborting. Default: 8000 (basic), 3000 per shard (sharded).

Total throughput ≈ BBO × SPEEDUP BBO rows/sec (rough guide). At the default BBO=100000 SPEEDUP=100 that's ~10M rows/sec — enough to fill the basic RDB to its cap within ~20 s. For a demo where the climb is gradual enough to watch (see Observing the Performance Difference), dial the volume down.

# Run the clock faster — same volume, more buckets per second
SPEEDUP=10 ./up-basic.sh

# Heavier volume per bucket, same clock speed
BBO=5000 TRADES=250 SPEEDUP=100 ./up-basic.sh      # single RDB climbs to the cap
BBO=5000 TRADES=250 SPEEDUP=100 ./up-sharded.sh    # per-shard RDBs stay flat

# Run the clock faster — same volume, more buckets per second
SPEEDUP=10 ./up-basic.sh

# Heavier volume per bucket — same clock speed
BBO=500000 TRADES=25000 ./up-basic.sh

# Lighter load for low-power machines
BBO=10000 TRADES=500 ./up-sharded.sh

# Raise the basic RDB cap to delay the drop (default 8000)
RDB_MAXMB=16000 BBO=5000 TRADES=250 SPEEDUP=100 ./up-basic.sh

Observing the Performance Difference

A primary benefit of the sharded/writedown architecture is bounded RDB memory. The basic-mode RDB holds the entire trading day in memory. It never writes down intraday, so its footprint climbs continuously under load until it hits the -maxmb soft cap and starts dropping batches. The sharded RDBs flush older data to the WDB/IDB writedown path every cycle, so each holds only a short retention window and stays flat no matter how long the feed runs.

Two signals make this concrete: RDB memory footprint (showing RAM pressure) and transit lag / throughput (the latency view). Run the same volume in each mode and compare.

Signal 1: RDB memory footprint (.gw.footprint)

.gw.footprint reports the live workspace (MB) and rows held by each in-memory tier. Drive both modes at the same rate — BBO=5000 TRADES=250 SPEEDUP=100 is ≈ 500K bbo + 25K trade rows/sec, gentle enough to watch the climb.

Basic — single RDB, memory climbs to the cap:

BBO=5000 TRADES=250 SPEEDUP=100 ./up-basic.sh
q)gw:hopen 5014
q)gw (`.gw.footprint;::)
tier used_mb rows
-----------------------
rdb  5604    60065250

Sample it a few seconds apart and the climb is unmistakable (measured, single TP keeping up throughout):

elapsed used_mb rows bbo p95 lag
8s 90 1.0M 3.4 ms
24s 1258 9.5M 3.4 ms
72s 2812 34.8M 3.6 ms
120s 5604 60.1M 4.8 ms
~145s hits the -maxmb cap (8000) → starts dropping new batches

Throughput holds ≈ 500K bbo/sec and p95 lag stays ~3–5 ms the whole time — the TP keeps up, yet memory still climbs, because nothing is written down. (used_mb sawtooths a little with garbage collection; rows is the clean monotonic signal.) Left running, the RDB pins at the cap and sheds load.

When we increase the ingestion rate, we begin to see how throughput and transitLag begin to fall behind while the RDB memory fills up much quicker.

Sharded — per-shard RDB + IDB, RDB stays flat:

BBO=5000 TRADES=250 SPEEDUP=100 ./up-sharded.sh
q)gw:hopen 5030
q)gw (`.gw.footprint;::)
shard tier used_mb rows
------------------------------
0     rdb  336     3466218
1     rdb  332     2706571
0     idb  1321    13854419
1     idb  682     8765833

At the same 500K rows/sec, each RDB holds only its short retention window (~90–360 MB, well under the 3000 MB cap) and never grows — the WDB flushes older data to the IDB every cycle. The IDB absorbs the day's volume (and is cleared at EOD); the RDBs stay flat indefinitely. Same feed, same rate: basic pins at 8 GB and drops in ~2 min; sharded sits at a few hundred MB throughout the lifecycle of the running system.

Signal 2: transit lag and throughput

Each RDB carries a metrics table (see metrics/met.q). The feed publishes a batch_sent row per bucket with a per-feed batchId; the RDB stamps batch_arrived on receipt. The gateway aggregates batch_arrived.time − batch_sent.time per table.

Step 1: Basic Mode

./up-basic.sh
q)gw:hopen 5014
q)gw (`.gw.transitLag;"n"$.z.p-0D00:01)

Returns a row per table with n, avg_us, p50_us, p95_us, max_us.

Step 2: Sharded Mode

./up-sharded.sh
q)gw:hopen 5030
q)gw (`.gw.transitLag;"n"$.z.p-0D00:01)

Per-shard breakdown. With A+N at default replay speed the single-TP basic setup keeps up (sub-ms median) — the sharding benefit is visible under full A-Z (commercial license) or time-compressed replay, where basic-mode p95/max climb while sharded stays flat.

The sharding benefit becomes visible when the single TP in basic mode can't drain the feed queue fast enough. Use BBO=100000 SPEEDUP=1000 (or higher) to drive it into saturation.

Usage Examples

Basic Mode Queries

./up-basic.sh

$ q
q)gw:hopen 5014
q)gw (`.gw.footprint;::)                     / RDB workspace (MB) + rows held
q)gw (`.gw.transitLag; "n"$.z.p-0D00:01)    / end-to-end lag, last 60s
q)gw (`.gw.throughput; 0D00:05)              / rows/sec over last 5 min
q)\\

$ q
q)rdb:hopen 5011
q)rdb "count bbo"
q)rdb "count trade"
q)rdb "select count i by sym,metric from metrics"  / raw metrics table
q)\\

Sharded Mode Queries

./up-sharded.sh

$ q
q)gw:hopen 5030
q)gw (`.gw.help;::)
q)gw (`.gw.lastTrade;`AAPL)                  / Routes to Shard0 (A-M)
q)gw (`.gw.lastTrade;`TSLA)                  / Routes to Shard1 (N-Z)
q)gw (`.gw.tradeStats;`AAPL)
q)gw (`.gw.tradeCountByShard;::)
q)gw (`.gw.health;::)
q)gw (`.gw.footprint;::)                      / per-shard RDB + IDB workspace (MB) + rows
q)gw (`.gw.transitLag;"n"$.z.p-0D00:01)      / per-shard lag, last 60s

$ q
q)rdb0:hopen 5011    / Shard0 — A-M only
q)rdb0 "exec distinct first each string sym from trade"
q)rdb1:hopen 5021    / Shard1 — N-Z only
q)rdb1 "exec distinct first each string sym from trade"
q)\\

KX Dashboards Integration

KX Dashboards (packaged with KDB-X) can subscribe to live ingestion metrics and render a real-time latency/throughput view that works identically in basic and sharded modes.

What it shows: end-to-end transit lag (feed → TP → RDB, in microseconds) alongside ingestion rate (rows/sec, batches/sec, MB/sec), per table and — in sharded mode — per shard. When the single-TP basic setup overruns, the p95/max lag climbs on the chart; sharded stays flat because each shard has its own TP handling roughly half the load.

Architecture. dashboards/ingestion.q runs as a separate process that:

  1. Opens an IPC connection to the gateway (:5014 basic, :5030 sharded)
  2. Polls .gw.transitLag and .gw.throughput every second (configurable via CLI parameter)
  3. Tags rows with a shard symbol (`basic / `shard0 / `shard1) so the same chart works for both modes
  4. Republishes via the kdb-tick pub/sub framework (~/.kx/dashboards/sample/tick/u.q) as lag and throughput tables that KX Dashboards streaming subscriptions auto-render

The bridge is pure an observer — it reads from the existing gateway so that ingestion path itself is unchanged.

          ┌─ Gateway ─┐                    ┌─ KX Dashboards ─┐
 TP/RDBs ─┤ transitLag├──poll 1/sec──►  ingestion.q  ──stream──► line charts
          └─throughput┘                  (port 6812)           (port 10001)

Setup (one-time)

  1. Install KX Dashboards from developer.kx.com so the sample tick framework lands at ~/.kx/dashboards/sample/tick/u.q. Your ~/.kx license must include the dsh (Dashboards) entitlement.

  2. Install the repo-checked-in connection + any shared dashboards into your local KX Dashboards data directory:

    ./install-dashboards.sh           # symlink (default — repo edits propagate live)
./install-dashboards.sh --copy    # or copy, to pin a snapshot
./install-dashboards.sh --uninstall

    This places a pre-configured ingestion-bridge connection pointing at localhost:6812. Any dashboards in dashboards/kx/ are installed the same way, and will show up in your Dashboards document list already wired to the bridge.

    Optional: --isolate hides the ~22 demo dashboards/connections that ship with KX, leaving only the two repo dashboards ("Basic Mode" and "Sharded Mode" Ingestion Monitor) visible. Fully reversible — see Hide stock dashboards below.

Run

One command. ./down.sh tears down everything it started.

./up-basic.sh              # or ./up-sharded.sh — pipeline + bridge + webapp

Open http://localhost:10001.

What each script does:

  • up-basic.sh / up-sharded.sh — start the ingestion pipeline (TP/RDB/HDB/Gateway + feeds), the dashboards/ingestion.q bridge on port 6812, and the KX Dashboards webapp on port 10001 (via start-dashboards.sh). The bridge and webapp launch automatically when ~/.kx/dashboards/dash.q is present, since dashboards need streaming pipeline data anyway. Opt out of both with DASHBOARD=0; change the bridge port with DASH_PORT=<port> and the webapp port with DASH_WEB_PORT=<port>. Logs at logs/dashboard_bridge.log and logs/dashboards_web.log.
  • start-dashboards.sh — direct webapp launcher. Called automatically from up-*.sh, but can also be invoked standalone (e.g. to bring the webapp back up after restarting it independently). Idempotent: if the webapp is already running on DASH_WEB_PORT, it reuses that instance instead of starting a second one. Uses nohup + disown so accidental Ctrl-Z in the spawning terminal can't SIGSTOP it (which presents as "stuck and never loads" in the browser).

Note: this intentionally does not launch KX's sample/demo.q — it binds port 6812, the same port our ingestion bridge uses. If you want the built-in KX sample dashboards, run DASHBOARD=0 ./up-basic.sh and start demo.q manually.

Stop everything with ./down.sh. It reads .pids_basic, .pids_sharded, and .pids_dashboards, so pipeline and webapp come down together. Also scans the project's known ports (5010–5030, 6812, 10001) and kills any orphaned q listeners that aren't in a pid file — catches processes reparented to init after a crashed parent shell.

Heads up on disk usage: TP transaction logs (data/sym<YYYY.MM.DD> in basic mode, data/segment{0,1}/sym<YYYY.MM.DD> in sharded) grow with every published batch and can hit hundreds of GB under sustained throughput. up-*.sh deletes them at next startup, but they linger in the gap between ./down.sh and your next run. Add --clean-tp-logs to wipe them as part of shutdown:

./down.sh --clean-tp-logs    # stop + delete leftover sym<DATE> logs

Open the bundled Ingestion Monitor dashboards

Two ready-made dashboards ship with the repo — one tailored to each mode:

./install-dashboards.sh picks both up automatically. Open the one that matches the mode you've started — both are already wired to the ingestion-bridge connection. They're produced from a shared template so layout and styling stay consistent.

The same layout on each — identical units and gauge scales in both modes, so dial positions compare directly across the two dashboards:

Row Tile What it shows
Top (gauges, 4 across) batches sent (M rows/s, max 2.5) · batches arrived (M rows/s, max 2.5) · p95 lag (ms, max 50) · max lag (ms, max 250) Windowed over the last 15 s. Sent = offered load (what the feed published); arrived = durably ingested (batches that actually landed — dropped batches never record arrival). Same scale, so the gap between the two dials is the headline: in basic they diverge hard once the RDB caps (sent stays high, arrived collapses); in sharded they track each other. Lag dials peg to max when load is offered but nothing arrives (a stalled RDB means unbounded lag, not "no data"), and read null when idle
Middle (left) In-memory footprint (MB), fixed 0–10000 axis Basic: RDB line climbing to the red 8 GB cap baseline (overshoots to ~10 GB before freezing). Sharded: per-RDB lines staying bounded below the 3 GB cap baseline, dashed IDB lines absorbing the writedown
Middle (center) Ingest throughput (K rows/s), dual Y axes bbo on the left axis, trade on the right (dashed) — trade volume is ~20× smaller and was invisible on a shared axis
Middle (right) p95 transit lag (ms, log Y-axis, fixed 1–1000) Per-source lines plus amber (15 ms) and red (50 ms) baseline thresholds
Bottom (left) Throughput grid rows/s (with Sum footer), batches/s, MB/s per (shard, tbl)
Bottom (right) Lag grid n, avg, p50, p95, max ms per (shard, tbl), with amber/red text highlighting at the same thresholds as the gauges

Lag thresholds everywhere (gauge bands, chart baselines, grid highlights): green < 15 ms p95, amber ≤ 35 ms, red above — max-lag equivalents at 75/150/250 ms.

The scales are tuned to the recommended dashboard demo load:

BBO=30000 TRADES=1500 SPEEDUP=100   # ≈ 3.1M rows/s offered

At that load (measured on an M-series laptop): basic ingests ~40 s with sent and arrived dials tracking together at green lag, then the RDB blows through its 8 GB cap and stalls — the sent dial stays pegged near 3 M rows/s while the arrived dial collapses toward zero (the feed keeps publishing, but nothing is durably ingested), every lag dial pegs red, and the memory line rides the cap baseline. Sharded backpressures the feed to a sustainable ~2 M rows/s where sent and arrived stay locked together — every published row is ingested and written down — holding p95 ≈ 8–13 ms in the green, with the per-shard RDBs staying flat and bounded while the writedown flows to the IDB lines. Full defaults (BBO=100000) saturate both modes on a laptop; BBO=5000 never stresses basic-mode lag at all.

The sent vs arrived split is the clearest single signal in the demo: a high "batches sent" number next to a collapsed "batches arrived" number is basic mode dropping data on the floor. .gw.throughput now returns arrived_rows_per_sec alongside rows_per_sec — sent is derived from batch_sent (feed publish), arrived from batch_arrived (RDB receipt, never stamped for a dropped batch).

⚠️ This demo velocity is a stress burst, not a full trading session

BBO=30000 SPEEDUP=100 is deliberately violent: its only job is to drive the basic-mode RDB into its cap within ~40 s so the sent-vs-arrived divergence is immediate and obvious. It is not meant to replay a whole trading day in one sitting. Two things follow:

  • The sharded RDBs stay bounded and lossless indefinitely at this load — that fix is real and permanent (each RDB sawtooths ~0.6–2.5 GB under its 3 GB cap and never drops, regardless of IDB health).
  • The sharded IDBs, by design, keep growing for the entire trading day. They hold the intraday history in memory until EOD (when the WDB merges it to the HDB and clears them). At BBO=30000 a full 6.5 h session is ~46 GB of IDB data, so on any laptop the IDBs will reach the device's RAM ceiling well before EOD. That is expected: you are cramming a full day into a few minutes of wall-clock. Watch the contrast for a minute or two, then ./down.sh — don't leave it running to "EOD".

If you want the sharded tier to stand up for a full day in memory, cap the per-second volume so the whole day fits in RAM. Total IDB day ≈ BBO × 23,400 event-seconds × ~66 bytes/row. A safe full-session load:

BBO=5000 TRADES=250 SPEEDUP=100    # ≈ 0.5 M rows/s offered; full-day IDB ≈ 8 GB

At this velocity a full 6.5 h session replays in ~4 min of wall-clock, the sharded IDBs peak around 8 GB (comfortable on a 16 GB+ machine) and clear at EOD, and basic mode still hits its 8 GB RDB cap — just after ~2 min instead of ~40 s. SPEEDUP only sets the wall-clock pace (lower = calmer to watch); it does not change the IDB's peak size — only BBO/TRADES (rows per event-second) do.

./install-dashboards.sh    # one-time, after each clone (or pull that touched dashboards/kx/)
./up-basic.sh              # or ./up-sharded.sh — pipeline + bridge + webapp all start together
# Open http://localhost:10001 → click "Basic Mode" or "Sharded Mode" Ingestion Monitor

The data sources poll the bridge's ring buffers (.dash.bufThroughput, .dash.bufLag) every second. The bridge holds ~5 minutes of back-history in those buffers, so the line charts pre-populate immediately on open and then animate live.

Hide stock dashboards (optional)

KX Dashboards ships with ~22 demo dashboards ("Demo", "Demo Maps", "Demo Data", etc.) and several demo connections that clutter the document list. To make the two repo dashboards ("Basic Mode" and "Sharded Mode" Ingestion Monitor) the only things visible at http://localhost:10001:

./install-dashboards.sh --isolate         # move stock entries to a backup dir
./install-dashboards.sh --restore-stock   # ...and restore them later

What --isolate actually does. Anything in ~/.kx/dashboards/data/{dashboards,connections}/ that is not a symlink into this repo gets moved (not deleted) to ~/.kx/dashboards/data/.stock_backup/. Repo-managed symlinks (i.e. the ones ./install-dashboards.sh created) stay in place, so "Ingestion Monitor" remains visible and ingestion-bridge remains connectable. Refresh http://localhost:10001 and the document list now contains a single entry.

Fully reversible. --restore-stock moves every file from the backup dir back to its original location. Both operations are idempotent — safe to re-run. Nothing is deleted; the user's KX install is never destructively altered.

Each clone is independent: --isolate only affects the local ~/.kx/dashboards/ install. A colleague who clones the repo and wants the same clean view runs ./install-dashboards.sh --isolate on their machine.

Building a chart in KX Dashboards

  1. New data source → pick the pre-installed ingestion-bridge connection
  2. Query type Streaming, table lag (or throughput), subscription filter ` (empty — subscribe to all rows)
  3. Drag a Line Chart onto the canvas
  4. Bind the data source:
    • X axis: time
    • Y axis: avg_us (or p95_us, max_us)
    • Series / group by: shard or tbl (both work; pick one for trellising)
  5. Save. The chart updates every second as new rows arrive via pub/sub.

For throughput, repeat with the throughput table and bind rows_per_sec or mb_per_sec on the Y axis.

Sharing a dashboard via git

Dashboards are plain JSON files at ~/.kx/dashboards/data/dashboards/<uuid>.json. To share one through this repo so it appears on every clone:

# After saving in the UI, copy the JSON into the repo:
cp ~/.kx/dashboards/data/dashboards/<uuid>.json dashboards/kx/ingestion-metrics.json
git add dashboards/kx/ingestion-metrics.json && git commit

Other users then just run ./install-dashboards.sh and the dashboard shows up in their document list at http://localhost:10001, already pointed at their local bridge via the shared ingestion-bridge connection name.

What you'll see

Running basic mode at the default 100× replay speed against A+N data, both charts stay roughly flat — single TP keeps up, p95 lag is sub-millisecond. Drive it harder (SPEEDUP=500 ./up-basic.sh, or the full --full A-Z dataset on a commercial license) and the basic-mode lag curve climbs while throughput plateaus. Restart in sharded mode and the two per-shard series stay flat — the sharding benefit is visible as a direct chart comparison.

The clearest single signal is the RDB memory (MB) chart. Run BBO=5000 TRADES=250 SPEEDUP=100 ./up-basic.sh and its one line climbs steadily toward the -maxmb cap; restart in sharded mode at the same volume and the two per-shard lines sit flat at a few hundred MB — the writedown path holding RDB memory bounded, live on screen.

CLI reference (advanced)

q dashboards/ingestion.q -gw :localhost:5014 -mode basic   [-p 6812] [-interval 1000] [-window 0D00:00:05]
q dashboards/ingestion.q -gw :localhost:5030 -mode sharded [-p 6812] [-interval 1000] [-window 0D00:00:05]
  • -gw: gateway IPC endpoint to poll (required)
  • -mode: basic or sharded (required; tags the shard column)
  • -p: bridge listener port (default 6812)
  • -interval: poll period in milliseconds (default 1000)
  • -window: trailing time window for each poll as timespan (default 0D00:00:05)

Integrations

Data Generation

datagen/datagen.q — Streaming market data generator

  • Vendored and extended from KxSystems/datagen capmkts/init.q (Apache 2.0) — see that repo for broader use cases
  • Correlated log-normal random walk (Cholesky decomposition, 51-symbol price universe)
  • Per-bucket API: .dg.init[cfg], .dg.mkBBO[idx;n], .dg.mkTrade[idx;n], .dg.resetDay[]
  • Runs indefinitely — regenerates price path each simulated day
  • Self-contained: datagen/master.csv (51-symbol universe) vendored alongside
  • See datagen/README.md for full API and schema docs

Feeds (feeds/)

All three feeds support both basic and sharded modes via --mode=basic|sharded.

feeds/datagen_feed.q — Single-process synthetic feed using datagen

  • Emits full 23-col BBO + 15-col trade schemas (TAQ-compatible)
  • Auto-subdivides buckets to keep tick interval ≤ 100 ms (clean behaviour at low speedup)
  • Routes to single TP (basic) or two TPs by symbol initial (sharded)
  • CLI: --mode=basic|sharded, --tp=:port / --tp0 --tp1, --speedup N, --bbo N, --trades N

feeds/bbo_feed.q and feeds/trade_feed.q — TAQ replay feeds

  • Read from time-sorted replay HDB (data/taq_replay/); used only with SOURCE=taq
  • Same routing and metrics instrumentation as datagen_feed.q

Vendored (tick/)

From KxSystems/kdb-architecture-course:

tick/tick.q — Tickerplant with logging and EOD processing

tick/rdb.q — Standard real-time database

  • In-memory data store, subscribes to TP, log replay on startup
  • .Q.hdpf at EOD: save partitions, clear memory, signal HDB reload

tick/hdb.q — Historical database loader

tick/gw.q — Standard gateway (basic mode: queries HDB + RDB)

tick/sym.q — TAQ schema (bbo 23-col, trade 15-col)

Sharded (sharded/)

Custom implementation for distributed ingestion:

sharded/rdb.q — Sharded RDB (thin wrapper)

  • Loads vendored tick/rdb.q, overrides .u.end to clear memory only
  • WDB owns all persistence — RDB never writes to disk

sharded/wdb.q — Write Database

  • Subscribes to TP alongside RDB (second subscriber)
  • Flushes rows older than now - interval to int-partitions (today/<i>/<table>/)
  • Enumerates sym against shared HDB root (-hdbdir)
  • Signals IDB to reload after each flush via .idb.reload[] async IPC
  • At EOD: merges int-partitions with xasc/p#sym, saves date partition, signals HDB + IDB

sharded/idb.q — Intraday Database

  • Loads flushed int-partitions from WDB staging dir into memory
  • Serves selectFunc queries (same API as RDB/HDB)
  • Reloads on signal from WDB, clears after EOD (HDB takes over)

sharded/shardedgw.q — Shard-aware gateway

  • Routes queries by first letter of symbol to appropriate shard
  • Queries three tiers per shard: RDB (real-time) + IDB (intraday) + HDB (historical)
  • Fan-out/merge for cross-shard queries; IDB connections optional (graceful degradation)
  • API: .gw.lastTrade[], .gw.lastBBO[], .gw.tradeStats[], .gw.tradeCountByShard[], .gw.health[]

metrics/met.q — Metrics module kernel

  • Narrow metrics table (time,sym,metric,val,tags) with per-metric dimensions in tags
  • .met.record publishes via TP; .met.instrument wraps subscriber upd to stamp batch_arrived on receipt
  • Enables .gw.transitLag and .gw.throughput queries via the gateway
  • Design notes: metrics/README.md

sharded/config.q — Shard routing function (A-M→0, N-Z→1)

Orchestration

up-basic.sh — Start basic mode (4 processes + 1 feed)

  • TP (5010) + RDB (5011) + HDB (5012) + Gateway (5014)
  • SOURCE=synthetic (default) uses datagen_feed.q; SOURCE=taq uses bbo_feed.q + trade_feed.q

up-sharded.sh — Start sharded mode (9 processes + 1 feed)

  • Per shard: TP + RDB + WDB + IDB; shared: HDB + Gateway
  • SOURCE=synthetic (default) uses datagen_feed.q --mode sharded; SOURCE=taq uses the two TAQ replay feeds
  • Cleans stale WDB staging dirs on startup

down.sh — Graceful shutdown of all processes

  • Reads .pids_{basic,sharded,dashboards}; orphan port-scan catches q listeners that escaped pid-file tracking
  • --clean-tp-logs also wipes leftover data/**/sym<DATE> transaction logs (can be hundreds of GB after heavy runs)

Tools

tools/ — PyKX-based utilities. hdb-to-parquet.py exports an HDB date partition to Hive-partitioned Parquet (consumable by DuckDB, Spark, Polars, the kdb-x parquet module, etc.). See tools/README.md for setup and usage.

Project Structure

.
├── README.md                    # This file
├── CLAUDE.md                    # AI assistant guidance
├── up-basic.sh                  # Basic mode startup (SOURCE=synthetic default)
├── up-sharded.sh                # Sharded mode startup (SOURCE=synthetic default)
├── down.sh                      # Graceful shutdown
├── download.sh                  # Optional: download real NYSE TAQ data + build HDB
├── arch/                        # Architecture diagrams
│   ├── basic-mode.drawio.png    # Basic mode architecture diagram
│   └── sharded-mode.drawio.png  # Sharded mode architecture diagram
├── datagen/                     # Streaming synthetic market data generator
│   ├── datagen.q                # Price model + per-bucket BBO/trade generators
│   └── README.md                # API docs, schemas, upstream PR notes
├── tick/                        # Vendored from kdb-architecture-course
│   ├── tick.q                   # Tickerplant with logging and EOD
│   ├── rdb.q                    # Real-time database (standard)
│   ├── hdb.q                    # Historical database loader
│   ├── gw.q                     # Standard gateway (basic mode)
│   ├── sym.q                    # TAQ schema: bbo and trade tables
│   └── u.q                      # Subscriber utility
├── metrics/                     # Metrics module (see metrics/README.md)
│   ├── met.q                    # Kernel: schema + helpers + subscriber wrap
│   └── rdb-metrics.q            # tick/rdb.q + metrics instrumentation
├── feeds/                       # Feed handlers (basic + sharded)
│   ├── datagen_feed.q           # Synthetic feed (TAQ-compatible schema, indefinite)
│   ├── bbo_feed.q               # TAQ replay feed (used with SOURCE=taq)
│   └── trade_feed.q             # TAQ replay feed (used with SOURCE=taq)
├── sharded/                     # Custom distributed implementation
│   ├── config.q                 # Shard routing: A-M→0, N-Z→1
│   ├── rdb.q                    # Sharded RDB (thin wrapper, clear-only EOD)
│   ├── wdb.q                    # Write Database (disk I/O, flush, merge)
│   ├── idb.q                    # Intraday Database (loads flushed data)
│   └── shardedgw.q              # Shard-aware gateway (RDB+IDB+HDB)
├── taq/                         # TAQ post-build tools (optional, SOURCE=taq only)
│   ├── sort-hdb.q               # In-place re-sort
│   └── resort-hdb.q             # Scalable re-sort → new sorted HDB
├── tools/                       # PyKX-based utilities (see tools/README.md)
│   └── hdb-to-parquet.py        # HDB → Hive-partitioned Parquet exporter
├── docs/                        # Design documents
│   ├── wdb-idb-spec.md          # WDB/IDB architecture spec
│   └── q-style-guide.md         # q coding style guide
├── data/
│   ├── sym/                     # Basic mode output HDB
│   ├── hdb/                     # Sharded mode HDB root (par.txt + sym)
│   ├── segment0/sym/            # Shard0 segment (A-M data)
│   └── segment1/sym/            # Shard1 segment (N-Z data)
└── logs/                        # Process logs

Real NYSE TAQ Data (Optional)

To replay actual NYSE trade and BBO data instead of synthetic, use SOURCE=taq:

Additional Prerequisites

  • kx.taq module (requires kx.printf): 
    git clone https://github.com/KxSystems/taq.git
mkdir -p ~/.kx/mod/kx && cp -r taq/taq ~/.kx/mod/kx/

Install kx.printf (kx.taq dependency — and used directly by this repo)

(from KxSystems/taq install.md — trailing \ removed after LATEST= lines)

LATEST=$(curl -s https://api.github.com/repos/KxSystems/printf/releases/latest | grep 'tag_name' | cut -d '"' -f 4)
curl -L https://github.com/KxSystems/printf/archive/refs/tags/$LATEST.zip -o printf.zip && \
unzip -j printf.zip "printf-$LATEST/printf.q" -d $HOME/.kx/mod/kx/ && \
rm printf.zip

export QPATH="$QPATH:$HOME/.kx/mod"  # add to .bashrc/.zshrc

Verify kx.taq load cleanly

@[use;`kx.taq;{-2 "FAIL: kx.taq: ",x; exit 1}];
.log.info "kx.taq + kx.printf OK";

  • Standard Unix tools: bash, curl, gzip

    • Available by default on macOS/Linux

  • NYSE TAQ Data: Downloaded automatically by download.sh

    • Sample files from NYSE FTP
    • Or provide your own TAQ files in data/raw/

Components

Data Pipeline

download.sh - Download NYSE TAQ data and build input HDBs

  • Downloads BBO splits (A+N default, A-Z with --full), trade, and master files
  • Parses via kx.taq parseToDisk — produces data/taq_hdb/ with 23-col quote, 15-col trade, 41-col master (canonical, p#sym applied)
  • Then calls taq/resort_hdb.q -sortby time to produce data/taq_replay/ — a time-sorted copy the feeds use for realistic replay (bin-based O(log N) bucket lookups)
  • Feeds publish the native kx.taq columns directly — no projection or type casting (xcols hoists time,sym for tick.q compatibility)
  • The two-step split will collapse into one once KxSystems/taq PR #1 (time-sort support in kx.taq) lands — then kx.taq can emit the replay HDB directly and resort_hdb.q drops out of the pipeline
  • --full A-Z requires a commercial license (see Prerequisites)

Vendored (tick/)

From KxSystems/kdb-architecture-course:

tick/tick.q - Tickerplant with logging and EOD processing

tick/rdb.q - Standard real-time database

  • In-memory data store, subscribes to TP, log replay on startup
  • .Q.hdpf at EOD: save partitions, clear memory, signal HDB reload
  • Used directly in basic mode

tick/hdb.q - Historical database loader

tick/gw.q - Standard gateway (basic mode: queries HDB + RDB)

tick/sym.q - TAQ schema (trade and bbo tables)

Sharded (sharded/)

Custom implementation for distributed ingestion:

sharded/rdb.q - Sharded RDB (thin wrapper)

  • Loads vendored tick/rdb.q, overrides .u.end to clear memory only
  • WDB owns all persistence — RDB never writes to disk

sharded/wdb.q - Write Database

  • Subscribes to TP alongside RDB (second subscriber)
  • Flushes rows older than now - interval to int-partitions (today/<i>/<table>/)
  • Enumerates sym against shared HDB root (-hdbdir)
  • Signals IDB to reload after each flush via .idb.reload[] async IPC
  • At EOD: merges int-partitions with xasc/p#sym, saves date partition, signals HDB + IDB

sharded/idb.q - Intraday Database

  • Loads flushed int-partitions from WDB staging dir into memory
  • Serves selectFunc queries (same API as RDB/HDB)
  • Reloads on signal from WDB, clears after EOD (HDB takes over)

sharded/shardedgw.q - Shard-aware gateway

  • Routes queries by first letter of symbol to appropriate shard
  • Queries three tiers per shard: RDB (real-time) + IDB (intraday) + HDB (historical)
  • Fan-out/merge for cross-shard queries
  • IDB connections optional (graceful degradation)
  • API: .gw.lastTrade[], .gw.lastBBO[], .gw.tradeStats[], .gw.tradeCountByShard[], .gw.health[]

feeds/bbo_feed.q and feeds/trade_feed.q - TAQ replay feeds

  • Time-slice batching (1 s event-time buckets; replay clock driven by --speedup) — no per-row loops
  • Vectorized alphabetic symbol routing for sharded mode
  • Publishes batch_sent metric per batch for transit-lag correlation

metrics/met.q - Metrics module kernel

  • Narrow metrics table (time,sym,metric,val,tags) with per-metric dimensions in tags
  • .met.record publishes via TP; .met.instrument wraps subscriber upd to stamp batch_arrived on receipt
  • Enables .gw.transitLag query via the gateway (basic and sharded)
  • Design notes: metrics/README.md

sharded/config.q - Shard routing function (A-M→0, N-Z→1)

Orchestration

up-basic.sh - Start basic mode (4 processes + 2 feeds)

  • TP (5010) + RDB (5011) + HDB (5012) + Gateway (5014)

up-sharded.sh - Start sharded mode (11 processes + 2 feeds)

  • Per shard: TP + RDB + WDB + IDB
  • Shared: HDB + Gateway
  • Cleans stale WDB staging dirs on startup

Replay speedup: both feeds default to 100× realtime (a full trading day replays in ~4 minutes). This is baked into the feed scripts; override via SPEEDUP=N ./up-basic.sh (1 = realtime, ceiling ~1000× set by kdb+'s 1 ms timer floor against the feed's 1 s event-time bucket). With the full 23-col bbo / 15-col trade schema, the per-tick HDB bucket materialization (~60K rows × 23 cols, ~170 ms) becomes the bottleneck rather than the TP — effective replay speed is ~5–6× regardless of --speedup.

down.sh - Graceful shutdown of all processes

Tools

tools/ — PyKX-based utilities. Currently includes hdb-to-parquet.py, which exports an HDB date partition to a Hive-partitioned Parquet dataset (consumable by DuckDB, Spark, Polars, the kdb-x parquet module, etc.). See tools/README.md for setup and usage.

Usage Examples

Basic Mode

./download.sh                  # auto-picks the latest NYSE sample date
# ./download.sh 2026.01.02     # ...or pin a specific date

Produces two HDBs:

  • data/taq_hdb/ — canonical p#sym-sorted (queries, Parquet export)
  • data/taq_replay/ — time-sorted copy for feed replay (built by resort-hdb.q -sortby time)

Default A+N download: ~5 GB compressed, ~25 GB decompressed, ~250 M rows. Expect 30–60 minutes.

Note: NYSE rotates the daily TAQ sample window — older dates fall off. Check what's available:

curl -s "https://ftp.nyse.com/Historical%20Data%20Samples/DAILY%20TAQ/" \
  | grep -oE 'SPLITS_US_ALL_BBO_A_[0-9]{8}\.gz'

Full A-Z (--full) produces ~2.3B quote rows (18.5 GB sym column). Applying p#sym to that exceeds kdb-x Community Edition's 16 GB workspace cap — requires a commercial license.

Run with TAQ data

SOURCE=taq ./up-basic.sh       # auto-detects date from data/taq_replay/
SOURCE=taq ./up-sharded.sh

Project Structure

.
├── README.md                    # This file
├── CLAUDE.md                    # AI assistant guidance
├── log.q                        # kx.log initializer (loaded by every q process)
├── download.sh                  # Download TAQ data + build input HDB
├── up-basic.sh                  # Basic mode startup (4 processes + bridge)
├── up-sharded.sh                # Sharded mode startup (11 processes + bridge)
├── start-dashboards.sh          # KX Dashboards webapp launcher (port 10001)
├── install-dashboards.sh        # Install repo-checked-in dashboards/connections
├── down.sh                      # Cleanup script (pipeline + dashboards)
├── arch/                        # Architecture diagrams
│   ├── basic-mode.drawio.png    # Basic mode architecture diagram
│   └── sharded-mode.drawio.png  # Sharded mode architecture diagram
├── tick/                        # Vendored from kdb-architecture-course
│   ├── tick.q                   # Tickerplant with logging and EOD
│   ├── rdb.q                    # Real-time database (standard)
│   ├── hdb.q                    # Historical database loader
│   ├── gw.q                     # Standard gateway (basic mode)
│   ├── sym.q                    # TAQ schema: trade and bbo tables
│   └── u.q                      # Subscriber utility
├── metrics/                     # Metrics module (see metrics/README.md)
│   ├── met.q                    # Kernel: schema + helpers + subscriber wrap
│   └── rdb_metrics.q            # tick/rdb.q + metrics instrumentation
├── sharded/                     # Custom distributed implementation
│   ├── config.q                 # Shard routing: A-M→0, N-Z→1
│   ├── rdb.q                    # Sharded RDB (thin wrapper, clear-only EOD)
│   ├── wdb.q                    # Write Database (disk I/O, flush, merge)
│   ├── idb.q                    # Intraday Database (loads flushed data)
│   ├── shardedgw.q              # Shard-aware gateway (RDB+IDB+HDB)
│   ├── bbo_feed.q               # BBO replay with symbol routing
│   └── trade_feed.q             # Trade replay with symbol routing
├── taq/                         # TAQ post-build tools (parsing via kx.taq module)
│   ├── sort_hdb.q               # In-place re-sort (usually not needed; kx.taq sorts)
│   └── resort_hdb.q             # Scalable re-sort → new sorted HDB (-sortby time for replay, -sortby sym for queries)
├── tools/                       # PyKX-based utilities (see tools/README.md)
│   └── hdb-to-parquet.py        # HDB → Hive-partitioned Parquet exporter
├── dashboards/                  # KX Dashboards integration
│   ├── ingestion.q              # Bridge: polls GW, republishes via kdb-tick pub/sub
│   └── kx/                      # Checked-in dashboards + connections
│       ├── connections/         # Pre-configured connections (ingestion-bridge)
│       └── *.json               # Shared dashboards (install via install-dashboards.sh)
├── docs/                        # Design documents
│   ├── wdb-idb-spec.md          # WDB/IDB architecture spec
│   └── q-style-guide.md         # q coding style guide
├── data/
│   ├── taq_hdb/                 # Canonical input HDB (p#sym sorted)
│   ├── taq_replay/              # Time-sorted replay HDB (feed source)
│   ├── sym/                     # Basic mode output HDB
│   ├── hdb/                     # Sharded mode HDB root (par.txt + sym)
│   ├── segment0/sym/            # Shard0 segment (A-M data)
│   └── segment1/sym/            # Shard1 segment (N-Z data)
└── logs/                        # Process logs

Architecture Credits

This blueprint builds on two excellent KX reference implementations:

  1. KxSystems/kdb-architecture-course

    • Provided: tick.q, rdb.q, hdb.q, gw.q, u.q (vendored to tick/)
    • We adapted: feed.q → bbo_feed.q + trade_feed.q, gw.q → shardedgw.q, rdb.q → sharded/rdb.q + wdb.q

  2. KxSystems/taq (kx.taq module)

    • Parses NYSE TAQ PSV files to date-partitioned kdb+ HDB
    • Used by download.sh for the ingest pipeline (parseToDisk)

  3. KX Intraday Writedown Whitepaper

    • Inspiration for the WDB/IDB intraday writedown architecture

Our novel contributions:

  • Time-slice TAQ replay with per-batch end-to-end transit lag (metrics module)
  • Alphabetic symbol-based shard routing (A-M vs N-Z)
  • Separated write path (WDB/IDB) for non-blocking intraday persistence
  • Three-tier gateway queries (RDB + IDB + HDB) with optional IDB degradation

Troubleshooting

q not found:

export PATH=$PATH:/path/to/q/installation

Port already in use:

./down.sh
lsof -i :5010

Process won't stop:

./down.sh
pkill -9 q

Architecture Credits

  1. KxSystems/kdb-architecture-course

    • Vendored: tick.q, rdb.q, hdb.q, gw.q, u.q (tick/)
    • Adapted: feed.q → feeds/datagen_feed.q + feeds/bbo_feed.q + feeds/trade_feed.q; gw.q → shardedgw.q; rdb.q → sharded/rdb.q + wdb.q

  2. KxSystems/datagen (Apache 2.0)

    • Price model vendored and extended from capmkts/init.q into datagen/datagen.q; master.csv included

  3. KxSystems/taq (kx.taq module)

    • Used by download.sh for NYSE TAQ ingestion (parseToDisk)

  4. KX Intraday Writedown Whitepaper

    • Inspiration for the WDB/IDB intraday writedown architecture

Novel contributions:

  • Synthetic TAQ-compatible data generator (per-bucket streaming API, datagen/datagen.q)
  • Time-slice replay with per-batch end-to-end transit lag (metrics module)
  • Alphabetic symbol-based shard routing (A-M vs N-Z)
  • Separated write path (WDB/IDB) for non-blocking intraday persistence
  • Three-tier gateway queries (RDB + IDB + HDB) with optional IDB degradation

Further Reading

About

No description, website, or topics provided.

Resources

Readme
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages