Inside Quantitative Development: Systems, Strategy, and Infrastructure

Background and Path into Quant Finance

The presenter is a quantitative developer with four years of experience, holding an engineering degree from the University of Waterloo and six prior internships in big tech and startups — spanning software development and machine learning engineering — but with no prior finance background before entering the quant world. After graduation, hundreds of applications led to a role at a quant firm trading global equities, obtained through a six-round interview process.

---

The Interview Process

Round 1: Portfolio Management System

• Asked to build a system that could track orders, execute orders, and maintain positions across stocks
• Required familiarity with features like stop losses and limit orders
• Key takeaway: Communication of thought process is as important as technical correctness — half of what interviewers assess is how you explain your reasoning, not just whether the code is right

Round 2: Combinatorics and Statistics Problem

• Involved building an algorithm around combinations in a modified 52-card deck
• Tested the ability to apply statistics to programming
• The presenter only reached ~70% of a correct solution, yet advanced — again reinforcing that clear communication carries significant weight

Rounds 3–6: System Design, Behavioral, and Partner Interviews

• Round 3 involved a deep dive into resume projects with two partners, including drawing out system architecture on screen and answering questions about databases, data engineering, and ML algorithms (gradient boosted trees)
• Rounds 4–6 rotated through different partners across sub-teams, covering system design, behavioral questions, database knowledge, and high-level questions about global equity trading
• Advice: Be prepared to discuss your own projects in depth for up to an hour, with confidence and technical precision

Interview Advice Summary

1. Practice problems by talking through them out loud first
2. Write pseudo-code before implementing
3. Build the habit of explaining your reasoning concisely
4. Prioritize communication skill — it often outweighs raw coding ability in quant interviews

---

The Multi-Factor Trading Model: End-to-End Architecture

This is the foundational framework underpinning nearly everything at top quant firms. The recommended reference text is Active Portfolio Management by Richard Grinold and Ronald Kahn.

The Core Purpose: Generating Alpha

• Clients can buy an index cheaply, so a quant firm's value lies entirely in outperforming the benchmark
• Alpha = the return above the benchmark that comes from active decisions, not from market movement
• The market already prices in collective consensus; the job is to identify where the model disagrees with consensus and be right more often than competitors
• Active management is fundamentally about forecasting the market's errors

Return Decomposition

A stock's return on any given day is not one thing — it is a stack of components:

• Broad market movement
• Country exposure
• Industry/sector exposure (e.g., oil sector)
• Idiosyncratic (company-specific) residual — this is what a quant firm actually has a view on

This decomposition is performed via regression: factor exposures explain common drivers, and the unexplained residual is the stock-specific return. Notably, even for stocks with strong sector ties, the idiosyncratic component is typically large.

---

The Multi-Factor Pipeline: Block by Block

Block 1: Data and Signals

• All downstream quality depends entirely on input data quality
• Sources include both traditional financial data and alternative data (social media, supply chain, news, credit card transactions, broker reports)
• A trading universe must be defined first — not all stocks are worth trading, especially illiquid ones that cause adverse price impact when entering or exiting positions
• Raw signals (value, momentum, size, quality, and more exotic alternatives) represent individual measurable views on a stock

Block 2: Alpha Construction

An alpha is a refined, tradeable forecast of return. Each signal is transformed into an alpha by multiplying:

1. The stock's volatility
2. The signal's genuine predictive skill (information coefficient)
3. The current signal strength for that specific stock

Multiple alphas run simultaneously — each capturing a different, ideally independent edge.

Block 3: Risk Model

• The risk model quantifies what the alpha could cost in volatility, and runs in parallel with alpha construction rather than after it
• Uses the same factor decomposition: each stock's return = exposure to common factors (industries, size, value, volatility, momentum) + specific return

Why use factors instead of direct stock-to-stock covariance?

• A 1,400-stock universe requires ~980,000 pairwise covariance estimates — statistically intractable
• By expressing each stock as exposures to ~65 common factors, only ~2,000 factor covariances are needed
• This collapses an impossible problem into a computationally tractable one

Two critical risk intuitions:

1. Diversification works on specific risk only. Portfolio risk is less than the weighted average of its parts because stocks don't all move together. Specific risk averages away with enough uncorrelated names. But systematic (market) risk never diversifies away — this is the insight behind CAPM and Arbitrage Pricing Theory
2. Risk scales with the square root of time, not linearly. Variance adds linearly across time periods; since risk = √variance, monthly volatility scales to annual by multiplying by √12, not 12. Getting this wrong corrupts all risk calculations

Block 4: Portfolio Construction

• An optimizer balances: maximize expected return, subtract a risk penalty, subtract transaction costs, while respecting constraints (position limits, sector neutrality, turnover caps, leverage limits)
• Output: the target portfolio — exact holdings for each name

Block 5: Implementation and Trading

• Philosophy: subtract as little value as possible
• Sources of trading cost (all leak alpha):
  • Commission: per-share broker fee
  • Bid-ask spread: cost of buying at ask and selling at bid
  • Market impact: large orders move the price against you as you trade — described as the "finance version of the Heisenberg principle"
  • Opportunity cost: waiting for a better price while the stock moves away
• Performance measured via implementation shortfall: comparing a hypothetical zero-cost paper portfolio to the real one

Block 6: Performance Analysis

• Decomposes realized returns into: intended factor bets, constraints, and noise
• Goal is to separate skill from luck and identify where genuine skill resides
• Feeds directly back into research, since factors decay as edges get discovered and crowded out

---

The Fundamental Law of Active Management

Two numbers tie the entire system together:

• Information Ratio (IR) = Active Return ÷ Active Risk — the primary performance report card
• Fundamental Law: IR ≈ Skill × √Breadth
  • Skill = predictive accuracy per bet
  • Breadth = number of independent bets made

The critical insight about independence: Five long positions all in retail and five short positions all in energy is not 10 bets — it is 2 bets. Real breadth requires genuinely distinct decisions.

The entire purpose of a multi-factor model is to function as a breadth machine — making thousands of small, independent, slightly better-than-even bets across the entire market every day. Modest per-bet skill compounds into a serious edge through breadth.

---

Data Engineering in Practice

Data Scrubbing

Before any analysis: cross-checking outliers, filling gaps, fixing formats, reconciling vendor differences in identifiers (CUSIP, ISIN, Bloomberg IDs), and handling restated historical figures.

Security Matching

• The process of mapping entities in a purchased dataset to the firm's internal identifiers
• Example: mapping "apple.com" to Apple's official financial identifiers
• Point-in-time accuracy is mandatory to prevent look-ahead bias — identifiers are only valid for specific time periods, and they can change (e.g., due to mergers and acquisitions)
• Datasets are often multiple terabytes, requiring distributed computing and fast transformation tools such as Polars, Spark, or NumPy

Data Loaders

• Each vendor has a unique data loader that pulls updated data, performs transformations and aggregations, and saves it to the firm's internal database
• Data loaders are production-critical: if a loader fails, the trading factor depending on it is halted
• Update frequencies vary by vendor: daily, weekly, or monthly

Data Auditing

• Cannot rely on vendors to provide consistently correct data
• Auditing systems use statistical metrics on key columns and coverage checks on securities in each new data file
• Discrepancies beyond a threshold trigger alerts for investigation
• Goal: identify data errors as early and as far upstream as possible

---

Productionizing Trading Factors

Research-to-Production Handoff

• Senior researchers develop factor methodology; quant developers convert research code (typically written in R) into production-ready code in Python, Spark, or KDB
• KDB is a legacy but highly optimized language and database widely used at hedge funds
• Distributed computing and vectorization (NumPy) are essential since every factor must be run over 5–15 years of historical data

Production Timing Constraints

• Multi-factor models update every trading day
• For global equity, there is a narrow window between the New York market close and the Japan market open to run the full model on fresh data
• Distributed computing is not optional — it is what makes this window achievable

Documentation and Team Structure

• Each project produces a Word document detailing all research methodology — serves as the central reference for everyone on the project
• Projects typically involve a researcher, developer, and tester, plus senior partners providing oversight
• Factors always decay over time; portfolio managers decide whether a factor warrants a new research cycle to improve performance
• High-performing, well-coordinated pods are kept together by management for future projects

---

Cloud Infrastructure and Apache Spark

Migration from On-Premises to Cloud

• The firm originally ran everything on on-premises servers; growing team size, data volumes, and compute demands forced a cloud migration
• Primary tools: AWS (EC2, ECR, S3) and Databricks (notebooks, scheduled workflows)

Apache Spark Architecture

• A distributed, in-memory data processing engine designed to spread computations across a cluster of machines working in parallel
• Key components:
  • Driver: the program brain — holds the application logic and builds the execution plan
  • Executors: worker nodes distributed across the cluster that process data partitions in parallel
  • Cluster manager: allocates machines to the job
  • Partitions: the dataset is split into chunks; each executor works on its own partitions simultaneously

Lazy Evaluation and the Catalyst Optimizer

• Spark does not execute transformations immediately — it records each requested operation and builds a directed acyclic graph (DAG) of steps
• Execution only begins when an action is triggered (e.g., writing results, counting rows)
• This laziness enables Spark's Catalyst query optimizer to inspect the entire plan before running anything, applying optimizations such as:
  • Pushing filters down to the data source so less data is read
  • Reordering joins for efficiency
  • Eliminating unnecessary computation

In-Memory Processing

• Unlike the older MapReduce model, which wrote intermediate results to disk between every step, Spark keeps data in memory across the entire pipeline
• This is the primary reason Spark is often an order of magnitude faster for multi-step analytical jobs

Fault Tolerance via Lineage

• Spark tracks the complete lineage (the recipe of transformations) that produced any piece of data
• If a machine fails and a data partition is lost, Spark recomputes only that chunk from its lineage rather than failing the entire job

The Shuffle Problem

• Operations like joins and group-bys require data residing on different machines to be physically moved across the network so related rows end up together
• This movement — the shuffle — is the most expensive operation in Spark
• Most Spark optimization work comes down to minimizing and controlling shuffles

A Practical Spark Example: Processing Alternative Data

A concrete end-to-end example illustrates how these concepts apply in production:

1. Read: Spark reads several terabytes of alternative data stored as partitioned files in Amazon S3, spreading partitions across executors
2. Filter push-down: Because of lazy evaluation, date and column filters are pushed all the way down to the source — only the needed data is actually read
3. Parallel transformation: Each executor independently cleans, aggregates, and computes features on its own partitions simultaneously
4. Broadcast join: When attaching internal security identifiers, the mapping table is small enough to be broadcast to every executor, so the join happens locally with no shuffle
5. Write: Output is partitioned by date and written into a Delta Lake table, ready for the trading factor to consume
6. Result: A historical computation that would take days on a single machine completes in hours

---

Storage Architecture: Where Data Lives

The choice of storage technology is not an afterthought — it directly determines system performance. Two database types dominate quant infrastructure and are asked about in roughly 80% of quant developer interviews.

Parquet and Delta Lake: The Research and Batch Warehouse

Parquet: Columnar File Format

• Unlike row-oriented databases (which store all fields of a record together), Parquet stores each column together
• For quant workloads — which typically touch a few columns across millions of rows — this yields major benefits:
  • Only the requested columns are read from disk
  • Compression is excellent because similar values are stored adjacently
  • Entire chunks of a file can be skipped if they cannot match the filter (predicate pushdown)

Delta Lake: Transactions on Top of Parquet

• A plain folder of Parquet files has no mechanism for consistent, all-or-nothing changes
• Delta Lake adds a transaction log that provides full ACID guarantees:
  • Atomicity: a write either fully completes or does not happen at all
  • Consistency/Isolation: readers never see a half-written table; concurrent jobs cannot corrupt each other
  • Durability: committed data persists
• Delta also enforces schema, preventing bad data from silently entering the system

Why This Combination is Ideal for Quant Data

• Data is partitioned by date: queries for a specific historical window scan only the relevant date partitions and only the needed columns — nothing else
• Daily updates are simple, cheap appends of a new date partition
• Batch computation is a single large parallel scan — highly efficient

Time Travel: The Most Valuable Feature

• Because Delta maintains a complete transaction log, tables can be queried as they appeared on any past date
• This directly enforces point-in-time accuracy: when running a factor over history, the data reflects what was known at that moment, not how it has since been restated — preventing look-ahead bias
• Makes historical runs fully reproducible
• If a vendor restates historical data, only the affected partitions need to be overwritten — not the entire table

Scalability

• Everything sits on cheap object storage (e.g., S3), and read/write throughput scales with cluster size rather than being bottlenecked by a single machine

---

KDB and Q: The High-Performance Time Series Engine

What KDB Is

• An in-memory, columnar time series database built specifically for the kind of data finance generates: enormous, continuous streams of timestamped ticks and quotes
• Considered old and niche, but still widely used at hedge funds because nothing matches its raw speed on timestamped data

The Q Language

• KDB's query language is terse and vectorized — small expressions operate on entire columns at once, similar to NumPy
• Runs extremely close to the hardware, minimizing overhead

The As-Of Join: KDB's Killer Feature

• The as-of join — for every trade, finding the quote that was in effect at that exact moment — is ubiquitous in finance and painfully slow in conventional databases
• In KDB it is a first-class, lightning-fast primitive, which is the primary reason firms continue to use it

Production Usage

• At the presenter's firm, KDB served as the backbone of the live trading model
• It was paired with HTCondor, a job scheduler that distributes computational work across a grid of machines
• KDB held and served time series data at speed; HTCondor distributed the model computation on top of it

---

Two Databases, Two Different Jobs

	Parquet + Delta Lake	KDB
Best for	Research, batch processing, historical storage	Live trading, real-time time series
Key strength	Cheap, massive, versioned, parallelizable	Extreme speed on timestamped data
Scale model	Scales out across cluster	In-memory, runs close to the metal
Typical use	Factor research, historical runs, data warehouse	Tick data, live model execution

Most firms use both, because they solve fundamentally different problems.

---

The Unified Picture

Every component described here is one block of the same connected system:

• The data block expands into security matching, data scrubbing, and auditing
• The alpha block expands into the research-to-production pipeline
• Underneath everything sits the infrastructure and databases — Spark for computation, Parquet/Delta for historical storage, KDB for live execution — that make the entire model run within the tight daily production window

The defining characteristic of working inside a quant firm is that every piece of tooling exists for a precise reason within this larger machine. Understanding how the parts connect is what separates effective quant developers from those who only understand their own isolated component.