How We Build AI

A Plazo Sullivan Roche Capital Field Manual for Operators Who Actually Ship

​

Every quarter, a hedge fund or bank walks into a room, announces they are "building an AI platform," and walks out eighteen months later with a Jupyter notebook, a burned-out data science team, and a PowerPoint deck describing what could have been.

​

Here is the uncomfortable truth Plazo Sullivan Roche Capital tells every founder and CTO before they spend a dollar: financial AI does not fail on the model. It fails on the architecture of decision-making around the model. The math is usually fine. The governance, the data pipeline discipline, the hardware sizing, and the feedback loop with the market are where capital goes to die.

​

The market does not reward effort. It rewards correctly placed pressure. And in financial AI, pressure is placed in exactly five spots: strategy, framework, infrastructure, team quality control, and public-facing validation. Miss one, and the other four collapse under it like a table with a short leg.

​

This guide is the full blueprint, the one we hand to portfolio companies before they touch a GPU cluster or hire a single quant. It is long because the topic deserves it. Read it once for the map, then again as a checklist while you build.

​

The Opportunity: Why Now, Why This Matters

​

Financial AI in 2026 is no longer a research curiosity, it is a capital allocation weapon. Alternative data ingestion, LLM-driven research synthesis, agentic execution layers, and real-time risk modeling have moved from "innovation lab" to "core infrastructure" at every serious fund, bank, and fintech. The firms sitting this out are not being conservative. They are being outcompeted in latency, insight generation, and cost structure.

​

What's changed:

​

• Compute got cheap enough that mid-sized funds can run models that used to require a Renaissance-sized balance sheet.

• Foundation models commoditized the "intelligence" layer, which means the actual moat has shifted to data engineering, workflow design, and domain-specific fine-tuning — exactly the layers most teams neglect.

• Regulators are watching closely, which means firms that build with governance baked in from day one will out-license, out-audit, and out-partner the ones bolting it on later.

 

What beginners misunderstand: they think financial AI is a modeling problem. It is a systems problem wearing a modeling costume.

​

What sophisticated operators see: the winning firm isn't the one with the smartest model. It's the one with the tightest loop between hypothesis, data, execution, and feedback repeated faster and more disciplined than anyone else in the market.

​

Who This Guide Is For

​

Best fit:

​

• Fund managers, fintech founders, or bank innovation leads with genuine P&L or capital-allocation authority, not just a mandate to "explore AI."

• Teams with at least one person who understands market microstructure and one who understands ML infrastructure. You need both fluencies in the room, not just one.

• Operators comfortable making six-figure infrastructure decisions before there is a finished product to point to.

 

Resources required:

​

• Capital: realistically $250K–$2M for a serious first build (compute, data licensing, senior hires), scaling from there.

• Time: 6–12 months to a defensible production system; expect 90 days just to get the data layer trustworthy.

• Risk tolerance: high on process, disciplined on capital deployment — you're comfortable killing a model that doesn't validate, but not comfortable skipping validation.

 

Warning signs this isn't for you yet:

​

• You want a chatbot for client FAQs. That's a vendor problem, not this playbook.

• You don't have clean historical data and no budget to acquire or clean it. Fix that first.

• Your organization treats "AI" as a marketing line item rather than an infrastructure investment. You will burn the budget and produce nothing defensible.

 

Mindset and Strategic Foundation

​

Think of financial AI not as a product you build once, but as a factory that manufactures validated hypotheses at increasing speed. The output isn't a model. The output is a repeatable process for turning a market question into a tested, deployable, monitored decision system.

​

What to obsess over: data lineage, latency budgets, and the feedback loop between model output and real-world outcome.

 

What to ignore: which foundation model is "best" this month. That's noise. The frontier model race is not your race; your race is turning whatever model you use into something that knows your market better than a generic API call ever could.

​

A bad offer is a beautiful restaurant with no door. A bad financial AI system is a brilliant model with no pipeline feeding it clean data and no downstream process acting on its output. Most beginners build a model. Professionals build a decision infrastructure — the model is one component in a machine that repeatedly turns data into signal, and signal into capital efficiency.

​

Common traps:

​

• Chasing model sophistication before data integrity is solved.

• Letting the data science team operate without a risk officer or compliance partner in the room from week one.

• Treating "backtested well" as equivalent to "production ready."

 

Avoid shiny-object syndrome by anchoring every technical decision to one question: does this reduce the time between hypothesis and validated, monitored deployment? If a tool, model, or framework doesn't answer yes, it's a distraction dressed as progress.

​

The Step-by-Step Execution Plan

​

Step 1: Set the Big Picture and Strategy

​

Objective: Define precisely what problem the AI system solves and how success will be measured before any code is written.

 

Why it matters: Financial AI initiatives die most often not from bad models but from scope that expands to fill available compute. A team without a locked strategic thesis will build five half-finished systems instead of one production-grade one.

 

Exact actions:

​

1. Write a one-page strategic charter answering: What decision does this AI make or support? Who acts on its output? What is the cost of a false positive vs. a false negative in this specific use case?

2. Classify the initiative into one of three buckets: alpha generation (research/signal discovery), risk and compliance (monitoring, fraud, exposure), or operational efficiency (execution, reconciliation, client service). Each bucket has different latency, accuracy, and governance requirements — conflating them is the single most common strategic error.

3. Define your kill criteria in writing before you start: the specific performance thresholds under which this project gets shut down, not iterated on forever.

4. Get sign-off from compliance, risk, and the P&L owner in the same room — not sequentially. Sequential sign-off is how strategy gets diluted three times before deployment.

 

Tools: Notion or Confluence for the charter; a simple decision-matrix template (we use a weighted scoring model across impact, feasibility, and regulatory exposure).

​

Time estimate: 2–3 weeks. Cost estimate: Largely internal time; budget for a compliance/legal consult if internal counsel isn't AI-fluent (~$10K–$25K).

​

Mistakes to avoid: Starting with "let's see what the model can do" instead of "here is the decision we need this model to improve."

​

Success metric: A one-page charter every stakeholder has signed, with explicit kill criteria.

Deliverable: Strategic Charter Document.

​

Step 2: Establish the Framework

​

Objective: Build the governance and architectural skeleton the entire system will hang on — before a single model is trained.

 

Why it matters: In financial AI, the framework is the moat. Two firms can use the same foundation model and get wildly different outcomes because one has a framework that enforces data lineage, versioning, and auditability, and the other doesn't.

​

Exact actions:

​

1. Define your data governance layer first: source-of-truth registry, data lineage tracking, and a versioning system (tools like DVC or LakeFS) so every model prediction can be traced back to the exact data snapshot that produced it.

2. Build a model risk management (MRM) framework aligned to whatever regulatory regime applies (SR 11-7 style model risk principles are a strong default even outside the U.S. banking context) — documentation standards, validation requirements, and a model inventory from day one.

3. Choose your experimentation framework: MLflow or Weights & Biases for experiment tracking, so every hypothesis tested is logged, comparable, and auditable.

4. Define your latency and deployment tiers upfront — batch research models, near-real-time risk models, and ultra-low-latency execution models require entirely different architectures. Decide which tier you're building for before choosing infrastructure.

 

Tools: MLflow (experiment tracking, essential), DVC/LakeFS (data versioning, essential), a model inventory spreadsheet or dedicated MRM platform like ValidMind (essential for regulated environments).

​

Time estimate: 4–6 weeks running in parallel with Step 1's later phases. Cost estimate: $15K–$60K depending on whether you build governance tooling in-house or license a platform.

​

Mistakes to avoid: Treating governance as a compliance afterthought bolted on before an audit. It should be load-bearing infrastructure, not decoration.

​

Success metric: Every model artifact traceable to source data, code version, and the human who approved deployment.

Deliverable: A documented Model Risk Framework and versioned data/experiment pipeline.

​

Step 3: Identify the Key Hardware and Software Stack

​

Objective: Match your infrastructure precisely to your latency tier and budget — not to what's trending on tech Twitter.

 

Why it matters: Overbuying GPU infrastructure for a batch research use case is one of the most common capital-destroying mistakes in this space. Underbuying for a real-time execution system is the other.

​

Exact actions:

​

1. For research and alpha discovery: cloud GPU (A100/H100 instances on AWS, GCP, or a specialized provider like Lambda Labs or CoreWeave) rented on-demand. Do not buy hardware for this tier — rent until usage patterns justify reserved instances.

2. For risk and compliance monitoring: mid-tier CPU/GPU hybrid infrastructure with strong uptime SLAs; prioritize reliability over raw throughput.

3. For execution-adjacent, low-latency systems: colocated or on-premise infrastructure becomes justified — this is the one tier where owning hardware near exchange infrastructure can matter.

4. Software stack baseline: Python (PyTorch or JAX for modeling), a feature store (Feast or Tecton) to prevent training/serving skew, a vector database (Pinecone, Weaviate, or pgvector) if you're doing retrieval-augmented research synthesis, and Kubernetes for orchestration once you're past prototype stage.

5. Build vs. buy decision: license foundation model APIs (Anthropic, OpenAI) for general reasoning and research synthesis tasks; fine-tune smaller open-weight models only where you have proprietary data that creates real differentiation.

 

Tools: AWS/GCP/CoreWeave (compute, essential), Feast (feature store, essential once beyond prototype), Pinecone/pgvector (retrieval, situational), Kubernetes (orchestration, essential at scale, optional at prototype).

​

Time estimate: 3–4 weeks to select and provision; ongoing optimization thereafter. Cost estimate: $30K–$300K+ annually depending on tier and scale — this is the widest variance line item in the entire build.

​

Mistakes to avoid: Buying hardware before your latency requirements are proven with real usage data. Rent first, always.

Success metric: Infrastructure cost per validated hypothesis trending down quarter over quarter.

​

Deliverable: A provisioned, documented technical stack matched to each use-case tier.

​

Step 4: Establish the Team and Quality Control

​

Objective: Build a team structure where no single point of failure — technical or judgment-based — can quietly corrupt output.

​

Why it matters: The single most expensive failure mode in financial AI is a brilliant, isolated quant whose model nobody else can validate, review, or challenge.

​

Exact actions:

​

1. Structure the team around three non-negotiable roles even at small scale: a quantitative/ML lead (builds), an independent model validator (someone who did not build the model, stress-tests it), and a domain expert (trader, risk officer, or analyst who understands what "reasonable output" looks like in this market).

2. Implement a four-eyes principle: no model goes to production without sign-off from someone other than its builder.

3. Establish a quality control cadence: weekly model performance reviews, monthly drift analysis, quarterly full model revalidation.

4. Build a red-team function, even if it's one person 20% of the time — their job is to actively try to break the model with adversarial inputs, edge-case market regimes, and data poisoning scenarios.

 

Tools: ValidMind or a custom validation dashboard (essential for MRM); Slack/Notion for review cadence documentation; Evidently AI or WhyLabs for automated drift detection (essential once in production).

​

Time estimate: Ongoing from month one; expect 2–3 months to fully staff the independent validation function. Cost estimate: Senior ML hire $180K–$350K; independent validator $150K–$250K; can be fractional/consulting initially.

​

Mistakes to avoid: Letting the builder also be the validator "temporarily" — temporary always becomes permanent, and it's how blown-up models get to production.

​

Success metric: Zero production deployments without independent sign-off, tracked as a hard compliance gate.

Deliverable: A staffed team with documented, enforced validation gates.

​

Step 5: Execution, Testing, and Public Feedback

​

Objective: Deploy in a way that generates real-world validation quickly, safely, and with a feedback loop back into the model.

 

Why it matters: A model that performs well in backtest and never gets deployed teaches you nothing. A model deployed carelessly teaches you an expensive lesson. The goal is controlled exposure that generates signal.

​

Exact actions:

​

1. Deploy first in shadow mode: the model runs live, generates predictions, but does not act on capital. Compare its output to actual outcomes for a defined period (minimum 4–8 weeks for most use cases).

2. Move to limited live deployment with strict capital or exposure caps, monitored daily.

3. Build a public or client-facing feedback channel where appropriate — a research note distribution, a client dashboard, or a limited external pilot. Public feedback (from LPs, clients, or counterparties) surfaces blind spots your internal team is too close to see.

4. Formalize the feedback loop: every deployed prediction gets logged, every outcome gets reconciled against it, and drift/error analysis feeds back into the next model iteration on a fixed schedule, not ad hoc.

 

Tools: Evidently AI or WhyLabs (monitoring, essential), a client-facing dashboard (Retool or custom build, situational), Airtable or a lightweight CRM to track qualitative feedback from pilot users.

​

Time estimate: 8–12 weeks shadow-to-limited-live; ongoing feedback loop thereafter. Cost estimate: $20K–$50K for monitoring tooling and dashboard build.

​

Mistakes to avoid: Skipping shadow mode because "the backtest looked great." Backtests lie by omission — they cannot show you what the model does in a regime they didn't sample.

​

Success metric: Model performance in live/shadow mode within an acceptable variance band of backtested expectations; if it isn't, that gap is your most valuable data point.

​

Deliverable: A live-monitored deployment with a documented, recurring feedback-to-retraining loop.

​

Tools, Software, and Resources — By Category

​

Research & Data: Bloomberg/Refinitiv terminals (essential for market data), Alpha Vantage or Polygon.io (cost-effective alternative data), DVC/LakeFS (data versioning, essential).

​

Strategy & Governance: ValidMind (model risk management, essential in regulated environments), Notion/Confluence (documentation, essential).

​

Modeling: PyTorch/JAX (essential), Feast (feature store, essential at scale), MLflow/Weights & Biases (experiment tracking, essential).

​

Infrastructure: AWS/GCP/CoreWeave (compute, essential), Kubernetes (orchestration, essential at scale, skip at prototype stage).

​

Monitoring & QC: Evidently AI, WhyLabs (drift detection, essential post-deployment).

​

Communication & Feedback: Retool (internal/client dashboards, optional but high-leverage), Airtable (qualitative feedback tracking, optional).

​

Compliance: Specialized fintech compliance counsel (essential, not optional, in any regulated deployment).

​

Offer and Revenue Architecture

​

If you're building this as a platform or service rather than purely internal infrastructure, package it in tiers:

​

• Entry-level: A research-augmentation tool for smaller funds — signal synthesis and risk flagging, $2K–$10K/month per seat.

• Premium: Fully integrated risk monitoring with custom model validation support, $25K–$100K/month per client.

• Recurring/platform: Licensed infrastructure with ongoing model monitoring-as-a-service, $50K–$500K annually.

• Done-for-you: Full build-and-operate engagement for a fund without internal ML capability, $500K–$3M project fee.

• Advisory: Fractional CTO/quant advisory on AI strategy and governance, $15K–$40K/month.

• High-ticket: White-glove build for a large institution with bespoke regulatory requirements, $2M–$10M+ engagements.

 

Revenue Scenarios (Illustrative, Not Guaranteed)

​

Conservative: 3 advisory clients at $20K/month = $720K annually. Assumes slow sales cycle, high-touch delivery.

 

Moderate: 2 platform clients at $75K/month + 5 advisory clients at $20K/month = $2.7M annually. Assumes credible case study exists.

​

Aggressive: 1 done-for-you build ($1.5M) + 4 platform clients at $150K/month = $8.7M annually. Requires a proven, referenceable deployment.

​

Premium/high-performance: One flagship institutional engagement at $5M+ plus recurring monitoring revenue. Requires a public or semi-public track record and regulatory credibility.

​

What must be true: a working, validated, monitored deployment you can reference — without that, none of these numbers are reachable regardless of how good your sales deck is.

​

Marketing and Deployment Timeline

​

First 24 hours: Finalize the strategic charter and kill criteria; brief compliance and risk stakeholders.

First 7 days: Provision compute, stand up data versioning, begin independent validator hiring search.

First 30 days: Complete the governance framework; begin model development in shadow-only mode.

First 90 days: Shadow-mode results reviewed; go/no-go decision on limited live deployment.

First 12 months: Full feedback loop operational, first external reference client or pilot secured, revenue model tested against real client conversations.

​

Positioning line for market-facing communication: "We don't sell you a model. We sell you the machine that keeps your model honest."

​

Common Mistakes

​

Mistake: Building the model before the governance framework. Fix: Framework first, always — it's cheaper to build early than retrofit under audit pressure.

Mistake: Letting the builder validate their own model. Fix: Hard four-eyes rule, no exceptions, no "just this once."

Mistake: Skipping shadow mode. Fix: Minimum 4–8 weeks of shadow deployment before any capital is at risk.

Mistake: Over-provisioning hardware before usage patterns justify it. Fix: Rent compute until data proves you need to own it.

​

Advanced Strategies

​

The real moat isn't the model — it's proprietary data lineage and a validation cadence competitors can't easily replicate. Pair that with pricing power that comes from regulatory credibility (being the vendor an institution's compliance team already trusts), and you have distribution advantages that compound.

 

Final Playbook

​

The big idea: financial AI is not a modeling exercise, it's a governed manufacturing process for validated decisions. The fastest path to execution is strategy → framework → stack → team → shadow deployment, in that exact order, never skipped. The first move today: write the one-page strategic charter with explicit kill criteria and get it signed by every stakeholder before touching infrastructure.

​

The firms that win this decade won't be the ones with the smartest model. They'll be the ones whose machine never stops learning from the market's answer.