Huu Tai Le Rust data infrastructure for Solana

For 158.9 hours my indexer reported perfect health and wrote empty rows.

Containers up, checkpoints advancing, four-second median lag, rows arriving on schedule. Nothing in the monitoring suite asked whether the data was right, so 64,650 rows went missing in silence. Everything I have built since answers the question liveness monitoring structurally cannot.

Open to Rust data engineering and Solana infrastructure roles.taihl.work@gmail.com

Dataflow of the Kamino Lend indexer and its integrity checker Solana mainnet streams account updates over Yellowstone gRPC at 7.45 KB per second into a Rust ingest service, which writes batched inserts into ClickHouse. The karat checker samples indexed rows, refetches them from Solana RPC with getMultipleAccounts and getBlocks as the source of truth, and returns a non-zero exit code to the CI gate when any check does not pass. Solana mainnet the source of truth Yellowstone gRPC 7.45 KB/s ingest (Rust, tokio) discriminators derived at runtime, write timeout batched inserts ClickHouse, hot store 1.04M updates, 140K positions, 11 days sampled rows karat, three checks shape, reconcile, completeness Fail and Skipped both fail re-read the chain getMultipleAccounts getBlocks byte-diff stored against on-chain exit code CI gate, or pipeline step
The system as it runs today. The lit path is the part most indexers leave out: re-reading the chain to check what was stored against what is actually there.
karat67Sep 2026 –
present

Rust, MIT
open source

An indexer can be perfectly live and completely wrong. karat67 is the check for that.

An open-source Rust crate and CLI that tests indexed Solana data against the chain itself. Three checks ship, each returning one uniform result and a non-zero exit code on anything short of a pass, so it drops into a pipeline step or a CI job with no wrapper around it.

  • Shape. Validates account data length and the 8-byte Anchor discriminator against the program IDL. This is the failure class that emptied my own Obligation rows while every dashboard stayed green.
  • Reconciliation. Samples indexed accounts, refetches them over getMultipleAccounts, and byte-diffs stored against on-chain. Slot-lag tolerance is explicit: a row inside the window is reported Skipped with its reason, never quietly passed.
  • Completeness. Compares indexed slots against getBlocks rather than a contiguous range, so slots the leader skipped are not miscounted as data loss.

Design choices that matter to whoever inherits this: ureq and rustls instead of the full solana-client stack, so a build needs a Rust toolchain and nothing else, no system OpenSSL. An AccountFetcher trait with a mock implementation, so reconciliation tests run offline and deterministically. A KaratIntegrityProcessor that sits after account decode in a Carbon pipeline, adding checks to an existing indexer without depending on the Carbon crate or rewriting the pipeline.

$ karat shape --account-type UserMetadata --len 1032
{
  "check": "shape(UserMetadata)",
  "status": "Pass",
  "detail": null
}
exit 0

$ karat shape --account-type UserMetadata --len 0
{
  "check": "shape",
  "status": "Fail",
  "detail": "empty payload: 0 bytes, every allocated
             account starts with an 8-byte discriminator"
}
exit 1  the 158.9-hour incident, in one call
github.com/SGDlab67/karat67 Upstream proposal: Carbon #591
Rust 2024clapserdeureq + rustlsSolana JSON-RPCAnchor IDLCarbon adapterCI exit codes
klend-indexerJul – Aug 2026

Frozen dataset
on GCS

The pipeline that taught me the difference between flowing and correct.

A continuous Rust indexer streaming Kamino Lend account state from Solana mainnet over Yellowstone gRPC into ClickHouse, then out again into columnar cold storage. It ran unattended on GCP and it broke in two instructive ways.

1.04M
account updates ingested
140K
decoded lending positions
4 s
median ingest lag
11 days
unattended in production
$548 to <$1
monthly analytical storage
50
unit and regression tests

Architecture

a hot path for freshness, a cold path for history
  • Yellowstone gRPC into a tokio ingest service, batched inserts into ClickHouse for the live window. Anchor discriminators are derived at runtime from the account name rather than transcribed as constants, so a program upgrade cannot silently invalidate a hardcoded byte string.
  • Slot-gap detection keyed to a provider-independent clock instead of a contiguous slot range, which keeps it correct under any provider replay or backfill behaviour.
  • A second, cold analytical path built to escape both the ReplacingMergeTree FINAL read ceiling and a managed-OLAP bill driven by write cadence rather than data volume: 4.25 GB of warehouse history exported into 91 MB of Hive-partitioned Parquet on Google Cloud Storage in 65 seconds, served through DataFusion. A gap audit re-derived from the Parquet matched the live detector exactly.
  • Chose Alchemy pay-as-you-go at $1–$2 per month after comparing Alchemy, Helius, and Triton on cost and fit against $499–$2,900 per month plans, which is the difference between a project that runs for a year and one that gets switched off.

Incident one

an 8.4-hour silent stall
  • Traced to a database write with no timeout sitting on a half-open connection. The task was not dead, it was blocked forever, which no liveness probe reported.
  • Shipped an insert timeout plus an external freshness watchdog outside the process, because a process cannot be the sole judge of its own health.

Incident two

64,650 rows lost across 158.9 hours, every signal green
  • An accounts_data_slice request was returning empty payloads. Rows kept arriving on schedule, so freshness, throughput, and liveness all read healthy while the payloads carried nothing.
  • Found by a distribution audit on the data itself, not by a monitor. Wrote regression tests that replay both incidents, plus boundary cases including off-by-one gap tolerance, 128-bit overflow, and malformed account data.
  • This incident is the entire reason karat67 exists, and why its shape check refuses to treat a zero-length payload as anything but a failure.
github.com/SGDlab67/klend-indexer Frozen dashboard
Rust, tokioYellowstone gRPCClickHouseApache ArrowParquetDataFusionDockersystemd timersGoogle Cloud
On-chainFeb – Aug 2026

Anchor, Token-2022
Solidity, Foundry

I read indexed account layouts the way I do because I have written the programs that produce them.

Deployed programs on Solana and Ethereum testnets, each with the client that talks to it. Knowing what a program guarantees on-chain is what tells you which invariants an indexer is allowed to assume.

Mentik World

staking pool and Token-2022 economy, Solana devnet
  • A non-custodial SOL staking pool using a u128 fixed-point reward index with Flexible, 7-day, and 30-day lock tiers, plus an in-place account-layout migration for stake accounts created before the change, executed without asking users to withdraw.
  • Launched the MENTIK token on Token-2022 with a Metadata Pointer, a Permanent Delegate, a PDA mint authority, and an admin-set SOL rate held on-chain, exposing buy, force-transfer, and force-burn instructions.
  • Built the React 19 and Next.js client with Phantom and Solflare flows, holding a strict client-server boundary so no amount of client state can mint value on-chain.
SourceLive on devnet

Decentralized stablecoin

Solidity, deployed and verified on Sepolia
  • An overcollateralized ERC-20 whose only owner is the engine, so no wallet controls supply, paired with a DSCEngine that holds wETH and wBTC collateral and enforces 200% overcollateralization.
  • Chainlink price feeds read through an OracleLib that reverts on stale data rather than transacting on a frozen number, then tested with unit, fuzz, and invariant suites in Foundry using mocks for failed transfers and stale feeds.
Source

Real-world asset vault

Solana Developer Bootcamp 2026 cohort project
  • A PDA-isolated vault program in Rust and Anchor on devnet, with deposit and withdrawal paths and user-isolated authority derivation.
  • Wired to a Next.js and TypeScript client through a Codama-generated type-safe SDK, so the client surface is regenerated from the IDL rather than maintained by hand.
ResearchAug 2026 –
present

Oregon State
University

Measuring Solana's validator gossip network on a schedule.

An independent Rust observatory that samples Solana's gossip network on scheduled runs, built with research direction from Dr. Zane Ma at Oregon State University.

  • Checking Anza's protocol documentation against Agave validator source and recording where documented behaviour has drifted from what the code actually does, which is the same discipline as the indexer work: trust the system of record, not the description of it.
  • Collection runs and repository maintenance are driven by a self-hosted agent on AWS Lightsail running open-weight models, with runtime stability, API spend, and job outcomes monitored across network timeouts.
RustSolana gossip protocolAgave sourcescheduled collectionPythonAWS Lightsail
Stackwhat I reach
for, and why

Tools I have shipped something real with.

Languages
Rust, Python, TypeScript, JavaScript, Solidity, SQL, Bash
Data and storage
ClickHouse, Apache Arrow, Parquet, Apache DataFusion, Yellowstone gRPC, DuckDB
Testing and quality
Unit, integration, regression, fuzz, and invariant testing; boundary and negative-case design; data-quality audits; root-cause analysis; monitoring and alerting
Solana
Anchor, Pinocchio, Token-2022, PDAs and CPI, SPL, Web3.js, Codama IDL generation, JSON-RPC, Carbon
Ethereum
Solidity, Foundry, Ethers.js, Chainlink price feeds
Infrastructure
Docker, systemd timers, Google Cloud and GCS, AWS Lightsail, Git and GitHub, Linux (Ubuntu)
Frontend
Next.js, React 19, Tailwind CSS, Zustand, @solana/react-hooks, FastAPI
Backgroundeducation and
training

Education, programs, certifications.

2026 – 2028 expected
Oregon State University. M.S. Computer Science. Coursework in data structures and web development.
2022 – 2026
University of South Florida. B.S. Hospitality Management, minor in Finance. The finance side is what makes lending-protocol data legible rather than abstract.
Spring 2026
Solana Developer Bootcamp 2026. Solana Foundation official cohort. Low-level Solana architecture, Rust compute-unit optimization, advanced state management in Anchor.
Summer 2026
Solana Summer School. Led by Andres Correia. Token-2022 extensions including metadata pointer, permanent delegate, and transfer hooks; escrow and vault internals compared across Anchor and Pinocchio.
Certifications
Solidity Smart Contract Development, Chainlink Fundamentals, Foundry Fundamentals, Advanced Foundry, Blockchain Basics
2024
Tan Le Quang Jewelry. Sales and digital marketing apprentice. Built and deployed tanlequangjewelry.com as a discovery-to-visit funnel and grew social traffic 30% with content aimed at store visits rather than online checkout.
ContactCorvallis, OR
United States

If your pipeline's health signals and your data have ever disagreed, we should talk.

Looking for Rust data engineering and Solana infrastructure work: indexers, warehouses, columnar storage, and the correctness layer around all three. Happy to walk through either incident in detail.

taihl.work@gmail.com Résumé (PDF) GitHub LinkedIn