OpenAIR Data Ingestion & Processing Pipeline
Back to projectsData engineeringEntity matching, data validation, AI-assisted review and replay-safe writes across several sources.TypeScript · Inngest · PostgreSQL · Drizzle ORM · Vitest Backend & DevOpsThe conflict review API, the team's GitLab CI and security scanning.Next.js · GitLab CI · Vercel · Semgrep · Gitleaks · npm audit
OpenAIR Data Ingestion & Processing Pipeline
In DevelopmentTypeScriptNext.jsInngestPostgreSQLDrizzle ORMVitestGitLab CI
An ingestion pipeline for WA AI Hub that keeps a register of AI systems accurate when its sources update and disagree.
- Client
- Josh Tedjasaputra, Co-Founder & Chair · WA AI Hub
- Role
- Owned matching, conflict review, the write path and CI/CD
- Team size
- 5
- Period
- Sep – Nov 2026
Data engineering
- No silent merges of different systems: replaced fuzzy vendor matching with identity-first matching and stored aliases. Cases the rules can't settle go to a reviewer instead of being guessed.
- AI-assisted review of ambiguous matches: an optional LLM judge assesses the matches the rules leave open. Its answer is shown to the reviewer as evidence and never merges records on its own.
- Safe to re-run: retried or repeated runs don't create duplicate entries or alerts, and no register change is saved without its decision history.
- Failures surfaced, not hidden: bad records are rejected one by one with a reason, and a run with nothing usable is marked failed instead of reported as an empty success.
Backend & DevOps
- Reviewer decisions that stick: when sources disagree, neither value is overwritten. A reviewer settles it through an admin API, the decision is recorded with who made it, and later runs respect it.
- Every merge request type-checked, built and tested: built the team's GitLab CI pipeline, with tests against a real Postgres and Vercel deploy jobs.
- Vulnerable dependencies block the merge: added SAST, secret detection and a dependency gate that fails the pipeline on unreviewed vulnerabilities in production dependencies, then patched 6 critical and high ones, including an RCE in Next.js.
The code is not public. I’m happy to walk through the design in person though :).






