OpenAIR Data Ingestion & Processing Pipeline

Back to projects
OpenAIR Data Ingestion & Processing Pipeline
In Development
TypeScript/Next.js/Inngest/PostgreSQL/Drizzle ORM/Vitest/GitLab CI

An ingestion pipeline for WA AI Hub that keeps a register of AI systems accurate when its sources update and disagree.

Data engineering

  • No silent merges of different systems: replaced fuzzy vendor matching with identity-first matching and stored aliases. Cases the rules can't settle go to a reviewer instead of being guessed.
  • AI-assisted review of ambiguous matches: an optional LLM judge assesses the matches the rules leave open. Its answer is shown to the reviewer as evidence and never merges records on its own.
  • Safe to re-run: retried or repeated runs don't create duplicate entries or alerts, and no register change is saved without its decision history.
  • Failures surfaced, not hidden: bad records are rejected one by one with a reason, and a run with nothing usable is marked failed instead of reported as an empty success.

Backend & DevOps

  • Reviewer decisions that stick: when sources disagree, neither value is overwritten. A reviewer settles it through an admin API, the decision is recorded with who made it, and later runs respect it.
  • Every merge request type-checked, built and tested: built the team's GitLab CI pipeline, with tests against a real Postgres and Vercel deploy jobs.
  • Vulnerable dependencies block the merge: added SAST, secret detection and a dependency gate that fails the pipeline on unreviewed vulnerabilities in production dependencies, then patched 6 critical and high ones, including an RCE in Next.js.

The code is not public. I’m happy to walk through the design in person though :).

Profile Image of the Author
nonameTAT
olleH :)
Site Statistics
Posts
2
Total Words
3,039
Running Days
0 days
Last Activity
0 days ago
Site Info
Build Platform
GitHub Actions
Blog Version
Firefly v6.16.9
License
All Rights Reserved
Table of Contents