The goal
Keep Nodo.ia's Spanish news section fresh every day without an editor manually curating — fully unattended pipeline that pulls, rewrites, verifies, images, and opens a PR against the content repo, with a weekly analytics digest on top.
The solution
A Node.js pipeline module (src/newsPipeline/) wired into the Nodo.ia backend that runs on node-cron (default 06:15 America/Santo_Domingo), reads articles/categories from GitHub with local/MinIO fallback, fetches RSS candidates, scrapes sources, rewrites with gpt-4o-mini under a verify-and-repair loop, produces webp-optimized images, appends to nodo-lab-articles.json, and opens a PR via GitHub App or PAT. A separate weekly digest job summarizes top content via analytics.
Highlights
- Verify-and-repair LLM loop eliminates most hallucinations before publish
- Editor-owned delivery: PRs instead of direct writes
- Same module deploys to dev/staging/prod unchanged — everything is ENV-driven
- Mobile-optimized .webp images with 150KB budget
The challenge
Naïvely asking an LLM to rewrite a scraped article introduces hallucinations — wrong names, dates, numbers.
Two-step verify-and-repair loop: after the rewrite, a second LLM call verifies claims against the source; failures trigger a repair prompt up to N times, otherwise the article is dropped.
The team wanted automation but not unsupervised writes to the production content repo.
Pipeline opens a PR against the configured base branch (staging in dev, main in prod) with the diff — editors review and merge. Auth via GitHub App (dynamic token) with PAT fallback.
Dev/staging/prod expect slightly different article lists — syncing them manually is error-prone.
Load-order chain GitHub → local → MinIO → defaults, with NEWS_GITHUB_SOURCE_BRANCH aligned to the PR base branch so each env reads its own state.
Raw RSS/scraped images are oversized and slow app header loads.
Every run exports images as .webp with max 1024px width, 576 height, and a 150KB hard cap, uploaded to MinIO (or local in dev) with feature flags to override.
Iterating locally racked up LLM spend and noisy logs.
Mock mode + NEWS_FIXTURE_ITEMS_PATH deterministic fixtures, plus a [news]-only log file for focused debugging.
What was delivered
- src/newsPipeline/ module wired into Nodo.ia backend server lifecycle
- RSS candidate fetcher and source scraper
- LLM rewrite + verify-and-repair layer on gpt-4o-mini
- GitHub PR flow with App/PAT auth and per-env base branches
- .webp image optimizer + MinIO/local storage integration
- Weekly analytics digest job (analyticsService integration)
- Mock/dry-run + fixture support
- README documenting the full NEWS_* env surface
- Dedicated [news] log channel
Results
Daily 06:15
Cadence
GitHub PR/run
Delivery
≤150KB .webp
Image budget
Weekly LLM
Digest
What it taught me
- A verify-and-repair LLM loop is mandatory for any pipeline that publishes to end-users
- Delivering via PR instead of direct writes preserves editor control
- Config-first design (NEWS_* env vars) meant the same pipeline runs unchanged across dev/staging/prod