Building a Self-Healing GitHub Trending API in One Day

What happens when you refuse to accept "the source is down" as an answer. The problem GitHub's trending page is where millions of developers find new projects every day. But GitHub has never shipped an official API for it — if you want that data programmatically, you have to scrape the HTML. Every existing scraper I found had the same design: one source, one point of failure. Small HTML change from GitHub → your service is 503 for a week. Rate limited from Cloudflare → same result. I wanted a trending API that stays up when GitHub's page is down. So I built Hydra, and put it live at hydra9.dev. The architecture (in one picture) ┌───────────────┐ Request ───▶ │ Router │ └───────┬───────┘ │ tries in order, breaker-guarded ┌───────────────┼────────────────┬─────────────────┐ ▼ ▼ ▼ ▼ ┌────────┐ ┌──────────┐ ┌──────────────┐ ┌──────────────┐ │Track A │ │ Track B │ │ Track C │ │ Track D │ │HTML │ │Community │ │Search API │ │gitstar- │ │scrape │ │mirror │ │(created:>7d) │ │ranking.com │ └────────┘ └──────────┘ └──────────────┘ └──────────────┘ Any track dead? Circuit breaker opens, next request routes past it. All tracks dead? Serve most recent Postgres snapshot + fire Prometheus alert. Four independent sources. A track that fails 3 times in a rolling window gets its breaker opened for 60 seconds — the router jumps to the next track without adding latency to the user's request. Why 4 sources and not 2 Two sources feels like enough right up until they share a failure mode. The four I picked are architecturally independent: Track A (github.com/trending HTML): freshest data, but the format changes without warning Track B (ghapi.huchen.dev community mirror): a nice person maintains this, but it can go quiet for hours Track C (GitHub Search API created:>7d sort:stars): the most stable, but it's reconstructed trending — not what appears on the actual page Track D (gitstar-ranking.com): different data model entirely (lifetime stars), useful only as a last-resort cross-check If A breaks because of a GitHub HTML change, B doesn't (it has its own parser). If B breaks because the maintainer's server goes down, C is a completely different infrastructure at GitHub. If C rate-limits you, D is a totally different site. Historical data — the thing GitHub won't give you The official trending page is a snapshot in time. If you refresh, yesterday's list is gone forever. Hydra runs a Celery beat task every 15 minutes that stores a fresh snapshot to Postgres. That unlocks four analytics endpoints the official page can't offer: Repos that showed up today but weren't there yesterday curl https://hydra9.dev/api/v1/trending/new-stars Repos that just fell off the list curl https://hydra9.dev/api/v1/trending/dropped Ranked by stars/hour instead of cumulative fame curl https://hydra9.dev/api/v1/trending/velocity Look at any point in time curl https://hydra9.dev/api/v1/trending/history?at=2026-07-14T00:00:00Z velocity in particular has been my favorite — it surfaces projects that are exploding right now, not projects that were famous 6 months ago and still coasting. The stack (deliberately boring) API: FastAPI (Python 3.11+) Task queue: Celery + Redis (broker on db 1, results on db 2) Storage: PostgreSQL 16 + Alembic HTTP client: httpx (async everywhere) HTML parser: selectolax (faster than BeautifulSoup) Observability: Prometheus + Grafana + 7 alert rules Deploy: Docker Compose (single file) The whole thing is 52 Python files and 47 tests that run in 6 seconds. One lesson: @lru_cache and Pydantic don't mix The single bug that took me from "green in dev" to "500 in prod" was this innocent-looking function: @lru_cache def track_singletons(settings: Settings) -> dict[TrackId, Track]: return { TrackId.A: TrackA(settings), TrackId.B: TrackB(settings), TrackId.C: TrackC(settings), TrackId.D: TrackD(settings), } Pydantic's BaseSettings is mutable and doesn't implement __hash_. lru_cache needs hashable arguments. Every real request threw: TypeError: unhashable type: 'Settings' Fix: since get_settings() is itself an @lru_cache'd singleton, the parameter was redundant. Call it inside: @lru_cache def _track_singletons() -> dict[TrackId, Track]: settings = get_settings() return {...} Silly bug. The tests didn't catch it because they patched Settings with a mock that was hashable. Lesson: your test doubles should mirror the real thing's flaws, not paper over them. Try it Live: https://hydra9.dev API docs: https://hydra9.dev/docs Source (MIT): https://github.com/loly-baby/Hydra Self-host in 4 commands: git clone https://github.com/loly-baby/Hydra cd Hydra && cp .env.example .env docker compose -f infra/docker-compose.prod.yml up -d --build curl http://localhost:8000/api/v1/trending If it's useful, a ⭐ on the repo goes a long way. If you have a use case where you'd want more analytics (release velocity? topic clustering? language-specific momentum?), open an issue. Hydra runs on a single 6GB VPS shared with another project. If the public instance gets hammered, self-hosting is a Docker Compose away.