Software Engineer, Data Mining

Software Engineer, Data Mining
About Nation Graph

Nation Graph is building the data and intelligence layer for the public sector.

• More than 110,000 state and local government agencies across the U.S. independently publish information about:

  • How they operate

  • What they buy

  • Who they work with

  • What problems they are trying to solve

• That information is fragmented across millions of websites, documents, databases, procurement systems, meeting records, and public records.

• Nation Graph turns that information into structured, connected, actionable intelligence for businesses selling to government.

• Founded in 2024, Nation Graph is dedicated to making uncommon knowledge common
  , because public data should actually be public.

We’re looking for a Software Engineer, Data Mining to own one of the most important technical problems at Nation Graph:
building the systems that acquire public-sector information from across the internet at massive scale.

Our goal is to operate hundreds of thousands, and eventually millions, of scrapers covering every level of government across the U.S. and Canada, and eventually worldwide.

This is not a role focused on manually building individual scrapers. You’ll own the infrastructure, abstractions, and automation that allow us to create, deploy, monitor, and maintain an enormous fleet of scrapers reliably.

You’ll work across:

• Web crawling and scraping

• Browser automation

• Distributed systems

• Data extraction

• Infrastructure and orchestration

• LLMs and agents

• Monitoring and observability

• Own our scraping infrastructure end-to-end

  • Build systems for creating, deploying, scheduling, monitoring, and maintaining hundreds of thousands of scrapers.

  • Design abstractions that allow us to scale toward millions of sources without scaling engineering effort linearly.

• Build for the messy internet

  • Work across government websites, APIs, procurement systems, PDFs, spreadsheets, meeting records, and legacy systems.

  • Handle changing websites, undocumented APIs, rate limits, broken sources, and countless edge cases.

• Make scraping a distributed systems problem

  • Build for orchestration, concurrency, retries, backfills, change detection, observability, cost management, and failure recovery.

  • Ensure we know when sources break, data disappears, or extraction silently becomes incorrect.

• Use AI to rethink scraping

  • Work with our ML Research team to use LLMs and agents to:

    • Discover new sources

    • Understand unfamiliar websites

    • Generate scraping logic

    • Detect source changes

    • Diagnose and repair failures

    • Validate extracted data

• Build systems that get better with scale

  • Identify common platforms and patterns that can unlock thousands of government agencies at once.

  • Make new sources increasingly cheap and automated to onboard.

• Expand our coverage globally

  • Help comprehensively map public-sector information across the U.S. and Canada.

  • Build the foundation to eventually acquire public-sector information worldwide.

• You’re an unusually strong engineer who enjoys figuring out how things work.

• You’ve built production web crawlers, scraping systems, browser automation, or large-scale external data pipelines.

• You’re strong in Python, Go, Type Script, or another backend/systems language.

• You understand the realities of scraping modern websites, including:

  • JavaScript rendering

  • Sessions and cookies

  • Rate limits

  • Proxies

  • Authentication

  • Changing schemas and websites

• You understand distributed systems, including:

  • Orchestration

  • Queues and concurrency

  • Idempotency

  • Retries

  • Backfills

  • Observability

  • Failure recovery

• You care deeply about data quality, correctness, and reliability.

• You’re excited about using LLMs and agents to automate traditionally manual scraping work.

• You naturally think about leverage:
  not how to scrape one website, but how to build a…