Kori StroudOperations automation
Available

TradeOps Data · Python data pipeline

Scattered bid pages become one reviewable report

Public construction opportunities live on dozens of inconsistent agency pages and portals. TradeOps collects them into a single research report that keeps sources, duplicates, and usage rights visible — so an estimator spends time judging work, not assembling links.

See how it works Discuss a similar workflowView source on GitHub

The problem

What kept going wrong

Finding public construction work means checking city portals, county feeds, and procurement platforms that all describe the same kinds of opportunities differently. Titles, deadlines, documents, and access rules vary page by page. The same bid shows up in two places under two names.

An estimator doing this by hand loses hours to assembly work — and still ends up with a spreadsheet that has no record of where each fact came from or whether the source allows reuse.

Who it's for

Who would use this

Contractors and estimating teams who research public construction bids by trade and region — and anyone whose research workflow means re-checking the same public sources every week.

The build

What I built

A Python pipeline that reads opportunity pages, converts each one into a single canonical record, classifies the relevant trade, collapses near-duplicates, remembers what it has already seen, and applies a per-source usage-rights decision before anything is released into a report.

The output is a set of local files an estimator can actually review: an HTML summary page plus JSON and CSV exports, with quarantined records preserved and explained rather than silently dropped.

The workflow

How a run actually goes

This is the demonstrated sequence from the offline demo — five synthetic source records in, four reviewable opportunities out.

  1. Read the source pages

    The full system collects from live pages using async HTTP, static extraction, platform-aware adapters, and a Playwright fallback for dynamic sites. The standalone public demo reads project-owned HTML fixtures instead — the same workflow shape, with no network path.

  2. Normalize into one record shape

    Pydantic models standardize identifiers, titles, deadlines, documents, locations, and trade codes, so every downstream step works with the same structure regardless of the source.

  3. Classify the trade

    A trade filter tags each record with relevant trade categories — and records its reasons and disqualifiers, so a surprising match can be traced instead of trusted.

  4. Collapse duplicates and skip the already-seen

    Near-duplicate records merge into one opportunity. SQLite tracks previously seen bid IDs: in the demo, the first pass records four new IDs and a second pass skips all four.

  5. Apply the source-rights decision

    Every source starts quarantined. Only records from sources with an explicitly reviewed usage policy are released; the rest stay preserved internally with a reason code.

  6. Produce the reviewable output

    Approved records land in an HTML review page plus JSON and CSV exports. Quarantined records are listed with their reason — visible, not vanished.

Worth a closer look

The most useful parts

  • Provenance on every recordEach opportunity keeps its source link and collection context, so any fact in the report can be traced back to the page it came from.
  • Duplicate prevention across runsSeen-state in SQLite means re-running the pipeline doesn't resurface old records as new — the demo proves the second pass skips everything from the first.
  • A rights gate that defaults to noUnreviewed sources can't leak into a report by accident. Quarantine is the default state, and release requires an explicit per-source decision.
  • Evidence over confidenceWeak or missing evidence becomes a visible hold instead of confident copy. Disqualifiers are recorded alongside matches.

The real screens

What it looks like in use

Real output from the verified offline demo — synthetic records on the reserved .test domain, no real agencies or contractors.

TradeOps demo reportsynthetic data
The demo report: four unique synthetic opportunities, each with its trade code, deadline, evidence, and source link.
TradeOps demo report listing four synthetic construction opportunities with trade codes, deadlines, and source links

The demo report: four unique synthetic opportunities, each with its trade code, deadline, evidence, and source link.

Source-rights reviewsynthetic data
The rights gate at work: three records released, one source held back with its reason code visible.
TradeOps rights review view showing three exported records and one quarantined source with a reason code

The rights gate at work: three records released, one source held back with its reason code visible.

Demo output file (index.html)synthetic data
The actual demo output file rendered in a browser — this is a data/output view of the supplied artifact, not a separate product screen.
Full TradeOps demo output page rendered in a browser, showing demo counts, the public-safe review feed, and the rights-aware exclusion table

The actual demo output file rendered in a browser — this is a data/output view of the supplied artifact, not a separate product screen.

Technical verification

What I tested

These numbers describe the local test suite and the reproducible offline demo — engineering evidence, not customer outcomes.

1,457tests passed (1 optional dependency skip)
5 → 4synthetic records deduplicated in the demo
3 + 1records released + quarantined by the rights gate
0network requests in the demo run

The full local suite passed 1,457 tests with one skip (an optional Excel-export test that needs openpyxl). The portfolio demo runs offline against project-owned fixtures and asserts the counts above, including the second-pass duplicate skip and the rights-gate outcome. The demo manifest also records that no email, SMS, scheduler, CRM, feed, or deployment action was possible in the demo path.

Current scope

  • New source portals are added and reviewed individually; the pipeline is not a universal crawler.
  • The report organizes opportunities and evidence. It does not score contractor fit or make bid decisions.
  • The portfolio example uses invented records. Production setup would start with the sources, trades, and review rules a team actually uses.
  • Source-rights rules enforce a team's documented policy; they are not legal advice or blanket permission to reuse data.

Keep exploring

Next case study: ATS Copilot