Data pipelines and connectors for legal documents

We build customized pipelines for your dockets, transcripts, contracts and other legal documents so that your lawyers can securely access them anywhere
556k
U.S. Code sections parsed from USLM and published as open data
150k+
Congressional bills ingested from govinfo.gov, split into sections
1,000+
Legislative subject labels an LLM pipeline assigns and scores
300k
Bill sections compared against each other for near-duplicate text

Legal data pipelines

Each lawyer, court, and agency produces documents in their own formats on their own schedules: PDFs, Word, Google Docs, scanned images, hearing video, and paper -- so much paper. A single legal document can run to thousands of pages, exist in dozens of versions, and cite sources that have since been changed.

Generic ETL tooling treats these documents as blobs of text. That loses the structure — formatting, citations, effective dates, and other critical metadata that legal work depends on. We are lawyers and technologists (and lawyers who are technologists) who put the work in to understand your unique workflows and data sources, and build pipelines that preserve that structure and character-level accuracy, while allowing you to use the latest tools.

  • Structure preserved, not flattened
  • Reproducible, re-runnable loads
  • References traceable to the source

What we connect

A connector is a customized set of transformations that keep your data in sync.

Sources

  • MS Word, Sharepoint, Google Docs, email
  • PDFs and scans, including OCR for older records
  • Hearing audio and video transcripts
  • Document management systems, S3 buckets, shared drives

Pipeline

  • Parsing into structured data
  • Version tracking and redlining
  • Near-duplicate detection with locality-sensitive hashing
  • LLM classification, labeling, and summarization
  • Embeddings for semantic and hybrid search
  • Evaluation harnesses that measure output quality

Destinations

  • PostgreSQL and pgvector
  • Elasticsearch and vector stores — Weaviate, LanceDB, FAISS
  • Custom APIs
  • Retrieval layers behind AI assistants and chat interfaces
  • Warehouses and managed datasets for analytics
  • Existing case, matter, and docket systems

How we build a pipeline

1

Map the source

We read the publisher's schema and its exceptions: which fields are reliable, where the schema is violated in practice, how often the source updates, and what breaks when it does. This is where most integration projects fail, so it is where we start.

2

Parse and normalize

Documents are split into their real units — titles, chapters, sections, subsections — with identifiers, ancestry, citations, and status flags carried along. Both the clean text and the original markup are kept, so nothing has to be re-derived later.

3

Enrich

Classification, labeling, summarization, similarity fingerprints, and embeddings are added as pipeline stages with their own evaluation metrics. Each enrichment is measured against known-good data before it ships, and re-measured when a model or prompt changes.

4

Index and serve

Keyword indexes, vector indexes, and citation lookups are combined so a query returns the right document whether the user typed the exact statutory phrase or described the idea in their own words. Results are delivered through an API you own.

5

Run it

Scheduled loads, backfills, alerting on source changes, and dashboards that show what came in and what failed. The pipeline is code in your repository, deployed on your infrastructure, documented for your engineers.

Our Open Legislative Pipelines

We publish the output of several of our legislative pipelines as open datasets on Hugging Face, under CC0. They show some of the work we've done on high value data sets — you can download them, read the schemas, and judge the work for yourself.

uscode

556k rows

The United States Code parsed from USLM 1.0.15 into sections, with citations, ancestry chains, repeal status, and both plain text and XML for each section.

View dataset

bill_text_us

151k rows

Congressional bill text from govinfo.gov, Congresses 107–117, normalized into structured sections with headers and identifiers alongside the full document.

View dataset

bill_labels_us

120k rows

Bills joined to their policy area and legislative subjects — training and evaluation data for classification against more than a thousand possible subjects.

View dataset

bill_summary_us

125k rows

Bills paired with their Congressional Research Service summaries, the reference set for evaluating machine-generated legislative analysis.

View dataset

bill_committees_us

126k rows

Committee referrals for Congressional bills, the metadata needed to route a bill to the people and processes that will act on it.

View dataset

All datasets

Seven public datasets, downloaded thousands of times by researchers, journalists, and civic technologists.

huggingface.co/dreamproit

What we have built with them

Hybrid search

Smart Search for InstaTrac

We built a smart search -- combining keyword matching and semantic search -- so MassTrac users can find relevant bills and follow statutory changes even when the searched keywords don't appear directly.

Read the announcement
  • Keyword and semantic results ranked together in one list
  • Citation-aware retrieval across bills and the General Laws
  • Match provenance shown on every result
  • Meeting transcripts processed and prepared for analysis
  • Agenda items identified within each hearing
  • Summaries generated to each jurisdiction's format
Document pipeline

Legislative summarization at volume

A legislative tracking company was writing meeting summaries by hand, with different requirements in every jurisdiction and a growing backlog. We built a custom pipeline that processes the transcript, identifies the agenda items discussed, and drafts summaries in each client's required format. The backlog cleared and the volume of hearings they can cover no longer depends on headcount.

Read the full story
Similarity and classification

Finding the bill that came before

Tens of thousands of bills are introduced each year, many of them reworked versions of earlier text. Comparing 50,000 bills against 50,000 bills, and 300,000 sections against 300,000 sections, is not a job for full-text search. We fingerprint each document with locality-sensitive hashing, store the fingerprints in Postgres, and match on them — turning days of comparison into seconds, served through a public API built with FastAPI in collaboration with Demand Progress.

The same corpus supports classification. Our labeling pipeline assigns a policy area and legislative subjects to a bill and reports accuracy, precision, recall, and F1 against the Congressional Research Service's own labels, so the quality of a run is a number rather than an impression.

Read the results
  • Near-duplicate detection across billions of document pairs
  • Policy area and legislative subject labeling from bill text
  • Precision, recall, and F1 reported for every model change
Michael Segal

As a legislative information and bill tracking company, we struggled with manual summarization of state capitol events such as legislative committee hearings for our state-based service, and for generating meeting minutes for our municipal government line of work. DreamProIT provided us with a custom AI pipeline that not only streamlined our summarization and minutes generating processes. They also ensured the timely delivery of precise summaries, helping us to overcome a significant backlog. The results exceeded our expectations. And our clients are thrilled with the results, too. We highly recommend DreamProIT to anyone seeking efficient and reliable AI solutions.

DreamProIT team
For law firms and legal teams

Hire us to build your integration

If your firm needs a source of legal or legislative data connected to the systems your lawyers already use — a research tool, a matter database, a drafting workflow, an internal AI assistant that has to cite real authority — we will scope it, estimate it in time and cost, and build it.

  • A written estimate before the work starts
  • A working prototype on your own documents
  • Production deployment, with the code handed over to you
Talk to our team
Top