Lead Generation Pipeline
Automated organisation-profile construction from a dozen-plus public web sources, each field attributed and confidence-scored
A web-scraping and information-extraction pipeline that builds structured organisation profiles (address, contacts, leadership, industry) by crawling and cross-referencing public business directories, search results and company websites.
- Python
- Flask
- Web scraping
- MongoDB
- Azure (VMs, Storage Queues)
- NLP (keyword/topic extraction, text classification)
Problem
Building a usable profile of an organisation (address, phone number, leadership, industry) from the public web meant checking a dozen-plus separate sources by hand: search results, business directories, the company’s own site. That doesn’t scale past a handful of organisations.
Approach
A pipeline of purpose-built extractors, each targeting one source: Crunchbase, OpenCorporates, D&B, Avention, Google search results, Google address and phone lookups, a company’s own contact page, LinkedIn and Owler profiles surfaced through search, and a deep crawl of the company website itself. These feed into:
- a company-type predictor built on keyword and topic extraction (word clouds, TextRank, RAKE, LDA) run over the deep-crawled site content
- a consolidation component that merges every extractor’s output into one profile
- a confidence-and-source tagging component, so each field in the final export carries where it came from and how much to trust it
Running each source as an independent extractor, rather than one combined scraper, meant a single source going down or changing its page structure degraded one field instead of breaking the whole profile.
Result
An automated pipeline on Azure (VMs and Storage Queues, with MongoDB as the profile store) that replaced manual, source-by-source research with a per-organisation profile ready to export, each field attributed and confidence-scored.
What I’d do differently
Confidence and source tagging was added after most of the extractors already existed, once conflicting values from different sources became a real problem. I’d design the schema to carry provenance from the first extractor onward; retrofitting it meant re-touching every component instead of building it once.