Data & crawling · Built for a client · 2020–2021
Address-directory crawling estate
Crawling and address-extraction system that builds a directory of the physical locations of internet businesses from public web pages.
View in the worldWhat I did
- Built the StormCrawler topology and a rule that picks few, likely contact links.
- Split address extraction from crawling so extraction logic could improve without recrawling.
- Moved location detection to named-entity recognition and designed the Geonames model in MongoDB.
Skills
Terraform
Java
PHPThrow me a paper plane
Write a line; I’ll find it in my inbox.