Some years back, we worked on a vertical search engine for job postings, and it quickly grew to 10MM domains. To avoid showing expired jobs data, the system was mostly monitoring (re-crawling) about 250,000+ sites with actual jobs, or where jobs could potentially appear in the near future. A lot of data attributes were extracted from unstructured web and processed through filters, classifiers, Machine Learning and custom heuristics, resulting