Skip to content
My Glossary Project

Web Scraping

TL;DR

Section titled “TL;DR”
  • Automated or manual extraction of data from websites using specialized tools.
  • Commonly used to gather large amounts of online data quickly for industries like retail, finance, and marketing.
  • Can access data not available via APIs but may raise performance, legal, or terms-of-service concerns.

Definition

Section titled “Definition”

Web scraping is the process of extracting data from a website.

Explanation

Section titled “Explanation”

Web scraping can be performed manually or, more commonly, with specialized software or tools that visit websites and collect data automatically. It is used across industries such as retail, finance, and marketing to gather large volumes of data quickly and to automate data-collection tasks. Web scraping can also retrieve information that is not available through APIs or other methods.

The practice requires care because excessive requests from a scraper can negatively affect the performance of the target website, and certain uses of scraped data (for example, obtaining sensitive or confidential information, or engaging in spamming) can be illegal or violate a website’s terms of service.

Examples

Section titled “Examples”

Price comparison websites

Section titled “Price comparison websites”

Price comparison websites use web scraping to gather pricing information from multiple online retailers. The scraping software visits each retailer’s website, collects the pricing information for a specific product, and displays it on the comparison site so consumers can compare prices from different retailers.

Real estate listings

Section titled “Real estate listings”

Real estate agents and brokers use web scraping to collect information about available properties and their prices. The scraping software visits real estate websites and gathers details such as location, price, and property features. Agents then use that information to find properties that match clients’ needs and budgets.

Use cases

Section titled “Use cases”
  • Retail
  • Finance
  • Marketing

Notes or pitfalls

Section titled “Notes or pitfalls”
  • Performance impact: Excessive requests from scrapers can slow down or crash the target website.
  • Legal and policy risks: Scraping can be unlawful if used to obtain sensitive or confidential information or to engage in malicious activities (such as spamming). Scraping may also violate a website’s terms of service, potentially leading to legal action.
Section titled “Related terms”
  • API
  • Terms of service
  • Web scraping software (web scraper)