Rampage Blogs

Despite being somewhat confused as the same thing, there are distinct differences between the two processes and applications. In this post we'll be breaking down the two processes and how they work; discussing more on specific use cases.

Web Scraping

Using HTML identifiers, web scraping is used to extract data from web pages. When a scraper is given its target, it will collect all data stored inside these target identifiers to be parsed and analysed later. Web scraping is no small feat. It is used from anywhere to small scale personal projects to industrial uses. For example, take two e-commerce websites utilising dynamic pricing. While both utilising the pricing strategy, it is likely they frequent scrape the competitor site to gain insight on their pricing; allowing them to provide competitive pricing opportunities based on real time pricing. This can be considered equivalent to a game of "pricing chess", where a move from one party heavily influences the next.

To deter this, site 2 may employ antibot measures to prevent scrapers successfully collecting data by restricting access and blocking too many subsequent requests from an IP address. To circumvent this, site 1 will deploy proxies to rotate IP addresses and continue scraping without restrictions. Sites may also geo-restrict, so by utilising a geo-located proxy akin to the host region the competition can ensure they can collect from where ever they are in the world. This is particularly useful during instances where companies are based in different time zones. The competitive advantage lies not just in being able to scrape data; but in doing so efficiently and ethically. Utilising proxies responsibly ensures that the scraping activities don't disrupt the target website by abuse of request and remain within the bounds of ethical considerations in which scraping is used. In this case, we highly recommend you check out this post on some of the best practises used in web scraping and how you can be a competitive yet ethical internet citizen.

Web Crawling

Albeit somewhat similar, in that this process also collects information from web pages, web crawling has a key difference. Crawling is used for automatically indexing a sites content to learn what information the pages hold. If a website is to appear well on a search engine, it must be indexed. Without indexing, including regular updates to this, the search engine will be less likely or unable to locate the information on that site and display it to a user whos query it could potentially answer. Having a site crawled and indexed creates the entries required for search engines.

However, sites can refute crawling. Using tools such as their robot.txt files, sites can request against crawling and indexing, or only give permission for partial indexing. This can be for a variety of reasons. Take server load for example, a site can request against crawling and indexing to decrease the traffic from bots on and subsequently reducing load on the servers. Too high of a load can negatively impact the experience a genuine use might have. This can also be the case for anti-DDOS (Distributed Denial of Service), again looking to mitigate against such attacks by reducing load.

While the two processes sound similar (in terms of connecting, collecting, and storing), crawling is an entirely different process itself. Instead of a target HTML attribute, a crawler is given a "seed". This is a list of URL's to begin the crawl on. The crawler will then categorise the pages, paying careful attention the robots.txt file mentioned above. The crawler can from the seed navigate to other pages based on internal links it identifies and follows. While operating it will copy and store the data inside the meta tags. Essentially, the meta tags provide structured metadata about a website. It's important to note the importance of optimising these themselves, in which case we recommend you read more about that here and the importance of it in SEO management. The data stored from a crawler taken from the meta tags is what's used to index a page for the search engine to scan key words.

The indexed pages and stored keywords (along with some engine specific algorithm wizardry) are what dictates whether a page is returned when a search engine is queried. A search engine will return a list of indexed pages, ranked on what it deems most important.

In conclusion, while web scraping and web crawling may be mistakenly perceived as interchangeable, this post has shed light on their distinct differences, functionalities, and applications.