Web scraping and web crawling. Ever heard of these techniques? Well, you’re more familiar with them than you think. One of the perhaps lesser-known tech jargon, this technique is an exceptionally powerful tool used across many different industries. Simply put, web scraping allows an organisation to access websites in order to read and extract relevant data. Take SkyScanner for instance. The company uses web scraping tools to provide you with the best flight prices. But how exactly does the process work? What current web scraping software is available? Is it legal?
In this week’s Q&A, we meet with a tech leader who helps businesses to realise their full potential by harnessing the power of data through ethical web scraping. Introducing Julius Černiauskas, CEO and Founder of Oxylabs. Founded in 2015, Julius has taken the public web data gathering solutions company from strength to strength, transforming it from a small startup run by five staff members to one of the biggest companies in the data collection industry.
Thanks for joining us here at EM360 Julius! To start, can you tell us about the lightbulb moment behind Oxylabs? What made you believe that the business technology market needed a solution such as this?
The story of Oxylabs began in 2015. Still taking baby steps, the proxy industry was in that sweet spot where the technology was mainly employed for several niche uses, but more of them were already emerging. It seemed like a perfect time to jump onboard.
We started our journey with data centre proxies. After a while, residential proxies seemed like a natural addition to our product list. Moving even further and looking deeper into the needs of our clients we understood that we do not want to limit ourselves to being an infrastructure provider only.
We started developing a solution that would make automated web data collection even more accessible. Thus, Real-Time Crawler was born – a solution that enables companies to get the data they need by simply sending a few requests, without building an in-house scraping infrastructure. With this product, we can adapt to the needs of a particular business even better.
We saw proxy technology as something that will eventually become very important for modern businesses. Yet, we could not truly predict the scope and speed in which proxies would turn out indispensable. Automated web data collection is used in so many areas now – from marketing to e-commerce, from travel to finance. It provides a competitive edge to many companies and even is a foundation of their business for some.
Web scraping, data scraping, and web crawling are more than often used interchangeably. Do they hold the same meaning? If yes/no, please explain why.
While web scraping and data scraping usually describe the same thing, web-data crawling differs from them.
The internet is full of public data which when utilized, can help businesses make more effective decisions. It depends on the particular needs of a company, whether that data requires web crawling or scraping, but usually, these processes go hand-in-hand. While web crawling gathers pages to create indices or collections, web scraping downloads pages to extract specific sets of data.
Web scraping is an automated process in which publicly available information is gathered from websites and that data is then stored for future access and analysis. A very simple example of how it works could be a person looking for a new household appliance online and writing down its different characteristics in an excel sheet. Web scraping does exactly that, except the process is fully automated and you get those characteristics without the lengthy manual action.
Meanwhile data scraping, a term often used interchangeably with web scraping, means taking any publicly available data, whether from the web or your computer and importing the specific information from it into a file on your computer. The main difference from web scraping is that data scraping usually also involves other sources than websites.
Web crawling describes a process in which web crawlers go into targeted websites and find required data, e.g., price and description of the product. When the found data is downloaded, it then becomes web scraping; not web crawling.
How has the global perception of web scraping and its associated technologies evolved since its inception?
Initially, web scraping was primarily associated with several industries – flight fare and hotel booking aggregators, giant tech businesses, venture capital, and hedge funds. However, as access to technology is getting cheaper, more industries are finding public data collection useful to boost revenue.
We have recently conducted a survey in the UK’s finance sector where we asked senior data decision-makers about their everyday challenges. Eighty-three percent of respondents claimed that their data needs increased in the past 12 months. Sixty-three percent responded saying they use alternative data to meet those growing demands. This shows that the collection of publicly available data has become a mainstream practice in the finance industry.
Even though we do not have the numbers from other industries, many would look similar. Data-reliant departments, such as marketing, sales, R&D and others are now attempting to gather data from a larger pool of sources to get a more comprehensive view for their decision making.
As web scraping is becoming a mainstream technology, it is slowly becoming more understood. In the early days, there were so many myths and misunderstandings surrounding web scraping - starting from whether it is even legal. Therefore, we are putting a lot of effort into educating the market on what web scraping is and how it should be done.
It is also becoming a testing field for innovation as data collection practices are being constantly improved. Artificial intelligence and machine learning technologies are shaping the latest developments and it will be interesting to see how it will tackle even the most complicated issues.
In what ways can organisations harness the power of data through web scraping and why should they consider doing it today?
Data-driven businesses outperform their competitors. For example, by understanding their customer behaviour better, they can adapt to their needs and meet their expectations.
There are so many ways to use publicly available data. Its usefulness only depends on business specificity. First, web scrapers provide automation, therefore for those businesses who still collect large amounts of data manually, it could save time and human resources. Another very common use is market research, which entails monitoring many websites on a large scale whenever required.
Some companies use web scraping for brand protection purposes, checking if their products are not being counterfeited. On the other hand, online retailers depend on price monitoring, which is automated and reliable with web scraping. Many businesses are acquiring information based on geolocation settings to get localized and relevant data.
These are just a few most common ways of benefiting from automated public data collection, but the opportunities with web scraping technology are endless.
When it comes to the implementation and utilisation of web scraping, what best practices do business leaders need to be aware of?
The first thing a company should define is the purpose of collecting publicly available data. Web scraping is a powerful tool that can provide tons of insights for better decision-making. However, to benefit from it, one must have clear goals on what to do with all that information. Before collecting the data, businesses must have effective data management practices in place.
Next, a company should decide whether it wants to build its own web scraping solution or outsource it. The decision mainly depends on your organization’s resources. Our research in the UK’s finance industry revealed that the same amount (36%) of surveyed companies have an in-house team for web scraping or outsource web scraping partners. Others combine both methods.
Larger companies (over 250 employees) have a slight preference towards having in-house teams. Collecting data in-house provides them with more flexibility – they only need supporting tools, such as proxies, but tasks are performed by their own developers, system administrators, and data analysts. However, having such a team in-house is quite expensive and difficult to maintain, as it needs to be fully dedicated to the technical aspects of data acquisition. For this reason, it’s much easier for many companies to outsource a web scraping team.
Another and probably the simplest way to benefit from this technology is by using ready-to-use solutions. The process is very simple – a client sends a request to such tools like our Real-Time Crawler, it collects the information and allows the client to retrieve it in several file formats. Using solutions like this, your infrastructure costs will be lower and you’ll require less human resources.
No matter which option is chosen, utilizing web scraping will definitely bring more progress and efficiency to the organization, so it’s only about deciding what works best for your company.
You’ve written countless articles about ethical web scraping, including a recent LinkedIn post on ‘rebuilding public trust in data gathering’. What does it mean to be an ethical web scraper and how does Oxylabs work to be one?
Data needs to be treated sensitively – there are both legal and moral implications. However, as it happens with all new technologies, there were and unfortunately still are some market players that conduct web scraping for unethical purposes or even in illegal ways. This, of course, does not serve the industry image well. There are also many myths associated with web scraping out of not knowing the technology, for example, people fear that their private data can be affected, while web scraping is only about collecting publicly available data.
We want the industry to earn the public’s trust back, thus we put a lot of emphasis on ethics. We hold ourselves and our partners to very high standards. To us, the ethics of web scraping start with diligently respecting the law: only collecting the data that is publicly available, not downloading copyrighted data. And it goes far beyond those basics.
One of the most important things to consider when choosing a provider for your data acquisition practices is how they acquire proxies. Residential proxies are the only proxy types that are directly linked to the physical devices of actual people. Therefore it’s of utmost importance to make sure that those who participate in the proxy network are well informed and consent to it.
We have developed a grading system for all non-ISP residential proxy acquisition practices in the industry.
The highest-grade Tier A+ means that a platform provides financial reward for end-users in exchange for participation in the residential proxy network. Tier A means that the end-users are also fully informed and consent with the participation in the network, the only difference from Tier A+ is the absence of the financial reward.
We only consider these two proxy acquisition practices ethical, and the majority of our network is composed of tier A+ proxies. This way we ensure that the participants in our proxy network are well informed, and most are even rewarded.
Oxylabs’s web scraping conference, OxyCon, is back this August. What can attendees expect to engage in this year and where can they sign-up?
OxyCon is an opportunity for the web scraping community to connect and share recent joys and challenges. It’s also a great opportunity to get acquainted with the field for those who are planning to enrich their data collection practices.
We wanted this event to be as universal as possible because there are so many different aspects to data collection. OxyCon’s focus will be divided into three major themes: data collection for business, the future of web data gathering, and web scraping for developers. Participants will be able to navigate through informative presentations by global business leaders, round-table discussions, Q&A sessions and seminars.
Our speakers will investigate public data collection from different angles. They will present how external data is changing the business and why companies should start using it now. They will also talk about the challenges that come with web scraping at scale and offer solutions.
One of the most beloved speakers from the last OxyCon – data consultant and celebrated speaker Allen O’Neill will dive into the topic of data quality, the biggest test for data scientists. Pujaa Rajan, Machine learning (ML) engineer at Stripe and member of the AI&ML advisory board at Oxylabs will share her invaluable experience on building ML infrastructure. Another great speaker - data consultant Adi Andrei, whose experience include gigs with NASA, Unilever, SIXT, British Gas and others will delve into the specific topic of entity detection in parsed HTML.
Anyone interested in more technical issues will hear from those who tackle web scraping challenges every day. The topics will vary from augmenting web scraping with machine learning to best practices of web scraper monitoring.
OxyCon will take place online on 25-26 August. Registration to the event is free and available here.
Interested in learning more about web scraping, web crawling and data scraping? Subscribe to the YouTube Channel for more educational content in enterprise technology.
