Scrapy Masterclass: Learn Web Scraping With Scrapy Framework

About Scrapy Masterclass: Learn Web Scraping With Scrapy Framework

Web scraping is the process of scraping websites and extracting desired data from the same, and in this course, you'll learn and master web scraping using python and scrapy framework with a step-by-step and in-depth guide.

A Step-By-Step Guide

Assuming that you know nothing about web scraping, web crawling, scrapy framework, web scraping, or even web scrapping, we will start from the complete basics. In the first section, you'll learn about the

web scraping process step-by-step

(with infographics - no code), how to scrape data from websites and how to use scrapy for the same (i.e. scrapy meaning). After getting the basics clear and having an idea of

how web scraping works

, we will start web scraping using python & scrapy framework! Again, we'll move step-by-step and perform each step learned in the basics with

bite-sized lessons

. We'll take it slow so that it's easier for you to understand every step involved in scraping and extracting data from websites.

Web Scraping & Scrapy Essentials

Having built an actual web scraper, you'll get an idea of how web scraping works first-hand. Now it's crucial to cover the essential concepts of web scraping and scrapy, which we will do next.

CSS Selectors to _select_ web elements

XPath to select web elements.

Scrapy Shell

to test & verify selectors.

Items to organise extracted data

Load Items with

ItemLoaders

with input & output Processors

Export data to JSON, CSV, XLSX (_Excel_) & XML file formats

Save extracted data to online databases like MongoDB using

ItemPipelines.

Master Web Scraping In-Depth

Learning how to scrape websites and the essentials already makes you a complete web scraper, but we'll take this even further and learn the advanced web scraping techniques to

become an expert

!

Follow links in a webpage to another page.

Crawl multiple pages and extract data, i.e.

Pagination.

Scrape data using Regular Expressions (RegEx)

Extract Data From HTML Tables

Login

Into Websites Using Scrapy FormRequest

Bypass CSRF-protected Login forms.

Scrape Dynamic or

JavaScript Rendered Websites

using Scrapy Playwright.

Interact

with web elements like fill forms, click buttons, etc.

Handle Infinite Scroll websites.

Wait For Elements when contents/data take time to load

Take

Screenshots

of websites.

Save websites as

PDFs.

Identify API calls from websites and scrape data from

API

s

Use middleware in a scrapy project.

Configure settings in a scrapy project

Use and Rotate

User-Agents

&

Proxies

Web scraping Best Practices

Real-World Projects

After mastering web scraping and web crawling, we need projects to start! That's why you'll perform three projects as well:

Champions League Table [

ESPN

]

Product Tracker [

Amazon

]

Scraper Application [

GUI

] Join us in this in-depth course, where you'll learn about web scraping from scratch and master the process of extracting data from websites step-by-step. Check out the preview lessons to learn how web scraping works! See you there~