My go-to language for web scraping is Python, as it has well-integrated libraries that can generally handle all of the functionality required. And sure enough, a Selenium library exists for Python.

But it quickly graduates to general-purpose web scraping and even web automation. These techniques can be very useful for testing browser-based applications, too.

To get us started, you will need to start a new Python3 project with and install Scrapy (a web scraping and web crawling library for Python). I’m using pipenv for this tutorial, but you can use ...