What tools or libraries do you use or know for automating data collection from websites?

Tools and libraries commonly used in Python for automating data collection from websites include:

• Requests — for sending HTTP requests and retrieving HTML pages.
• BeautifulSoup — for parsing and extracting data from HTML or XML.
• Scrapy — a powerful framework for creating complex web crawlers and parsers.
• Selenium — for browser automation, useful for sites with dynamic JavaScript content.

Example of a simple script using Requests and BeautifulSoup:

import requests
from bs4 import BeautifulSoup

url = 'https://example.com'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')

for item in soup.find_all('h2'):
    print(item.text)

This code downloads the page and prints the text of all h2 headers.