如何用python抓取网页数据连续十页

如何用Python抓取网页数据连续十页

要用Python抓取网页数据连续十页，可以使用requests库、BeautifulSoup库、Scrapy框架、循环结构等工具和方法。本文将详细介绍如何使用这些工具和方法进行网页数据抓取，并提供代码示例和操作步骤。

一、安装必要的库和工具

要开始抓取网页数据，首先需要安装一些Python库：

pip install requests pip install beautifulsoup4 pip install lxml

requests库用于发送HTTP请求，BeautifulSoup库用于解析HTML文档，lxml作为解析器提高解析速度和性能。

二、理解目标网页的结构

在抓取网页之前，必须先了解目标网页的结构。以某个分页的新闻网站为例，每一页的URL可能是类似如下的形式：

https://example.com/news?page=1
https://example.com/news?page=2
...
https://example.com/news?page=10

三、编写Python脚本抓取单页数据

首先编写一个Python脚本抓取单页的数据，以确保能正确获取内容：

import requests
from bs4 import BeautifulSoup
def fetch_page(url):
    response = requests.get(url)
    if response.status_code == 200:
        return response.text
    else:
        return None
def parse_page(html):
    soup = BeautifulSoup(html, 'lxml')
    titles = soup.find_all('h2', class_='news-title')
    for title in titles:
        print(title.get_text())
url = "https://example.com/news?page=1"
html = fetch_page(url)
if html:
    parse_page(html)
else:
    print("Failed to retrieve the page")

四、循环抓取连续十页数据

在确保单页抓取功能正常后，使用循环结构抓取连续十页的数据：

import requests
from bs4 import BeautifulSoup
def fetch_page(url):
    response = requests.get(url)
    if response.status_code == 200:
        return response.text
    else:
        return None
def parse_page(html):
    soup = BeautifulSoup(html, 'lxml')
    titles = soup.find_all('h2', class_='news-title')
    for title in titles:
        print(title.get_text())
base_url = "https://example.com/news?page="
for i in range(1, 11):
    url = base_url + str(i)
    print(f"Fetching page {i}: {url}")
    html = fetch_page(url)
    if html:
        parse_page(html)
    else:
        print(f"Failed to retrieve page {i}")

五、存储抓取的数据

为了保存抓取的数据，可以将结果写入文件或数据库。以下代码示例将抓取的标题存储到CSV文件中：

import requests
from bs4 import BeautifulSoup
import csv
def fetch_page(url):
    response = requests.get(url)
    if response.status_code == 200:
        return response.text
    else:
        return None
def parse_page(html):
    soup = BeautifulSoup(html, 'lxml')
    titles = soup.find_all('h2', class_='news-title')
    return [title.get_text() for title in titles]
base_url = "https://example.com/news?page="
all_titles = []
for i in range(1, 11):
    url = base_url + str(i)
    print(f"Fetching page {i}: {url}")
    html = fetch_page(url)
    if html:
        titles = parse_page(html)
        all_titles.extend(titles)
    else:
        print(f"Failed to retrieve page {i}")
with open('news_titles.csv', 'w', newline='', encoding='utf-8') as csvfile:
    writer = csv.writer(csvfile)
    writer.writerow(['Title'])
    for title in all_titles:
        writer.writerow([title])
print("Data has been written to news_titles.csv")

六、处理异常和错误

在抓取网页数据时，可能会遇到各种异常和错误，如网络超时、页面不存在等。为了提高脚本的健壮性，需要处理这些异常：

import requests
from bs4 import BeautifulSoup
import csv
import time
def fetch_page(url):
    try:
        response = requests.get(url, timeout=10)
        response.raise_for_status()
        return response.text
    except requests.exceptions.HTTPError as http_err:
        print(f"HTTP error occurred: {http_err}")
    except requests.exceptions.ConnectionError as conn_err:
        print(f"Connection error occurred: {conn_err}")
    except requests.exceptions.Timeout as timeout_err:
        print(f"Timeout error occurred: {timeout_err}")
    except requests.exceptions.RequestException as req_err:
        print(f"An error occurred: {req_err}")
    return None
def parse_page(html):
    soup = BeautifulSoup(html, 'lxml')
    titles = soup.find_all('h2', class_='news-title')
    return [title.get_text() for title in titles]
base_url = "https://example.com/news?page="
all_titles = []
for i in range(1, 11):
    url = base_url + str(i)
    print(f"Fetching page {i}: {url}")
    html = fetch_page(url)
    if html:
        titles = parse_page(html)
        all_titles.extend(titles)
    else:
        print(f"Failed to retrieve page {i}")
    time.sleep(1)  # To avoid being blocked by the website
with open('news_titles.csv', 'w', newline='', encoding='utf-8') as csvfile:
    writer = csv.writer(csvfile)
    writer.writerow(['Title'])
    for title in all_titles:
        writer.writerow([title])
print("Data has been written to news_titles.csv")

七、使用Scrapy框架进行抓取

Scrapy是一个强大的网页抓取框架，适用于更复杂的抓取任务。以下是使用Scrapy框架抓取连续十页数据的示例：

安装Scrapy：

pip install scrapy

创建Scrapy项目：

scrapy startproject news_scraper

创建爬虫：

cd news_scraper scrapy genspider news_spider example.com

编辑news_spider.py文件：

import scrapy
class NewsSpider(scrapy.Spider):
    name = 'news_spider'
    start_urls = [f'https://example.com/news?page={i}' for i in range(1, 11)]
    def parse(self, response):
        for title in response.css('h2.news-title::text').getall():
            yield {'title': title}