如何用python爬取豆瓣top250

使用Python爬取豆瓣Top250的方法包括：使用requests库进行网页请求、使用BeautifulSoup库解析网页内容、提取电影信息并保存数据。接下来，我将详细展开其中一个步骤：使用BeautifulSoup解析网页内容。

使用BeautifulSoup解析网页内容涉及创建BeautifulSoup对象并使用其方法来查找和提取特定的HTML元素。具体来说，首先要获取网页的HTML内容，然后利用BeautifulSoup进行解析。通过分析网页结构，可以定位到需要提取的信息，如电影名称、评分、评论等。

下面是详细的实现步骤：

一、安装所需库

首先，确保安装了requests和BeautifulSoup库。如果没有安装，可以使用以下命令进行安装：

pip install requests pip install beautifulsoup4

二、发送HTTP请求获取网页内容

使用requests库发送HTTP请求，获取网页的HTML内容：

import requests
url = "https://movie.douban.com/top250"
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3"
}
response = requests.get(url, headers=headers)
html_content = response.text

三、解析网页内容

使用BeautifulSoup解析网页内容，提取电影信息：

from bs4 import BeautifulSoup
soup = BeautifulSoup(html_content, 'html.parser')
movies = soup.find_all('div', class_='item')
for movie in movies:
    title = movie.find('span', class_='title').text
    rating = movie.find('span', class_='rating_num').text
    quote = movie.find('span', class_='inq').text if movie.find('span', class_='inq') else ""
    print(f"Title: {title}, Rating: {rating}, Quote: {quote}")

四、处理分页

豆瓣Top250有多页内容，需要处理分页来获取所有数据：

import time
all_movies = []
for i in range(0, 250, 25):
    url = f"https://movie.douban.com/top250?start={i}"
    response = requests.get(url, headers=headers)
    soup = BeautifulSoup(response.text, 'html.parser')
    movies = soup.find_all('div', class_='item')
    for movie in movies:
        title = movie.find('span', class_='title').text
        rating = movie.find('span', class_='rating_num').text
        quote = movie.find('span', class_='inq').text if movie.find('span', class_='inq') else ""
        all_movies.append({
            "title": title,
            "rating": rating,
            "quote": quote
        })
    time.sleep(1)  # 避免过于频繁的请求导致被封禁
打印所有电影信息
for movie in all_movies:
    print(movie)

五、保存数据

将提取到的数据保存到CSV文件：

import csv
with open('douban_top250.csv', 'w', newline='', encoding='utf-8') as csvfile:
    fieldnames = ['Title', 'Rating', 'Quote']
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
    writer.writeheader()
    for movie in all_movies:
        writer.writerow({'Title': movie['title'], 'Rating': movie['rating'], 'Quote': movie['quote']})

六、代码完整性与异常处理

为了确保代码的健壮性，添加异常处理：

import requests
from bs4 import BeautifulSoup
import csv
import time
def get_html(url, headers):
    try:
        response = requests.get(url, headers=headers)
        response.raise_for_status()
        return response.text
    except requests.RequestException as e:
        print(f"Error fetching {url}: {e}")
        return None
def parse_html(html):
    soup = BeautifulSoup(html, 'html.parser')
    movies = soup.find_all('div', class_='item')
    movie_list = []
    for movie in movies:
        title = movie.find('span', class_='title').text
        rating = movie.find('span', class_='rating_num').text
        quote = movie.find('span', class_='inq').text if movie.find('span', class_='inq') else ""
        movie_list.append({
            "title": title,
            "rating": rating,
            "quote": quote
        })
    return movie_list
def save_to_csv(movies, filename):
    with open(filename, 'w', newline='', encoding='utf-8') as csvfile:
        fieldnames = ['Title', 'Rating', 'Quote']
        writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
        writer.writeheader()
        for movie in movies:
            writer.writerow({'Title': movie['title'], 'Rating': movie['rating'], 'Quote': movie['quote']})
def main():
    url_template = "https://movie.douban.com/top250?start={}"
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3"
    }
    all_movies = []
    for i in range(0, 250, 25):
        url = url_template.format(i)
        html = get_html(url, headers)
        if html:
            movies = parse_html(html)
            all_movies.extend(movies)
        time.sleep(1)  # 避免过于频繁的请求导致被封禁
    save_to_csv(all_movies, 'douban_top250.csv')
if __name__ == "__main__":
    main()