python如何爬取豆瓣t250

Python 爬取豆瓣TOP250电影的方法包括使用requests库进行网页请求、使用BeautifulSoup库进行网页解析、以及处理分页和数据存储。

其中，使用requests库进行网页请求、处理分页是爬取过程中较为关键的步骤。下面将详细介绍如何实现这些步骤。

一、使用requests库进行网页请求

requests库是一个非常简单易用的HTTP库，用于发送HTTP请求和获取响应。首先，我们需要安装requests库：

pip install requests

接下来，我们使用requests库发送请求并获取网页内容：

import requests
url = 'https://movie.douban.com/top250'
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'}
response = requests.get(url, headers=headers)
if response.status_code == 200:
    page_content = response.text
    print(page_content)
else:
    print(f"Failed to retrieve the page. Status code: {response.status_code}")

在上面的代码中，我们设置了User-Agent头，以模拟浏览器的请求，防止被网站检测到是爬虫程序。

二、使用BeautifulSoup库进行网页解析

BeautifulSoup库用于解析HTML文档，并提取我们需要的数据。首先，安装BeautifulSoup库：

pip install beautifulsoup4

接下来，使用BeautifulSoup解析网页内容并提取电影信息：

from bs4 import BeautifulSoup
soup = BeautifulSoup(page_content, 'html.parser')
movies = soup.find_all('div', class_='item')
for movie in movies:
    title = movie.find('span', class_='title').text
    rating = movie.find('span', class_='rating_num').text
    print(f"Title: {title}, Rating: {rating}")

三、处理分页

豆瓣TOP250页面有多个分页，每页25部电影。我们需要遍历所有分页，获取完整的TOP250数据。可以通过修改URL中的start参数来实现分页处理：

base_url = 'https://movie.douban.com/top250'
movies = []
for start in range(0, 250, 25):
    url = f"{base_url}?start={start}"
    response = requests.get(url, headers=headers)
    if response.status_code == 200:
        soup = BeautifulSoup(response.text, 'html.parser')
        items = soup.find_all('div', class_='item')
        for item in items:
            title = item.find('span', class_='title').text
            rating = item.find('span', class_='rating_num').text
            movies.append({'title': title, 'rating': rating})
    else:
        print(f"Failed to retrieve the page. Status code: {response.status_code}")
for movie in movies:
    print(f"Title: {movie['title']}, Rating: {movie['rating']}")

四、数据存储

获取数据后，可以将数据存储到本地文件中，例如CSV文件：

import csv
with open('douban_top250.csv', 'w', newline='', encoding='utf-8') as csvfile:
    fieldnames = ['title', 'rating']
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
    writer.writeheader()
    for movie in movies:
        writer.writerow(movie)

五、完整代码示例

import requests
from bs4 import BeautifulSoup
import csv
def get_movies(url, headers):
    response = requests.get(url, headers=headers)
    if response.status_code == 200:
        soup = BeautifulSoup(response.text, 'html.parser')
        items = soup.find_all('div', class_='item')
        movies = []
        for item in items:
            title = item.find('span', class_='title').text
            rating = item.find('span', class_='rating_num').text
            movies.append({'title': title, 'rating': rating})
        return movies
    else:
        print(f"Failed to retrieve the page. Status code: {response.status_code}")
        return []
def main():
    base_url = 'https://movie.douban.com/top250'
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'
    }
    all_movies = []
    for start in range(0, 250, 25):
        url = f"{base_url}?start={start}"
        movies = get_movies(url, headers)
        all_movies.extend(movies)
    with open('douban_top250.csv', 'w', newline='', encoding='utf-8') as csvfile:
        fieldnames = ['title', 'rating']
        writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
        writer.writeheader()
        for movie in all_movies:
            writer.writerow(movie)
if __name__ == "__main__":
    main()

六、总结

通过以上步骤，我们可以使用Python爬取豆瓣TOP250电影数据，并将其存储到本地文件中。使用requests库进行网页请求、使用BeautifulSoup库进行网页解析、处理分页和数据存储是实现这一任务的关键步骤。希望通过本文的介绍，您能顺利实现对豆瓣TOP250电影数据的爬取，并将其应用到实际项目中。