首页 > 编程语言 > 详细

Python数据解析实例--获取网站上小说内容

时间:2021-06-09 18:40:26      阅读:24      评论:0      收藏:0      [点我收藏+]
from bs4 import BeautifulSoup
import requests
if __name__ == ‘__main__‘:
headers = {‘User-Agent‘: ‘Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36‘}
url = ‘https://www.shicimingju.com/book/sanguoyanyi.html‘
page_text = requests.get(url=url,headers=headers).content.decode("utf-8")
soup = BeautifulSoup(page_text,‘lxml‘)
li_list = soup.select(‘.book-mulu > ul > li‘)
print(li_list)
fp = open(‘./三国演义小说.txt‘,‘w‘,encoding=‘utf-8‘)
for li in li_list:
title = li.a.string
detail_url = ‘https://www.shicimingju.com‘+li.a[‘href‘]
detail_page_text = requests.get(url=detail_url,headers=headers).content.decode("utf-8")
detail_soup = BeautifulSoup(detail_page_text, ‘lxml‘)
div_tag = detail_soup.find(‘div‘,class_=‘chapter_content‘)
content = div_tag.text
fp.write(‘\n‘ + title + ‘:‘ + content +‘\n‘)
print(title,‘爬取成功‘)

Python数据解析实例--获取网站上小说内容

原文:https://www.cnblogs.com/xxm2017/p/14867278.html

(0)
(0)
   
举报
评论 一句话评论(0
关于我们 - 联系我们 - 留言反馈 - 联系我们:wmxa8@hotmail.com
© 2014 bubuko.com 版权所有
打开技术之扣,分享程序人生!