首页 > 编程语言 > 详细

python线程池应用场景-爬虫

时间:2019-02-04 10:26:12      阅读:217      评论:0      收藏:0      [点我收藏+]
import requests
from bs4 import BeautifulSoup
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor

def task(url):
    print(url)
    r1 = requests.get(
        url=url,
        headers={
            'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.92 Safari/537.36'
        }
    )

    # 查看下载下来的文本信息
    soup = BeautifulSoup(r1.text, 'html.parser')
    print(soup.text)
    content_list = soup.find('div',attrs={'id':'content-list'})
    for item in content_list.find_all('div',attrs={'class':'item'}):
        title = item.find('a').text.strip()
        target_url = item.find('a').get('href')
        print(title,target_url)


def run():
    pool = ThreadPoolExecutor(5)
    for i in range(1, 50):
        pool.submit(task, 'https://dig.chouti.com/all/hot/recent/%s' % i)


if __name__ == '__main__':
    run()

python线程池应用场景-爬虫

原文:https://www.cnblogs.com/apollo1616/p/10351487.html

(0)
(0)
   
举报
评论 一句话评论(0
关于我们 - 联系我们 - 留言反馈 - 联系我们:wmxa8@hotmail.com
© 2014 bubuko.com 版权所有
打开技术之扣,分享程序人生!