前言
本文的文字及圖片來源于網路,僅供學習、交流使用,不具有任何商業用途,如有問題請及時聯系我們以作處理，
以下文章來源于青燈編程 ，作者：清風
Python爬蟲、資料分析、網站開發等案例教程視頻免費在線觀看
https://space.bilibili.com/523606542
對于這個外包給予的網站資訊，那么我們就爬取選擇第一個吧<獵聘網>，也就簡單實作，并不會進行打包處理，以及輸入關鍵字爬取，
本篇文章就使用三種爬蟲模式爬取相關資料
1、常規爬取資料
2、多執行緒爬取資料
3、scrapy框架爬取資料
基本開發環境
- Python 3.6
- Pycharm
相關模塊的使用
常規爬取資料
import requests import parsel import csv
多執行緒爬取資料
import requests import parsel import csv import threading
scrapy框架爬取資料
import scrapy import csv
目標網頁分析
爬取python招聘資訊資料
資料獲取：
1、標題
2、薪資
3、所在城市
4、學歷要求
5、作業經驗要求
6、公司名字
7、公司福利
8、公司融資情況
9、簡歷發布時間
......
該網頁是比較簡單的，靜態網頁沒有什么可以過多的分析，還是比較簡單的，
1、模擬瀏覽器請求網頁，獲取網頁源代碼資料
2、決議網頁源代碼，提取想要的資料內容
3、將提取的資料內容保存成csv檔案，或者其他形式
都說比較簡單了，那為什么這個外包還價值1000呢？難道外包賺錢真的這么簡單么，是不難，但是不意味著1K的外包就很好賺，畢竟別人只是簡單的給出幾個網站，首先看你是否能爬取其中的資料，其次甲方的要求肯定不至于此，資料量也不簡單，所以今天就以三個版本的爬蟲爬取資料，
外包的價格高低因素：
- 任務的難易程度
- 爬取的資料量
- 是否緊急需要
- 是否需要原始碼
- 后期是否需要更新代碼
...
常規爬蟲代碼
import requests import parsel import csv f = open('data.csv', mode='a', encoding='utf-8', newline='') csv_writer = csv.DictWriter(f, fieldnames=['標題', '薪資', '城市', '學歷', '作業經驗', '公司名字', '融資情況', '公司福利', '招聘時間', '簡歷反饋時間' ]) csv_writer.writeheader() for page in range(0, 10): url = 'https://www.liepin.com/zhaopin/' params = { 'compkind': '', 'dqs': '', 'pubTime': '', 'pageSize': '40', 'salary': '', 'compTag': '', 'sortFlag': '', 'degradeFlag': '0', 'compIds': '', 'subIndustry': '', 'jobKind': '', 'industries': '', 'compscale': '', 'key': 'python', 'siTag': 'I-7rQ0e90mv8a37po7dV3Q~fA9rXquZc5IkJpXC-Ycixw', 'd_sfrom': 'search_fp', 'd_ckId': 'cd74f9fdbdb63c6d462bad39feddc7f1', 'd_curPage': '2', 'd_pageSize': '40', 'd_headId': 'cd74f9fdbdb63c6d462bad39feddc7f1', 'curPage': page, } headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36'} response = requests.get(url=url, params=params, headers=headers) selector = parsel.Selector(response.text) lis = selector.css('div.job-content div:nth-child(1) ul li') for li in lis: title = li.css('.job-info h3 a::text').get().strip() money = li.css('.condition span.text-warning::text').get() city = li.css('.condition .area::text').get() edu = li.css('.condition .edu::text').get() experience = li.css('.condition span:nth-child(4)::text').get() company = li.css('.company-name a::text').get() financing = li.css('.field-financing span::text').get() temptation_list = li.css('p.temptation.clearfix span::text').getall() temptation_str = '|'.join(temptation_list) release_time = li.css('p.time-info.clearfix time::text').get() feedback_time = li.css('p.time-info.clearfix span::text').get() dit = { '標題': title, '薪資': money, '城市': city, '學歷': edu, '作業經驗': experience, '公司名字': company, '融資情況': financing, '公司福利': temptation_str, '招聘時間': release_time, '簡歷反饋時間': feedback_time, } csv_writer.writerow(dit) print(dit)
實作效果
多執行緒爬蟲
import requests import parsel import csv import threading f = open('data_1.csv', mode='a', encoding='utf-8', newline='') csv_writer = csv.DictWriter(f, fieldnames=['標題', '薪資', '城市', '學歷', '作業經驗', '公司名字', '融資情況', '公司福利', '招聘時間', '簡歷反饋時間' ]) csv_writer.writeheader() def get_response(html_url, p): headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36'} response = requests.get(url=html_url, params=p, headers=headers) return response def get_parsing(html_data): selector = parsel.Selector(html_data) return selector def main(p): url = 'https://www.liepin.com/zhaopin/' html_data = get_response(url, p).text selector = get_parsing(html_data) lis = selector.css('div.job-content div:nth-child(1) ul li') for li in lis: title = li.css('.job-info h3 a::text').get().strip() money = li.css('.condition span.text-warning::text').get() city = li.css('.condition .area::text').get() edu = li.css('.condition .edu::text').get() experience = li.css('.condition span:nth-child(4)::text').get() company = li.css('.company-name a::text').get() financing = li.css('.field-financing span::text').get() temptation_list = li.css('p.temptation.clearfix span::text').getall() temptation_str = '|'.join(temptation_list) release_time = li.css('p.time-info.clearfix time::text').get() feedback_time = li.css('p.time-info.clearfix span::text').get() dit = { '標題': title, '薪資': money, '城市': city, '學歷': edu, '作業經驗': experience, '公司名字': company, '融資情況': financing, '公司福利': temptation_str, '招聘時間': release_time, '簡歷反饋時間': feedback_time, } csv_writer.writerow(dit) print(dit) if __name__ == '__main__': for page in range(0, 10): params = { 'compkind': '', 'dqs': '', 'pubTime': '', 'pageSize': '40', 'salary': '', 'compTag': '', 'sortFlag': '', 'degradeFlag': '0', 'compIds': '', 'subIndustry': '', 'jobKind': '', 'industries': '', 'compscale': '', 'key': 'python', 'siTag': 'I-7rQ0e90mv8a37po7dV3Q~fA9rXquZc5IkJpXC-Ycixw', 'd_sfrom': 'search_fp', 'd_ckId': 'cd74f9fdbdb63c6d462bad39feddc7f1', 'd_curPage': '2', 'd_pageSize': '40', 'd_headId': 'cd74f9fdbdb63c6d462bad39feddc7f1', 'curPage': page, } main_thread = threading.Thread(target=main, args=(params,)) main_thread.start()
實作效果
scrapy爬蟲框架
items.py
import scrapy class LiepingwangItem(scrapy.Item): title = scrapy.Field() money = scrapy.Field() city = scrapy.Field() edu = scrapy.Field() experience = scrapy.Field() company = scrapy.Field() financing = scrapy.Field() temptation_str = scrapy.Field() release_time = scrapy.Field() feedback_time = scrapy.Field()
middlewares.py
class LiepingwangDownloaderMiddleware: def process_request(self, request, spider): request.headers.update( { 'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36', } ) return Noe
pipelines.py
import csv class LiepingwangPipeline: def __init__(self): self.file = open('data_2.csv', mode='a', encoding='utf-8', newline='') self.csv_file = csv.DictWriter(self.file, fieldnames=['title', 'money', 'city', 'edu', 'experience', 'company', 'financing', 'temptation_str', 'release_time', 'feedback_time' ]) self.csv_file.writeheader() def process_item(self, item, spider): dit = dict(item) dit['financing'] = dit['financing'].strip() dit['title'] = dit['title'].strip() self.csv_file.writerow(dit) return item def spider_closed(self, spider): self.file.close()
settings.py
ROBOTSTXT_OBEY = False DOWNLOADER_MIDDLEWARES = { 'liepingwang.middlewares.LiepingwangDownloaderMiddleware': 543, } ITEM_PIPELINES = { 'liepingwang.pipelines.LiepingwangPipeline': 300,
爬蟲檔案
import scrapy from ..items import LiepingwangItem class ZpinfoSpider(scrapy.Spider): name = 'zpinfo' allowed_domains = ['liepin.com'] start_urls = ['https://www.liepin.com/zhaopin/?sfrom=click-pc_homepage-centre_searchbox-search_new&d_sfrom=search_fp&key=python'] def parse(self, response): lis = response.css('div.job-content div:nth-child(1) ul li') for li in lis: title = li.css('.job-info h3 a::text').get().strip() money = li.css('.condition span.text-warning::text').get() city = li.css('.condition .area::text').get() edu = li.css('.condition .edu::text').get() experience = li.css('.condition span:nth-child(4)::text').get() company = li.css('.company-name a::text').get() financing = li.css('.field-financing span::text').get() temptation_list = li.css('p.temptation.clearfix span::text').getall() temptation_str = '|'.join(temptation_list) release_time = li.css('p.time-info.clearfix time::text').get() feedback_time = li.css('p.time-info.clearfix span::text').get() yield LiepingwangItem(title=title, money=money, city=city, edu=edu, experience=experience, company=company, financing=financing, temptation_str=temptation_str, release_time=release_time, feedback_time=feedback_time) href = response.css('div.job-content div:nth-child(1) a:nth-child(9)::attr(href)').get() if href: next_url = 'https://www.liepin.com' + href yield scrapy.Request(url=next_url, callback=self.parse)
實作效果
轉載請註明出處，本文鏈接：https://www.uj5u.com/houduan/245057.html
標籤：其他