Daily sharing ：
A man's life is short , But if it's boring to live this life , It's too long .
I haven't experienced the pain of others , Don't persuade others to be magnanimous .
one ,scrapy.Request Parameters of
scrapy.Request（url[,callback,method="GET",headers,body,cookies,meta,dont_filter=Fallse]）
Parameter interpretation ：
* The parameters in brackets are optional , Can write but not write
* callback： Indicates the current url Which function does the response go to （ Default to parse function ）
*
meta： Realize the transfer of data in different analytic functions ,meta Default with partial data , For example, Download delay , Request depth, etc （ Used for data transfer between parsing methods , It is often used when a piece of data is scattered in multiple pages with different structures ）
*
dont_filter： Default to False, Requests will be filtered url address , I.e. requested url The address will not continue to be requested , For requests that need to be repeated url The address can be set to True,start_urls The address in will be repeatedly requested , Otherwise, the program will not start
* headers： Receive a dictionary , Not included cookies
* cookies： Receive a dictionary , Special placement cookies
* method： appoint POST or GET request
* body： receive json character string , by post Data transmission payload_post request
two ,meta parameter
meta Role of ：meta It can realize the transfer of data in different analytic functions
In the crawler file parse Method , Add a function parse_detail function （ Used to parse another page ）：
def parse(self,response):
...
yield scrapy.Request(detail_url,
callback=self.parse_detail,meta={"item":item})
...
def parse_detail(self,response):
# Get previously passed in item
item = resposne.meta["item"]
It's equivalent to , hold parse The data parsed in is saved to meta In the dictionary , Corresponding key by item; And in another function （parse_detail） in , adopt meta Dictionary key：item To extract parse Data in , So as to realize the splicing of different page data
be careful ：
* meta Parameter is a dictionary
* meta There is a fixed key in the dictionary proxy, Represent agent ip
Technology
Daily Recommendation