Daily sharing :

A man's life is short , But if it's boring to live this life , It's too long .

I haven't experienced the pain of others , Don't persuade others to be magnanimous .

one ,scrapy.Request Parameters of


Parameter interpretation :

* The parameters in brackets are optional , Can write but not write
* callback: Indicates the current url Which function does the response go to ( Default to parse function )
meta: Realize the transfer of data in different analytic functions ,meta Default with partial data , For example, Download delay , Request depth, etc ( Used for data transfer between parsing methods , It is often used when a piece of data is scattered in multiple pages with different structures )
dont_filter: Default to False, Requests will be filtered url address , I.e. requested url The address will not continue to be requested , For requests that need to be repeated url The address can be set to True,start_urls The address in will be repeatedly requested , Otherwise, the program will not start
* headers: Receive a dictionary , Not included cookies
* cookies: Receive a dictionary , Special placement cookies
* method: appoint POST or GET request
* body: receive json character string , by post Data transmission payload_post request
two ,meta parameter

meta Role of :meta It can realize the transfer of data in different analytic functions

In the crawler file parse Method , Add a function parse_detail function ( Used to parse another page ):

def parse(self,response):
    yield scrapy.Request(detail_url,

def parse_detail(self,response):
    # Get previously passed in item
    item = resposne.meta["item"]

It's equivalent to , hold parse The data parsed in is saved to meta In the dictionary , Corresponding key by item; And in another function (parse_detail) in , adopt meta Dictionary key:item To extract parse Data in , So as to realize the splicing of different page data

be careful :

* meta Parameter is a dictionary
* meta There is a fixed key in the dictionary proxy, Represent agent ip

©2019-2020 Toolsou All rights reserved,
C++ of string of compare usage Python Study notes ( one )evo Tool usage problems ——Degenerate covariance rank, Umeyama alignment is not possibleRISC-V_GD32VF103-TIMER0 timer interrupt java Array subscript variable _Java Basic grammar : array be based on stm32 Control four-wheel trolley motor drive ( one ) be based on redis Design of liking function Software engineering career planning mysql Query random data by conditions _MySQL Random query of several qualified records centos7 install RabbitMq