scrapy API Reference
Full documentation for scrapy (2.17.0)
scrapy.Field
Python Library Documentation: class Field in module scrapy.item class Field(builtins.dict) | Container of field metadata | | Method resolution order: | Field | builtins.dict | builtins.object | | Data descriptors defined here: | | __dict__ | dictionary for instance variables | | __weakref__ | list of weak references to the object | | ---------------------------------------------------------------------- | Data and other attributes defined here: | | __orig_bases__ = (dict[str, typing.Any],) | | ---------------------------------------------------------------------- | Methods inherited from builtins.dict: | | __contains__(self, key, /) | True if the dictionary has the specified key, else False. | | __delitem__(self, key, /) | Delete self[key]. | | __eq__(self, value, /) | Return self==value. | | __ge__(self, value, /) | Return self>=value. | | __getattribute__(self, name, /) | Return getattr(self, name). | | __getitem__(self, key, /) | Return self[key]. | | __gt__(self, value, /) | Return self>value. | | __init__(self, /, *args, **kwargs) | Initialize self. See help(type(self)) for accurate signature. | | __ior__(self, value, /) | Return self|=value. | | __iter__(self, /) | Implement iter(self). | | __le__(self, value, /) | Return self<=value. | | __len__(self, /) | Return len(self). | | __lt__(self, value, /) | Return self<value. | | __ne__(self, value, /) | Return self!=value. | | __or__(self, value, /) | Return self|value. | | __repr__(self, /) | Return repr(self). | | __reversed__(self, /) | Return a reverse iterator over the dict keys. | | __ror__(self, value, /) | Return value|self. | | __setitem__(self, key, value, /) | Set self[key] to value. | | __sizeof__(...) | D.__sizeof__() -> size of D in memory, in bytes | | clear(...) | D.clear() -> None. Remove all items from D. | | copy(...) | D.copy() -> a shallow copy of D | | get(self, key, default=None, /) | Return the value for key if key is in the dictionary, else default. | | items(...) | D.items() -> a set-like object providing a view on D's items | | keys(...) | D.keys() -> a set-like object providing a view on D's keys | | pop(...) | D.pop(k[,d]) -> v, remove specified key and return the corresponding value. | | If the key is not found, return the default if given; otherwise, | raise a KeyError. | | popitem(self, /) | Remove and return a (key, value) pair as a 2-tuple. | | Pairs are returned in LIFO (last-in, first-out) order. | Raises KeyError if the dict is empty. | | setdefault(self, key, default=None, /) | Insert key with a value of default if key is not in the dictionary. | | Return the value for key if key is in the dictionary, else default. | | update(...) | D.update([E, ]**F) -> None. Update D from dict/iterable E and F. | If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] | If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v | In either case, this is followed by: for k in F: D[k] = F[k] | | values(...) | D.values() -> an object providing a view on D's values | | ---------------------------------------------------------------------- | Class methods inherited from builtins.dict: | | __class_getitem__(...) | See PEP 585 | | fromkeys(iterable, value=None, /) | Create a new dictionary with keys from iterable and values set to value. | | ---------------------------------------------------------------------- | Static methods inherited from builtins.dict: | | __new__(*args, **kwargs) class method of builtins.dict | Create and return a new object. See help(type) for accurate signature. | | ---------------------------------------------------------------------- | Data and other attributes inherited from builtins.dict: | | __hash__ = None
scrapy.FormRequest
Python Library Documentation: class FormRequest in module scrapy.http.request.form
class FormRequest(scrapy.http.request.Request)
| FormRequest(*args: 'Any', formdata: 'FormdataType' = None, **kwargs: 'Any') -> 'None'
|
| A :class:`~scrapy.Request` subclass with a ``formdata`` parameter that
| url-encodes the given data and assigns it to the request, which makes it
| convenient to send arbitrary form data via HTTP POST or GET without an HTML
| ``<form>`` element to parse.
|
| .. note:: To build a request from an HTML ``<form>`` element found in a
| response, use :doc:`form2request <form2request:index>` instead. See
| :ref:`form`.
|
| The remaining arguments are the same as for the :class:`~scrapy.Request`
| class and are not documented here.
|
| :param formdata: a dictionary (or iterable of (key, value) tuples)
| containing HTML form data which will be url-encoded. If
| :attr:`~scrapy.Request.method` is not given and ``formdata`` is
| provided, the method is set to ``"POST"`` and the data is assigned to
| the request body; if the method is ``"GET"``, the data is added to the
| URL query string instead.
| :type formdata: dict or collections.abc.Iterable
|
| To send data via HTTP POST, simulating an HTML form submission, return a
| :class:`~scrapy.FormRequest` object from your spider:
|
| .. skip: next
| .. code-block:: python
|
| return [
| FormRequest(
| url="http://www.example.com/post/action",
| formdata={"name": "John Doe", "age": "27"},
| callback=self.after_post,
| )
| ]
|
| To send the data in the URL query string instead, use the ``GET`` method:
|
| .. skip: next
| .. code-block:: python
|
| return [
| FormRequest(
| url="http://www.example.com/search",
| method="GET",
| formdata={"q": "keyword", "page": "1"},
| callback=self.parse_results,
| )
| ]
|
| Method resolution order:
| FormRequest
| scrapy.http.request.Request
| scrapy.utils.trackref.object_ref
| builtins.object
|
| Methods defined here:
|
| __init__(self, *args: 'Any', formdata: 'FormdataType' = None, **kwargs: 'Any') -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| from_response(response: 'TextResponse', formname: 'str | None' = None, formid: 'str | None' = None, formnumber: 'int' = 0, formdata: 'FormdataType' = None, clickdata: 'dict[str, str | int] | None' = None, dont_click: 'bool' = False, formxpath: 'str | None' = None, formcss: 'str | None' = None, **kwargs: 'Any') -> 'Self'
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'valid_form_methods': 'ClassVar[list[str]]'}
|
| valid_form_methods = ['GET', 'POST']
|
| ----------------------------------------------------------------------
| Methods inherited from scrapy.http.request.Request:
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| copy(self) -> 'Self'
|
| replace(self, *args: 'Any', cls: 'type[Request] | None' = None, **kwargs: 'Any') -> 'Request'
| Create a new Request with the same attributes except for those given new values
|
| to_dict(self, *, spider: 'scrapy.Spider | None' = None) -> 'dict[str, Any]'
| Return a dictionary containing the Request's data.
|
| Use :func:`~scrapy.utils.request.request_from_dict` to convert back into a :class:`~scrapy.Request` object.
|
| If a spider is given, this method will try to find out the name of the spider methods used as callback
| and errback and include them in the output dict, raising an exception if they cannot be found.
|
| ----------------------------------------------------------------------
| Class methods inherited from scrapy.http.request.Request:
|
| from_curl(curl_command: 'str', ignore_unknown_options: 'bool' = True, **kwargs: 'Any') -> 'Self'
| Create a Request object from a string containing a `cURL
| <https://curl.se/>`_ command. It populates the HTTP method, the
| URL, the headers, the cookies and the body. It accepts the same
| arguments as the :class:`Request` class, taking preference and
| overriding the values of the same arguments contained in the cURL
| command.
|
| Unrecognized options are ignored by default. To raise an error when
| finding unknown options call this method by passing
| ``ignore_unknown_options=False``.
|
| .. caution:: Using :meth:`from_curl` from :class:`~scrapy.Request`
| subclasses, such as :class:`~scrapy.http.JsonRequest`, or
| :class:`~scrapy.http.XmlRpcRequest`, as well as having
| :ref:`downloader middlewares <topics-downloader-middleware>`
| and
| :ref:`spider middlewares <topics-spider-middleware>`
| enabled, such as
| :class:`~scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware`,
| :class:`~scrapy.downloadermiddlewares.useragent.UserAgentMiddleware`,
| or
| :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`,
| may modify the :class:`~scrapy.Request` object.
|
| To translate a cURL command into a Scrapy request,
| you may use `curl2scrapy <https://michael-shub.github.io/curl2scrapy/>`_.
|
| ----------------------------------------------------------------------
| Readonly properties inherited from scrapy.http.request.Request:
|
| body
|
| cb_kwargs
|
| encoding
|
| meta
|
| url
|
| ----------------------------------------------------------------------
| Data descriptors inherited from scrapy.http.request.Request:
|
| __weakref__
| list of weak references to the object
|
| callback
|
| cookies
|
| dont_filter
|
| errback
|
| flags
|
| headers
|
| method
|
| priority
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from scrapy.http.request.Request:
|
| attributes = ('url', 'headers', 'body', 'cookies', 'meta', 'encoding',...
|
| ----------------------------------------------------------------------
| Static methods inherited from scrapy.utils.trackref.object_ref:
|
| __new__(cls, *args: 'Any', **kwargs: 'Any') -> 'Self'
| Create and return a new object. See help(type) for accurate signature.
scrapy.Item
Python Library Documentation: class Item in module scrapy.item
class Item(collections.abc.MutableMapping, scrapy.utils.trackref.object_ref)
| Item(*args: 'Any', **kwargs: 'Any')
|
| Base class for scraped items.
|
| In Scrapy, an object is considered an ``item`` if it's supported by the
| `itemadapter`_ library. For example, when the output of a spider callback
| is evaluated, only such objects are passed to :ref:`item pipelines
| <topics-item-pipeline>`. :class:`Item` is one of the classes supported by
| `itemadapter`_ by default.
|
| Items must declare :class:`Field` attributes, which are processed and stored
| in the ``fields`` attribute. This restricts the set of allowed field names
| and prevents typos, raising ``KeyError`` when referring to undefined fields.
| Additionally, fields can be used to define metadata and control the way
| data is processed internally. Please refer to the :ref:`documentation
| about fields <topics-items-fields>` for additional information.
|
| Unlike instances of :class:`dict`, instances of :class:`Item` may be
| :ref:`tracked <topics-leaks-trackrefs>` to debug memory leaks.
|
| .. _itemadapter: https://github.com/scrapy/itemadapter
|
| Method resolution order:
| Item
| collections.abc.MutableMapping
| collections.abc.Mapping
| collections.abc.Collection
| collections.abc.Sized
| collections.abc.Iterable
| collections.abc.Container
| scrapy.utils.trackref.object_ref
| builtins.object
|
| Methods defined here:
|
| __delitem__(self, key: 'str') -> 'None'
|
| __getattr__(self, name: 'str') -> 'NoReturn'
|
| __getitem__(self, key: 'str') -> 'Any'
|
| __hash__(self, /) from builtins.object
| Return hash(self).
|
| __init__(self, *args: 'Any', **kwargs: 'Any')
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self) -> 'Iterator[str]'
|
| __len__(self) -> 'int'
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| __setattr__(self, name: 'str', value: 'Any') -> 'None'
| Implement setattr(self, name, value).
|
| __setitem__(self, key: 'str', value: 'Any') -> 'None'
|
| copy(self) -> 'Self'
|
| deepcopy(self) -> 'Self'
| Return a :func:`~copy.deepcopy` of this item.
|
| keys(self) -> 'KeysView[str]'
| D.keys() -> a set-like object providing a view on D's keys
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| __annotations__ = {'fields': 'dict[str, Field]'}
|
| __orig_bases__ = (collections.abc.MutableMapping[str, typing.Any], <cl...
|
| fields = {}
|
| ----------------------------------------------------------------------
| Methods inherited from collections.abc.MutableMapping:
|
| clear(self)
| D.clear() -> None. Remove all items from D.
|
| pop(self, key, default=<object object at 0x0000025A356081F0>)
| D.pop(k[,d]) -> v, remove specified key and return the corresponding value.
| If key is not found, d is returned if given, otherwise KeyError is raised.
|
| popitem(self)
| D.popitem() -> (k, v), remove and return some (key, value) pair
| as a 2-tuple; but raise KeyError if D is empty.
|
| setdefault(self, key, default=None)
| D.setdefault(k[,d]) -> D.get(k,d), also set D[k]=d if k not in D
|
| update(self, other=(), /, **kwds)
| D.update([E, ]**F) -> None. Update D from mapping/iterable E and F.
| If E present and has a .keys() method, does: for k in E: D[k] = E[k]
| If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v
| In either case, this is followed by: for k, v in F.items(): D[k] = v
|
| ----------------------------------------------------------------------
| Methods inherited from collections.abc.Mapping:
|
| __contains__(self, key)
|
| __eq__(self, other)
| Return self==value.
|
| get(self, key, default=None)
| D.get(k[,d]) -> D[k] if k in D, else d. d defaults to None.
|
| items(self)
| D.items() -> a set-like object providing a view on D's items
|
| values(self)
| D.values() -> an object providing a view on D's values
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from collections.abc.Mapping:
|
| __reversed__ = None
|
| ----------------------------------------------------------------------
| Class methods inherited from collections.abc.Collection:
|
| __subclasshook__(C)
| Abstract classes can override this to customize issubclass().
|
| This is invoked early on by abc.ABCMeta.__subclasscheck__().
| It should return True, False or NotImplemented. If it returns
| NotImplemented, the normal algorithm is used. Otherwise, it
| overrides the normal algorithm (and the outcome is cached).
|
| ----------------------------------------------------------------------
| Class methods inherited from collections.abc.Iterable:
|
| __class_getitem__ = GenericAlias(...)
| Represent a PEP 585 generic type
|
| E.g. for t = list[int], t.__origin__ is list and t.__args__ is (int,).
|
| ----------------------------------------------------------------------
| Static methods inherited from scrapy.utils.trackref.object_ref:
|
| __new__(cls, *args: 'Any', **kwargs: 'Any') -> 'Self'
| Create and return a new object. See help(type) for accurate signature.
scrapy.Request
Python Library Documentation: class Request in module scrapy.http.request
class Request(scrapy.utils.trackref.object_ref)
| Request(url: 'str', callback: 'CallbackT | None' = None, method: 'str' = 'GET', headers: 'Mapping[AnyStr, Any] | Iterable[tuple[AnyStr, Any]] | None' = None, body: 'bytes | str | None' = None, cookies: 'CookiesT | None' = None, meta: 'dict[str, Any] | None' = None, encoding: 'str' = 'utf-8', priority: 'int' = 0, dont_filter: 'bool' = False, errback: 'Callable[[Failure], Any] | None' = None, flags: 'list[str] | None' = None, cb_kwargs: 'dict[str, Any] | None' = None) -> 'None'
|
| Represents an HTTP request, which is usually generated in a Spider and
| executed by the Downloader, thus generating a :class:`~scrapy.http.Response`.
|
| Method resolution order:
| Request
| scrapy.utils.trackref.object_ref
| builtins.object
|
| Methods defined here:
|
| __init__(self, url: 'str', callback: 'CallbackT | None' = None, method: 'str' = 'GET', headers: 'Mapping[AnyStr, Any] | Iterable[tuple[AnyStr, Any]] | None' = None, body: 'bytes | str | None' = None, cookies: 'CookiesT | None' = None, meta: 'dict[str, Any] | None' = None, encoding: 'str' = 'utf-8', priority: 'int' = 0, dont_filter: 'bool' = False, errback: 'Callable[[Failure], Any] | None' = None, flags: 'list[str] | None' = None, cb_kwargs: 'dict[str, Any] | None' = None) -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| copy(self) -> 'Self'
|
| replace(self, *args: 'Any', cls: 'type[Request] | None' = None, **kwargs: 'Any') -> 'Request'
| Create a new Request with the same attributes except for those given new values
|
| to_dict(self, *, spider: 'scrapy.Spider | None' = None) -> 'dict[str, Any]'
| Return a dictionary containing the Request's data.
|
| Use :func:`~scrapy.utils.request.request_from_dict` to convert back into a :class:`~scrapy.Request` object.
|
| If a spider is given, this method will try to find out the name of the spider methods used as callback
| and errback and include them in the output dict, raising an exception if they cannot be found.
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| from_curl(curl_command: 'str', ignore_unknown_options: 'bool' = True, **kwargs: 'Any') -> 'Self'
| Create a Request object from a string containing a `cURL
| <https://curl.se/>`_ command. It populates the HTTP method, the
| URL, the headers, the cookies and the body. It accepts the same
| arguments as the :class:`Request` class, taking preference and
| overriding the values of the same arguments contained in the cURL
| command.
|
| Unrecognized options are ignored by default. To raise an error when
| finding unknown options call this method by passing
| ``ignore_unknown_options=False``.
|
| .. caution:: Using :meth:`from_curl` from :class:`~scrapy.Request`
| subclasses, such as :class:`~scrapy.http.JsonRequest`, or
| :class:`~scrapy.http.XmlRpcRequest`, as well as having
| :ref:`downloader middlewares <topics-downloader-middleware>`
| and
| :ref:`spider middlewares <topics-spider-middleware>`
| enabled, such as
| :class:`~scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware`,
| :class:`~scrapy.downloadermiddlewares.useragent.UserAgentMiddleware`,
| or
| :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`,
| may modify the :class:`~scrapy.Request` object.
|
| To translate a cURL command into a Scrapy request,
| you may use `curl2scrapy <https://michael-shub.github.io/curl2scrapy/>`_.
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| body
|
| cb_kwargs
|
| encoding
|
| meta
|
| url
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| callback
|
| cookies
|
| dont_filter
|
| errback
|
| flags
|
| headers
|
| method
|
| priority
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'attributes': 'tuple[str, ...]'}
|
| attributes = ('url', 'headers', 'body', 'cookies', 'meta', 'encoding',...
|
| ----------------------------------------------------------------------
| Static methods inherited from scrapy.utils.trackref.object_ref:
|
| __new__(cls, *args: 'Any', **kwargs: 'Any') -> 'Self'
| Create and return a new object. See help(type) for accurate signature.
scrapy.Selector
Python Library Documentation: class Selector in module scrapy.selector.unified
class Selector(parsel.selector.Selector, scrapy.utils.trackref.object_ref)
| Selector(response: 'TextResponse | None' = None, text: 'str | None' = None, type: 'SelectorType | None' = None, root: 'Any | None' = <object object at 0x0000025A356096E0>, **kwargs: 'Any')
|
| An instance of :class:`Selector` is a wrapper over response to select
| certain parts of its content.
|
| ``response`` is an :class:`~scrapy.http.HtmlResponse` or an
| :class:`~scrapy.http.XmlResponse` object that will be used for selecting
| and extracting data.
|
| ``text`` is a unicode string or utf-8 encoded text for cases when a
| ``response`` isn't available. Using ``text`` and ``response`` together is
| undefined behavior.
|
| ``type`` defines the selector type, it can be ``"html"``, ``"xml"``,
| ``"json"``, ``"text"`` or ``None`` (default). It's passed to
| :class:`parsel.Selector` and its meaning is defined there. However, when
| ``type`` is ``None``, it is set to ``"xml"`` for an
| :class:`~scrapy.http.XmlResponse` and to ``"html"`` otherwise before
| passing it to :class:`parsel.Selector`.
|
| .. note:: JSON selector support requires ``parsel`` 1.8.0 or higher. With
| older versions setting ``type`` to ``"json"`` or ``"text"`` is not
| supported.
|
| Method resolution order:
| Selector
| parsel.selector.Selector
| scrapy.utils.trackref.object_ref
| builtins.object
|
| Methods defined here:
|
| __init__(self, response: 'TextResponse | None' = None, text: 'str | None' = None, type: 'SelectorType | None' = None, root: 'Any | None' = <object object at 0x0000025A356096E0>, **kwargs: 'Any')
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| response
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| selectorlist_cls = <class 'scrapy.selector.unified.SelectorList'>
| The :class:`SelectorList` class is a subclass of the builtin ``list``
| class, which provides a few additional methods.
|
|
| ----------------------------------------------------------------------
| Methods inherited from parsel.selector.Selector:
|
| __bool__(self) -> 'bool'
| Return ``True`` if there is any real content selected or ``False``
| otherwise. In other words, the boolean value of a :class:`Selector` is
| given by the contents it selects.
|
| __getstate__(self) -> 'Any'
| Helper for pickle.
|
| __nonzero__ = __bool__(self) -> 'bool'
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| __str__(self) -> 'str'
| Return str(self).
|
| css(self, query: 'str') -> 'SelectorList[Self]'
| Apply the given CSS selector and return a :class:`SelectorList` instance.
|
| ``query`` is a string containing the CSS selector to apply.
|
| In the background, CSS queries are translated into XPath queries using
| `cssselect`_ library and run ``.xpath()`` method.
|
| .. _cssselect: https://pypi.python.org/pypi/cssselect/
|
| drop(self) -> 'None'
| Drop matched nodes from the parent element.
|
| extract = get(self) -> 'Any'
|
| get(self) -> 'Any'
| Serialize and return the matched nodes.
|
| For HTML and XML, the result is always a string, and percent-encoded
| content is unquoted.
|
| getall(self) -> 'list[str]'
| Serialize and return the matched node in a 1-element list of strings.
|
| jmespath(self, query: 'str', **kwargs: 'Any') -> 'SelectorList[Self]'
| Find objects matching the JMESPath ``query`` and return the result as a
| :class:`SelectorList` instance with all elements flattened. List
| elements implement :class:`Selector` interface too.
|
| ``query`` is a string containing the `JMESPath
| <https://jmespath.org/>`_ query to apply.
|
| Any additional named arguments are passed to the underlying
| ``jmespath.search`` call, e.g.::
|
| selector.jmespath('author.name', options=jmespath.Options(dict_cls=collections.OrderedDict))
|
| re(self, regex: 'str | Pattern[str]', replace_entities: 'bool' = True) -> 'list[str]'
| Apply the given regex and return a list of strings with the
| matches.
|
| ``regex`` can be either a compiled regular expression or a string which
| will be compiled to a regular expression using ``re.compile(regex)``.
|
| By default, character entity references are replaced by their
| corresponding character (except for ``&`` and ``<``).
| Passing ``replace_entities`` as ``False`` switches off these
| replacements.
|
| re_first(self, regex: 'str | Pattern[str]', default: 'str | None' = None, replace_entities: 'bool' = True) -> 'str | None'
| Apply the given regex and return the first string which matches. If
| there is no match, return the default value (``None`` if the argument
| is not provided).
|
| By default, character entity references are replaced by their
| corresponding character (except for ``&`` and ``<``).
| Passing ``replace_entities`` as ``False`` switches off these
| replacements.
|
| register_namespace(self, prefix: 'str', uri: 'str') -> 'None'
| Register the given namespace to be used in this :class:`Selector`.
| Without registering namespaces you can't select or extract data from
| non-standard namespaces. See :ref:`selector-examples-xml`.
|
| remove_namespaces(self) -> 'None'
| Remove all namespaces, allowing to traverse the document using
| namespace-less xpaths. See :ref:`removing-namespaces`.
| For JSON selectors, this method does nothing.
|
| xpath(self, query: 'str', namespaces: 'Mapping[str, str] | None' = None, **kwargs: 'Any') -> 'SelectorList[Self]'
| Find nodes matching the xpath ``query`` and return the result as a
| :class:`SelectorList` instance with all elements flattened. List
| elements implement :class:`Selector` interface too.
|
| ``query`` is a string containing the XPATH query to apply.
|
| ``namespaces`` is an optional ``prefix: namespace-uri`` mapping (dict)
| for additional prefixes to those registered with ``register_namespace(prefix, uri)``.
| Contrary to ``register_namespace()``, these prefixes are not
| saved for future calls.
|
| Any additional named arguments can be used to pass values for XPath
| variables in the XPath expression, e.g.::
|
| selector.xpath('//a[href=$url]', url="http://www.example.com")
|
| ----------------------------------------------------------------------
| Readonly properties inherited from parsel.selector.Selector:
|
| attrib
| Return the attributes dictionary for underlying element.
| For JSON selectors, return an empty dict.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from parsel.selector.Selector:
|
| __weakref__
| list of weak references to the object
|
| body
|
| namespaces
|
| root
|
| type
|
| ----------------------------------------------------------------------
| Static methods inherited from scrapy.utils.trackref.object_ref:
|
| __new__(cls, *args: 'Any', **kwargs: 'Any') -> 'Self'
| Create and return a new object. See help(type) for accurate signature.
scrapy.Spider
Python Library Documentation: class Spider in module scrapy.spiders
class Spider(scrapy.utils.trackref.object_ref)
| Spider(name: 'str | None' = None, **kwargs: 'Any')
|
| Base class that any spider must subclass.
|
| It provides a default :meth:`start` implementation that sends
| requests based on the :attr:`start_urls` class attribute and calls the
| :meth:`parse` method for each response.
|
| Method resolution order:
| Spider
| scrapy.utils.trackref.object_ref
| builtins.object
|
| Methods defined here:
|
| __init__(self, name: 'str | None' = None, **kwargs: 'Any')
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| log(self, message: 'Any', level: 'int' = 10, **kw: 'Any') -> 'None'
| Log the given message at the given log level
|
| This helper wraps a log call to the logger within the spider, but you
| can use it directly (e.g. Spider.logger.info('msg')) or use any other
| Python logger too.
|
| parse(self, response: 'Response', **kwargs: 'Any') -> 'Any'
|
| async start(self) -> 'AsyncIterator[Any]'
| Yield the initial :class:`~scrapy.Request` objects to send.
|
| .. versionadded:: 2.13
|
| For example:
|
| .. code-block:: python
|
| from scrapy import Request, Spider
|
|
| class MySpider(Spider):
| name = "myspider"
|
| async def start(self):
| yield Request("https://toscrape.com/")
|
| The default implementation reads URLs from :attr:`start_urls` and
| yields a request for each with :attr:`~scrapy.Request.dont_filter`
| enabled. It is functionally equivalent to:
|
| .. code-block:: python
|
| async def start(self):
| for url in self.start_urls:
| yield Request(url, dont_filter=True)
|
| You can also yield :ref:`items <topics-items>`. For example:
|
| .. code-block:: python
|
| async def start(self):
| yield {"foo": "bar"}
|
| To write spiders that work on Scrapy versions lower than 2.13,
| define also a synchronous ``start_requests()`` method that returns an
| iterable. For example:
|
| .. code-block:: python
|
| def start_requests(self):
| yield Request("https://toscrape.com/")
|
| .. seealso:: :ref:`start-requests`
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| from_crawler(crawler: 'Crawler', *args: 'Any', **kwargs: 'Any') -> 'Self'
|
| handles_request(request: 'Request') -> 'bool'
|
| update_settings(settings: 'BaseSettings') -> 'None'
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| close(spider: 'Spider', reason: 'str') -> 'Deferred[None] | None'
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| logger
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'custom_settings': 'dict[str, Any] | None', 'name':...
|
| custom_settings = None
|
| ----------------------------------------------------------------------
| Static methods inherited from scrapy.utils.trackref.object_ref:
|
| __new__(cls, *args: 'Any', **kwargs: 'Any') -> 'Self'
| Create and return a new object. See help(type) for accurate signature.
scrapy.exceptions
Python Library Documentation: module scrapy.exceptions in scrapy
NAME
scrapy.exceptions - Scrapy core exceptions
DESCRIPTION
These exceptions are documented in docs/topics/exceptions.rst. Please don't add
new exceptions here without documenting them there.
CLASSES
builtins.AssertionError(builtins.Exception)
ContractFail
builtins.Exception(builtins.BaseException)
CannotResolveHostError
CloseSpider
DontCloseSpider
DownloadCancelledError
DownloadConnectionRefusedError
DownloadFailedError
DownloadTimeoutError
DropItem
IgnoreRequest
NotConfigured
NotSupported
ResponseDataLossError
StopDownload
UnsupportedURLSchemeError
UsageError
builtins.Warning(builtins.Exception)
ScrapyDeprecationWarning
class CannotResolveHostError(builtins.Exception)
| Indicates that the provided hostname cannot be resolved.
|
| Method resolution order:
| CannotResolveHostError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class CloseSpider(builtins.Exception)
| CloseSpider(reason: 'str' = 'cancelled')
|
| Raise this from callbacks to request the spider to be closed
|
| Method resolution order:
| CloseSpider
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Methods defined here:
|
| __init__(self, reason: 'str' = 'cancelled')
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class ContractFail(builtins.AssertionError)
| Error raised in case of a failing contract
|
| Method resolution order:
| ContractFail
| builtins.AssertionError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.AssertionError:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.AssertionError:
|
| __new__(*args, **kwargs) class method of builtins.AssertionError
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class DontCloseSpider(builtins.Exception)
| Request the spider not to be closed yet
|
| Method resolution order:
| DontCloseSpider
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class DownloadCancelledError(builtins.Exception)
| Indicates that a request download was cancelled.
|
| Method resolution order:
| DownloadCancelledError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class DownloadConnectionRefusedError(builtins.Exception)
| Indicates that a connection was refused by the server.
|
| Method resolution order:
| DownloadConnectionRefusedError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class DownloadFailedError(builtins.Exception)
| Indicates that a request download has failed.
|
| Method resolution order:
| DownloadFailedError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class DownloadTimeoutError(builtins.Exception)
| Indicates that a request download has timed out.
|
| Method resolution order:
| DownloadTimeoutError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class DropItem(builtins.Exception)
| DropItem(message: 'str', log_level: 'str | None' = None)
|
| Drop item from the item pipeline
|
| Method resolution order:
| DropItem
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Methods defined here:
|
| __init__(self, message: 'str', log_level: 'str | None' = None)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class IgnoreRequest(builtins.Exception)
| Indicates a decision was made not to process a request
|
| Method resolution order:
| IgnoreRequest
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class NotConfigured(builtins.Exception)
| Indicates a missing configuration situation
|
| Method resolution order:
| NotConfigured
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class NotSupported(builtins.Exception)
| Indicates a feature or method is not supported
|
| Method resolution order:
| NotSupported
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class ResponseDataLossError(builtins.Exception)
| Indicates that Scrapy couldn't get a complete response.
|
| Method resolution order:
| ResponseDataLossError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class ScrapyDeprecationWarning(builtins.Warning)
| Warning category for deprecated features, since the default
| :exc:`DeprecationWarning` is silenced.
|
| Method resolution order:
| ScrapyDeprecationWarning
| builtins.Warning
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Warning:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Warning:
|
| __new__(*args, **kwargs) class method of builtins.Warning
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class StopDownload(builtins.Exception)
| StopDownload(*, fail: 'bool' = True)
|
| Stop the download of the body for a given response.
| The 'fail' boolean parameter indicates whether or not the resulting partial response
| should be handled by the request errback. Note that 'fail' is a keyword-only argument.
|
| Method resolution order:
| StopDownload
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Methods defined here:
|
| __init__(self, *, fail: 'bool' = True)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'response': 'Response | None'}
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class UnsupportedURLSchemeError(builtins.Exception)
| Indicates that the URL scheme is not supported.
|
| Method resolution order:
| UnsupportedURLSchemeError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.Exception:
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
class UsageError(builtins.Exception)
| UsageError(*a: 'Any', **kw: 'Any')
|
| To indicate a command-line usage error
|
| Method resolution order:
| UsageError
| builtins.Exception
| builtins.BaseException
| builtins.object
|
| Methods defined here:
|
| __init__(self, *a: 'Any', **kw: 'Any')
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.Exception:
|
| __new__(*args, **kwargs) class method of builtins.Exception
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.BaseException:
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __reduce__(...)
| Helper for pickle.
|
| __repr__(self, /)
| Return repr(self).
|
| __setstate__(...)
|
| __str__(self, /)
| Return str(self).
|
| add_note(...)
| Exception.add_note(note) --
| add a note to the exception
|
| with_traceback(...)
| Exception.with_traceback(tb) --
| set self.__traceback__ to tb and return self.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.BaseException:
|
| __cause__
| exception cause
|
| __context__
| exception context
|
| __dict__
|
| __suppress_context__
|
| __traceback__
|
| args
DATA
TYPE_CHECKING = False
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\scrapy\exceptions.py
scrapy.http
Python Library Documentation: package scrapy.http in scrapy
NAME
scrapy.http - Module containing all HTTP related classes
DESCRIPTION
Use this module (instead of the more specific ones) when importing Headers,
Request and Response outside this module.
PACKAGE CONTENTS
cookies
headers
request (package)
response (package)
CLASSES
scrapy.utils.datatypes.CaselessDict(builtins.dict)
scrapy.http.headers.Headers
scrapy.utils.trackref.object_ref(builtins.object)
scrapy.http.request.Request
scrapy.http.request.form.FormRequest
scrapy.http.request.json_request.JsonRequest
scrapy.http.request.rpc.XmlRpcRequest
scrapy.http.response.Response
scrapy.http.response.text.TextResponse
scrapy.http.response.html.HtmlResponse
scrapy.http.response.json.JsonResponse
scrapy.http.response.xml.XmlResponse
class FormRequest(scrapy.http.request.Request)
| FormRequest(*args: 'Any', formdata: 'FormdataType' = None, **kwargs: 'Any') -> 'None'
|
| A :class:`~scrapy.Request` subclass with a ``formdata`` parameter that
| url-encodes the given data and assigns it to the request, which makes it
| convenient to send arbitrary form data via HTTP POST or GET without an HTML
| ``<form>`` element to parse.
|
| .. note:: To build a request from an HTML ``<form>`` element found in a
| response, use :doc:`form2request <form2request:index>` instead. See
| :ref:`form`.
|
| The remaining arguments are the same as for the :class:`~scrapy.Request`
| class and are not documented here.
|
| :param formdata: a dictionary (or iterable of (key, value) tuples)
| containing HTML form data which will be url-encoded. If
| :attr:`~scrapy.Request.method` is not given and ``formdata`` is
| provided, the method is set to ``"POST"`` and the data is assigned to
| the request body; if the method is ``"GET"``, the data is added to the
| URL query string instead.
| :type formdata: dict or collections.abc.Iterable
|
| To send data via HTTP POST, simulating an HTML form submission, return a
| :class:`~scrapy.FormRequest` object from your spider:
|
| .. skip: next
| .. code-block:: python
|
| return [
| FormRequest(
| url="http://www.example.com/post/action",
| formdata={"name": "John Doe", "age": "27"},
| callback=self.after_post,
| )
| ]
|
| To send the data in the URL query string instead, use the ``GET`` method:
|
| .. skip: next
| .. code-block:: python
|
| return [
| FormRequest(
| url="http://www.example.com/search",
| method="GET",
| formdata={"q": "keyword", "page": "1"},
| callback=self.parse_results,
| )
| ]
|
| Method resolution order:
| FormRequest
| scrapy.http.request.Request
| scrapy.utils.trackref.object_ref
| builtins.object
|
| Methods defined here:
|
| __init__(self, *args: 'Any', formdata: 'FormdataType' = None, **kwargs: 'Any') -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| from_response(response: 'TextResponse', formname: 'str | None' = None, formid: 'str | None' = None, formnumber: 'int' = 0, formdata: 'FormdataType' = None, clickdata: 'dict[str, str | int] | None' = None, dont_click: 'bool' = False, formxpath: 'str | None' = None, formcss: 'str | None' = None, **kwargs: 'Any') -> 'Self'
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'valid_form_methods': 'ClassVar[list[str]]'}
|
| valid_form_methods = ['GET', 'POST']
|
| ----------------------------------------------------------------------
| Methods inherited from scrapy.http.request.Request:
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| copy(self) -> 'Self'
|
| replace(self, *args: 'Any', cls: 'type[Request] | None' = None, **kwargs: 'Any') -> 'Request'
| Create a new Request with the same attributes except for those given new values
|
| to_dict(self, *, spider: 'scrapy.Spider | None' = None) -> 'dict[str, Any]'
| Return a dictionary containing the Request's data.
|
| Use :func:`~scrapy.utils.request.request_from_dict` to convert back into a :class:`~scrapy.Request` object.
|
| If a spider is given, this method will try to find out the name of the spider methods used as callback
| and errback and include them in the output dict, raising an exception if they cannot be found.
|
| ----------------------------------------------------------------------
| Class methods inherited from scrapy.http.request.Request:
|
| from_curl(curl_command: 'str', ignore_unknown_options: 'bool' = True, **kwargs: 'Any') -> 'Self'
| Create a Request object from a string containing a `cURL
| <https://curl.se/>`_ command. It populates the HTTP method, the
| URL, the headers, the cookies and the body. It accepts the same
| arguments as the :class:`Request` class, taking preference and
| overriding the values of the same arguments contained in the cURL
| command.
|
| Unrecognized options are ignored by default. To raise an error when
| finding unknown options call this method by passing
| ``ignore_unknown_options=False``.
|
| .. caution:: Using :meth:`from_curl` from :class:`~scrapy.Request`
| subclasses, such as :class:`~scrapy.http.JsonRequest`, or
| :class:`~scrapy.http.XmlRpcRequest`, as well as having
| :ref:`downloader middlewares <topics-downloader-middleware>`
| and
| :ref:`spider middlewares <topics-spider-middleware>`
| enabled, such as
| :class:`~scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware`,
| :class:`~scrapy.downloadermiddlewares.useragent.UserAgentMiddleware`,
| or
| :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`,
| may modify the :class:`~scrapy.Request` object.
|
| To translate a cURL command into a Scrapy request,
| you may use `curl2scrapy <https://michael-shub.github.io/curl2scrapy/>`_.
|
| ----------------------------------------------------------------------
| Readonly properties inherited from scrapy.http.request.Request:
|
| body
|
| cb_kwargs
|
| encoding
|
| meta
|
| url
|
| ----------------------------------------------------------------------
| Data descriptors inherited from scrapy.http.request.Request:
|
| __weakref__
| list of weak references to the object
|
| callback
|
| cookies
|
| dont_filter
|
| errback
|
| flags
|
| headers
|
| method
|
| priority
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from scrapy.http.request.Request:
|
| attributes = ('url', 'headers', 'body', 'cookies', 'meta', 'encoding',...
|
| ----------------------------------------------------------------------
| Static methods inherited from scrapy.utils.trackref.object_ref:
|
| __new__(cls, *args: 'Any', **kwargs: 'Any') -> 'Self'
| Create and return a new object. See help(type) for accurate signature.
class Headers(scrapy.utils.datatypes.CaselessDict)
| Headers(seq: 'Mapping[AnyStr, Any] | Iterable[tuple[AnyStr, Any]] | None' = None, encoding: 'str' = 'utf-8')
|
| Case insensitive http headers dictionary
|
| Method resolution order:
| Headers
| scrapy.utils.datatypes.CaselessDict
| builtins.dict
| builtins.object
|
| Methods defined here:
|
| __copy__(self) -> 'Self'
|
| __getitem__(self, key: 'AnyStr') -> 'bytes | None'
| Return self[key].
|
| __init__(self, seq: 'Mapping[AnyStr, Any] | Iterable[tuple[AnyStr, Any]] | None' = None, encoding: 'str' = 'utf-8')
| Initialize self. See help(type(self)) for accurate signature.
|
| appendlist(self, key: 'AnyStr', value: 'Iterable[_RawValue]') -> 'None'
|
| copy = __copy__(self) -> 'Self'
|
| get(self, key: 'AnyStr', def_val: 'Any' = None) -> 'bytes | None'
| Return the value for key if key is in the dictionary, else default.
|
| getlist(self, key: 'AnyStr', def_val: 'Any' = None) -> 'list[bytes]'
|
| items(self) -> 'Iterable[tuple[bytes, list[bytes]]]'
| D.items() -> a set-like object providing a view on D's items
|
| normkey(self, key: 'AnyStr') -> 'bytes'
| Normalize key to bytes
|
| normvalue(self, value: '_RawValue | Iterable[_RawValue]') -> 'list[bytes]'
| Normalize values to bytes
|
| setlist(self, key: 'AnyStr', list_: 'Iterable[_RawValue]') -> 'None'
|
| setlistdefault(self, key: 'AnyStr', default_list: 'Iterable[_RawValue]' = ()) -> 'Any'
|
| to_string(self) -> 'bytes'
|
| to_tuple_list(self) -> 'list[tuple[str, str]]'
| Return headers as a list of ``(key, value)`` tuples.
|
| Multiple values are represented as multiple tuples with the same key.
|
| to_unicode_dict(self) -> 'CaseInsensitiveDict'
| Return headers as a CaseInsensitiveDict with str keys
| and str values. Multiple values are joined with ','.
|
| update(self, seq: 'Mapping[AnyStr, Any] | Iterable[tuple[AnyStr, Any]]') -> 'None'
| D.update([E, ]**F) -> None. Update D from dict/iterable E and F.
| If E is present and has a .keys() method, then does: for k in E: D[k] = E[k]
| If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v
| In either case, this is followed by: for k in F: D[k] = F[k]
|
| values(self) -> 'list[bytes | None]'
| D.values() -> an object providing a view on D's values
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from scrapy.utils.datatypes.CaselessDict:
|
| __contains__(self, key: 'AnyStr') -> 'bool'
| True if the dictionary has the specified key, else False.
|
| __delitem__(self, key: 'AnyStr') -> 'None'
| Delete self[key].
|
| __setitem__(self, key: 'AnyStr', value: 'Any') -> 'None'
| Set self[key] to value.
|
| has_key = __contains__(self, key: 'AnyStr') -> 'bool'
|
| pop(self, key: 'AnyStr', *args: 'Any') -> 'Any'
| D.pop(k[,d]) -> v, remove specified key and return the corresponding value.
|
| If the key is not found, return the default if given; otherwise,
| raise a KeyError.
|
| setdefault(self, key: 'AnyStr', def_val: 'Any' = None) -> 'Any'
| Insert key with a value of default if key is not in the dictionary.
|
| Return the value for key if key is in the dictionary, else default.
|
| ----------------------------------------------------------------------
| Class methods inherited from scrapy.utils.datatypes.CaselessDict:
|
| fromkeys(keys: 'Iterable[AnyStr]', value: 'Any' = None) -> 'Self'
| Create a new dictionary with keys from iterable and values set to value.
|
| ----------------------------------------------------------------------
| Static methods inherited from scrapy.utils.datatypes.CaselessDict:
|
| __new__(cls, *args: 'Any', **kwargs: 'Any') -> 'Self'
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.dict:
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __gt__(self, value, /)
| Return self>value.
|
| __ior__(self, value, /)
| Return self|=value.
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __or__(self, value, /)
| Return self|value.
|
| __repr__(self, /)
| Return repr(self).
|
| __reversed__(self, /)
| Return a reverse iterator over the dict keys.
|
| __ror__(self, value, /)
| Return value|self.
|
| __sizeof__(...)
| D.__sizeof__() -> size of D in memory, in bytes
|
| clear(...)
| D.clear() -> None. Remove all items from D.
|
| keys(...)
| D.keys() -> a set-like object providing a view on D's keys
|
| popitem(self, /)
| Remove and return a (key, value) pair as a 2-tuple.
|
| Pairs are returned in LIFO (last-in, first-out) order.
| Raises KeyError if the dict is empty.
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.dict:
|
| __class_getitem__(...)
| See PEP 585
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from builtins.dict:
|
| __hash__ = None
class HtmlResponse(scrapy.http.response.text.TextResponse)
| HtmlResponse(*args: 'Any', **kwargs: 'Any')
|
| Method resolution order:
| HtmlResponse
| scrapy.http.response.text.TextResponse
| scrapy.http.response.Response
| scrapy.utils.trackref.object_ref
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Methods inherited from scrapy.http.response.text.TextResponse:
|
| __init__(self, *args: 'Any', **kwargs: 'Any')
| Initialize self. See help(type(self)) for accurate signature.
|
| css(self, query: 'str') -> 'SelectorList'
| Shortcut method implemented only by responses whose content
| is text (subclasses of TextResponse).
|
| follow(self, url: 'str | Link | parsel.Selector', callback: 'CallbackT | None' = None, method: 'str' = 'GET', headers: 'Mapping[AnyStr, Any] | Iterable[tuple[AnyStr, Any]] | None' = None, body: 'bytes | str | None' = None, cookies: 'CookiesT | None' = None, meta: 'dict[str, Any] | None' = None, encoding: 'str | None' = None, priority: 'int' = 0, dont_filter: 'bool' = False, errback: 'Callable[[Failure], Any] | None' = None, cb_kwargs: 'dict[str, Any] | None' = None, flags: 'list[str] | None' = None) -> 'Request'
| Return a :class:`~.Request` instance to follow a link ``url``.
| It accepts the same arguments as ``Request.__init__()`` method,
| but ``url`` can be not only an absolute URL, but also
|
| * a relative URL
| * a :class:`~scrapy.link.Link` object, e.g. the result of
| :ref:`topics-link-extractors`
| * a :class:`~scrapy.Selector` object for a ``<link>`` or ``<a>`` element, e.g.
| ``response.css('a.my_link')[0]``
| * an attribute :class:`~scrapy.Selector` (not SelectorList), e.g.
| ``response.css('a::attr(href)')[0]`` or
| ``response.xpath('//img/@src')[0]``
|
| See :ref:`response-follow-example` for usage examples.
|
| follow_all(self, urls: 'Iterable[str | Link] | parsel.SelectorList[Any] | None' = None, callback: 'CallbackT | None' = None, method: 'str' = 'GET', headers: 'Mapping[AnyStr, Any] | Iterable[tuple[AnyStr, Any]] | None' = None, body: 'bytes | str | None' = None, cookies: 'CookiesT | None' = None, meta: 'dict[str, Any] | None' = None, encoding: 'str | None' = None, priority: 'int' = 0, dont_filter: 'bool' = False, errback: 'Callable[[Failure], Any] | None' = None, cb_kwargs: 'dict[str, Any] | None' = None, flags: 'list[str] | None' = None, css: 'str | None' = None, xpath: 'str | None' = None) -> 'Iterable[Request]'
| A generator that produces :class:`~.Request` instances to follow all
| links in ``urls``. It accepts the same arguments as the :class:`~.Request`'s
| ``__init__()`` method, except that each ``urls`` element does not need to be
| an absolute URL, it can be any of the following:
|
| * a relative URL
| * a :class:`~scrapy.link.Link` object, e.g. the result of
| :ref:`topics-link-extractors`
| * a :class:`~scrapy.Selector` object for a ``<link>`` or ``<a>`` element, e.g.
| ``response.css('a.my_link')[0]``
| * an attribute :class:`~scrapy.Selector` (not SelectorList), e.g.
| ``response.css('a::attr(href)')[0]`` or
| ``response.xpath('//img/@src')[0]``
|
| In addition, ``css`` and ``xpath`` arguments are accepted to perform the link extraction
| within the ``follow_all()`` method (only one of ``urls``, ``css`` and ``xpath`` is accepted).
|
| Note that when passing a ``SelectorList`` as argument for the ``urls`` parameter or
| using the ``css`` or ``xpath`` parameters, this method will not produce requests for
| selectors from which links cannot be obtained (for instance, anchor tags without an
| ``href`` attribute)
|
| jmespath(self, query: 'str', **kwargs: 'Any') -> 'SelectorList'
| Shortcut method implemented only by responses whose content
| is text (subclasses of TextResponse).
|
| json(self) -> 'Any'
| Deserialize a JSON document to a Python object.
|
| urljoin(self, url: 'str') -> 'str'
| Join this Response's url with a possible relative url to form an
| absolute interpretation of the latter.
|
| xpath(self, query: 'str', **kwargs: 'Any') -> 'SelectorList'
| Shortcut method implemented only by responses whose content
| is text (subclasses of TextResponse).
|
| ----------------------------------------------------------------------
| Readonly properties inherited from scrapy.http.response.text.TextResponse:
|
| encoding
|
| selector
|
| text
| Body as unicode
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from scrapy.http.response.text.TextResponse:
|
| attributes = ('url', 'headers', 'body', 'flags', 'status', 'request', ...
|
| ----------------------------------------------------------------------
| Methods inherited from scrapy.http.response.Response:
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| copy(self) -> 'Self'
| Return a copy of this Response
|
| replace(self, *args: 'Any', cls: 'type[Response] | None' = None, **kwargs: 'Any') -> 'Response'
| Create a new Response with the same attributes except for those given new values
|
| ----------------------------------------------------------------------
| Readonly properties inherited from scrapy.http.response.Response:
|
| body
|
| cb_kwargs
|
| meta
|
| url
|
| ----------------------------------------------------------------------
| Data descriptors inherited from scrapy.http.response.Response:
|
| __weakref__
| list of weak references to the object
|
| certificate
|
| flags
|
| headers
|
| ip_address
|
| protocol
|
| request
|
| status
|
| ----------------------------------------------------------------------
| Static methods inherited from scrapy.utils.trackref.object_ref:
|
| __new__(cls, *args: 'Any', **kwargs: 'Any') -> 'Self'
| Create and return a new object. See help(type) for accurate signature.
class JsonRequest(scrapy.http.request.Request)
| JsonRequest(*args: 'Any', dumps_kwargs: 'dict[str, Any] | None' = None, **kwargs: 'Any') -> 'None'
|
| Method resolution order:
| JsonRequest
| scrapy.http.request.Request
| scrapy.utils.trackref.object_ref
| builtins.object
|
| Methods defined here:
|
| __init__(self, *args: 'Any', dumps_kwargs: 'dict[str, Any] | None' = None, **kwargs: 'Any') -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| replace(self, *args: 'Any', cls: 'type[Request] | None' = None, **kwargs: 'Any') -> 'Request'
| Create a new Request with the same attributes except for those given new values
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| dumps_kwargs
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'attributes': 'tuple[str, ...]'}
|
| attributes = ('url', 'headers', 'body', 'cookies', 'meta', 'encoding',...
|
| ----------------------------------------------------------------------
| Methods inherited from scrapy.http.request.Request:
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| copy(self) -> 'Self'
|
| to_dict(self, *, spider: 'scrapy.Spider | None' = None) -> 'dict[str, Any]'
| Return a dictionary containing the Request's data.
|
| Use :func:`~scrapy.utils.request.request_from_dict` to convert back into a :class:`~scrapy.Request` object.
|
| If a spider is given, this method will try to find out the name of the spider methods used as callback
| and errback and include them in the output dict, raising an exception if they cannot be found.
|
| ----------------------------------------------------------------------
| Class methods inherited from scrapy.http.request.Request:
|
| from_curl(curl_command: 'str', ignore_unknown_options: 'bool' = True, **kwargs: 'Any') -> 'Self'
| Create a Request object from a string containing a `cURL
| <https://curl.se/>`_ command. It populates the HTTP method, the
| URL, the headers, the cookies and the body. It accepts the same
| arguments as the :class:`Request` class, taking preference and
| overriding the values of the same arguments contained in the cURL
| command.
|
| Unrecognized options are ignored by default. To raise an error when
| finding unknown options call this method by passing
| ``ignore_unknown_options=False``.
|
| .. caution:: Using :meth:`from_curl` from :class:`~scrapy.Request`
| subclasses, such as :class:`~scrapy.http.JsonRequest`, or
| :class:`~scrapy.http.XmlRpcRequest`, as well as having
| :ref:`downloader middlewares <topics-downloader-middleware>`
| and
| :ref:`spider middlewares <topics-spider-middleware>`
| enabled, such as
| :class:`~scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware`,
| :class:`~scrapy.downloadermiddlewares.useragent.UserAgentMiddleware`,
| or
| :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`,
| may modify the :class:`~scrapy.Request` object.
|
| To translate a cURL command into a Scrapy request,
| you may use `curl2scrapy <https://michael-shub.github.io/curl2scrapy/>`_.
|
| ----------------------------------------------------------------------
| Readonly properties inherited from scrapy.http.request.Request:
|
| body
|
| cb_kwargs
|
| encoding
|
| meta
|
| url
|
| ----------------------------------------------------------------------
| Data descriptors inherited from scrapy.http.request.Request:
|
| __weakref__
| list of weak references to the object
|
| callback
|
| cookies
|
| dont_filter
|
| errback
|
| flags
|
| headers
|
| method
|
| priority
|
| ----------------------------------------------------------------------
| Static methods inherited from scrapy.utils.trackref.object_ref:
|
| __new__(cls, *args: 'Any', **kwargs: 'Any') -> 'Self'
| Create and return a new object. See help(type) for accurate signature.
class JsonResponse(scrapy.http.response.text.TextResponse)
| JsonResponse(*args: 'Any', **kwargs: 'Any')
|
| Method resolution order:
| JsonResponse
| scrapy.http.response.text.TextResponse
| scrapy.http.response.Response
| scrapy.utils.trackref.object_ref
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Methods inherited from scrapy.http.response.text.TextResponse:
|
| __init__(self, *args: 'Any', **kwargs: 'Any')
| Initialize self. See help(type(self)) for accurate signature.
|
| css(self, query: 'str') -> 'SelectorList'
| Shortcut method implemented only by responses whose content
| is text (subclasses of TextResponse).
|
| follow(self, url: 'str | Link | parsel.Selector', callback: 'CallbackT | None' = None, method: 'str' = 'GET', headers: 'Mapping[AnyStr, Any] | Iterable[tuple[AnyStr, Any]] | None' = None, body: 'bytes | str | None' = None, cookies: 'CookiesT | None' = None, meta: 'dict[str, Any] | None' = None, encoding: 'str | None' = None, priority: 'int' = 0, dont_filter: 'bool' = False, errback: 'Callable[[Failure], Any] | None' = None, cb_kwargs: 'dict[str, Any] | None' = None, flags: 'list[str] | None' = None) -> 'Request'
| Return a :class:`~.Request` instance to follow a link ``url``.
| It accepts the same arguments as ``Request.__init__()`` method,
| but ``url`` can be not only an absolute URL, but also
|
| * a relative URL
| * a :class:`~scrapy.link.Link` object, e.g. the result of
| :ref:`topics-link-extractors`
| * a :class:`~scrapy.Selector` object for a ``<link>`` or ``<a>`` element, e.g.
| ``response.css('a.my_link')[0]``
| * an attribute :class:`~scrapy.Selector` (not SelectorList), e.g.
| ``response.css('a::attr(href)')[0]`` or
| ``response.xpath('//img/@src')[0]``
|
| See :ref:`response-follow-example` for usage examples.
|
| follow_all(self, urls: 'Iterable[str | Link] | parsel.SelectorList[Any] | None' = None, callback: 'CallbackT | None' = None, method: 'str' = 'GET', headers: 'Mapping[AnyStr, Any] | Iterable[tuple[AnyStr, Any]] | None' = None, body: 'bytes | str | None' = None, cookies: 'CookiesT | None' = None, meta: 'dict[str, Any] | None' = None, encoding: 'str | None' = None, priority: 'int' = 0, dont_filter: 'bool' = False, errback: 'Callable[[Failure], Any] | None' = None, cb_kwargs: 'dict[str, Any] | None' = None, flags: 'list[str] | None' = None, css: 'str | None' = None, xpath: 'str | None' = None) -> 'Iterable[Request]'
| A generator that produces :class:`~.Request` instances to follow all
| links in ``urls``. It accepts the same arguments as the :class:`~.Request`'s
| ``__init__()`` method, except that each ``urls`` element does not need to be
| an absolute URL, it can be any of the following:
|
| * a relative URL
| * a :class:`~scrapy.link.Link` object, e.g. the result of
| :ref:`topics-link-extractors`
| * a :class:`~scrapy.Selector` object for a ``<link>`` or ``<a>`` element, e.g.
| ``response.css('a.my_link')[0]``
| * an attribute :class:`~scrapy.Selector` (not SelectorList), e.g.
| ``response.css('a::attr(href)')[0]`` or
| ``response.xpath('//img/@src')[0]``
|
| In addition, ``css`` and ``xpath`` arguments are accepted to perform the link extraction
| within the ``follow_all()`` method (only one of ``urls``, ``css`` and ``xpath`` is accepted).
|
| Note that when passing a ``SelectorList`` as argument for the ``urls`` parameter or
| using the ``css`` or ``xpath`` parameters, this method will not produce requests for
| selectors from which links cannot be obtained (for instance, anchor tags without an
| ``href`` attribute)
|
| jmespath(self, query: 'str', **kwargs: 'Any') -> 'SelectorList'
| Shortcut method implemented only by responses whose content
| is text (subclasses of TextResponse).
|
| json(self) -> 'Any'
| Deserialize a JSON document to a Python object.
|
| urljoin(self, url: 'str') -> 'str'
| Join this Response's url with a possible relative url to form an
| absolute interpretation of the latter.
|
| xpath(self, query: 'str', **kwargs: 'Any') -> 'SelectorList'
| Shortcut method implemented only by responses whose content
| is text (subclasses of TextResponse).
|
| ----------------------------------------------------------------------
| Readonly properties inherited from scrapy.http.response.text.TextResponse:
|
| encoding
|
| selector
|
| text
| Body as unicode
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from scrapy.http.response.text.TextResponse:
|
| attributes = ('url', 'headers', 'body', 'flags', 'status', 'request', ...
|
| ----------------------------------------------------------------------
| Methods inherited from scrapy.http.response.Response:
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| copy(self) -> 'Self'
| Return a copy of this Response
|
| replace(self, *args: 'Any', cls: 'type[Response] | None' = None, **kwargs: 'Any') -> 'Response'
| Create a new Response with the same attributes except for those given new values
|
| ----------------------------------------------------------------------
| Readonly properties inherited from scrapy.http.response.Response:
|
| body
|
| cb_kwargs
|
| meta
|
| url
|
| ----------------------------------------------------------------------
| Data descriptors inherited from scrapy.http.response.Response:
|
| __weakref__
| list of weak references to the object
|
| certificate
|
| flags
|
| headers
|
| ip_address
|
| protocol
|
| request
|
| status
|
| ----------------------------------------------------------------------
| Static methods inherited from scrapy.utils.trackref.object_ref:
|
| __new__(cls, *args: 'Any', **kwargs: 'Any') -> 'Self'
| Create and return a new object. See help(type) for accurate signature.
class Request(scrapy.utils.trackref.object_ref)
| Request(url: 'str', callback: 'CallbackT | None' = None, method: 'str' = 'GET', headers: 'Mapping[AnyStr, Any] | Iterable[tuple[AnyStr, Any]] | None' = None, body: 'bytes | str | None' = None, cookies: 'CookiesT | None' = None, meta: 'dict[str, Any] | None' = None, encoding: 'str' = 'utf-8', priority: 'int' = 0, dont_filter: 'bool' = False, errback: 'Callable[[Failure], Any] | None' = None, flags: 'list[str] | None' = None, cb_kwargs: 'dict[str, Any] | None' = None) -> 'None'
|
| Represents an HTTP request, which is usually generated in a Spider and
| executed by the Downloader, thus generating a :class:`~scrapy.http.Response`.
|
| Method resolution order:
| Request
| scrapy.utils.trackref.object_ref
| builtins.object
|
| Methods defined here:
|
| __init__(self, url: 'str', callback: 'CallbackT | None' = None, method: 'str' = 'GET', headers: 'Mapping[AnyStr, Any] | Iterable[tuple[AnyStr, Any]] | None' = None, body: 'bytes | str | None' = None, cookies: 'CookiesT | None' = None, meta: 'dict[str, Any] | None' = None, encoding: 'str' = 'utf-8', priority: 'int' = 0, dont_filter: 'bool' = False, errback: 'Callable[[Failure], Any] | None' = None, flags: 'list[str] | None' = None, cb_kwargs: 'dict[str, Any] | None' = None) -> 'None'
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| copy(self) -> 'Self'
|
| replace(self, *args: 'Any', cls: 'type[Request] | None' = None, **kwargs: 'Any') -> 'Request'
| Create a new Request with the same attributes except for those given new values
|
| to_dict(self, *, spider: 'scrapy.Spider | None' = None) -> 'dict[str, Any]'
| Return a dictionary containing the Request's data.
|
| Use :func:`~scrapy.utils.request.request_from_dict` to convert back into a :class:`~scrapy.Request` object.
|
| If a spider is given, this method will try to find out the name of the spider methods used as callback
| and errback and include them in the output dict, raising an exception if they cannot be found.
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| from_curl(curl_command: 'str', ignore_unknown_options: 'bool' = True, **kwargs: 'Any') -> 'Self'
| Create a Request object from a string containing a `cURL
| <https://curl.se/>`_ command. It populates the HTTP method, the
| URL, the headers, the cookies and the body. It accepts the same
| arguments as the :class:`Request` class, taking preference and
| overriding the values of the same arguments contained in the cURL
| command.
|
| Unrecognized options are ignored by default. To raise an error when
| finding unknown options call this method by passing
| ``ignore_unknown_options=False``.
|
| .. caution:: Using :meth:`from_curl` from :class:`~scrapy.Request`
| subclasses, such as :class:`~scrapy.http.JsonRequest`, or
| :class:`~scrapy.http.XmlRpcRequest`, as well as having
| :ref:`downloader middlewares <topics-downloader-middleware>`
| and
| :ref:`spider middlewares <topics-spider-middleware>`
| enabled, such as
| :class:`~scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware`,
| :class:`~scrapy.downloadermiddlewares.useragent.UserAgentMiddleware`,
| or
| :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`,
| may modify the :class:`~scrapy.Request` object.
|
| To translate a cURL command into a Scrapy request,
| you may use `curl2scrapy <https://michael-shub.github.io/curl2scrapy/>`_.
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| body
|
| cb_kwargs
|
| encoding
|
| meta
|
| url
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| callback
|
| cookies
|
| dont_filter
|
| errback
|
| flags
|
| headers
|
| method
|
| priority
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'attributes': 'tuple[str, ...]'}
|
| attributes = ('url', 'headers', 'body', 'cookies', 'meta', 'encoding',...
|
| ----------------------------------------------------------------------
| Static methods inherited from scrapy.utils.trackref.object_ref:
|
| __new__(cls, *args: 'Any', **kwargs: 'Any') -> 'Self'
| Create and return a new object. See help(type) for accurate signature.
class Response(scrapy.utils.trackref.object_ref)
| Response(url: 'str', status: 'int' = 200, headers: 'Mapping[AnyStr, Any] | Iterable[tuple[AnyStr, Any]] | None' = None, body: 'bytes' = b'', flags: 'list[str] | None' = None, request: 'Request | None' = None, certificate: 'Any' = None, ip_address: 'IPv4Address | IPv6Address | None' = None, protocol: 'str | None' = None)
|
| An object that represents an HTTP response, which is usually
| downloaded (by the Downloader) and fed to the Spiders for processing.
|
| Method resolution order:
| Response
| scrapy.utils.trackref.object_ref
| builtins.object
|
| Methods defined here:
|
| __init__(self, url: 'str', status: 'int' = 200, headers: 'Mapping[AnyStr, Any] | Iterable[tuple[AnyStr, Any]] | None' = None, body: 'bytes' = b'', flags: 'list[str] | None' = None, request: 'Request | None' = None, certificate: 'Any' = None, ip_address: 'IPv4Address | IPv6Address | None' = None, protocol: 'str | None' = None)
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| copy(self) -> 'Self'
| Return a copy of this Response
|
| css(self, *a: 'Any', **kw: 'Any') -> 'SelectorList'
| Shortcut method implemented only by responses whose content
| is text (subclasses of TextResponse).
|
| follow(self, url: 'str | Link', callback: 'CallbackT | None' = None, method: 'str' = 'GET', headers: 'Mapping[AnyStr, Any] | Iterable[tuple[AnyStr, Any]] | None' = None, body: 'bytes | str | None' = None, cookies: 'CookiesT | None' = None, meta: 'dict[str, Any] | None' = None, encoding: 'str | None' = 'utf-8', priority: 'int' = 0, dont_filter: 'bool' = False, errback: 'Callable[[Failure], Any] | None' = None, cb_kwargs: 'dict[str, Any] | None' = None, flags: 'list[str] | None' = None) -> 'Request'
| Return a :class:`~.Request` instance to follow a link ``url``.
| It accepts the same arguments as ``Request.__init__()`` method,
| but ``url`` can be a relative URL or a :class:`~scrapy.link.Link` object,
| not only an absolute URL.
|
| :class:`~.TextResponse` provides a :meth:`~.TextResponse.follow`
| method which supports selectors in addition to absolute/relative URLs
| and Link objects.
|
| follow_all(self, urls: 'Iterable[str | Link]', callback: 'CallbackT | None' = None, method: 'str' = 'GET', headers: 'Mapping[AnyStr, Any] | Iterable[tuple[AnyStr, Any]] | None' = None, body: 'bytes | str | None' = None, cookies: 'CookiesT | None' = None, meta: 'dict[str, Any] | None' = None, encoding: 'str | None' = 'utf-8', priority: 'int' = 0, dont_filter: 'bool' = False, errback: 'Callable[[Failure], Any] | None' = None, cb_kwargs: 'dict[str, Any] | None' = None, flags: 'list[str] | None' = None) -> 'Iterable[Request]'
| Return an iterable of :class:`~.Request` instances to follow all links
| in ``urls``. It accepts the same arguments as ``Request.__init__()`` method,
| but elements of ``urls`` can be relative URLs or :class:`~scrapy.link.Link` objects,
| not only absolute URLs.
|
| :class:`~.TextResponse` provides a :meth:`~.TextResponse.follow_all`
| method which supports selectors in addition to absolute/relative URLs
| and Link objects.
|
| jmespath(self, *a: 'Any', **kw: 'Any') -> 'SelectorList'
| Shortcut method implemented only by responses whose content
| is text (subclasses of TextResponse).
|
| replace(self, *args: 'Any', cls: 'type[Response] | None' = None, **kwargs: 'Any') -> 'Response'
| Create a new Response with the same attributes except for those given new values
|
| urljoin(self, url: 'str') -> 'str'
| Join this Response's url with a possible relative url to form an
| absolute interpretation of the latter.
|
| xpath(self, *a: 'Any', **kw: 'Any') -> 'SelectorList'
| Shortcut method implemented only by responses whose content
| is text (subclasses of TextResponse).
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| body
|
| cb_kwargs
|
| meta
|
| text
| For subclasses of TextResponse, this will return the body
| as str
|
| url
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __weakref__
| list of weak references to the object
|
| certificate
|
| flags
|
| headers
|
| ip_address
|
| protocol
|
| request
|
| status
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'attributes': 'tuple[str, ...]'}
|
| attributes = ('url', 'headers', 'body', 'flags', 'status', 'request', ...
|
| ----------------------------------------------------------------------
| Static methods inherited from scrapy.utils.trackref.object_ref:
|
| __new__(cls, *args: 'Any', **kwargs: 'Any') -> 'Self'
| Create and return a new object. See help(type) for accurate signature.
class TextResponse(scrapy.http.response.Response)
| TextResponse(*args: 'Any', **kwargs: 'Any')
|
| Method resolution order:
| TextResponse
| scrapy.http.response.Response
| scrapy.utils.trackref.object_ref
| builtins.object
|
| Methods defined here:
|
| __init__(self, *args: 'Any', **kwargs: 'Any')
| Initialize self. See help(type(self)) for accurate signature.
|
| css(self, query: 'str') -> 'SelectorList'
| Shortcut method implemented only by responses whose content
| is text (subclasses of TextResponse).
|
| follow(self, url: 'str | Link | parsel.Selector', callback: 'CallbackT | None' = None, method: 'str' = 'GET', headers: 'Mapping[AnyStr, Any] | Iterable[tuple[AnyStr, Any]] | None' = None, body: 'bytes | str | None' = None, cookies: 'CookiesT | None' = None, meta: 'dict[str, Any] | None' = None, encoding: 'str | None' = None, priority: 'int' = 0, dont_filter: 'bool' = False, errback: 'Callable[[Failure], Any] | None' = None, cb_kwargs: 'dict[str, Any] | None' = None, flags: 'list[str] | None' = None) -> 'Request'
| Return a :class:`~.Request` instance to follow a link ``url``.
| It accepts the same arguments as ``Request.__init__()`` method,
| but ``url`` can be not only an absolute URL, but also
|
| * a relative URL
| * a :class:`~scrapy.link.Link` object, e.g. the result of
| :ref:`topics-link-extractors`
| * a :class:`~scrapy.Selector` object for a ``<link>`` or ``<a>`` element, e.g.
| ``response.css('a.my_link')[0]``
| * an attribute :class:`~scrapy.Selector` (not SelectorList), e.g.
| ``response.css('a::attr(href)')[0]`` or
| ``response.xpath('//img/@src')[0]``
|
| See :ref:`response-follow-example` for usage examples.
|
| follow_all(self, urls: 'Iterable[str | Link] | parsel.SelectorList[Any] | None' = None, callback: 'CallbackT | None' = None, method: 'str' = 'GET', headers: 'Mapping[AnyStr, Any] | Iterable[tuple[AnyStr, Any]] | None' = None, body: 'bytes | str | None' = None, cookies: 'CookiesT | None' = None, meta: 'dict[str, Any] | None' = None, encoding: 'str | None' = None, priority: 'int' = 0, dont_filter: 'bool' = False, errback: 'Callable[[Failure], Any] | None' = None, cb_kwargs: 'dict[str, Any] | None' = None, flags: 'list[str] | None' = None, css: 'str | None' = None, xpath: 'str | None' = None) -> 'Iterable[Request]'
| A generator that produces :class:`~.Request` instances to follow all
| links in ``urls``. It accepts the same arguments as the :class:`~.Request`'s
| ``__init__()`` method, except that each ``urls`` element does not need to be
| an absolute URL, it can be any of the following:
|
| * a relative URL
| * a :class:`~scrapy.link.Link` object, e.g. the result of
| :ref:`topics-link-extractors`
| * a :class:`~scrapy.Selector` object for a ``<link>`` or ``<a>`` element, e.g.
| ``response.css('a.my_link')[0]``
| * an attribute :class:`~scrapy.Selector` (not SelectorList), e.g.
| ``response.css('a::attr(href)')[0]`` or
| ``response.xpath('//img/@src')[0]``
|
| In addition, ``css`` and ``xpath`` arguments are accepted to perform the link extraction
| within the ``follow_all()`` method (only one of ``urls``, ``css`` and ``xpath`` is accepted).
|
| Note that when passing a ``SelectorList`` as argument for the ``urls`` parameter or
| using the ``css`` or ``xpath`` parameters, this method will not produce requests for
| selectors from which links cannot be obtained (for instance, anchor tags without an
| ``href`` attribute)
|
| jmespath(self, query: 'str', **kwargs: 'Any') -> 'SelectorList'
| Shortcut method implemented only by responses whose content
| is text (subclasses of TextResponse).
|
| json(self) -> 'Any'
| Deserialize a JSON document to a Python object.
|
| urljoin(self, url: 'str') -> 'str'
| Join this Response's url with a possible relative url to form an
| absolute interpretation of the latter.
|
| xpath(self, query: 'str', **kwargs: 'Any') -> 'SelectorList'
| Shortcut method implemented only by responses whose content
| is text (subclasses of TextResponse).
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| encoding
|
| selector
|
| text
| Body as unicode
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'attributes': 'tuple[str, ...]'}
|
| attributes = ('url', 'headers', 'body', 'flags', 'status', 'request', ...
|
| ----------------------------------------------------------------------
| Methods inherited from scrapy.http.response.Response:
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| copy(self) -> 'Self'
| Return a copy of this Response
|
| replace(self, *args: 'Any', cls: 'type[Response] | None' = None, **kwargs: 'Any') -> 'Response'
| Create a new Response with the same attributes except for those given new values
|
| ----------------------------------------------------------------------
| Readonly properties inherited from scrapy.http.response.Response:
|
| body
|
| cb_kwargs
|
| meta
|
| url
|
| ----------------------------------------------------------------------
| Data descriptors inherited from scrapy.http.response.Response:
|
| __weakref__
| list of weak references to the object
|
| certificate
|
| flags
|
| headers
|
| ip_address
|
| protocol
|
| request
|
| status
|
| ----------------------------------------------------------------------
| Static methods inherited from scrapy.utils.trackref.object_ref:
|
| __new__(cls, *args: 'Any', **kwargs: 'Any') -> 'Self'
| Create and return a new object. See help(type) for accurate signature.
class XmlResponse(scrapy.http.response.text.TextResponse)
| XmlResponse(*args: 'Any', **kwargs: 'Any')
|
| Method resolution order:
| XmlResponse
| scrapy.http.response.text.TextResponse
| scrapy.http.response.Response
| scrapy.utils.trackref.object_ref
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Methods inherited from scrapy.http.response.text.TextResponse:
|
| __init__(self, *args: 'Any', **kwargs: 'Any')
| Initialize self. See help(type(self)) for accurate signature.
|
| css(self, query: 'str') -> 'SelectorList'
| Shortcut method implemented only by responses whose content
| is text (subclasses of TextResponse).
|
| follow(self, url: 'str | Link | parsel.Selector', callback: 'CallbackT | None' = None, method: 'str' = 'GET', headers: 'Mapping[AnyStr, Any] | Iterable[tuple[AnyStr, Any]] | None' = None, body: 'bytes | str | None' = None, cookies: 'CookiesT | None' = None, meta: 'dict[str, Any] | None' = None, encoding: 'str | None' = None, priority: 'int' = 0, dont_filter: 'bool' = False, errback: 'Callable[[Failure], Any] | None' = None, cb_kwargs: 'dict[str, Any] | None' = None, flags: 'list[str] | None' = None) -> 'Request'
| Return a :class:`~.Request` instance to follow a link ``url``.
| It accepts the same arguments as ``Request.__init__()`` method,
| but ``url`` can be not only an absolute URL, but also
|
| * a relative URL
| * a :class:`~scrapy.link.Link` object, e.g. the result of
| :ref:`topics-link-extractors`
| * a :class:`~scrapy.Selector` object for a ``<link>`` or ``<a>`` element, e.g.
| ``response.css('a.my_link')[0]``
| * an attribute :class:`~scrapy.Selector` (not SelectorList), e.g.
| ``response.css('a::attr(href)')[0]`` or
| ``response.xpath('//img/@src')[0]``
|
| See :ref:`response-follow-example` for usage examples.
|
| follow_all(self, urls: 'Iterable[str | Link] | parsel.SelectorList[Any] | None' = None, callback: 'CallbackT | None' = None, method: 'str' = 'GET', headers: 'Mapping[AnyStr, Any] | Iterable[tuple[AnyStr, Any]] | None' = None, body: 'bytes | str | None' = None, cookies: 'CookiesT | None' = None, meta: 'dict[str, Any] | None' = None, encoding: 'str | None' = None, priority: 'int' = 0, dont_filter: 'bool' = False, errback: 'Callable[[Failure], Any] | None' = None, cb_kwargs: 'dict[str, Any] | None' = None, flags: 'list[str] | None' = None, css: 'str | None' = None, xpath: 'str | None' = None) -> 'Iterable[Request]'
| A generator that produces :class:`~.Request` instances to follow all
| links in ``urls``. It accepts the same arguments as the :class:`~.Request`'s
| ``__init__()`` method, except that each ``urls`` element does not need to be
| an absolute URL, it can be any of the following:
|
| * a relative URL
| * a :class:`~scrapy.link.Link` object, e.g. the result of
| :ref:`topics-link-extractors`
| * a :class:`~scrapy.Selector` object for a ``<link>`` or ``<a>`` element, e.g.
| ``response.css('a.my_link')[0]``
| * an attribute :class:`~scrapy.Selector` (not SelectorList), e.g.
| ``response.css('a::attr(href)')[0]`` or
| ``response.xpath('//img/@src')[0]``
|
| In addition, ``css`` and ``xpath`` arguments are accepted to perform the link extraction
| within the ``follow_all()`` method (only one of ``urls``, ``css`` and ``xpath`` is accepted).
|
| Note that when passing a ``SelectorList`` as argument for the ``urls`` parameter or
| using the ``css`` or ``xpath`` parameters, this method will not produce requests for
| selectors from which links cannot be obtained (for instance, anchor tags without an
| ``href`` attribute)
|
| jmespath(self, query: 'str', **kwargs: 'Any') -> 'SelectorList'
| Shortcut method implemented only by responses whose content
| is text (subclasses of TextResponse).
|
| json(self) -> 'Any'
| Deserialize a JSON document to a Python object.
|
| urljoin(self, url: 'str') -> 'str'
| Join this Response's url with a possible relative url to form an
| absolute interpretation of the latter.
|
| xpath(self, query: 'str', **kwargs: 'Any') -> 'SelectorList'
| Shortcut method implemented only by responses whose content
| is text (subclasses of TextResponse).
|
| ----------------------------------------------------------------------
| Readonly properties inherited from scrapy.http.response.text.TextResponse:
|
| encoding
|
| selector
|
| text
| Body as unicode
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from scrapy.http.response.text.TextResponse:
|
| attributes = ('url', 'headers', 'body', 'flags', 'status', 'request', ...
|
| ----------------------------------------------------------------------
| Methods inherited from scrapy.http.response.Response:
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| copy(self) -> 'Self'
| Return a copy of this Response
|
| replace(self, *args: 'Any', cls: 'type[Response] | None' = None, **kwargs: 'Any') -> 'Response'
| Create a new Response with the same attributes except for those given new values
|
| ----------------------------------------------------------------------
| Readonly properties inherited from scrapy.http.response.Response:
|
| body
|
| cb_kwargs
|
| meta
|
| url
|
| ----------------------------------------------------------------------
| Data descriptors inherited from scrapy.http.response.Response:
|
| __weakref__
| list of weak references to the object
|
| certificate
|
| flags
|
| headers
|
| ip_address
|
| protocol
|
| request
|
| status
|
| ----------------------------------------------------------------------
| Static methods inherited from scrapy.utils.trackref.object_ref:
|
| __new__(cls, *args: 'Any', **kwargs: 'Any') -> 'Self'
| Create and return a new object. See help(type) for accurate signature.
class XmlRpcRequest(scrapy.http.request.Request)
| XmlRpcRequest(*args: 'Any', encoding: 'str | None' = None, **kwargs: 'Any')
|
| Method resolution order:
| XmlRpcRequest
| scrapy.http.request.Request
| scrapy.utils.trackref.object_ref
| builtins.object
|
| Methods defined here:
|
| __init__(self, *args: 'Any', encoding: 'str | None' = None, **kwargs: 'Any')
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Methods inherited from scrapy.http.request.Request:
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| copy(self) -> 'Self'
|
| replace(self, *args: 'Any', cls: 'type[Request] | None' = None, **kwargs: 'Any') -> 'Request'
| Create a new Request with the same attributes except for those given new values
|
| to_dict(self, *, spider: 'scrapy.Spider | None' = None) -> 'dict[str, Any]'
| Return a dictionary containing the Request's data.
|
| Use :func:`~scrapy.utils.request.request_from_dict` to convert back into a :class:`~scrapy.Request` object.
|
| If a spider is given, this method will try to find out the name of the spider methods used as callback
| and errback and include them in the output dict, raising an exception if they cannot be found.
|
| ----------------------------------------------------------------------
| Class methods inherited from scrapy.http.request.Request:
|
| from_curl(curl_command: 'str', ignore_unknown_options: 'bool' = True, **kwargs: 'Any') -> 'Self'
| Create a Request object from a string containing a `cURL
| <https://curl.se/>`_ command. It populates the HTTP method, the
| URL, the headers, the cookies and the body. It accepts the same
| arguments as the :class:`Request` class, taking preference and
| overriding the values of the same arguments contained in the cURL
| command.
|
| Unrecognized options are ignored by default. To raise an error when
| finding unknown options call this method by passing
| ``ignore_unknown_options=False``.
|
| .. caution:: Using :meth:`from_curl` from :class:`~scrapy.Request`
| subclasses, such as :class:`~scrapy.http.JsonRequest`, or
| :class:`~scrapy.http.XmlRpcRequest`, as well as having
| :ref:`downloader middlewares <topics-downloader-middleware>`
| and
| :ref:`spider middlewares <topics-spider-middleware>`
| enabled, such as
| :class:`~scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware`,
| :class:`~scrapy.downloadermiddlewares.useragent.UserAgentMiddleware`,
| or
| :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`,
| may modify the :class:`~scrapy.Request` object.
|
| To translate a cURL command into a Scrapy request,
| you may use `curl2scrapy <https://michael-shub.github.io/curl2scrapy/>`_.
|
| ----------------------------------------------------------------------
| Readonly properties inherited from scrapy.http.request.Request:
|
| body
|
| cb_kwargs
|
| encoding
|
| meta
|
| url
|
| ----------------------------------------------------------------------
| Data descriptors inherited from scrapy.http.request.Request:
|
| __weakref__
| list of weak references to the object
|
| callback
|
| cookies
|
| dont_filter
|
| errback
|
| flags
|
| headers
|
| method
|
| priority
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from scrapy.http.request.Request:
|
| attributes = ('url', 'headers', 'body', 'cookies', 'meta', 'encoding',...
|
| ----------------------------------------------------------------------
| Static methods inherited from scrapy.utils.trackref.object_ref:
|
| __new__(cls, *args: 'Any', **kwargs: 'Any') -> 'Self'
| Create and return a new object. See help(type) for accurate signature.
DATA
__all__ = ['FormRequest', 'Headers', 'HtmlResponse', 'JsonRequest', 'J...
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\scrapy\http\__init__.py
scrapy.item
Python Library Documentation: module scrapy.item in scrapy
NAME
scrapy.item - Scrapy Item
DESCRIPTION
See documentation in docs/topics/items.rst
CLASSES
abc.ABCMeta(builtins.type)
ItemMeta
builtins.dict(builtins.object)
Field
collections.abc.MutableMapping(collections.abc.Mapping)
Item(collections.abc.MutableMapping, scrapy.utils.trackref.object_ref)
scrapy.utils.trackref.object_ref(builtins.object)
Item(collections.abc.MutableMapping, scrapy.utils.trackref.object_ref)
class Field(builtins.dict)
| Container of field metadata
|
| Method resolution order:
| Field
| builtins.dict
| builtins.object
|
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __orig_bases__ = (dict[str, typing.Any],)
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.dict:
|
| __contains__(self, key, /)
| True if the dictionary has the specified key, else False.
|
| __delitem__(self, key, /)
| Delete self[key].
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __getitem__(self, key, /)
| Return self[key].
|
| __gt__(self, value, /)
| Return self>value.
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| __ior__(self, value, /)
| Return self|=value.
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __or__(self, value, /)
| Return self|value.
|
| __repr__(self, /)
| Return repr(self).
|
| __reversed__(self, /)
| Return a reverse iterator over the dict keys.
|
| __ror__(self, value, /)
| Return value|self.
|
| __setitem__(self, key, value, /)
| Set self[key] to value.
|
| __sizeof__(...)
| D.__sizeof__() -> size of D in memory, in bytes
|
| clear(...)
| D.clear() -> None. Remove all items from D.
|
| copy(...)
| D.copy() -> a shallow copy of D
|
| get(self, key, default=None, /)
| Return the value for key if key is in the dictionary, else default.
|
| items(...)
| D.items() -> a set-like object providing a view on D's items
|
| keys(...)
| D.keys() -> a set-like object providing a view on D's keys
|
| pop(...)
| D.pop(k[,d]) -> v, remove specified key and return the corresponding value.
|
| If the key is not found, return the default if given; otherwise,
| raise a KeyError.
|
| popitem(self, /)
| Remove and return a (key, value) pair as a 2-tuple.
|
| Pairs are returned in LIFO (last-in, first-out) order.
| Raises KeyError if the dict is empty.
|
| setdefault(self, key, default=None, /)
| Insert key with a value of default if key is not in the dictionary.
|
| Return the value for key if key is in the dictionary, else default.
|
| update(...)
| D.update([E, ]**F) -> None. Update D from dict/iterable E and F.
| If E is present and has a .keys() method, then does: for k in E: D[k] = E[k]
| If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v
| In either case, this is followed by: for k in F: D[k] = F[k]
|
| values(...)
| D.values() -> an object providing a view on D's values
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.dict:
|
| __class_getitem__(...)
| See PEP 585
|
| fromkeys(iterable, value=None, /)
| Create a new dictionary with keys from iterable and values set to value.
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.dict:
|
| __new__(*args, **kwargs) class method of builtins.dict
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from builtins.dict:
|
| __hash__ = None
class Item(collections.abc.MutableMapping, scrapy.utils.trackref.object_ref)
| Item(*args: 'Any', **kwargs: 'Any')
|
| Base class for scraped items.
|
| In Scrapy, an object is considered an ``item`` if it's supported by the
| `itemadapter`_ library. For example, when the output of a spider callback
| is evaluated, only such objects are passed to :ref:`item pipelines
| <topics-item-pipeline>`. :class:`Item` is one of the classes supported by
| `itemadapter`_ by default.
|
| Items must declare :class:`Field` attributes, which are processed and stored
| in the ``fields`` attribute. This restricts the set of allowed field names
| and prevents typos, raising ``KeyError`` when referring to undefined fields.
| Additionally, fields can be used to define metadata and control the way
| data is processed internally. Please refer to the :ref:`documentation
| about fields <topics-items-fields>` for additional information.
|
| Unlike instances of :class:`dict`, instances of :class:`Item` may be
| :ref:`tracked <topics-leaks-trackrefs>` to debug memory leaks.
|
| .. _itemadapter: https://github.com/scrapy/itemadapter
|
| Method resolution order:
| Item
| collections.abc.MutableMapping
| collections.abc.Mapping
| collections.abc.Collection
| collections.abc.Sized
| collections.abc.Iterable
| collections.abc.Container
| scrapy.utils.trackref.object_ref
| builtins.object
|
| Methods defined here:
|
| __delitem__(self, key: 'str') -> 'None'
|
| __getattr__(self, name: 'str') -> 'NoReturn'
|
| __getitem__(self, key: 'str') -> 'Any'
|
| __hash__(self, /) from builtins.object
| Return hash(self).
|
| __init__(self, *args: 'Any', **kwargs: 'Any')
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self) -> 'Iterator[str]'
|
| __len__(self) -> 'int'
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| __setattr__(self, name: 'str', value: 'Any') -> 'None'
| Implement setattr(self, name, value).
|
| __setitem__(self, key: 'str', value: 'Any') -> 'None'
|
| copy(self) -> 'Self'
|
| deepcopy(self) -> 'Self'
| Return a :func:`~copy.deepcopy` of this item.
|
| keys(self) -> 'KeysView[str]'
| D.keys() -> a set-like object providing a view on D's keys
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __abstractmethods__ = frozenset()
|
| __annotations__ = {'fields': 'dict[str, Field]'}
|
| __orig_bases__ = (collections.abc.MutableMapping[str, typing.Any], <cl...
|
| fields = {}
|
| ----------------------------------------------------------------------
| Methods inherited from collections.abc.MutableMapping:
|
| clear(self)
| D.clear() -> None. Remove all items from D.
|
| pop(self, key, default=<object object at 0x0000025A356081F0>)
| D.pop(k[,d]) -> v, remove specified key and return the corresponding value.
| If key is not found, d is returned if given, otherwise KeyError is raised.
|
| popitem(self)
| D.popitem() -> (k, v), remove and return some (key, value) pair
| as a 2-tuple; but raise KeyError if D is empty.
|
| setdefault(self, key, default=None)
| D.setdefault(k[,d]) -> D.get(k,d), also set D[k]=d if k not in D
|
| update(self, other=(), /, **kwds)
| D.update([E, ]**F) -> None. Update D from mapping/iterable E and F.
| If E present and has a .keys() method, does: for k in E: D[k] = E[k]
| If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v
| In either case, this is followed by: for k, v in F.items(): D[k] = v
|
| ----------------------------------------------------------------------
| Methods inherited from collections.abc.Mapping:
|
| __contains__(self, key)
|
| __eq__(self, other)
| Return self==value.
|
| get(self, key, default=None)
| D.get(k[,d]) -> D[k] if k in D, else d. d defaults to None.
|
| items(self)
| D.items() -> a set-like object providing a view on D's items
|
| values(self)
| D.values() -> an object providing a view on D's values
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from collections.abc.Mapping:
|
| __reversed__ = None
|
| ----------------------------------------------------------------------
| Class methods inherited from collections.abc.Collection:
|
| __subclasshook__(C)
| Abstract classes can override this to customize issubclass().
|
| This is invoked early on by abc.ABCMeta.__subclasscheck__().
| It should return True, False or NotImplemented. If it returns
| NotImplemented, the normal algorithm is used. Otherwise, it
| overrides the normal algorithm (and the outcome is cached).
|
| ----------------------------------------------------------------------
| Class methods inherited from collections.abc.Iterable:
|
| __class_getitem__ = GenericAlias(...)
| Represent a PEP 585 generic type
|
| E.g. for t = list[int], t.__origin__ is list and t.__args__ is (int,).
|
| ----------------------------------------------------------------------
| Static methods inherited from scrapy.utils.trackref.object_ref:
|
| __new__(cls, *args: 'Any', **kwargs: 'Any') -> 'Self'
| Create and return a new object. See help(type) for accurate signature.
class ItemMeta(abc.ABCMeta)
| ItemMeta(class_name: 'str', bases: 'tuple[type, ...]', attrs: 'dict[str, Any]') -> 'ItemMeta'
|
| Metaclass_ of :class:`Item` that handles field definitions.
|
| .. _metaclass: https://realpython.com/python-metaclasses
|
| Method resolution order:
| ItemMeta
| abc.ABCMeta
| builtins.type
| builtins.object
|
| Static methods defined here:
|
| __new__(mcs, class_name: 'str', bases: 'tuple[type, ...]', attrs: 'dict[str, Any]') -> 'ItemMeta'
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| ----------------------------------------------------------------------
| Methods inherited from abc.ABCMeta:
|
| __instancecheck__(cls, instance)
| Override for isinstance(instance, cls).
|
| __provides__
|
| __subclasscheck__(cls, subclass)
| Override for issubclass(subclass, cls).
|
| register(cls, subclass)
| Register a virtual subclass of an ABC.
|
| Returns the subclass, to allow usage as a class decorator.
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from abc.ABCMeta:
|
| __implemented__ = classImplements(ABCMeta)
|
| __providedBy__ = classImplements(type)
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.type:
|
| __call__(self, /, *args, **kwargs)
| Call self as a function.
|
| __delattr__(self, name, /)
| Implement delattr(self, name).
|
| __dir__(self, /)
| Specialized __dir__ implementation for types.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| __or__(self, value, /)
| Return self|value.
|
| __repr__(self, /)
| Return repr(self).
|
| __ror__(self, value, /)
| Return value|self.
|
| __setattr__(self, name, value, /)
| Implement setattr(self, name, value).
|
| __sizeof__(self, /)
| Return memory consumption of the type object.
|
| __subclasses__(self, /)
| Return a list of immediate subclasses.
|
| mro(self, /)
| Return a type's method resolution order.
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.type:
|
| __prepare__(...)
| __prepare__() -> dict
| used to create the namespace for the class statement
|
| ----------------------------------------------------------------------
| Data descriptors inherited from builtins.type:
|
| __abstractmethods__
|
| __dict__
|
| __text_signature__
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from builtins.type:
|
| __base__ = <class 'abc.ABCMeta'>
| Metaclass for defining Abstract Base Classes (ABCs).
|
| Use this metaclass to create an ABC. An ABC can be subclassed
| directly, and then acts as a mix-in class. You can also register
| unrelated concrete classes (even built-in classes) and unrelated
| ABCs as 'virtual subclasses' -- these and their descendants will
| be considered subclasses of the registering ABC by the built-in
| issubclass() function, but the registering ABC won't show up in
| their MRO (Method Resolution Order) nor will method
| implementations defined by the registering ABC be callable (not
| even via super()).
|
|
| __bases__ = (<class 'abc.ABCMeta'>,)
|
| __basicsize__ = 920
|
| __dictoffset__ = 264
|
| __flags__ = 2156420608
|
| __itemsize__ = 40
|
| __mro__ = (<class 'scrapy.item.ItemMeta'>, <class 'abc.ABCMeta'>, <cla...
|
| __type_params__ = ()
|
| __weakrefoffset__ = 368
DATA
NoReturn = typing.NoReturn
Special type indicating functions that never return.
Example::
from typing import NoReturn
def stop() -> NoReturn:
raise Exception('no way')
NoReturn can also be used as a bottom type, a type that
has no values. Starting in Python 3.11, the Never type should
be used for this concept instead. Type checkers should treat the two
equivalently.
TYPE_CHECKING = False
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\scrapy\item.py
scrapy.link
Python Library Documentation: module scrapy.link in scrapy
NAME
scrapy.link - This module defines the Link object used in Link extractors.
DESCRIPTION
For actual link extractors implementation see scrapy.linkextractors, or
its documentation in: docs/topics/link-extractors.rst
CLASSES
builtins.object
Link
class Link(builtins.object)
| Link(url: str, text: str = '', fragment: str = '', nofollow: bool = False)
|
| Link objects represent an extracted link by the LinkExtractor.
|
| Using the anchor tag sample below to illustrate the parameters:
|
| .. code-block:: html
|
| <a href="https://example.com/nofollow.html#foo" rel="nofollow">Dont follow this one</a>
|
| :param url: the absolute url being linked to in the anchor tag.
| From the sample, this is ``https://example.com/nofollow.html``.
|
| :param text: the text in the anchor tag. From the sample, this is ``Dont follow this one``.
|
| :param fragment: the part of the url after the hash symbol. From the sample, this is ``foo``.
|
| :param nofollow: an indication of the presence or absence of a nofollow value in the ``rel`` attribute
| of the anchor tag.
|
| Methods defined here:
|
| __eq__(self, other: object) -> bool
| Return self==value.
|
| __hash__(self) -> int
| Return hash(self).
|
| __init__(self, url: str, text: str = '', fragment: str = '', nofollow: bool = False)
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self) -> str
| Return repr(self).
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| fragment
|
| nofollow
|
| text
|
| url
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\scrapy\link.py
scrapy.linkextractors
Python Library Documentation: package scrapy.linkextractors in scrapy
NAME
scrapy.linkextractors - scrapy.linkextractors
DESCRIPTION
This package contains a collection of Link Extractors.
For more info see docs/topics/link-extractors.rst
PACKAGE CONTENTS
lxmlhtml
CLASSES
builtins.object
scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor
LinkExtractor = class LxmlLinkExtractor(builtins.object)
| LinkExtractor(allow: '_RegexOrSeveral' = (), deny: '_RegexOrSeveral' = (), allow_domains: 'str | Iterable[str]' = (), deny_domains: 'str | Iterable[str]' = (), restrict_xpaths: 'str | Iterable[str]' = (), tags: 'str | Iterable[str]' = ('a', 'area'), attrs: 'str | Iterable[str]' = ('href',), canonicalize: 'bool' = False, unique: 'bool' = True, process_value: 'Callable[[Any], Any] | None' = None, deny_extensions: 'str | Iterable[str] | None' = None, restrict_css: 'str | Iterable[str]' = (), strip: 'bool' = True, restrict_text: '_RegexOrSeveral | None' = None, deny_tags: 'str | Iterable[str]' = (), deny_attrs: 'str | Iterable[str]' = ())
|
| LxmlLinkExtractor is the recommended link extractor with handy filtering
| options. It is implemented using lxml's robust HTMLParser.
|
| :param allow: a single regular expression (or list of regular expressions)
| that the (absolute) urls must match in order to be extracted. If not
| given (or empty), it will match all links.
| :type allow: str or list
|
| :param deny: a single regular expression (or list of regular expressions)
| that the (absolute) urls must match in order to be excluded (i.e. not
| extracted). It has precedence over the ``allow`` parameter. If not
| given (or empty) it won't exclude any links.
| :type deny: str or list
|
| :param allow_domains: a single value or a list of string containing
| domains which will be considered for extracting the links
| :type allow_domains: str or list
|
| :param deny_domains: a single value or a list of strings containing
| domains which won't be considered for extracting the links
| :type deny_domains: str or list
|
| :param deny_extensions: a single value or list of strings containing
| extensions that should be ignored when extracting links.
| If not given, it will default to
| :data:`scrapy.linkextractors.IGNORED_EXTENSIONS`.
| :type deny_extensions: list
|
| :param restrict_xpaths: is an XPath (or list of XPath's) which defines
| regions inside the response where links should be extracted from.
| If given, only the text selected by those XPath will be scanned for
| links.
| :type restrict_xpaths: str or list
|
| :param restrict_css: a CSS selector (or list of selectors) which defines
| regions inside the response where links should be extracted from.
| Has the same behaviour as ``restrict_xpaths``.
| :type restrict_css: str or list
|
| :param restrict_text: a single regular expression (or list of regular
| expressions) that the link's text must match in order to be extracted.
| If not given (or empty), it will match all links. If a list of regular
| expressions is given, the link will be extracted if it matches at least
| one.
| :type restrict_text: str or list
|
| :param tags: a tag or a list of tags to consider when extracting links.
| Defaults to ``('a', 'area')``. Use ``'*'`` to consider every tag.
| :type tags: str or list
|
| :param attrs: an attribute or list of attributes which should be considered
| when looking for links to extract (only for those tags specified in the
| ``tags`` parameter). Defaults to ``('href',)``. Use ``'*'`` to consider
| every attribute.
| :type attrs: list
|
| :param deny_tags: a tag or a list of tags that should not be considered when
| extracting links. It has precedence over the ``tags`` parameter, so it
| can be combined with ``tags='*'`` to consider every tag except a few.
| Defaults to ``()`` (no tag is excluded).
|
| .. versionadded:: 2.17.0
| :type deny_tags: str or list
|
| :param deny_attrs: an attribute or a list of attributes that should not be
| considered when looking for links to extract. It has precedence over the
| ``attrs`` parameter, so it can be combined with ``attrs='*'`` to consider
| every attribute except a few. Defaults to ``()`` (no attribute is
| excluded).
|
| .. versionadded:: 2.17.0
| :type deny_attrs: str or list
|
| :param canonicalize: canonicalize each extracted url (using
| w3lib.url.canonicalize_url). Defaults to ``False``.
| Note that canonicalize_url is meant for duplicate checking;
| it can change the URL visible at server side, so the response can be
| different for requests with canonicalized and raw URLs. If you're
| using LinkExtractor to follow links it is more robust to
| keep the default ``canonicalize=False``.
| :type canonicalize: bool
|
| :param unique: whether duplicate filtering should be applied to extracted
| links.
| :type unique: bool
|
| :param process_value: a function which receives each value extracted from
| the tag and attributes scanned and can modify the value and return a
| new one, or return ``None`` to ignore the link altogether. If not
| given, ``process_value`` defaults to ``lambda x: x``.
|
| .. highlight:: html
|
| For example, to extract links from this code::
|
| <a href="javascript:goToPage('../other/page.html'); return false">Link text</a>
|
| .. highlight:: python
|
| You can use the following function in ``process_value``:
|
| .. code-block:: python
|
| def process_value(value):
| m = re.search(r"javascript:goToPage\('(.*?)'", value)
| if m:
| return m.group(1)
|
| :type process_value: collections.abc.Callable
|
| :param strip: whether to strip whitespaces from extracted attributes.
| According to HTML5 standard, leading and trailing whitespaces
| must be stripped from ``href`` attributes of ``<a>``, ``<area>``
| and many other elements, ``src`` attribute of ``<img>``, ``<iframe>``
| elements, etc., so LinkExtractor strips space chars by default.
| Set ``strip=False`` to turn it off (e.g. if you're extracting urls
| from elements or attributes which allow leading/trailing whitespaces).
| :type strip: bool
|
| Methods defined here:
|
| __init__(self, allow: '_RegexOrSeveral' = (), deny: '_RegexOrSeveral' = (), allow_domains: 'str | Iterable[str]' = (), deny_domains: 'str | Iterable[str]' = (), restrict_xpaths: 'str | Iterable[str]' = (), tags: 'str | Iterable[str]' = ('a', 'area'), attrs: 'str | Iterable[str]' = ('href',), canonicalize: 'bool' = False, unique: 'bool' = True, process_value: 'Callable[[Any], Any] | None' = None, deny_extensions: 'str | Iterable[str] | None' = None, restrict_css: 'str | Iterable[str]' = (), strip: 'bool' = True, restrict_text: '_RegexOrSeveral | None' = None, deny_tags: 'str | Iterable[str]' = (), deny_attrs: 'str | Iterable[str]' = ())
| Initialize self. See help(type(self)) for accurate signature.
|
| extract_links(self, response: 'TextResponse') -> 'list[Link]'
| Returns a list of :class:`~scrapy.link.Link` objects from the
| specified :class:`response <scrapy.http.Response>`.
|
| Only links that match the settings passed to the ``__init__`` method of
| the link extractor are returned.
|
| Duplicate links are omitted if the ``unique`` attribute is set to ``True``,
| otherwise they are returned.
|
| matches(self, url: 'str') -> 'bool'
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
DATA
IGNORED_EXTENSIONS = ['7z', '7zip', 'bz2', 'rar', 'tar', 'tar.gz', 'xz...
__all__ = ['IGNORED_EXTENSIONS', 'LinkExtractor']
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\scrapy\linkextractors\__init__.py
scrapy.selector
Python Library Documentation: package scrapy.selector in scrapy
NAME
scrapy.selector - Selectors
PACKAGE CONTENTS
unified
CLASSES
parsel.selector.Selector(builtins.object)
scrapy.selector.unified.Selector(parsel.selector.Selector, scrapy.utils.trackref.object_ref)
parsel.selector.SelectorList(builtins.list)
scrapy.selector.unified.SelectorList(parsel.selector.SelectorList, scrapy.utils.trackref.object_ref)
scrapy.utils.trackref.object_ref(builtins.object)
scrapy.selector.unified.Selector(parsel.selector.Selector, scrapy.utils.trackref.object_ref)
scrapy.selector.unified.SelectorList(parsel.selector.SelectorList, scrapy.utils.trackref.object_ref)
class Selector(parsel.selector.Selector, scrapy.utils.trackref.object_ref)
| Selector(response: 'TextResponse | None' = None, text: 'str | None' = None, type: 'SelectorType | None' = None, root: 'Any | None' = <object object at 0x0000025A356096E0>, **kwargs: 'Any')
|
| An instance of :class:`Selector` is a wrapper over response to select
| certain parts of its content.
|
| ``response`` is an :class:`~scrapy.http.HtmlResponse` or an
| :class:`~scrapy.http.XmlResponse` object that will be used for selecting
| and extracting data.
|
| ``text`` is a unicode string or utf-8 encoded text for cases when a
| ``response`` isn't available. Using ``text`` and ``response`` together is
| undefined behavior.
|
| ``type`` defines the selector type, it can be ``"html"``, ``"xml"``,
| ``"json"``, ``"text"`` or ``None`` (default). It's passed to
| :class:`parsel.Selector` and its meaning is defined there. However, when
| ``type`` is ``None``, it is set to ``"xml"`` for an
| :class:`~scrapy.http.XmlResponse` and to ``"html"`` otherwise before
| passing it to :class:`parsel.Selector`.
|
| .. note:: JSON selector support requires ``parsel`` 1.8.0 or higher. With
| older versions setting ``type`` to ``"json"`` or ``"text"`` is not
| supported.
|
| Method resolution order:
| Selector
| parsel.selector.Selector
| scrapy.utils.trackref.object_ref
| builtins.object
|
| Methods defined here:
|
| __init__(self, response: 'TextResponse | None' = None, text: 'str | None' = None, type: 'SelectorType | None' = None, root: 'Any | None' = <object object at 0x0000025A356096E0>, **kwargs: 'Any')
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| response
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| selectorlist_cls = <class 'scrapy.selector.unified.SelectorList'>
| The :class:`SelectorList` class is a subclass of the builtin ``list``
| class, which provides a few additional methods.
|
|
| ----------------------------------------------------------------------
| Methods inherited from parsel.selector.Selector:
|
| __bool__(self) -> 'bool'
| Return ``True`` if there is any real content selected or ``False``
| otherwise. In other words, the boolean value of a :class:`Selector` is
| given by the contents it selects.
|
| __getstate__(self) -> 'Any'
| Helper for pickle.
|
| __nonzero__ = __bool__(self) -> 'bool'
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| __str__(self) -> 'str'
| Return str(self).
|
| css(self, query: 'str') -> 'SelectorList[Self]'
| Apply the given CSS selector and return a :class:`SelectorList` instance.
|
| ``query`` is a string containing the CSS selector to apply.
|
| In the background, CSS queries are translated into XPath queries using
| `cssselect`_ library and run ``.xpath()`` method.
|
| .. _cssselect: https://pypi.python.org/pypi/cssselect/
|
| drop(self) -> 'None'
| Drop matched nodes from the parent element.
|
| extract = get(self) -> 'Any'
|
| get(self) -> 'Any'
| Serialize and return the matched nodes.
|
| For HTML and XML, the result is always a string, and percent-encoded
| content is unquoted.
|
| getall(self) -> 'list[str]'
| Serialize and return the matched node in a 1-element list of strings.
|
| jmespath(self, query: 'str', **kwargs: 'Any') -> 'SelectorList[Self]'
| Find objects matching the JMESPath ``query`` and return the result as a
| :class:`SelectorList` instance with all elements flattened. List
| elements implement :class:`Selector` interface too.
|
| ``query`` is a string containing the `JMESPath
| <https://jmespath.org/>`_ query to apply.
|
| Any additional named arguments are passed to the underlying
| ``jmespath.search`` call, e.g.::
|
| selector.jmespath('author.name', options=jmespath.Options(dict_cls=collections.OrderedDict))
|
| re(self, regex: 'str | Pattern[str]', replace_entities: 'bool' = True) -> 'list[str]'
| Apply the given regex and return a list of strings with the
| matches.
|
| ``regex`` can be either a compiled regular expression or a string which
| will be compiled to a regular expression using ``re.compile(regex)``.
|
| By default, character entity references are replaced by their
| corresponding character (except for ``&`` and ``<``).
| Passing ``replace_entities`` as ``False`` switches off these
| replacements.
|
| re_first(self, regex: 'str | Pattern[str]', default: 'str | None' = None, replace_entities: 'bool' = True) -> 'str | None'
| Apply the given regex and return the first string which matches. If
| there is no match, return the default value (``None`` if the argument
| is not provided).
|
| By default, character entity references are replaced by their
| corresponding character (except for ``&`` and ``<``).
| Passing ``replace_entities`` as ``False`` switches off these
| replacements.
|
| register_namespace(self, prefix: 'str', uri: 'str') -> 'None'
| Register the given namespace to be used in this :class:`Selector`.
| Without registering namespaces you can't select or extract data from
| non-standard namespaces. See :ref:`selector-examples-xml`.
|
| remove_namespaces(self) -> 'None'
| Remove all namespaces, allowing to traverse the document using
| namespace-less xpaths. See :ref:`removing-namespaces`.
| For JSON selectors, this method does nothing.
|
| xpath(self, query: 'str', namespaces: 'Mapping[str, str] | None' = None, **kwargs: 'Any') -> 'SelectorList[Self]'
| Find nodes matching the xpath ``query`` and return the result as a
| :class:`SelectorList` instance with all elements flattened. List
| elements implement :class:`Selector` interface too.
|
| ``query`` is a string containing the XPATH query to apply.
|
| ``namespaces`` is an optional ``prefix: namespace-uri`` mapping (dict)
| for additional prefixes to those registered with ``register_namespace(prefix, uri)``.
| Contrary to ``register_namespace()``, these prefixes are not
| saved for future calls.
|
| Any additional named arguments can be used to pass values for XPath
| variables in the XPath expression, e.g.::
|
| selector.xpath('//a[href=$url]', url="http://www.example.com")
|
| ----------------------------------------------------------------------
| Readonly properties inherited from parsel.selector.Selector:
|
| attrib
| Return the attributes dictionary for underlying element.
| For JSON selectors, return an empty dict.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from parsel.selector.Selector:
|
| __weakref__
| list of weak references to the object
|
| body
|
| namespaces
|
| root
|
| type
|
| ----------------------------------------------------------------------
| Static methods inherited from scrapy.utils.trackref.object_ref:
|
| __new__(cls, *args: 'Any', **kwargs: 'Any') -> 'Self'
| Create and return a new object. See help(type) for accurate signature.
class SelectorList(parsel.selector.SelectorList, scrapy.utils.trackref.object_ref)
| SelectorList(iterable=(), /)
|
| The :class:`SelectorList` class is a subclass of the builtin ``list``
| class, which provides a few additional methods.
|
| Method resolution order:
| SelectorList
| parsel.selector.SelectorList
| builtins.list
| scrapy.utils.trackref.object_ref
| builtins.object
|
| Data and other attributes defined here:
|
| __annotations__ = {}
|
| __orig_bases__ = (parsel.selector.SelectorList['Selector'], <class 'sc...
|
| ----------------------------------------------------------------------
| Methods inherited from parsel.selector.SelectorList:
|
| __getitem__(self, pos: 'SupportsIndex | slice') -> '_SelectorType | SelectorList[_SelectorType]'
| Return self[index].
|
| __getstate__(self) -> 'None'
| Helper for pickle.
|
| css(self, query: 'str') -> 'SelectorList[_SelectorType]'
| Call the ``.css()`` method for each element in this list and return
| their results flattened as another :class:`SelectorList`.
|
| ``query`` is the same argument as the one in :meth:`Selector.css`
|
| drop(self) -> 'None'
| Drop matched nodes from the parent for each element in this list.
|
| extract = getall(self) -> 'list[str]'
|
| extract_first = get(self, default: 'str | None' = None) -> 'Any'
|
| get(self, default: 'str | None' = None) -> 'Any'
| Return the result of ``.get()`` for the first element in this list.
| If the list is empty, return the default value.
|
| getall(self) -> 'list[str]'
| Call the ``.get()`` method for each element is this list and return
| their results flattened, as a list of strings.
|
| jmespath(self, query: 'str', **kwargs: 'Any') -> 'SelectorList[_SelectorType]'
| Call the ``.jmespath()`` method for each element in this list and return
| their results flattened as another :class:`SelectorList`.
|
| ``query`` is the same argument as the one in :meth:`Selector.jmespath`.
|
| Any additional named arguments are passed to the underlying
| ``jmespath.search`` call, e.g.::
|
| selector.jmespath('author.name', options=jmespath.Options(dict_cls=collections.OrderedDict))
|
| re(self, regex: 'str | Pattern[str]', replace_entities: 'bool' = True) -> 'list[str]'
| Call the ``.re()`` method for each element in this list and return
| their results flattened, as a list of strings.
|
| By default, character entity references are replaced by their
| corresponding character (except for ``&`` and ``<``.
| Passing ``replace_entities`` as ``False`` switches off these
| replacements.
|
| re_first(self, regex: 'str | Pattern[str]', default: 'str | None' = None, replace_entities: 'bool' = True) -> 'str | None'
| Call the ``.re()`` method for the first element in this list and
| return the result in an string. If the list is empty or the
| regex doesn't match anything, return the default value (``None`` if
| the argument is not provided).
|
| By default, character entity references are replaced by their
| corresponding character (except for ``&`` and ``<``.
| Passing ``replace_entities`` as ``False`` switches off these
| replacements.
|
| xpath(self, xpath: 'str', namespaces: 'Mapping[str, str] | None' = None, **kwargs: 'Any') -> 'SelectorList[_SelectorType]'
| Call the ``.xpath()`` method for each element in this list and return
| their results flattened as another :class:`SelectorList`.
|
| ``xpath`` is the same argument as the one in :meth:`Selector.xpath`
|
| ``namespaces`` is an optional ``prefix: namespace-uri`` mapping (dict)
| for additional prefixes to those registered with ``register_namespace(prefix, uri)``.
| Contrary to ``register_namespace()``, these prefixes are not
| saved for future calls.
|
| Any additional named arguments can be used to pass values for XPath
| variables in the XPath expression, e.g.::
|
| selector.xpath('//a[href=$url]', url="http://www.example.com")
|
| ----------------------------------------------------------------------
| Readonly properties inherited from parsel.selector.SelectorList:
|
| attrib
| Return the attributes dictionary for the first element.
| If the list is empty, return an empty dict.
|
| ----------------------------------------------------------------------
| Data descriptors inherited from parsel.selector.SelectorList:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Methods inherited from builtins.list:
|
| __add__(self, value, /)
| Return self+value.
|
| __contains__(self, key, /)
| Return bool(key in self).
|
| __delitem__(self, key, /)
| Delete self[key].
|
| __eq__(self, value, /)
| Return self==value.
|
| __ge__(self, value, /)
| Return self>=value.
|
| __getattribute__(self, name, /)
| Return getattr(self, name).
|
| __gt__(self, value, /)
| Return self>value.
|
| __iadd__(self, value, /)
| Implement self+=value.
|
| __imul__(self, value, /)
| Implement self*=value.
|
| __init__(self, /, *args, **kwargs)
| Initialize self. See help(type(self)) for accurate signature.
|
| __iter__(self, /)
| Implement iter(self).
|
| __le__(self, value, /)
| Return self<=value.
|
| __len__(self, /)
| Return len(self).
|
| __lt__(self, value, /)
| Return self<value.
|
| __mul__(self, value, /)
| Return self*value.
|
| __ne__(self, value, /)
| Return self!=value.
|
| __repr__(self, /)
| Return repr(self).
|
| __reversed__(self, /)
| Return a reverse iterator over the list.
|
| __rmul__(self, value, /)
| Return value*self.
|
| __setitem__(self, key, value, /)
| Set self[key] to value.
|
| __sizeof__(self, /)
| Return the size of the list in memory, in bytes.
|
| append(self, object, /)
| Append object to the end of the list.
|
| clear(self, /)
| Remove all items from list.
|
| copy(self, /)
| Return a shallow copy of the list.
|
| count(self, value, /)
| Return number of occurrences of value.
|
| extend(self, iterable, /)
| Extend list by appending elements from the iterable.
|
| index(self, value, start=0, stop=9223372036854775807, /)
| Return first index of value.
|
| Raises ValueError if the value is not present.
|
| insert(self, index, object, /)
| Insert object before index.
|
| pop(self, index=-1, /)
| Remove and return item at index (default last).
|
| Raises IndexError if list is empty or index is out of range.
|
| remove(self, value, /)
| Remove first occurrence of value.
|
| Raises ValueError if the value is not present.
|
| reverse(self, /)
| Reverse *IN PLACE*.
|
| sort(self, /, *, key=None, reverse=False)
| Sort the list in ascending order and return None.
|
| The sort is in-place (i.e. the list itself is modified) and stable (i.e. the
| order of two equal elements is maintained).
|
| If a key function is given, apply it once to each list item and sort them,
| ascending or descending, according to their function values.
|
| The reverse flag can be set to sort in descending order.
|
| ----------------------------------------------------------------------
| Class methods inherited from builtins.list:
|
| __class_getitem__(...)
| See PEP 585
|
| ----------------------------------------------------------------------
| Static methods inherited from builtins.list:
|
| __new__(*args, **kwargs) class method of builtins.list
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from builtins.list:
|
| __hash__ = None
DATA
__all__ = ['Selector', 'SelectorList']
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\scrapy\selector\__init__.py
scrapy.signals
Python Library Documentation: module scrapy.signals in scrapy
NAME
scrapy.signals - Scrapy signals
DESCRIPTION
These signals are documented in docs/topics/signals.rst. Please don't add new
signals here without documenting them there.
DATA
bytes_received = <object object>
engine_started = <object object>
engine_stopped = <object object>
feed_exporter_closed = <object object>
feed_slot_closed = <object object>
headers_received = <object object>
item_dropped = <object object>
item_error = <object object>
item_scraped = <object object>
memusage_warning_reached = <object object>
request_dropped = <object object>
request_left_downloader = <object object>
request_reached_downloader = <object object>
request_scheduled = <object object>
response_downloaded = <object object>
response_received = <object object>
scheduler_empty = <object object>
spider_closed = <object object>
spider_error = <object object>
spider_idle = <object object>
spider_opened = <object object>
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\scrapy\signals.py
scrapy.spiders
Python Library Documentation: package scrapy.spiders in scrapy
NAME
scrapy.spiders - Base class for Scrapy spiders
DESCRIPTION
See documentation in docs/topics/spiders.rst
PACKAGE CONTENTS
crawl
feed
sitemap
CLASSES
builtins.object
scrapy.spiders.crawl.Rule
scrapy.utils.trackref.object_ref(builtins.object)
Spider
scrapy.spiders.crawl.CrawlSpider
scrapy.spiders.feed.CSVFeedSpider
scrapy.spiders.feed.XMLFeedSpider
scrapy.spiders.sitemap.SitemapSpider
class CSVFeedSpider(scrapy.spiders.Spider)
| CSVFeedSpider(name: 'str | None' = None, **kwargs: 'Any')
|
| Spider for parsing CSV feeds.
| It receives a CSV file in a response; iterates through each of its rows,
| and calls parse_row with a dict containing each field's data.
|
| This spider also gives the opportunity to override adapt_response and
| process_results methods for pre and post-processing purposes.
|
| You can set some options regarding the CSV file, such as the delimiter, quotechar
| and the file's headers.
|
| Method resolution order:
| CSVFeedSpider
| scrapy.spiders.Spider
| scrapy.utils.trackref.object_ref
| builtins.object
|
| Methods defined here:
|
| adapt_response(self, response: 'Response') -> 'Response'
| This method has the same purpose as the one in XMLFeedSpider
|
| parse_row(self, response: 'Response', row: 'dict[str, str]') -> 'Any'
| Receives a response and a dict (representing each row) with a key for
| each provided (or detected) header of the CSV file.
|
| This method must be overridden with your custom spider functionality.
|
| parse_rows(self, response: 'Response') -> 'Any'
|
| process_results(self, response: 'Response', results: 'Iterable[Any]') -> 'Iterable[Any]'
| This method has the same purpose as the one in XMLFeedSpider
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'delimiter': 'str | None', 'headers': 'list[str] | ...
|
| delimiter = None
|
| headers = None
|
| quotechar = None
|
| ----------------------------------------------------------------------
| Methods inherited from scrapy.spiders.Spider:
|
| __init__(self, name: 'str | None' = None, **kwargs: 'Any')
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| log(self, message: 'Any', level: 'int' = 10, **kw: 'Any') -> 'None'
| Log the given message at the given log level
|
| This helper wraps a log call to the logger within the spider, but you
| can use it directly (e.g. Spider.logger.info('msg')) or use any other
| Python logger too.
|
| parse(self, response: 'Response', **kwargs: 'Any') -> 'Any'
|
| async start(self) -> 'AsyncIterator[Any]'
| Yield the initial :class:`~scrapy.Request` objects to send.
|
| .. versionadded:: 2.13
|
| For example:
|
| .. code-block:: python
|
| from scrapy import Request, Spider
|
|
| class MySpider(Spider):
| name = "myspider"
|
| async def start(self):
| yield Request("https://toscrape.com/")
|
| The default implementation reads URLs from :attr:`start_urls` and
| yields a request for each with :attr:`~scrapy.Request.dont_filter`
| enabled. It is functionally equivalent to:
|
| .. code-block:: python
|
| async def start(self):
| for url in self.start_urls:
| yield Request(url, dont_filter=True)
|
| You can also yield :ref:`items <topics-items>`. For example:
|
| .. code-block:: python
|
| async def start(self):
| yield {"foo": "bar"}
|
| To write spiders that work on Scrapy versions lower than 2.13,
| define also a synchronous ``start_requests()`` method that returns an
| iterable. For example:
|
| .. code-block:: python
|
| def start_requests(self):
| yield Request("https://toscrape.com/")
|
| .. seealso:: :ref:`start-requests`
|
| ----------------------------------------------------------------------
| Class methods inherited from scrapy.spiders.Spider:
|
| from_crawler(crawler: 'Crawler', *args: 'Any', **kwargs: 'Any') -> 'Self'
|
| handles_request(request: 'Request') -> 'bool'
|
| update_settings(settings: 'BaseSettings') -> 'None'
|
| ----------------------------------------------------------------------
| Static methods inherited from scrapy.spiders.Spider:
|
| close(spider: 'Spider', reason: 'str') -> 'Deferred[None] | None'
|
| ----------------------------------------------------------------------
| Readonly properties inherited from scrapy.spiders.Spider:
|
| logger
|
| ----------------------------------------------------------------------
| Data descriptors inherited from scrapy.spiders.Spider:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from scrapy.spiders.Spider:
|
| custom_settings = None
|
| ----------------------------------------------------------------------
| Static methods inherited from scrapy.utils.trackref.object_ref:
|
| __new__(cls, *args: 'Any', **kwargs: 'Any') -> 'Self'
| Create and return a new object. See help(type) for accurate signature.
class CrawlSpider(scrapy.spiders.Spider)
| CrawlSpider(*a: 'Any', **kw: 'Any')
|
| Method resolution order:
| CrawlSpider
| scrapy.spiders.Spider
| scrapy.utils.trackref.object_ref
| builtins.object
|
| Methods defined here:
|
| __init__(self, *a: 'Any', **kw: 'Any')
| Initialize self. See help(type(self)) for accurate signature.
|
| parse_start_url(self, response: 'Response', **kwargs: 'Any') -> 'Any'
|
| async parse_with_rules(self, response: 'Response', callback: 'CallbackT | None', cb_kwargs: 'dict[str, Any]', follow: 'bool' = True) -> 'AsyncIterator[Any]'
|
| process_results(self, response: 'Response', results: 'Iterable[Any]') -> 'Iterable[Any]'
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| from_crawler(crawler: 'Crawler', *args: 'Any', **kwargs: 'Any') -> 'Self'
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_follow_links': 'bool', '_rules': 'list[Rule]', 'r...
|
| rules = ()
|
| ----------------------------------------------------------------------
| Methods inherited from scrapy.spiders.Spider:
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| log(self, message: 'Any', level: 'int' = 10, **kw: 'Any') -> 'None'
| Log the given message at the given log level
|
| This helper wraps a log call to the logger within the spider, but you
| can use it directly (e.g. Spider.logger.info('msg')) or use any other
| Python logger too.
|
| parse(self, response: 'Response', **kwargs: 'Any') -> 'Any'
|
| async start(self) -> 'AsyncIterator[Any]'
| Yield the initial :class:`~scrapy.Request` objects to send.
|
| .. versionadded:: 2.13
|
| For example:
|
| .. code-block:: python
|
| from scrapy import Request, Spider
|
|
| class MySpider(Spider):
| name = "myspider"
|
| async def start(self):
| yield Request("https://toscrape.com/")
|
| The default implementation reads URLs from :attr:`start_urls` and
| yields a request for each with :attr:`~scrapy.Request.dont_filter`
| enabled. It is functionally equivalent to:
|
| .. code-block:: python
|
| async def start(self):
| for url in self.start_urls:
| yield Request(url, dont_filter=True)
|
| You can also yield :ref:`items <topics-items>`. For example:
|
| .. code-block:: python
|
| async def start(self):
| yield {"foo": "bar"}
|
| To write spiders that work on Scrapy versions lower than 2.13,
| define also a synchronous ``start_requests()`` method that returns an
| iterable. For example:
|
| .. code-block:: python
|
| def start_requests(self):
| yield Request("https://toscrape.com/")
|
| .. seealso:: :ref:`start-requests`
|
| ----------------------------------------------------------------------
| Class methods inherited from scrapy.spiders.Spider:
|
| handles_request(request: 'Request') -> 'bool'
|
| update_settings(settings: 'BaseSettings') -> 'None'
|
| ----------------------------------------------------------------------
| Static methods inherited from scrapy.spiders.Spider:
|
| close(spider: 'Spider', reason: 'str') -> 'Deferred[None] | None'
|
| ----------------------------------------------------------------------
| Readonly properties inherited from scrapy.spiders.Spider:
|
| logger
|
| ----------------------------------------------------------------------
| Data descriptors inherited from scrapy.spiders.Spider:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from scrapy.spiders.Spider:
|
| custom_settings = None
|
| ----------------------------------------------------------------------
| Static methods inherited from scrapy.utils.trackref.object_ref:
|
| __new__(cls, *args: 'Any', **kwargs: 'Any') -> 'Self'
| Create and return a new object. See help(type) for accurate signature.
class Rule(builtins.object)
| Rule(link_extractor: 'LinkExtractor | None' = None, callback: 'CallbackT | str | None' = None, cb_kwargs: 'dict[str, Any] | None' = None, follow: 'bool | None' = None, process_links: 'ProcessLinksT | str | None' = None, process_request: 'ProcessRequestT | str | None' = None, errback: 'Callable[[Failure], Any] | str | None' = None)
|
| Methods defined here:
|
| __init__(self, link_extractor: 'LinkExtractor | None' = None, callback: 'CallbackT | str | None' = None, cb_kwargs: 'dict[str, Any] | None' = None, follow: 'bool | None' = None, process_links: 'ProcessLinksT | str | None' = None, process_request: 'ProcessRequestT | str | None' = None, errback: 'Callable[[Failure], Any] | str | None' = None)
| Initialize self. See help(type(self)) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
class SitemapSpider(scrapy.spiders.Spider)
| SitemapSpider(*a: 'Any', **kw: 'Any')
|
| Method resolution order:
| SitemapSpider
| scrapy.spiders.Spider
| scrapy.utils.trackref.object_ref
| builtins.object
|
| Methods defined here:
|
| __init__(self, *a: 'Any', **kw: 'Any')
| Initialize self. See help(type(self)) for accurate signature.
|
| sitemap_filter(self, entries: 'Iterable[dict[str, Any]]') -> 'Iterable[dict[str, Any]]'
| This method can be used to filter sitemap entries by their
| attributes, for example, you can filter locs with lastmod greater
| than a given date (see docs).
|
| async start(self) -> 'AsyncIterator[Any]'
| Yield the initial :class:`~scrapy.Request` objects to send.
|
| .. versionadded:: 2.13
|
| For example:
|
| .. code-block:: python
|
| from scrapy import Request, Spider
|
|
| class MySpider(Spider):
| name = "myspider"
|
| async def start(self):
| yield Request("https://toscrape.com/")
|
| The default implementation reads URLs from :attr:`start_urls` and
| yields a request for each with :attr:`~scrapy.Request.dont_filter`
| enabled. It is functionally equivalent to:
|
| .. code-block:: python
|
| async def start(self):
| for url in self.start_urls:
| yield Request(url, dont_filter=True)
|
| You can also yield :ref:`items <topics-items>`. For example:
|
| .. code-block:: python
|
| async def start(self):
| yield {"foo": "bar"}
|
| To write spiders that work on Scrapy versions lower than 2.13,
| define also a synchronous ``start_requests()`` method that returns an
| iterable. For example:
|
| .. code-block:: python
|
| def start_requests(self):
| yield Request("https://toscrape.com/")
|
| .. seealso:: :ref:`start-requests`
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| from_crawler(crawler: 'Crawler', *args: 'Any', **kwargs: 'Any') -> 'Self'
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'_max_size': 'int', '_warn_size': 'int', 'sitemap_a...
|
| sitemap_alternate_links = False
|
| sitemap_follow = ['']
|
| sitemap_rules = [('', 'parse')]
|
| sitemap_urls = ()
|
| ----------------------------------------------------------------------
| Methods inherited from scrapy.spiders.Spider:
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| log(self, message: 'Any', level: 'int' = 10, **kw: 'Any') -> 'None'
| Log the given message at the given log level
|
| This helper wraps a log call to the logger within the spider, but you
| can use it directly (e.g. Spider.logger.info('msg')) or use any other
| Python logger too.
|
| parse(self, response: 'Response', **kwargs: 'Any') -> 'Any'
|
| ----------------------------------------------------------------------
| Class methods inherited from scrapy.spiders.Spider:
|
| handles_request(request: 'Request') -> 'bool'
|
| update_settings(settings: 'BaseSettings') -> 'None'
|
| ----------------------------------------------------------------------
| Static methods inherited from scrapy.spiders.Spider:
|
| close(spider: 'Spider', reason: 'str') -> 'Deferred[None] | None'
|
| ----------------------------------------------------------------------
| Readonly properties inherited from scrapy.spiders.Spider:
|
| logger
|
| ----------------------------------------------------------------------
| Data descriptors inherited from scrapy.spiders.Spider:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from scrapy.spiders.Spider:
|
| custom_settings = None
|
| ----------------------------------------------------------------------
| Static methods inherited from scrapy.utils.trackref.object_ref:
|
| __new__(cls, *args: 'Any', **kwargs: 'Any') -> 'Self'
| Create and return a new object. See help(type) for accurate signature.
class Spider(scrapy.utils.trackref.object_ref)
| Spider(name: 'str | None' = None, **kwargs: 'Any')
|
| Base class that any spider must subclass.
|
| It provides a default :meth:`start` implementation that sends
| requests based on the :attr:`start_urls` class attribute and calls the
| :meth:`parse` method for each response.
|
| Method resolution order:
| Spider
| scrapy.utils.trackref.object_ref
| builtins.object
|
| Methods defined here:
|
| __init__(self, name: 'str | None' = None, **kwargs: 'Any')
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| log(self, message: 'Any', level: 'int' = 10, **kw: 'Any') -> 'None'
| Log the given message at the given log level
|
| This helper wraps a log call to the logger within the spider, but you
| can use it directly (e.g. Spider.logger.info('msg')) or use any other
| Python logger too.
|
| parse(self, response: 'Response', **kwargs: 'Any') -> 'Any'
|
| async start(self) -> 'AsyncIterator[Any]'
| Yield the initial :class:`~scrapy.Request` objects to send.
|
| .. versionadded:: 2.13
|
| For example:
|
| .. code-block:: python
|
| from scrapy import Request, Spider
|
|
| class MySpider(Spider):
| name = "myspider"
|
| async def start(self):
| yield Request("https://toscrape.com/")
|
| The default implementation reads URLs from :attr:`start_urls` and
| yields a request for each with :attr:`~scrapy.Request.dont_filter`
| enabled. It is functionally equivalent to:
|
| .. code-block:: python
|
| async def start(self):
| for url in self.start_urls:
| yield Request(url, dont_filter=True)
|
| You can also yield :ref:`items <topics-items>`. For example:
|
| .. code-block:: python
|
| async def start(self):
| yield {"foo": "bar"}
|
| To write spiders that work on Scrapy versions lower than 2.13,
| define also a synchronous ``start_requests()`` method that returns an
| iterable. For example:
|
| .. code-block:: python
|
| def start_requests(self):
| yield Request("https://toscrape.com/")
|
| .. seealso:: :ref:`start-requests`
|
| ----------------------------------------------------------------------
| Class methods defined here:
|
| from_crawler(crawler: 'Crawler', *args: 'Any', **kwargs: 'Any') -> 'Self'
|
| handles_request(request: 'Request') -> 'bool'
|
| update_settings(settings: 'BaseSettings') -> 'None'
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| close(spider: 'Spider', reason: 'str') -> 'Deferred[None] | None'
|
| ----------------------------------------------------------------------
| Readonly properties defined here:
|
| logger
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'custom_settings': 'dict[str, Any] | None', 'name':...
|
| custom_settings = None
|
| ----------------------------------------------------------------------
| Static methods inherited from scrapy.utils.trackref.object_ref:
|
| __new__(cls, *args: 'Any', **kwargs: 'Any') -> 'Self'
| Create and return a new object. See help(type) for accurate signature.
class XMLFeedSpider(scrapy.spiders.Spider)
| XMLFeedSpider(name: 'str | None' = None, **kwargs: 'Any')
|
| This class intends to be the base class for spiders that scrape
| from XML feeds.
|
| You can choose whether to parse the file using the 'iternodes' iterator, an
| 'xml' selector, or an 'html' selector. In most cases, it's convenient to
| use iternodes, since it's a faster and cleaner.
|
| Method resolution order:
| XMLFeedSpider
| scrapy.spiders.Spider
| scrapy.utils.trackref.object_ref
| builtins.object
|
| Methods defined here:
|
| adapt_response(self, response: 'Response') -> 'Response'
| You can override this function in order to make any changes you want
| to into the feed before parsing it. This function must return a
| response.
|
| parse_node(self, response: 'Response', selector: 'Selector') -> 'Any'
| This method is called for the nodes matching the provided tag name
| (itertag). Receives the response and an Selector for each node.
|
| This method must return either an item, a request, or a list
| containing any of them.
|
| This method must be overridden with your custom spider functionality.
|
| parse_nodes(self, response: 'Response', nodes: 'Iterable[Selector]') -> 'Any'
| This method is called for the nodes matching the provided tag name
| (itertag). Receives the response and an iterable of Selectors.
|
| process_results(self, response: 'Response', results: 'Iterable[Any]') -> 'Iterable[Any]'
| This overridable method is called for each result (item or request)
| returned by the spider, and it's intended to perform any last time
| processing required before returning the results to the framework core,
| for example setting the item GUIDs. It receives a list of results and
| the response which originated that results. It must return a list of
| results (items or requests).
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'iterator': 'str', 'itertag': 'str', 'namespaces': ...
|
| iterator = 'iternodes'
|
| itertag = 'item'
|
| namespaces = ()
|
| ----------------------------------------------------------------------
| Methods inherited from scrapy.spiders.Spider:
|
| __init__(self, name: 'str | None' = None, **kwargs: 'Any')
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self) -> 'str'
| Return repr(self).
|
| log(self, message: 'Any', level: 'int' = 10, **kw: 'Any') -> 'None'
| Log the given message at the given log level
|
| This helper wraps a log call to the logger within the spider, but you
| can use it directly (e.g. Spider.logger.info('msg')) or use any other
| Python logger too.
|
| parse(self, response: 'Response', **kwargs: 'Any') -> 'Any'
|
| async start(self) -> 'AsyncIterator[Any]'
| Yield the initial :class:`~scrapy.Request` objects to send.
|
| .. versionadded:: 2.13
|
| For example:
|
| .. code-block:: python
|
| from scrapy import Request, Spider
|
|
| class MySpider(Spider):
| name = "myspider"
|
| async def start(self):
| yield Request("https://toscrape.com/")
|
| The default implementation reads URLs from :attr:`start_urls` and
| yields a request for each with :attr:`~scrapy.Request.dont_filter`
| enabled. It is functionally equivalent to:
|
| .. code-block:: python
|
| async def start(self):
| for url in self.start_urls:
| yield Request(url, dont_filter=True)
|
| You can also yield :ref:`items <topics-items>`. For example:
|
| .. code-block:: python
|
| async def start(self):
| yield {"foo": "bar"}
|
| To write spiders that work on Scrapy versions lower than 2.13,
| define also a synchronous ``start_requests()`` method that returns an
| iterable. For example:
|
| .. code-block:: python
|
| def start_requests(self):
| yield Request("https://toscrape.com/")
|
| .. seealso:: :ref:`start-requests`
|
| ----------------------------------------------------------------------
| Class methods inherited from scrapy.spiders.Spider:
|
| from_crawler(crawler: 'Crawler', *args: 'Any', **kwargs: 'Any') -> 'Self'
|
| handles_request(request: 'Request') -> 'bool'
|
| update_settings(settings: 'BaseSettings') -> 'None'
|
| ----------------------------------------------------------------------
| Static methods inherited from scrapy.spiders.Spider:
|
| close(spider: 'Spider', reason: 'str') -> 'Deferred[None] | None'
|
| ----------------------------------------------------------------------
| Readonly properties inherited from scrapy.spiders.Spider:
|
| logger
|
| ----------------------------------------------------------------------
| Data descriptors inherited from scrapy.spiders.Spider:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from scrapy.spiders.Spider:
|
| custom_settings = None
|
| ----------------------------------------------------------------------
| Static methods inherited from scrapy.utils.trackref.object_ref:
|
| __new__(cls, *args: 'Any', **kwargs: 'Any') -> 'Self'
| Create and return a new object. See help(type) for accurate signature.
DATA
__all__ = ['CSVFeedSpider', 'CrawlSpider', 'Rule', 'SitemapSpider', 'S...
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\scrapy\spiders\__init__.py
scrapy.utils
Python Library Documentation: package scrapy.utils in scrapy
NAME
scrapy.utils
PACKAGE CONTENTS
_compression
_deps_compat
_download_handlers
asyncgen
asyncio
benchserver
boto
conf
console
curl
datatypes
decorators
defer
deprecate
display
engine
ftp
gz
httpobj
iterators
job
log
misc
ossignal
project
python
reactor
reactorless
request
response
serialize
signal
sitemap
spider
ssl
template
test
trackref
url
versions
FILE
c:\users\youse\appdata\local\programs\python\python312\lib\site-packages\scrapy\utils\__init__.py
scrapy.version_info
Python Library Documentation: tuple object class tuple(object) | tuple(iterable=(), /) | | Built-in immutable sequence. | | If no argument is given, the constructor returns an empty tuple. | If iterable is specified the tuple is initialized from iterable's items. | | If the argument is a tuple, the return value is the same object. | | Built-in subclasses: | asyncgen_hooks | DNS_IDAttributes | DNSPatternAttributes | HostnameAddressAttributes | ... and 15 other subclasses | | Methods defined here: | | __add__(self, value, /) | Return self+value. | | __contains__(self, key, /) | Return bool(key in self). | | __eq__(self, value, /) | Return self==value. | | __ge__(self, value, /) | Return self>=value. | | __getattribute__(self, name, /) | Return getattr(self, name). | | __getitem__(self, key, /) | Return self[key]. | | __getnewargs__(self, /) | | __gt__(self, value, /) | Return self>value. | | __hash__(self, /) | Return hash(self). | | __iter__(self, /) | Implement iter(self). | | __le__(self, value, /) | Return self<=value. | | __len__(self, /) | Return len(self). | | __lt__(self, value, /) | Return self<value. | | __mul__(self, value, /) | Return self*value. | | __ne__(self, value, /) | Return self!=value. | | __repr__(self, /) | Return repr(self). | | __rmul__(self, value, /) | Return value*self. | | count(self, value, /) | Return number of occurrences of value. | | index(self, value, start=0, stop=9223372036854775807, /) | Return first index of value. | | Raises ValueError if the value is not present. | | ---------------------------------------------------------------------- | Class methods defined here: | | __class_getitem__(...) | See PEP 585 | | ---------------------------------------------------------------------- | Static methods defined here: | | __new__(*args, **kwargs) | Create and return a new object. See help(type) for accurate signature.