Dependencies

Required

The following packages are required for basic functionality.

• cron-converter

• msgspec

• scrapy

• selenium

• sqlalchemy

• pywin32 [Windows only]

Extras

The following extras can be installed to provide additional features or install them all using

pip install scrachy[all].

• content_extraction

  • beautifulsoup4

  • boilerpy3

• html_parsing

  • html5lib

  • lxml

• mysql

  • pymysql

• postgresql

  • psycopg2

• testing

  • pytest

  • pytest-twisted

  • python-dotenv

  • pyyaml

Any database driver supported by SqlAlchemy Dialects should work, but are not provided as extras.

Optional

To get the best performance out of the DynamicHashRequestFingerprinter you should install a fast hashing algorithm.

• xxhash

• spookyhash