scrachy.middleware.filter

Middleware for filtering (or ignoring) responses if they are fresh in the cache.

Classes

CachedResponseFilter(crawler)

Sometimes you scrape the same domains multiple times looking for new content.