CMD + K

crawler-cleaner

Community

Process json web logs and remove logs from detected crawlers/bots/scrapers

Installation

To install this package, run one of the following:

Conda
$conda install axiom-data-science::crawler-cleaner

Usage Tracking

0.1
1 / 8 versions selected
Total downloads: 0

Description

Processes json web log input from stdin (one json log object per line), removing any user agents determined to be crawlers/bot/scrapers. The project uses the crawler-user-agents project as the user agent database.

About

Summary

Process json web logs and remove logs from detected crawlers/bots/scrapers

Information Last Updated

Mar 25, 2025 at 16:24

Total Downloads

6

Platforms

Linux 64 Version: 0.1
macOS 64 Version: 0.1