Graph Algorithms in Rust: Breadth First Search (BFS)
coming soon
Reference Implementations
- HTTrack
- Cyotek WebCopy
- Content Grabber
- ParseHub
- OutWit Hub
- spidy
- Evine
https://github.com/rivermont/spidy https://hakin9.org/evine-interactive-cli-web-crawler/ https://docs.scrapy.org/en/latest/
Web crawler
features
- Nice Logging Features
- Portalbility
- Browser Spoofing: Make requests using User Agents from 4 popular web browsers, use a custom spidy bot one, or create your own!
- zip file output?