Basic Crawler that stuffs its results in a .tar.gz
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Erik Hollensbe e01d3c1a98 Move a lock to make the channel close a little more reliable; still broken though 3 weeks ago
.gitignore add gitignore 3 weeks ago
README.md Add readme 3 weeks ago
go.mod commit go packaging files 3 weeks ago
go.sum commit go packaging files 3 weeks ago
main.go Move a lock to make the channel close a little more reliable; still broken though 3 weeks ago

README.md

This is a specialized URL crawler suited for a handful of situations; it stuffs the results automatically in a .tar.gz for ease of use.

Do not depend on this software.