chore: updated README
4f57e3d7
1 file(s) · +9 −3
| 1 | 1 | # Gargantua |
|
| 2 | 2 | ||
| 3 | - | // COVER |
|
| 3 | + |  |
|
| 4 | 4 | ||
| 5 | - | > This little maneuver is going to cost us years |
|
| 5 | + | > This little maneuver's going to cost us 51 years |
|
| 6 | 6 | ||
| 7 | 7 | A simple scraper tarpit and maze written in Go |
|
| 8 | 8 | ||
| 48 | 48 | ||
| 49 | 49 | Then have that `/gargantua` path do a permanent redirect to a server running the program. Crawlers are supposed to obey the `robots.txt`, and by knowingly disregarding that rule, they are subject to Gargantua. |
|
| 50 | 50 | ||
| 51 | - | I also wanted to write this in Go instead of Lua to make it more resource efficient. Go really is amazing for this kind of project, as it uses goroutines by default to enable multithreaded performance, and the design of the server is to keep each crawl within its own goroutine. The server can handle upwards of 1000 connections at once and not break a sweat, making it ideal to run on smaller hardware like a Pi. |
|
| 51 | + | I also wanted to write this in Go instead of Lua to make it more resource efficient. Go's advantage here (apart from speed) is that blocking is cheap. Each tarpitted connection lives in its own goroutine, parked in a sleep or socket wait for the vast majority of its life at a few KB of memory each, with the runtime's netpoller handling I/O without dedicating threads. The Markov chain corpus is built once with `sync.OnceValue` and shared across every connection, and templates are embedded and parsed at compile time, so a request never touches disk. The result is a single static binary that makes the OS file-descriptor limit the ceiling on how many crawlers you can trap at once, not memory or CPU. |
|
| 52 | + | ||
| 53 | + | We may not have a chance to stand again the onslaught of big AI, but it's fun to try. |
|
| 54 | + | ||
| 55 | + | ## Licence |
|
| 56 | + | ||
| 57 | + | [GNU GPL v3.0](LICENSE) |
|