chore: update README f0aa817f
Steve · 2026-11-17 00:09 1 file(s) · +10 −4
README.md +10 −4
1 1
# Gargantua 
2 2
3 -
![cover](https://files.stevedylan.dev/gargantua-demo.jpg)
3 +
![cover](https://files.stevedylan.dev/gargantua-logo.png)
4 4
5 5
> This little maneuver's going to cost us 51 years
6 6
7 -
A simple scraper tarpit and maze written in Go
7 +
A simple scraper tar pit and maze written in ~400 lines of Go.
8 8
9 +
<details>
10 +
<summary>Demo<summary/>
11 +
12 +
![gif](https://files.stevedylan.dev/gargantua-demo.gif)
13 +
14 +
</details>
9 15
10 16
## Quick Start 
11 17
31 37
32 38
## Running as a daemon 
33 39
34 -
Ready-made service files for systemd and OpenRC live in `deploy/`. Both raise the file-descriptor limit to 1,000,000, since that limit is the ceiling on how many crawlers the tarpit can hold at once. You can read more about this in the [reasoning](#Reasoning).
40 +
Ready-made service files for systemd and OpenRC live in `deploy/`. Both raise the file-descriptor limit to 1,000,000, since that limit is the ceiling on how many crawlers the tar pit can hold at once. You can read more about this in the [reasoning](#Reasoning).
35 41
36 42
> [!NOTE]
37 43
> Gargantua only affects crawlers hitting your own server, but it intentionally holds large numbers of connections open for long periods. Check your hosting provider's acceptable-use policy before deploying, especially on shared infrastructure.
110 116
111 117
Crawlers are supposed to obey the `robots.txt`, and by knowingly disregarding that rule, they are subject to Gargantua. 
112 118
113 -
I also wanted to write this in Go instead of Lua. Go's advantage here (apart from speed) is that blocking is cheap. Each tarpitted connection lives in its own goroutine, parked in a sleep or socket wait for the vast majority of its life at a few KB of memory each, with the runtime's netpoller handling I/O without dedicating threads. The Markov chain corpus is built once with `sync.OnceValue` and shared across every connection, and templates are embedded and parsed at compile time, so a request never touches disk. The result is a single static binary that makes the OS file-descriptor limit the ceiling on how many crawlers you can trap at once, not memory or CPU. 
119 +
I also wanted to write this in Go instead of Lua. Go's advantage here (apart from speed) is that blocking is cheap. Each tar pitted connection lives in its own goroutine, parked in a sleep or socket wait for the vast majority of its life at a few KB of memory each, with the runtime's netpoller handling I/O without dedicating threads. The Markov chain corpus is built once with `sync.OnceValue` and shared across every connection, and templates are embedded and parsed at compile time, so a request never touches disk. The result is a single static binary that makes the OS file-descriptor limit the ceiling on how many crawlers you can trap at once, not memory or CPU. 
114 120
115 121
We may not have a chance to stand again the onslaught of big AI, but it's fun to try. 
116 122