chore: update README reasoning 66a3688e
Steve · 2026-11-16 23:36 1 file(s) · +16 −2
README.md +16 −2
87 87
88 88
## Reasoning
89 89
90 -
LLM scrapers love to eat content, so this gives them something to feed on. Heavily inspired by Nepenthes, this sever will create an endless loop of content and relative links that are dripped slowly to consume resources and time. Instead of writing a proxy to go on top of a site, the idea of Gargantua is to have your personal site `robots.txt` say something like: 
90 +
LLM scrapers love to eat content, so this gives them something to feed on. Heavily inspired by Nepenthes, this sever will create an endless loop of content and relative links that are dripped slowly to consume resources and time. Instead of writing a proxy to go on top of a site, the idea of Gargantua is to have you set a trap like this in your HTML: 
91 +
92 +
```html
93 +
<a
94 +
  href="/gargantua"
95 +
  style="display: none;"
96 +
  aria-hidden="true"
97 +
  tabindex="-1"
98 +
>
99 +
  Lots of high quality data here
100 +
</a>
101 +
```
102 +
103 +
104 +
Then have that `/gargantua` path do a permanent redirect to a server running the program. An important piece is that your `robots.txt` has these values: 
91 105
92 106
```
93 107
User-agent: *
94 108
Disallow: /gargantua
95 109
```
96 110
97 -
Then have that `/gargantua` path do a permanent redirect to a server running the program. Crawlers are supposed to obey the `robots.txt`, and by knowingly disregarding that rule, they are subject to Gargantua. 
111 +
Crawlers are supposed to obey the `robots.txt`, and by knowingly disregarding that rule, they are subject to Gargantua. 
98 112
99 113
I also wanted to write this in Go instead of Lua. Go's advantage here (apart from speed) is that blocking is cheap. Each tarpitted connection lives in its own goroutine, parked in a sleep or socket wait for the vast majority of its life at a few KB of memory each, with the runtime's netpoller handling I/O without dedicating threads. The Markov chain corpus is built once with `sync.OnceValue` and shared across every connection, and templates are embedded and parsed at compile time, so a request never touches disk. The result is a single static binary that makes the OS file-descriptor limit the ceiling on how many crawlers you can trap at once, not memory or CPU. 
100 114