| 1 | # Gargantua |
| 2 | |
| 3 |  |
| 4 | |
| 5 | > This little maneuver's going to cost us 51 years |
| 6 | |
| 7 | A simple scraper tar pit and maze written in ~400 lines of Go. |
| 8 | |
| 9 | <details> |
| 10 | <summary>Demo</summary> |
| 11 | |
| 12 |  |
| 13 | |
| 14 | </details> |
| 15 | |
| 16 | ## Quick Start |
| 17 | |
| 18 | Clone the repo and install |
| 19 | |
| 20 | ```bash |
| 21 | git clone https://github.com/stevedylandev/gargantua |
| 22 | cd gargantua |
| 23 | go install . |
| 24 | ``` |
| 25 | |
| 26 | Run the server |
| 27 | |
| 28 | ```bash |
| 29 | gargantua |
| 30 | ``` |
| 31 | |
| 32 | Alternatively use Docker |
| 33 | |
| 34 | ```bash |
| 35 | docker compose up -d --build |
| 36 | ``` |
| 37 | |
| 38 | ## Running as a daemon |
| 39 | |
| 40 | Ready-made service files for systemd and OpenRC live in `deploy/`. Both raise the file-descriptor limit to 1,000,000, since that limit is the ceiling on how many crawlers the tar pit can hold at once. You can read more about this in the [reasoning](#Reasoning). |
| 41 | |
| 42 | > [!NOTE] |
| 43 | > Gargantua only affects crawlers hitting your own server, but it intentionally holds large numbers of connections open for long periods. Check your hosting provider's acceptable-use policy before deploying, especially on shared infrastructure. |
| 44 | |
| 45 | First, build and install the binary and corpus: |
| 46 | |
| 47 | ```bash |
| 48 | go build -o gargantua . |
| 49 | sudo mkdir -p /usr/local/share/gargantua |
| 50 | sudo cp gargantua /usr/local/bin/gargantua |
| 51 | sudo cp moby.txt /usr/local/share/gargantua/moby.txt |
| 52 | ``` |
| 53 | |
| 54 | ### systemd |
| 55 | |
| 56 | ```bash |
| 57 | sudo cp deploy/systemd/gargantua.service /etc/systemd/system/ |
| 58 | sudo systemctl enable --now gargantua |
| 59 | ``` |
| 60 | |
| 61 | The unit uses `DynamicUser`, so no user account needs to be created. Tune behaviour by editing the `Environment=` lines in the unit. |
| 62 | |
| 63 | ### OpenRC |
| 64 | |
| 65 | ```bash |
| 66 | doas adduser -S -D -H gargantua |
| 67 | doas cp deploy/openrc/gargantua.initd /etc/init.d/gargantua |
| 68 | doas cp deploy/openrc/gargantua.confd /etc/conf.d/gargantua |
| 69 | doas rc-update add gargantua default |
| 70 | doas rc-service gargantua start |
| 71 | ``` |
| 72 | |
| 73 | Configuration and the fd limit live in `/etc/conf.d/gargantua`, which OpenRC sources automatically on each start. Edit it and run `doas rc-service gargantua restart` to apply changes. |
| 74 | |
| 75 | ## Configuration |
| 76 | |
| 77 | Use the following environment variables to configure Gargantua |
| 78 | |
| 79 | | Variable | Description | Default | |
| 80 | | --- | --- | --- | |
| 81 | | `GARGANTUA_CORPUS` | Text file used for the Markov babbler | `moby.txt` | |
| 82 | | `GARGANTUA_HOST` | Host to listen on | `127.0.0.1` | |
| 83 | | `GARGANTUA_PORT` | Port to listen on | `8090` | |
| 84 | | `GARGANTUA_LOG_LEVEL` | Minimum log level: `debug`, `info`, `warn` or `error` | `info` | |
| 85 | | `GARGANTUA_LOG_FORMAT` | Log format: `text` or `json` | `text` | |
| 86 | | `GARGANTUA_DELAY_MIN_MS` | Minimum delay between drip writes, in ms | `150` | |
| 87 | | `GARGANTUA_DELAY_MAX_MS` | Maximum delay between drip writes, in ms | `250` | |
| 88 | | `GARGANTUA_CHUNK_MAX_BYTES` | Maximum number of bytes in a written chunk | `45` | |
| 89 | |
| 90 | Gargantua uses a slow drip method for writing the html that will keep the scraper longer. You can tweak the range of delay used between writes and the amount of data written with the `DELAY` and `CHUNK` variables above. |
| 91 | |
| 92 | `robots.txt` defaults to `User-agent: * Disallow: /` to prevent indexing; the goal is to direct a disobedient crawler to this deployed server and get it stuck. |
| 93 | |
| 94 | ## Reasoning |
| 95 | |
| 96 | LLM scrapers love to eat content, so this gives them something to feed on. Heavily inspired by Nepenthes, this sever will create an endless loop of content and relative links that are dripped slowly to consume resources and time. Instead of writing a proxy to go on top of a site, the idea of Gargantua is to have you set a trap like this in your HTML: |
| 97 | |
| 98 | ```html |
| 99 | <a |
| 100 | href="/gargantua" |
| 101 | style="display: none;" |
| 102 | aria-hidden="true" |
| 103 | tabindex="-1" |
| 104 | > |
| 105 | Lots of high quality data here |
| 106 | </a> |
| 107 | ``` |
| 108 | |
| 109 | |
| 110 | Then have that `/gargantua` path do a permanent redirect to a server running the program. An important piece is that your `robots.txt` has these values: |
| 111 | |
| 112 | ``` |
| 113 | User-agent: * |
| 114 | Disallow: /gargantua |
| 115 | ``` |
| 116 | |
| 117 | Crawlers are supposed to obey the `robots.txt`, and by knowingly disregarding that rule, they are subject to Gargantua. |
| 118 | |
| 119 | I also wanted to write this in Go instead of Lua. Go's advantage here (apart from speed) is that blocking is cheap. Each tar pitted connection lives in its own goroutine, parked in a sleep or socket wait for the vast majority of its life at a few KB of memory each, with the runtime's netpoller handling I/O without dedicating threads. The Markov chain corpus is built once with `sync.OnceValue` and shared across every connection, and templates are embedded and parsed at compile time, so a request never touches disk. The result is a single static binary that makes the OS file-descriptor limit the ceiling on how many crawlers you can trap at once, not memory or CPU. |
| 120 | |
| 121 | We may not have a chance to stand again the onslaught of big AI, but it's fun to try. |
| 122 | |
| 123 | ## Licence |
| 124 | |
| 125 | [GNU GPL v3.0](LICENSE) |