This was kind of a necessity for me. e621.net has been blocked in my country for years, and now, with bans on VPN, "misfire" on Cloudflare and unsolvable captchas that do not load in, doing anything remotely useful on the internet has become a problem.
It was back then when I decided to download like 1% of all posts as a "fun little project". WHICH LASTED I KID YOU NOT 4 ENTIRE MONTHS (yeah my ISP kinda sucks) of my phone downloading over TOR 24/7.
In the end, using Zepiwolf's The Wolf's Stash, i've downloaded 221981 posts (~4% of e621) which, even compressed (transcoded into webp/avif and ditching the alpha-channel), take up 150 GB.
An obvious problem arose - how to organize this huge ass library.
TheWolfsStash is a great app for browsing e621 and downloading some posts, but it can't do much beyond that.
e621-ng would probably be literally perfect, but I'm too dumb to figure out Docker.
Jellyfin literally crashed when i told it to scan a folder with ~100000 images (not even half of my library!)
And other "stock" imageboards/webgalleries would probably also be either not suited for such scales, too hard to set up, have incompatibility in formats or require you to manually import and tag each image.
That's when I realised if I want a perfect offline e621 library manager, i would have to make it myself.
TheWolfsStash downloads posts with name format <artist>-<id>.jpg, which already was a good start, but it did not provide other tag info in a usable way.
Initially, i tried to code some bullshit to fetch tags by post id using e621 api, but that did not work because
1. I had no idea what am i even trying to do (normally I only make desktop software, not anything web-related)
2. Cloudflare captcha.
I spent quite a lot of time trying to replicate TheWolfsStash trick of swapping in a legit Android WebView cookie file to bypass Cloudflare, but with pure Python on Windows and Firefox. And it did not work, not even once, no matter how hard i tried.
And them I finally RTFM and learned that I can just download a complete DB dump. * Facepalm *
So, three days ago i finally set on a mission to finally make a self-hosted offline e621 knockoff.
And here it is, open-source.
gitlab.com/greycat8-dev/d-glut
(Had to put it on gitlab - cant even register on github because stupid captcha never loads in)
D-Glut, which i called so because it is intended as a partial mirror of e621. E621 is sodium glutamate, and D-glutamate is a normally rare mirrorred form of L-glutamate, so this name is also a meta-pun.
Rough prototype with very basic functionality was vibe-coded with duck.ai. Almost everything was then manually refined with many swear words and somewhat ugly-looking, but real and working code.
All done in pure python3, only standard library, no pip modules, no django, with SQLite3 for database.
Features:
- Uses e621 DB dumps (e621net/db_export) to provide tag info
- Mandatory API key access so nobody can randomly stumble onto your "homework folder" server.
- Can convert AVIF into JPG if your browser does not support it.
- Sorts your posts into folders on disk by artist
- 8 themes in the signature e621 and e6ai colors
- No cookies - all the little data it needs (theme, page, search term, posts per page) - is stored in query parameters
Anti-features (things not implemented, but planned for later)
- No tag panel, so memorize your favorite tags
- No individual post pages (yes, it is all just a single search page for now)
- No thumbnailing for images (your router is gonna have a bad time) and videos (enjoy [VIDEO PLACEHOLDER] rectangle for now)
- No parent/child relations for posts and no pools.
- Can't upload to website (you have to put posts in a folder, via ftp/smb/direct access to the machine)
Requirements:
- Linux (almost everything works on Windows too. Not all tho, but I plan on fixing soon)
- python3 (dunno starting from which subversion, but works fine on 3.11-3.13)
- avifdec (apt install libavif)
Adding images for the first time:
1. Create folders named with author names
2. Download images using The Wolf's Stash
3. If you want to save disk space, convert them to avif/webp (tested, up to 10x reduction for JPGs and 2x for PNGs).
I found this cool command-line tool on github (github.com/A-Sverdrup/minsizer) and it's mostly great, but some files produced by it (specifically some PNGs converted with cavif) cause "This image cannot be displayed, because it contains errors" in Firefox. The tradeoff of not using avif however was much worse so i stuck with it and just implemented AVIF decoding in D-Glut.
4. Put 'em into __unknown__
5. Run autosort.py
Setting up database:
1. Download e621 database dumps
2. Unpack csv's from gz's
3. Run depost.py
4. Run initfts.py
Setting up:
1. Open Notepad/Python IDLE/your other favorite IDE and open D-Glut.py
2. Change ADMIN_KEY
3. Change some other constants if you want.
4. Save.
5. Run D-Glut.py
6. Go to <your server>:<your_port>/admin?api_key=<ADMIN_KEY you just set>. If on same machine and default port, this is localhost:621/admin?api_key=<your ADMIN_KEY>
7. Generate and enroll api_keys to hand out to users (by the way, your ADMIN_KEY is also always a valid api_key)
8. Go to <your server>:<your_port>/admin?api_key=<Any of api keys you enrolled> and enjoy
Adding more images:
1. Create more folders for artists you want to sort by
2. Put new images into __unknown__
3. Run autosort.py manually or from admin panel ("Autosort library")
4. Run initfts.py manually or from admin panel ("Update search DB")
5. Click "Rescan" in admin panel or restart server