mcsib said:
Thanks for understanding the situation. There are a ton of bugs that have popped up recently, new feature requests I need to look through, and those "wonderful" dependency security bugs (have to love them) that have popped up. I can't give time estimates for when I can work on the project, but I will be able to soonish.

As for the question, I have considered making a toggle for the post limiting in the downloader. The only reason I haven't is due to scrapes. I don't know how good the servers on e621 are, but I'm reluctant to make that functionality easier as not to anger the e621 staff into emailing me about it (I mean that as a light joke). It more or less depends, if an admin or staff worker gives me a thumbs up that I'm good to implement that toggle, I can add it. I just remember way back then when other downloaders did the same thing with other sites, and then immediately DDOSed the servers with hundreds of users downloading 20k+ images. That's something I'd rather not do. The main worry, in this case, wouldn't be a DDOS situation (most modern servers have gotten better about it) but more so throttling.

I obviously can't answer for the staff but in the API FAQ there's no mention that I've seen about a max number of downloads, only that the request rate should be limited to 1/sec or less. If you decide to implement the toggle, maybe adding a throttle to it that goes slower than it currently does and letting the user know that the trade-off on large scrapes is a slower speed would be a good compromise?

drakkenfyre said:
I obviously can't answer for the staff but in the API FAQ there's no mention that I've seen about a max number of downloads, only that the request rate should be limited to 1/sec or less. If you decide to implement the toggle, maybe adding a throttle to it that goes slower than it currently does and letting the user know that the trade-off on large scrapes is a slower speed would be a good compromise?

oh sorry did not read it so it can download individuals tabs

Thanks for the tool, and sorry for asking! ... but what exactly is the command, to download all my favorited pictures? Thanks in advance.

I'm not tech savvy so I thought someone was making a system to put E621 images in the game Rust XD All I could think of is someone raiding a base only to be blasted by Knots XD

Anyway keep up the good work, seems like a real handy tool!

Something changed recently and broke... I think the ability to parse some tags? I haven't narrowed down exactly what's busted yet.

magnuseffect said:
Something changed recently and broke... I think the ability to parse some tags? I haven't narrowed down exactly what's busted yet.

It looks like it could just be unpopulated tags breaking it, but it came off as unusual that so many broke at the same time.

Edit: It was 2 recently-DNP artists and another tag I input wrong.

Thank you for this forum post, McSib!

Your awesome script saved me a lot of time and trouble.

Thank you so much! Your program has helped me a lot! ❤

hey there, first of all this seems so awesome and thanks for sharing it... I don't actually understand how to use it though... I'm not savvy in programing. I tried following your instructions and downloaded everything you said. I get lost right after downloading Git. I'm not sure how to do anything in the bullet on the second step of the windows install guide. I'm not sure if it is worth your time to explain this to novices or better yet my a tutorial video. I'd really love to be able to use a program like this, though.

edit: I gave up and uninstalled everything. When I went to uninstall the rust program and my vpn was completely fried. I had to uninstall and reinstall it to get it working again. I've run several antivirus programs since and nothing came up, but it was very unsettling.

Sorry，but who can tell me how to edit the directory？ I want to edit the directory to "E:/work", but I find if I write that in the config the downloader will not work.
How to edit the config？I mean how to write if my directory is "E:/work"？I should write "E:/work" or“E:work” or “work” or what？

Hello everyone.

So i basically using e621_downloader and it is a neat tool.
However I cant figure it out how to downlad images with specific character on it.
When i put string like

[general]
character: character_name

And all othger tags are empty, save tags.txt file and lauch the program, it asks me for safe mode, and then crashes without any download. What I'am doing wrong? Did i need to fill other tags as well, like [pools] etc?

roy_batty said:
Hello everyone.

So i basically using e621_downloader and it is a neat tool.
However I cant figure it out how to downlad images with specific character on it.
When i put string like

[general]
character: character_name

And all othger tags are empty, save tags.txt file and lauch the program, it asks me for safe mode, and then crashes without any download. What I'am doing wrong? Did i need to fill other tags as well, like [pools] etc?

dont do
[general]
character: character_name

just do
[general]
character_name

example
[general]
nicole_watterson

Ba-da-bump!

Still using your program, still the easiest and most efficient scraper I've come across. I know it's been a couple years since your last release, but if you're still doing anything with it, how feasible would it be to support e621's AI-centric site e6ai.net? AFAIK it's on the exact same platform and using the same API, just a different domain and a separate user database from the main site. I've seen artists and characters I'd love to do scrapes of using your wonderful program but of course it was written specifically for this site.

Greetings,

I seem to be having trouble using this tool to download the tags I want. For some reason, the program crashes. I even tried using the latest gallery-dl version, and I keep getting 401 errors. Is this a back-end error on e621?

sfmamateur said:
Greetings,

I seem to be having trouble using this tool to download the tags I want. For some reason, the program crashes. I even tried using the latest gallery-dl version, and I keep getting 401 errors. Is this a back-end error on e621?

I just used it again a few minutes ago and it worked fine?

drakkenfyre said:
I obviously can't answer for the staff but in the API FAQ there's no mention that I've seen about a max number of downloads, only that the request rate should be limited to 1/sec or less. If you decide to implement the toggle, maybe adding a throttle to it that goes slower than it currently does and letting the user know that the trade-off on large scrapes is a slower speed would be a good compromise?

The logic is in grabber.rs and can easily be modified to increase the limit FWIW

drakkenfyre said:
I just used it again a few minutes ago and it worked fine?

Strange, let me see what's going on. I thought it it may be on my account, yet I see my status isn't banned or anything.
Okay, I found the error: Apparently it's not being able to locate the file "tags.txt" but I have it right here! What's going on??? I barely changed anything, even. Was this a backend change?
17:57:05 [INFO] Creating config file...
17:57:05 [INFO] Tag file does not exist, crating tag file...
17:57:05 [INFO] The tag file is created, the application will close so you can include the artists, sets, pools, and individual posts you wish to download.
Press ENTER to close the application...

I decided to update to 1.7.2 and got this:
thread 'main' panicked at 'called Result::unwrap() on an Err value: Error("missing field fileNamingConvention", line: 3, column: 1)', src\e621\io\mod.rs:80:50
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

EDIT: Even tried regenerating my API Key, nothing happened.

sfmamateur said:
Strange, let me see what's going on. I thought it it may be on my account, yet I see my status isn't banned or anything.
Okay, I found the error: Apparently it's not being able to locate the file "tags.txt" but I have it right here! What's going on??? I barely changed anything, even. Was this a backend change?
17:57:05 [INFO] Creating config file...
17:57:05 [INFO] Tag file does not exist, crating tag file...
17:57:05 [INFO] The tag file is created, the application will close so you can include the artists, sets, pools, and individual posts you wish to download.
Press ENTER to close the application...

I decided to update to 1.7.2 and got this:
thread 'main' panicked at 'called Result::unwrap() on an Err value: Error("missing field fileNamingConvention", line: 3, column: 1)', src\e621\io\mod.rs:80:50
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

EDIT: Even tried regenerating my API Key, nothing happened.

It sounds like you have some white spaces in the tags. Check to make sure there's no errant spaces after any of the tags or spaces between them. I was having that problem at one time. Copy/pasting tags and extra spaces behind them snuck in. Any spaces whatsoever will cause it to fail.

Found a couple bugs to report while attempting to back up lists of artists and pools.
Don't know if it makes a difference, but I changed "fileNamingConvention": "id" instead of "md5" in config.json for all of these.

1st:
Entering the following into the 'pool' tags section causes the program to stall for a minute or two, then error out after attempting to allocate 137 gigabytes of memory.
28451 # ヒスイバクフーン×ショウ 漫画 by Kaniku
18159 # Backyard Fun — by Funkybun

E:\E6 Archive\e621_downloader_v1.7.2_x86_64_windows>e621_downloader.exe
Should enter safe mode? no
memory allocation of 137438953472 bytes failed

Oddly, if I remove the comments and just leave the pool Ids, 18159 suddenly works just fine, but 28451 fails with a different error:

Should enter safe mode? no
02:04:09 [INFO] "ヒスイバクフーン×ショウ_漫画_by_Kaniku" grabbed!
 [00:00:00] [-------------------------------------------------------------------------------------] 0B/15.51 MiB 0B/s 0sthread 'main' panicked at 'byte index 25 is not a char boundary; it is inside '×' (bytes 24..26) of `ヒスイバクフーン×ショウ_漫画_by_Kaniku`', src\e621\grabber.rs:206:34
stack backtrace:
   0: rust_begin_unwind
             at /rustc/53e4b9dd74c29cc9308b8d0f10facac70bb101a7/library\std\src/panicking.rs:575:5
   1: core::panicking::panic_fmt
             at /rustc/53e4b9dd74c29cc9308b8d0f10facac70bb101a7/library\core\src/panicking.rs:64:14
   2: core::str::slice_error_fail_rt
   3: core::str::slice_error_fail
             at /rustc/53e4b9dd74c29cc9308b8d0f10facac70bb101a7/library\core\src\str/mod.rs:86:9
   4: <e621_downloader::e621::grabber::PostCollection as e621_downloader::e621::grabber::Shorten<&str>>::shorten
   5: e621_downloader::program::Program::run
   6: e621_downloader::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

It definitely seems like a character parsing issue, not properly handling UTF-8 characters. Weirdly, this one other pool that I would expect to have similar issues does work fine though:
23309 # Axolotl Suddenly Hates Everything (急に全てが嫌になったウーパールーパー) by Purinharumaki Karameru

===============================================

2nd:
Apparently, many of my favorites no longer exist, as the artist and all their work has been removed. Most of the time, a non-existing artist just causes the program to error out gracefully, telling me which artist no longer exists. For this artist specifically though, it crashed hard with a different error:
Artist: kestelto
PostId that was in my favorites.json: post #3998522

Should enter safe mode? no
thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', src\e621\io\tag.rs:233:13
stack backtrace:
   0: rust_begin_unwind
             at /rustc/53e4b9dd74c29cc9308b8d0f10facac70bb101a7/library\std\src/panicking.rs:575:5
   1: core::panicking::panic_fmt
             at /rustc/53e4b9dd74c29cc9308b8d0f10facac70bb101a7/library\core\src/panicking.rs:64:14
   2: core::panicking::panic
             at /rustc/53e4b9dd74c29cc9308b8d0f10facac70bb101a7/library\core\src/panicking.rs:111:5
   3: e621_downloader::e621::io::tag::TagIdentifier::get_tag_from_alias
   4: e621_downloader::e621::io::tag::TagIdentifier::search_for_tag::{{closure}}
   5: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
   6: e621_downloader::e621::io::tag::parse_tag_file
   7: e621_downloader::program::Program::run
   8: e621_downloader::main

are you sure that tags are applied properly? when i supply a negative tag like "-3d_(artwork)", then the number of downloads doesn't shrink like a proper intersection. Sometimes the numbers appear to grow.

please save images as <post ID>.<format>. that preserves information useful for debugging, such as being able to easily navigate to the page that has the image. instead of the random image ID, which is not so easily searched, at least with the Web UI.

noticed a quirk in which tag queries beginning with a negated term (e.g. "-animated") result in an awkward download tree, with a directory beginning with a dash. this can cause the directory name to break common commands, as file and folder names beginning with dash are interpreted as flags. recommend using a prefix like "e621-" in the gallery name.

noticed a glitch in which any e621 tags containing slashes (e.g. "male/female") trigger altering the download tree, nesting in deeper directories. recommend replacing slashes with dashes in the gallery name.

plz publish the crate on https://crates.io/

produce mac and linux binaries https://github.com/cross-rs/cross

and add a configuration option to specify the safe vs nonsafe behavior, skipping the prompt every single time.

would love an option to restrict to a list of (case insensitive) file extensions, and an option to automatically download each images' full tags as <image ID basename>.txt, in preparation for LoRA generation https://education.civitai.com/using-civitai-the-on-site-lora-trainer/

a filename prefix of score-<value>- would help with LoRA generation, as well as general trimming to save space, by allowing people to quickly delete low score images after the download.