Topic: Another e621 downloader

Posted under e621 Tools and Applications

I'm learning python right now and I came with idea - why not make another e621 downloader while I'm doing it. It can be downloaded here. Currently it's command line, only supports downloading pools and it's not customable at all, but I'm going to add more features soon.

Edit:
And now it supports downloading pools and downloading tags up to page 750.

Updated

Version 0.3.1 is released.

Currently downloader supports both tag and pool downloading up to page 750. Login with API key can be used. There's also an executable file, so in theory you should be able to run it even if you don't have python installed. In theory because I haven't tested it yet outside my OS.

Next feature, for version 0.4, should be GUI.

Updated by anonymous

faux-pa@tux-client:~/e621_downloader$ python ./sources/Downloader.py POOL "1413" "./Pools"
  File "./sources/Downloader.py", line 18
    def get_conf_paths() -> dict:
                         ^
SyntaxError: invalid syntax

I'm using python 2.7.13-2, which is the latest version of python available for Ubuntu Artful Aardvark (17.10)

python is already the newest version (2.7.13-2).

Updated by anonymous

Faux-Pa said:

faux-pa@tux-client:~/e621_downloader$ python ./sources/Downloader.py POOL "1413" "./Pools"
  File "./sources/Downloader.py", line 18
    def get_conf_paths() -> dict:
                         ^
SyntaxError: invalid syntax

I'm using python 2.7.13-2, which is the latest version of python available for Ubuntu Artful Aardvark (17.10)

python is already the newest version (2.7.13-2).

you can ignore your package manager's version of python and compile python 3.6.2, which is incompatible with the 2.7 series but is necessary for progress going forward.

Updated by anonymous

Faux-Pa said:

faux-pa@tux-client:~/e621_downloader$ python ./sources/Downloader.py POOL "1413" "./Pools"
  File "./sources/Downloader.py", line 18
    def get_conf_paths() -> dict:
                         ^
SyntaxError: invalid syntax

I'm using python 2.7.13-2, which is the latest version of python available for Ubuntu Artful Aardvark (17.10)

python is already the newest version (2.7.13-2).

You'll find that you also have a copy of python3 installed, which should be able to run it. Ubuntu and a few other distros have a split between python 2 and python 3 because they are not syntactically compatible, and some programs require 2.x still.

Updated by anonymous

Was trying to use this as stated in the readme, but I got a message about config not being detected.

Edit: This was apparently caused by not changing the directory. Whoops.

Updated by anonymous

Sorry for not responding. I've been quite busy for several last days.

Faux-Pa said:

faux-pa@tux-client:~/e621_downloader$ python ./sources/Downloader.py POOL "1413" "./Pools"
  File "./sources/Downloader.py", line 18
    def get_conf_paths() -> dict:
                         ^
SyntaxError: invalid syntax

I'm using python 2.7.13-2, which is the latest version of python available for Ubuntu Artful Aardvark (17.10)

python is already the newest version (2.7.13-2).

As others said it need the newest python 3. I think that the newest feature I use is typing, which was introduced in 3.5 version. Maybe I'm going to try to make unix executable in addition to windows one.

Furrin_Gok said:
Was trying to use this as stated in the readme, but I got a message about config not being detected.

Edit: This was apparently caused by not changing the directory. Whoops.

By that you mean calling the program from path more distant then 1 level up from executable/script? I've just added fix, so it should detect config no matter from where you call it.

Updated by anonymous

It's alive!

Version 0.4 is ready.
Changes since previous version:
- Downloader is no longer a command line application, it has a GUI. It's ugly, but it's a GUI.
- Config should load properly even if the executable is called not from root directory.
- Two additional fields in config that allow customization of name given to downloaded files.
- Dropped executable file (at least for now) since it's not working with tkinter.
- I hate tkinter now.

Updated by anonymous

Granberia said:
It's alive!

Version 0.4 is ready.
Changes since previous version:
- Downloader is no longer a command line application, it has a GUI. It's ugly, but it's a GUI.
- Config should load properly even if the executable is called not from root directory.
- Two additional fields in config that allow customization of name given to downloaded files.
- Dropped executable file (at least for now) since it's not working with tkinter.
- I hate tkinter now.

Hey, this is awesome! Python can be the worst for standalone applications, but its really easy to work with and is fairly barebones.

Updated by anonymous

Granberia said:
As others said it need the newest python 3. I think that the newest feature I use is typing, which was introduced in 3.5 version. Maybe I'm going to try to make unix executable in addition to windows one.

If you need python3, ask for it explicitly in the shbang line, eg #!/usr/bin/env python3. That's what PEP 394 is about.

Otherwise (eg if your shbang line says #!/usr/bin/env python) you are saying "I don't care what Python I get". "python" can be either Py2 or Py3, according to the distro's choice.

Updated by anonymous

savageorange said:
If you need python3, ask for it explicitly in the shbang line, eg #!/usr/bin/env python3. That's what PEP 394 is about.

Otherwise (eg if your shbang line says #!/usr/bin/env python) you are saying "I don't care what Python I get". "python" can be either Py2 or Py3, according to the distro's choice.

I will add it in the next version, thanks.

BTW does anyone knows why https://e621.net/set/index.json?user_id=16311 seems to by slower than https://e621.net/set/index.xml?user_id=16311 by order of magnitude (With python script and firefox it's like about 1 second vs 10 seconds.) Api page states that "It takes significantly more work to process the XML data structures", so I don't understand that result even more.

Edit:
NVM - I found the reason.

Updated by anonymous

And there's version 0.5:

  • Instead of just downloading user can do multiple actions including
    • Downloading posts
    • Upvoting posts (Login must be used for this action)
    • Adding posts to favorites (Login must be used for this action)
    • Adding posts to set - currently only one set can be selected (Login must be used for this action)
  • Actions can be done also on popular posts
  • Using logs instead of stderr is configurable in config
  • Shebang for python 3
  • Loading time can after config (re)load can be really slow if you have a lot of posts in sets you own/maintain until the bug from forum #260768 is fixed

The next version is going to be 1.0
I plan to:

  • Make UI more responsive by doing actions in separate thread (Currently UI can't respond while, for example, big post is being downloaded)
  • Allow to add post to multiple sets in one go
  • Draw professional logo of the application in paint
  • Fix any bugs in existing features

I remember that my downloader does not handle downloading posts where list has more than 750 pages, but I currently don't plan to fix it.

Updated by anonymous

  • 1