Topic: E621 Extended search utility (With a nice GUI)

Posted under e621 Tools and Applications

pierogi z miesem

Member

E621 Extended Query

Some people want to search by more than 40 tags and some people want tags to look like a list, and not a long string with weird characters.
So I made this this tool.
It lets you query the entire E621 database, locally on your computer and displays results in a nice GUI.
You can query by infinite amount of tags and then sort the results with simple secondary filters.
Bonus: The page switching feels a lot smoother now.
If people express interest in this concept - I will add other search parameters, speed up the querying process by a lot and reduce RAM usage.

Processing speeds:

  • Saving the database to disk, so that you don't have to re-download it each time: ~80 seconds
  • Indexing the database for further lookups: ~30 seconds
  • Performing a query: ~30 seconds
  • Navigating results: Instant
  • Query results are stored on disk, so that they can be picked up after restart, because it would be stupid not to.

Download and docs: Github

Feedback and suggestions are very much welcome
Also feel free to contact me if you have any questions.

Yes, it's a one-click solution.
No, you don't have to do anything special.

GitHub Readme copypaste

How to use

  • Go to releases tab and download the latest release.
  • Place the .exe into an arbitrary folder on the fastest storage device you have (anything that doesn't has moving parts inside and is connected through USB 3.0 (if applicable). Basically - put it on an SSD)
  • Run the .exe file and do what it says. If it spits shit like "Traceback (most recent call last)", please repot this as an error.
  • If it asks "Allow python to access network" - click yes or something like that.
  • If everything goes well you should end up on a webpage, which should be quite intuitive to navigate.
  • Now look at the folder the .exe is in, there should be a file called "tag_match.py".
  • Open it as a text file, preferably with Notepad++ (pls use Notepad++, don't do it in default Windows notepad).
  • That file describes a query to filter posts by. It has instructions inside. Edit it to your liking (and don't forget to save).
  • Go back to the webpage and click "Re-Download export", wait for it to finish.
  • Then click "Execute Query", wait for it to finish.
  • The results will be displayed below.
  • GIF-like posts and videos are labeled as such with an icon in the top right corner.
  • Click on an image/video to enlarge it and click again or press "Escape" to close the fullres overlay.

You can quickly sort the reults with the corresponding buttons in the GUI.
All sorting methods are additive. This means, that if you sort by Score and then
sort by Oldest - posts will be ordered by rating from oldest to newest and so on.

Click on an image to view its tags and a link to the original post.

The amount of RAM used by the tool equals to roughly 50-60% of the size of the database
(as of 04-02-2024 the database is almost 4GB and growing exponentially).
Although this is only true when actively performing queries.
If you perform a query, close the software and open it again -
the amount of RAM used will be the size of the cached query (usually a few hundred megabytes at most).
If you perform a query again - the RAM usage will go back to the numbers mentioned above.

You can only have one webpage with this tool opened.

For now it's only possible to search by tags.
If at least someone finds this tool useful - support for other query types
will be added.

Other ways to run the tool

Alternatively, you can download the repository and run the tool directly.
Should work both on Linux and Windows.

For that you need:

Once all the requirements are satisfied - run launcher.cmd, if you're on Windows.
Or simply execute main.py if .cmd file is not an option for you.

For now, when running the tool directly (through main.py) the port is hardcoded to 8089.

Compiling to .exe yourself

Compiling to exe works out of the box, there are no any kind of special setups needed.

All you need is:

Once all the requirements are satisfied - run compile_exe.cmd, if you're on Windows.
Or simply execute compile_exe.py if .cmd file is not an option for you.

The tool can be easily extended, so feature requests, feedback or any other comments are very much welcome

Updated

savageorange

Member 
main.py said:

                # todo: somehow get the total amount of records
		# problem is, it's impossible to know how many records
		# are there without reading the entire database
		total_records = 4_000_000

https://e621.net/stats (Posts section). Presumably you'd want to query this just before database update.

pierogi z miesem

Member

savageorange said:

main.py said:

                # todo: somehow get the total amount of records
		# problem is, it's impossible to know how many records
		# are there without reading the entire database
		total_records = 4_000_000

https://e621.net/stats (Posts section). Presumably you'd want to query this just before database update.

Interesting. I wasn't aware of this section.
I'll definitely use it, thank you.

  • 1