Posted under e621 Tools and Applications
Hi! Wanted to share that I made a script that analyzes a user's public favorites data to generate an overview of their most liked tags, in the style of something like Spotify Wrapped. Here's an example of what that looks like:
The project's source code, alongside instructions of how to generate one of your own, can be found in the link below :3
Updated
I do have a few comments:
On a side note, you also might be able to request a full dump of the favorites db table? I know previous developers were willing to do it, I'm not sure about the current developer
donovan_dmc said:
Pretty neat toolI do have a few comments:
- Your requirements.txt seems to be missing some dependencies, you should try running it in a virtualenv or somewhere with nothing preinstalled beyond python, I had to install requests and matplotlib before it would run
- This could likely be made to work without an api key for users who have their favorites public and may have concerns about putting their api key into a random program
- Your instructions on the repo are likely too technical for the average user, I'd have trouble believing they know what clone means on github or how to clone a repository, downloading a zip of the repository would probably be the best for them
On a side note, you also might be able to request a full dump of the favorites db table? I know previous developers were willing to do it, I'm not sure about the current developer
Thanks for trying it out! I updated the repository based on your feedback
About the full dump of the favorites... that would be really cool for generating a more detailed and less biased average user profile! Do you know who I should contact about requesting this? (and do you know what kind of file size we're talking about?)
Updated
mormonara said:
About the full dump of the favorites... that would be really cool for generating a more detailed and less biased average user profile! Do you know who I should contact about requesting this? (and do you know what kind of file size we're talking about?)
The actual sysadmin that would do the dump would be Dari, but I'm not sure if he would check his dmails or even respond
Cinder is our developer, but he probably has more pressing matters to deal with
I'm not sure how good your chances are but you can still try
The favorites table is nearly 1.1 billion rows, with a presumed csv header of post_id,user_id and each column being 1-5 digits I'd put it at anywhere between 8 and 12 gigabytes
I like countershade_fur 8.7x more than the average user. Truly interesting.
Jokes aside, this is neat. I don't use my favorites so I had to make a quick change to look at my upvotes instead, which was easy as just using the posts.json endpoint instead.
https://i.imgur.com/re7vCay.png
my constant favoriting of my own sona has been laid bare
This is really cool! Here's mine: https://imgur.com/a/KfaqRzz
Unfortunately, mine isn't the most accurate since I've had to unfav 160k posts and move them to sets to make room for new favs (it's weird, don't question it, I have my system in my mind xD fav+upvote, move 10k to a new set when nearly out of favs, mass unfav those in the set) but regardless it is cool to see the output for what's currently in my favs!
However, I don't think the "% of your favs have this tag" is accurate. Some examples, the program says 4.4% of my favs have lay_the_dragon, I searched for lay_the_dragon fav:jerdude002 and got 165 results, and with 66682 favs, that comes out to 0.247%. non-mammal_breasts is 11.8% according to the program, search on my favs finds 2326 with that tag, percentage actually comes out to 3.488%, and prehistoric_species is 12.3% according to the program, search finds 1372 favs with that tag, actual percentage comes out to 2.057%.
On a funny note, the "your likes in one post" for me actually falls in my blacklist xP but I did already have it fav'd anyway
jerdude002 said:
However, I don't think the "% of your favs have this tag" is accurate. Some examples, the program says 4.4% of my favs have lay_the_dragon, I searched for lay_the_dragon fav:jerdude002 and got 165 results, and with 66682 favs, that comes out to 0.247%. non-mammal_breasts is 11.8% according to the program, search on my favs finds 2326 with that tag, percentage actually comes out to 3.488%, and prehistoric_species is 12.3% according to the program, search finds 1372 favs with that tag, actual percentage comes out to 2.057%.
It only fetches the most recent 3200 favorites (10 pages) by default, you can increase tbe number of pages but it will take longer (1 page per 320 favorites)
donovan_dmc said:
It only fetches the most recent 3200 favorites (10 pages) by default, you can increase tbe number of pages but it will take longer (1 page per 320 favorites)
Ahh, thanks! I figured there was some limit (at least by default) xP I adjusted it, this is less surprising of a result: https://imgur.com/a/aQKE9V4
I went and redid this with all of my favorites and upvotes
Unsurprising to me they aren't that different since I upvote all of my favorites
Also another note to mormonara, your shell script is not executable
I believe you need to use git update-index --chmod=+x generate.sh to have git actually keep the executable bit
This is really nicely done. Results for my account, apparently I really like prick_ears and whisker_spots.
Also, generate.sh looks for python3, but recommend.sh looks for python and so didn't work out of the box for me at least.
mahrai_ziller said:
This is really nicely done. Results for my account, apparently I really like prick_ears and whisker_spots.Also, generate.sh looks for python3, but recommend.sh looks for python and so didn't work out of the box for me at least.
Oh, thanks! Just updated the repository to make this consistent. How did you like the recommendation system? Haven't really gotten any feedback on it yet
donovan_dmc said:
I went and redid this with all of my favorites and upvotes
Unsurprising to me they aren't that different since I upvote all of my favoritesAlso another note to mormonara, your shell script is not executable
I believe you need to use git update-index --chmod=+x generate.sh to have git actually keep the executable bit
I thought the user always had to do that themselves before running... I had no idea....
"Could not find pip in the virtual environment. Something went wrong"
Python 3.10.0 installed
mormonara said:
Oh, thanks! Just updated the repository to make this consistent. How did you like the recommendation system? Haven't really gotten any feedback on it yet
I've tried it a little, and I can see how some of the things it finds are relevant.
A few of the things it returned were on my blacklist (and on the default blacklist), I wasn't using an API key for this though.
The default score threshold is 100 but the shell script doesn't mention or ask about that.
Thank you so much for this. It's so much fun! ;w;
And it even teases you at the end. hihihi >:3c
Hi I'm pretty terrible with programming but when I installed Python via the website (version 3.13.5) and double clicked generate.bat the cmd window says
- Don't let the console window scare you <:3
- I'm setting things up so we can generate your E621 Wrapped :3
- Creating virtual environment... please wait a sec :3c
Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Apps > Advanced app settings > App execution aliases.
The system cannot find the path specified.
- Could not find pip in the virtual environment. Something went wrong.
Press any key to continue . . .
Probably a noob mistake but why wasn't python found?
uninstall python then reinstall make sure you click install to path
neat apparently for my Mazus_(tinysmallman) 999.9x
funkwolfie said:
uninstall python then reinstall make sure you click install to path
Ty realised I needed to use the options in the advanced install
kamaz5795 said:
Ty realised I needed to use the options in the advanced install
glad i could help I honestly guessed
Any chance this'll be given MacOS support?
1800foxfuck said:
Any chance this'll be given MacOS support?
This isn't operating system specific, it will work anywhere
The only possible missing thing for the shell script would be bash, but I'm sure that can be installed
Else the OP can convert it into a script that uses sh which should be present anywhere
Is there any way to increase the amount of favorites checked? Had a little poke around, but I couldn't work it out; would love to see my holistic stats.
Awesome script by the way - a lot of fun for data nerds like myself
lucrenox said:
Is there any way to increase the amount of favorites checked? Had a little poke around, but I couldn't work it out; would love to see my holistic stats.
Awesome script by the way - a lot of fun for data nerds like myself
Just run python e621_wrapped.py -u <user_id> -p <number_of_pages> on the terminal! It defaults to 100 pages but it can go up to I think 750, which would be e621's limit
After clicking generate for the first time I got
ERROR: Failed building wheel for matplotlib Failed to build matplotlib error: failed-wheel-build-for-install × Failed to build installable wheels for some pyproject.toml based projects ╰─> matplotlib
but it still allowed me to continue.
When I gave my user ID:
ModuleNotFoundError: No module named 'requests'
After installing requests, uninstalling, installing again, checking it is installed and trying every other trick I could find, the result is still the same.
Updated
Oh wow, thank you so much for making this! I've been wanting some kind of tool to analyze my faves like this for ages now. I was considering trying it myself, but I've just been too busy, heh.
One thing though- when I tried running more than 100 pages (I have a looooot of faves, heh) with this:
python e621_wrapped.py -u 5358 -p 610
it gave me this error:
ModuleNotFoundError: No module named 'requests'
....same as Supina above me.
supina said:
After clicking generate for the first time I gotERROR: Failed building wheel for matplotlib Failed to build matplotlib error: failed-wheel-build-for-install × Failed to build installable wheels for some pyproject.toml based projects ╰─> matplotlibbut it still allowed me to continue.
When I gave my user ID:
ModuleNotFoundError: No module named 'requests'After installing requests, uninstalling, installing again, checking it is installed and trying every other trick I could find, the result is still the same.
lloxie said:
Oh wow, thank you so much for making this! I've been wanting some kind of tool to analyze my faves like this for ages now. I was considering trying it myself, but I've just been too busy, heh.One thing though- when I tried running more than 100 pages (I have a looooot of faves, heh) with this:
python e621_wrapped.py -u 5358 -p 610
it gave me this error:
ModuleNotFoundError: No module named 'requests'
....same as Supina above me.
That's weird! Have you tried running e621_wrapped.py from the terminal? What's the output when you run pip show requests?
Sorry, meant to post again after doing some poking around- I made a copy of the generate.bat file and changed the run line to
python e621_wrapped.py -u 5358 -p 610
and that worked! I think trying to run e621_wrapped.py outside of the virtual environment you set up in the generate file is the issue, as it doesn't seem to know where to look for the requests file locally. Or something like that.
Also some other thoughts I had- you might want to tweak it to ignore some rather big catch-all tags like "third-party_edit", "avoid_posting", "unknown_artist", and "fan character" etc. Maybe even have it set up so users can supply certain tags to tell it to ignore. ^^ Just a thought!
lloxie said:
Sorry, meant to post again after doing some poking around- I made a copy of the generate.bat file and changed the run line to
python e621_wrapped.py -u 5358 -p 610
and that worked! I think trying to run e621_wrapped.py outside of the virtual environment you set up in the generate file is the issue, as it doesn't seem to know where to look for the requests file locally. Or something like that.Also some other thoughts I had- you might want to tweak it to ignore some rather big catch-all tags like "third-party_edit", "avoid_posting", "unknown_artist", and "fan character" etc. Maybe even have it set up so users can supply certain tags to tell it to ignore. ^^ Just a thought!
Ooooh that's a pretty good idea. I've already removed "sound_warning" and "conditional_dnp" as those were appearing on my results. I'll take a look at some of the most used artist tags and try to remove some more
mormonara said:
Hi! Wanted to share that I made a script that analyzes a user's public favorites data to generate an overview of their most liked tags, in the style of something like Spotify Wrapped. Here's an example of what that looks like:The project's source code, alongside instructions of how to generate one of your own, can be found in the link below :3
Hello, well.. it seems that this script is only available for desktop device, unfortunately I can't download some requirements for because I'm mobile. U should make for mobile users tho, right?
idanilo said:
Hello, well.. it seems that this script is only available for desktop device, unfortunately I can't download some requirements for because I'm mobile. U should make for mobile users tho, right?
I think the easiest way to get this working on mobile might be to make it run on web, but that's not really my area of expertise...
mormonara said:
Ooooh that's a pretty good idea. I've already removed "sound_warning" and "conditional_dnp" as those were appearing on my results. I'll take a look at some of the most used artist tags and try to remove some more
Yeah, good call hehe. Again, thanks for this tool, it's really nifty! ^^
On Ubuntu Linux (also, props to you for making it Linux compatible!):
Trying to run the recommendation script, first of all it doesn't seem to run when I try to do "Run in Terminal" from trying to execute it normally (nothing happens, no terminal appears) - I have to execute it from within the terminal. And it gives the following error:
- Virtual environment already exists! Skipping creation and pip install. /home/[USER]/Desktop/e621-Wrapped-main/recommend.sh: line 26: .venv/bin/activate: No such file or directory - Could not find pip in the virtual environment. Something went wrong.
EDIT: Found the problem, I think. The .venv folder was created before I installed pip, and was even blocking the recommendation script from running in terminal without being directly invoked from it.
EDIT2: Okay, running it - is there a way that it can be made to obey the user's blacklist if they have their API key?
Updated
eldfjall said:
is there a way that it can be made to obey the user's blacklist if they have their API key?
Outside of manually adding the blacklist as a negated search term (which is manually possible w/ grouped searches now), no; blacklists function client-side, not server-side, that way you can toggle them on & off w/o needing to refresh the page to fetch them from the server.
mormonara said:
That's weird! Have you tried running e621_wrapped.py from the terminal?
Thanks, that did help! It still didn't work right away but at least I got a more useful error message, telling me I was missing something called tqdm. Once I installed it and tried the same again finally it worked!
Also, did you really have to call me out like that at the end of the run xD
- So you're into brother and sister (lore), huh... ;3
Was there anywhere that instructed that you needed to have Microsoft Visual Studio installed to make this work? Because i ended up having to bother people trying to make this work until we realized that MVS was the missing piece of the puzzle. If it is part the instructions it should be made more apparent. If it isn't, it really should be. It should be listed in the requirements just like python was.
Updated
burning_house said:
Was there anywhere that instructed that you needed to have Microsoft Visual Studio installed to make this work? Because i ended up having to bother people trying to make this work until we realized that MVS was the missing piece of the puzzle. If it is part the instructions it should be made more apparent. If it isn't, it really should be. It should be listed in the requirements just like python was.
Huh, from my understanding it shouldn't be required. Is it? Do you know what part of the process required it? I'm glad it ended up working out for you
mormonara said:
Huh, from my understanding it shouldn't be required. Is it? Do you know what part of the process required it? I'm glad it ended up working out for you
I installed, uninstalled, and reinstalled python 3 times just to make sure I did everything right. after I gave up someone pointed out that I needed Microsoft Visual Studio for this to work. I'm guessing because the image isn't going to just make itself. After I download MVS with the C++ workload and used the generate.bat file I was finally able to get results. I don't know if MVS comes pre-installed now with windows by my install of it is fairly old and didn't have it.
But yeah, Microsoft Visual Studio is required.
Updated
supina said:
After clicking generate for the first time I gotERROR: Failed building wheel for matplotlib Failed to build matplotlib error: failed-wheel-build-for-install × Failed to build installable wheels for some pyproject.toml based projects ╰─> matplotlibbut it still allowed me to continue.
When I gave my user ID:
ModuleNotFoundError: No module named 'requests'After installing requests, uninstalling, installing again, checking it is installed and trying every other trick I could find, the result is still the same.
For folks with similar issue above, delete the virtual environment .venv folder and try updating the requirements.txt line 3 to matplotlib==3.10.0 before running generate.bat
It's something with the installation of Numpy that's having the issue in mine.
Cheers to @mormonara for making this and helping introduce folks to Python! :D
Twas pretty accurate with some interesting top results!
Edit: Fixed spelling. Thanks @savageorange! :D
Updated
^*matplotlib :)
I think the MVC requirement is analogous to the GCC requirement on Linux. Currently building and it seems to invoke GCC to build the numpy module.
IOW, MVC probably is NOT "required" on Windows -- it's probably just one of two main options that provides 'a compiler' (the other being the Windows port of GCC. Either Pip, or Numpy's setup.py is probably responsible for this behaviour.
BTW, not sure if permissions are wrong in the github repo, or this is just how unzip operates by default, but on Linux, generate.sh (and maybe recommend.sh) do not have executable permissions after unpacking. This may mean that it's not as simple as 'just double click on generate.sh'. Though if you are in a position to 'double click on generate.sh' (I'm not, I'm in the terminal), your file manager may also allow you to adjust the permissions.
Anyway it seems to work OK. I like to view the graph after rotating the image 90°.
INVALID_ARTISTS = "sound_warning conditional_dnp censored unknown_artist anonymous_artist avoid_posting third-party_edit"
INVALID_ARTISTS = INVALID_ARTISTS.split(" ")
EDIT: No, it doesn't. Preceding a term with - causes it to be parsed as an option. I believe this is because of this line in recommend.sh:
python3 e621_recommendation_engine.py -u "$USER_ID" -p "$PAGES" -s "$STARS" -t "$TAGS" -a "$ADD"
.
Could probably be changed to be something like this:
# ideally this would be done using bash arrays rather than a single string; would be safer if user enters something stupid, but I don't remember how to do that off the top of my head. I mainly use fish, which has requires much less in the way of obscure incantations to do things with arrays. args="-u$USER_ID -p$PAGES -s$STARS -a$ADD" [ -n "$TAGS" ] && args="$args -t$TAGS" python3 e621_recommendation_engine.py $args
EDIT: As usual, difficulty nailing down 'safe' quoting in bash. Probably the best approach here would be combining set -- with "$@" - in fact, that should be a pretty safe way to handle any kind of 'commandline argument list building' problem.
For anyone not dev savvy who 'just wants negative terms to work in extra tags', here's a minimal modification that seems to work as long as you are entering SOME extra tags:
python3 e621_recommendation_engine.py -u "$USER_ID" -p "$PAGES" -s "$STARS" -t"$TAGS" -a "$ADD"
(yes, this is just removing the single space after -t, that's enough to guarantee no part of the value of TAGS will be parsed as an option.)
Updated
this is pretty cool! the favorite character section was skewed towards characters in comics I've favorited so that wasn't perfectly accurate. also had page_number in the top 5 tags lol, but other than that it was pretty accurate :p
savageorange said:
^*matplotlib :)I think the MVC requirement is analogous to the GCC requirement on Linux. Currently building and it seems to invoke GCC to build the numpy module.
IOW, MVC probably is NOT "required" on Windows -- it's probably just one of two main options that provides 'a compiler' (the other being the Windows port of GCC. Either Pip, or Numpy's setup.py is probably responsible for this behaviour.
BTW, not sure if permissions are wrong in the github repo, or this is just how unzip operates by default, but on Linux, generate.sh (and maybe recommend.sh) do not have executable permissions after unpacking. This may mean that it's not as simple as 'just double click on generate.sh'. Though if you are in a position to 'double click on generate.sh' (I'm not, I'm in the terminal), your file manager may also allow you to adjust the permissions.
Anyway it seems to work OK. I like to view the graph after rotating the image 90°.
- One thing that seems unclear to me is the values in 'How much less present than average'. For example I have a tick mark '0.14' -- is this supposed to be equivalent to '14% less than average', which as I understand it matches the scale relative_presence is specified with.. or something else?
- .. 'avoid_posting' and 'third-party_edit' should probably be added to the list of INVALID_ARTISTS. You could also make that line easier to maintain by using string splitting:
INVALID_ARTISTS = "sound_warning conditional_dnp censored unknown_artist anonymous_artist avoid_posting third-party_edit" INVALID_ARTISTS = INVALID_ARTISTS.split(" ")
- WRT recommendations: Does 'adding' extra tags which are negated work?
EDIT: No, it doesn't. Preceding a term with - causes it to be parsed as an option. I believe this is because of this line in recommend.sh:
python3 e621_recommendation_engine.py -u "$USER_ID" -p "$PAGES" -s "$STARS" -t "$TAGS" -a "$ADD".
Could probably be changed to be something like this:# ideally this would be done using bash arrays rather than a single string; would be safer if user enters something stupid, but I don't remember how to do that off the top of my head. I mainly use fish, which has requires much less in the way of obscure incantations to do things with arrays. args="-u$USER_ID -p$PAGES -s$STARS -a$ADD" [ -n "$TAGS" ] && args="$args -t$TAGS" python3 e621_recommendation_engine.py $argsEDIT: As usual, difficulty nailing down 'safe' quoting in bash. Probably the best approach here would be combining set -- with "$@" - in fact, that should be a pretty safe way to handle any kind of 'commandline argument list building' problem.
For anyone not dev savvy who 'just wants negative terms to work in extra tags', here's a minimal modification that seems to work as long as you are entering SOME extra tags:
python3 e621_recommendation_engine.py -u "$USER_ID" -p "$PAGES" -s "$STARS" -t"$TAGS" -a "$ADD"(yes, this is just removing the single space after -t, that's enough to guarantee no part of the value of TAGS will be parsed as an option.)
Wooooah thanks for the suggestions! Answering your question about how much less present than the average, I believe that's just the absolute difference in presence (average_presence - your_presence). I did it like this because using relative_presence (your_presence / average_presence) would give way too high of a value for tags that you simply don't have in your favorites, even if they're very niche. The way I did it also emphasizes tags with a higher average_presence, so it should be more interesting that you're not into it since it's more common. I ended up not finding these results very interesting either way, so that's why it's tucked away and unexplained in the detailed report :p
this shit is sick me and my bf just used it, although i had to go into the e621-Wrapped-main/.venv/scripts folder and install matplotlib, pillow, tqdm, requests, and i think numpy maybe manually into the venv with command prompt (by typing "pip3 install *the module names*") even though i already had those installed on my system, i think because it was using the included python executable in the virtual environment instead of the one on my system, in case anyone has the same issue of it saying those modules were not found, awesome work, the pngs it generates look really nice and professional! 10/10
Running interest_generator.py gives an error:
Traceback (most recent call last):
File "C:\Users\User\Apps\e621 Wrapped\interest_generator.py", line 1, in <module>
from e621_client import e621Client
File "C:\Users\User\Apps\e621 Wrapped\e621_client.py", line 4, in <module>
from PIL import Image
ModuleNotFoundError: No module named 'PIL'eightoflakes said:
Running interest_generator.py gives an error:Traceback (most recent call last): File "C:\Users\User\Apps\e621 Wrapped\interest_generator.py", line 1, in <module> from e621_client import e621Client File "C:\Users\User\Apps\e621 Wrapped\e621_client.py", line 4, in <module> from PIL import Image ModuleNotFoundError: No module named 'PIL'
You have to run it from the generated virtual environment. It should be .venv\Scripts\activate on Windows and source .venv/bin/activate on Linux
mormonara said:
Thanks for trying it out! I updated the repository based on your feedback
- Fixed requirements.txt
- Providing your API key is now optional. Public favorites should now be searchable without providing a credentials.json
- Simplified the instructions for running. The repo now comes with a .bat and .sh file that will run the entire setup and main program for you
About the full dump of the favorites... that would be really cool for generating a more detailed and less biased average user profile! Do you know who I should contact about requesting this? (and do you know what kind of file size we're talking about?)
Great tool; enjoyed it
Did you end up messaging about the favorites table dump? I have long thought about trying to get my hands on it and using it to screw around with implied preferences stuff, both just going posts -> posts and with some sort of bpr on all of the posts in the daily dumps they do, or something more weird that looks for similar artists/tags/public favorite sets or uses them as a latent space like yours...
Anyway, the point is that I have big plans, but I feel a little intimidated to just raw message the admins esp. if you recently did same.
jwaters said:
...I feel a little intimidated to just raw message the admins esp. if you recently did same.
The admins are scary, but most of them don't touch the code anyways. I can help on that front.
Short answer is that we don't & can't allow a raw dump of the favorites table, & letting people query it is similarly not feasible. Our current DB export tooling was made years ago by someone who's since stepped back from working on the site, so that'd be a pain, plus we'd have to filter by users who have their favorites hidden as well, which might not be feasible. As for querying the tables, that'd require some work, as that functionality doesn't exist, & it would really only be useful for data analysis anyways, so it's fairly low priority.
Anyways, in the future, don't be scared to ask me about any code questions; my road to site staff started with a third party client, & I'm always eager to help our community devs.
aacafah said:
The admins are scary, but most of them don't touch the code anyways. I can help on that front.Short answer is that we don't & can't allow a raw dump of the favorites table, & letting people query it is similarly not feasible.
Oh, I hadn't messaged anyone about it but it's a shame that it's not feasible for now. I'm sure there's lots of cool uses for it if it ever becomes available