Latest version of the thing.

Try it out and tell me your thoughts.
(I won't give any usage instructions so I could, maybe, get feedback on how it seems to a person that uses it for a first time)

preview

(old) some examples:
https://streamable.com/27i4d
https://streamable.com/oa7ru
https://streamable.com/mebtw

After launching it, I assumed that the textbox was a quick tag search tool(i.e. solo male anthro, etc). Pressed Enter and it did nothing, I clicked Navigate and I ended up getting a "Navigation to the webpage was canceled" error. I couldn't refresh the page and had to relaunch.

Next, I tried www.e621.net and it worked. The site loaded and I was able to browse like I normally would on Chrome. Entered a post and clicked Download. The image loaded in, right-clicked and there was no "Save image as..." option. Tried to find a "Back" button to return to the previous page but there was none. Clicked "Navigate" and it was back to square one.

Saw FlashGet but I don't even know where to use it(index page would, I assume, just download the thumbnails).

I honesty don't see what I could do with this tool, all of the features I see now can be easily replicated on Chrome or Firefox w/ extensions. From what I'm assuming it's a mass e6 downloader/broswer?

TheGreatWolfgang said:
After launching it, I assumed that the textbox was a quick tag search tool(i.e. solo male anthro, etc). Pressed Enter and it did nothing, I clicked Navigate and I ended up getting a "Navigation to the webpage was canceled" error. I couldn't refresh the page and had to relaunch.

Next, I tried www.e621.net and it worked. The site loaded and I was able to browse like I normally would on Chrome. Entered a post and clicked Download. The image loaded in, right-clicked and there was no "Save image as..." option. Tried to find a "Back" button to return to the previous page but there was none. Clicked "Navigate" and it was back to square one.

Saw FlashGet but I don't even know where to use it(index page would, I assume, just download the thumbnails).

I honesty don't see what I could do with this tool, all of the features I see now can be easily replicated on Chrome or Firefox w/ extensions. From what I'm assuming it's a mass e6 downloader/broswer?

Fair fair, try visiting some of your favorite artists on some of their media outlets.

I added some tooltips, should make things slightly clearer https://ufile.io/sv6s2
(It's working!)

Both files are clean, as checked by VirusTotal (just doing my part as an IP-Sec nut, especially since I've almost had a close call on this website myself).

Right off the bat, it demands that I install "2007 Office Data Connectivity Component" first, prompting me with a yes/no option. Already not off to a great start.

After downloading, installing, and restarting e621 bot.exe, it greeted me with a dialog box, informing me that it was an art scraper for Artists on social media, including FurAffinity (eww... but that's not because of this tool... FurAffinity is just gross). I put in the link to WolfBlade's FurAffinity, and hit navigate. After that, I did some clicking around, and created a database for him.

And now I'm lost. How do I populate this database? I'm afraid that without any kind of documentation, this tool serves me no purpose, not to mention that it can potentially do more damage to my account than good. If it really does what it claims to do, how is it supposed to know that a higher-res version of a picture that it's uploading under my name doesn't already exist? Is there any kind of code that some of our more Windows and python-oriented users can review? I had to log into my FurAffinity account to view WolfBlade's account. Are my login details safe? Why isn't this application signed?

This application is raising a lot of red-flags for me, and I'm sure you can understand why.

Perhaps give a bit of detail on what it does, at least a description. I'm not downloading something just titled "e621 bot.exe".

Chaser said:
Perhaps give a bit of detail on what it does, at least a description. I'm not downloading something just titled "e621 bot.exe".

It's supposed to be an image scraper for artst's social media pictures, so you don't have to go through the pictures one-by-one and upload the pictures.

Edit: Here's the VirusTotal report for the latest file:
https://www.virustotal.com/#/file/fc8f10724a2705aff6495174075013ad6adb6c157054cde0e9cc9fc23490b50b/detection

Faux-Pa said:
including FurAffinity (eww... but that's not because of this tool... FurAffinity is just gross).

Same.
Also usually worst place to get artwork.

All you said is fair, even though it hurts a bit it's ME after all =/

I guess I could give a bit of instructions.
Since it's your first use, go to some artists twitter for example and scrape it, switch to 2nd tab on the left and you will have a list of his tweets.
Now you can do multiple things here, but I gues you would want to double click on the image(url), what happens next should be self explanatory.
As a last step I guess you could try uploading something to e6.

A short preview:
https://streamable.com/27i4d

As far as the downloading goes, you need dot net 4.5 and microsoft access components. It used to use regular .txt file for storing but so only .net was needed but I changed it, looked nicer this way and I was trying out new things.

Faux-Pa said:
1. "2007 Office Data Connectivity Component" first, prompting me with a yes/no option. Already not off to a great start.

2. And now I'm lost. How do I populate this database? I'm afraid that without any kind of documentation, this tool serves me no purpose, not to mention that it can potentially do more damage to my account than good. If it really does what it claims to do,

3. how is it supposed to know that a higher-res version of a picture that it's uploading under my name doesn't already exist?

4. I had to log into my FurAffinity account to view WolfBlade's account. Are my login details safe?

1. If you click yes it will just send a web address to whichever is your default browser.

2. Once you're on artists page, when you click scrape it makes a list. You can then save that list to database under name you chosen.

3. Well it doesn't really know, it's the same like when you try to upload manually, you'll get error if it already exists though.

4. App doesn't have a browser of its own, it uses windows one (internet explorer xD ), that's why its so small. Safe as in any other browser. But good question.

DelurC said:
... it uses windows one (internet explorer xD ), that's why its so small...

https://www.youtube.com/watch?v=3s5o2uWhNe8

TheGreatWolfgang said:
https://www.youtube.com/watch?v=3s5o2uWhNe8

IE has been part of it for a decade already if not more, I hope they replace it, because they haven't got around to fix the memory leak for nearly a decade as well.

I tried doing something recently, and once before that, and because of that memory leak everything breaks after like half an hour. Memory usage reaches the limit of 4gbs and poof.

But that's only under extreme circumstances, for normal use its fine.

DelurC said:
1. If you click yes it will just send a web address to whichever is your default browser.

2. Once you're on artists page, when you click scrape it makes a list. You can then save that list to database under name you chosen.

3. Well it doesn't really know, it's the same like when you try to upload manually, you'll get error if it already exists though.

4. App doesn't have a browser of its own, it uses windows one (internet explorer xD ), that's why its so small. Safe as in any other browser. But good question.

1. Okay, I get the database part. Luckily I was able to install it on an external drive to save my already-dwindling on-board storage.

2. I don't see any kind of scrape button.

3.So it just blindly uploads something if e621's MD5 checking doesn't stop it? The same image from one source can have a completely different MD5, and if I were to upload something that already exists, that would affect my upload limit.

4. That's weird, considering that I don't even have Internet Explorer installed. I specifically remember removing it after installing Windows 10.

Faux-Pa said:
1. Okay, I get the database part. Luckily I was able to install it on an external drive to save my already-dwindling on-board storage.

2. I don't see any kind of scrape button.

3.So it just blindly uploads something if e621's MD5 checking doesn't stop it? The same image from one source can have a completely different MD5, and if I were to upload something that already exists, that would affect my upload limit.

4. That's weird, considering that I don't even have Internet Explorer installed. I specifically remember removing it after installing Windows 10.

2. Which page are you currently viewing?

4. I belive that needed componets are still there. I don't think it uses edge. Does it look like edge?

DelurC said:
2. Which page are you currently viewing?

4. I belive that needed componets are still there. I don't think it uses edge. Does it look like edge?

I was viewing https://furaffinity.net/user/wolfblade, and it didn't really look like Edge, but then again, without the browser's original GUI, one wouldn't be able to tell.

Faux-Pa said:
I was viewing https://furaffinity.net/user/wolfblade, and it didn't really look like Edge, but then again, without the browser's original GUI, one wouldn't be able to tell.

You need to visit their gallery or some submission.

Moved video to op and added another one.
Waiting for more peeps to say what they think.

No matter what FurAffinity gallery I go to, the scrape button does not appear. It appears on Twitter pages, but not on FurAffinity.

Also, I know that it's a mass uploader, but I feel that there should be a tags box for each individual post, and a tags box that will be applied to all posts being uploaded.

Example:

Tags to go on all posts:
artist:example canine exmaple_tag

Post 1 tags:
underwear solo bulge

Post 2 tags:
some_tag another_tag

And so on.

And I know that I'm really nit-picking, but some kind of integration with iqdb.harry.lu would be amazing... especially considering the fact that I just had all 3 of the posts that I uploaded through this tool flagged for deletion because they already existed on the site under a different md5sum.

Faux-Pa said:
No matter what FurAffinity gallery I go to, the scrape button does not appear. It appears on Twitter pages, but not on FurAffinity.

It's because of that ad in the middle of the screen, it's broken, takes a long time to report its loaded.
Will add FA to F12 button too then.
Edit: Ok, added new version to op, just set focus to form (for example, on tabview on the left, click on the browser [globe] icon then press F12 on keyboard).

Faux-Pa said:
And I know that I'm really nit-picking, but some kind of integration with iqdb.harry.lu would be amazing... especially considering the fact that I just had all 3 of the posts that I uploaded through this tool flagged for deletion because they already existed on the site under a different md5sum.

If the same image exists, you will get error since e6 check for that already.
As far as for checking for version, iqdb only gives out %, it doesn't say if its worse or better.
So if you would want to check every image, then it would have to go to iqdb check if they exists, if they exist then go check each one to see if they are bigger (resolution or file size) but it still can't say if its better. Need AI for "is image better" part, that's way beyond me.
So I left it on manual. Uploading from websites that have better image quality over the ones that have worse, like pixiv > twitter.

And also, resolution is not absolute, image can be upscaled, while image size can be bloated, so yea =/

DelurC said:
It's because of that ad in the middle of the screen, it's broken, takes a long time to report its loaded.
Will add FA to F12 button too then.
Edit: Ok, added new version to op, just set focus to form (for example, on tabview on the left, click on the browser [globe] icon then press F12 on keyboard).

For me, it just goes to a random picture, and nothing populates in the Database tab.

And I get the IQDB thing. I had seen jacklul use it for one of his GitHub projects, and I thought it would be a cool thing to try.

Faux-Pa said:
For me, it just goes to a random picture, and nothing populates in the Database tab.

And I get the IQDB thing. I had seen jacklul use it for one of his GitHub projects, and I thought it would be a cool thing to try.

Yeah hold on, I'll hove to do more than just give the option to show the button.
Because when you try to scrape gallery, when it navigates to submission to grab time and text, it takes a long time to get notification it finished loading because of that broken add.

Seems I got it workings, even though it's not the solution I planned to, but hey, it works.
So broken ads shouldn't be a problem anymore, though it will be a bit slower than normally.
Try it.

I think I removed every single ad now, added newest version to op. Try it now =/

So, I've fixed some stuff, maybe changed it a bit.
I would like to hear some other complaints that you may have.

A preview of something I'm working on:
https://i.imgur.com/OiMpDwt.png
https://streamable.com/s33wi

That is... pretty sexy looking, not gonna lie. Wow.

Are there any features that would make your life uploading to e6 easier?

DelurC said:
Are there any features that would make your life uploading to e6 easier?

Keeping in mind that I might not be seeing what you're intending to show off... (I haven't used your tool before!)

Well, I tend to upload a lot of art from artists who do non-furry stuff... sooo...

It'd be nice to have a way to... like... leave notes on specific posts in the program. At it's most simple, something like "Not suitable for uploading" ... as well, maybe some way to place a marker so that if you have a multiple day uploading project, you know where you left off. Regardless, if there isn't a way, a way to leave something off of the list. :D

That said, the more I peer at the preview, the more I"m not positive what it does. this isn't a bad thing as my new thoughts are actually more exciting than the old ones, but this is REALLY cool.

I'm not sure what the is out of the scope of this program... but.. more tagging tools are always delicious.

SnowWolf said:
Keeping in mind that I might not be seeing what you're intending to show off... (I haven't used your tool before!)

Well, I tend to upload a lot of art from artists who do non-furry stuff... sooo...

It'd be nice to have a way to... like... leave notes on specific posts in the program. At it's most simple, something like "Not suitable for uploading" ... as well, maybe some way to place a marker so that if you have a multiple day uploading project, you know where you left off. Regardless, if there isn't a way, a way to leave something off of the list. :D

That said, the more I peer at the preview, the more I"m not positive what it does. this isn't a bad thing as my new thoughts are actually more exciting than the old ones, but this is REALLY cool.

I'm not sure what the is out of the scope of this program... but.. more tagging tools are always delicious.

Give me some artist of yours.

DelurC said:
Give me some artist of yours.

Uhh...

juanmao1997 - https://twitter.com/juanmao1997/

I think she/he has a few hundred posts on twitter, I've only uploaded ~100 of them. The remaining posts are a mix of posts that would appropriate to upload here, but I didn't bother with... and posts of humans or 'really-close-to-humans' or 'humans and house cats' and stuff, so... Should be a good example.

hmm.. I missed a few good posts recently. *uploads*

Oh here's another 'feature request'...

They also have a tumblr. They don't upload the same things in both places.

I'm not even sure if it's possible, but... tumblr support? :) any form of compare and contrast across sites? Maybe a way to display two different sources (twitter and FA, for example) in one 'stream' so you can manually compare and contrast different sources?

I dunno, you asked. I know these are probably improbable, but there's no harm in mentioning, yeah?

SnowWolf said:
Uhh...

juanmao1997 - https://twitter.com/juanmao1997/

I think she/he has a few hundred posts on twitter, I've only uploaded ~100 of them. The remaining posts are a mix of posts that would appropriate to upload here, but I didn't bother with... and posts of humans or 'really-close-to-humans' or 'humans and house cats' and stuff, so... Should be a good example.

hmm.. I missed a few good posts recently. *uploads*

Oh here's another 'feature request'...

They also have a tumblr. They don't upload the same things in both places.

I'm not even sure if it's possible, but... tumblr support? :) any form of compare and contrast across sites? Maybe a way to display two different sources (twitter and FA, for example) in one 'stream' so you can manually compare and contrast different sources?

I dunno, you asked. I know these are probably improbable, but there's no harm in mentioning, yeah?

For example : https://streamable.com/iqcs6
edit: forgot https://streamable.com/nh26r

there was support for two tumblr themes, but since they added that safe bull I removed it and ahven't worked on it.

DelurC said:
For example : https://streamable.com/iqcs6[/quote]

*happy wiggle, watches*
I'm sorry, I giggled at the new bug. I could just imaging the "oh.. oh no, not in front of the camera.." :) ...I'm sorry to cause you trouble, but know it helps too :) so.. sorry not sorry? :)

edit: forgot https://streamable.com/nh26r[/quote]

Honestly that is sexy as hell. I hate paging through a twitter to try and find art-- it's fine when I follow them on twitter, but it's frustrating when I'm trying to upload everything the light touches when 90% of their posts are "Waiting in the DMV... yawn.." or reposting other people's art...

there was support for two tumblr themes, but since they added that safe bull I removed it and ahven't worked on it.

yeah, I can understand that. Tumblr can be a butt. Shame because that seems like, to me, the best place to get an artist's just-art, at often the best resolution/quality.

Okay.. so let's see... other suggestions now that I understand it better. (thank you by the way) :D

1. maybe a way to say "an image from https://twitter.com/juanmao1997/ will always be tagged juanmao1997" -- I do see the 'input tags for pictures, all images will share these tags' area, but might be an extra neat feature-- or redundant! I"m not sure. :) But some way to help with multiple usernames would be nice-- juanmao1997 on twitter is juanmaodepp on tumblr, is juanmao-depp on lofter... ...etc. :)

2. Tagging tools. Like... okay, shooting for the stars here, but, I'd love to see, like, some frequently used tags that I could just click on/or checkbox to add. Gender, anthro/feral/humanoid etc, solo/duo/etc... Maybe some drop down/collapsible style menus for things like species, positions, positions, fur colors, etc. That said: I know that's asking a lot. But something like that on a smaller scale would be great--when I upload on the website, I have my list of quick tags to help me out. So, something to replicate that experience would be pretty awesome. Even if it's jsut a list of user-inputted quick tags I can select from :)

(That said, if this sounds feasible and you want some default tags built in, I can totally help put together a list of tags--tags are a lot of work. :))

The bug was e6 not giving some information for deleted posts.

Considering only direct links to images are saved, to save artists name it would have to save website urls and link it with artist name and images.
Basically, it makes it exponentially complicated for almost no benefit (you can just type artist name in shared tags box) and it also consumes more memory.
Edit: There is a kind of hack way to do it, basically just remember websites when you scrape and database table when you load, but considering you can edit them as you please you can change their index and name so that won't work either. I mean it works if you don't change it but it's not "idiot proof" and can break, so it's a no go.

As far as the tags go, clicking a checkbox takes as much time as typing the tag, searching for to click multiple checkboxes takes more and creating a few checkboxes for tags would require a custom form. I don't think checkboxes are intuitive.
I would rather have an AI tag bot, but I don't see that happening this or next decade.

DelurC said:
Considering only direct links to images are saved, to save artists name it would have to save website urls and link it with artist name and images.
Basically it makes it exponentially complicated for almost no benefit (you can just type artist name in shared tags box) and it also consumes more memory.
Edit: There is a kind of hack way to do it, basically just remember websites when you scrape and database table when you load, but considering you can edit them as you please, you can change their index and name so that won't work either, I mean it works if you don't change it, but it's not "idiot proof" and can break, so it's a no go.

I'm no programmer so I didn't know :) But I did think of something like the hack ish way to do it, but if it's not doable then It's not doable. No big deal :) Just something I thought about :)

As far as the tags go,clicking a checkbox takes as much time as typing the tag, searching for to click multiple checkboxes takes more, and creating a few checkboxes for tags would require a custom form. I don't think checkboxes are intuitive.

Mmm... Respectfully, I disagree.

It's not just about the time to type a tag (which I admit isn't much for something like female, but Ambiguous_gender or ambiguous_species takes some finger work), but it's also about having a visual reminder that you should tag these things. I sometimes forget tags like 'solo' if I don't remind myself.

So, in my ideal world, I'd have something like:

Or even, again, a list of self-defined tags

... That said... everyone types at a different speed and everyone reads at a different speed. For me, I read quickly and tend to have strong muscle memory, so poking a bunch of checkmarks or quick tags or whatever is WAY faster than typing them all out. Speakign for myself and my own tagging experience. your mileage may varry though :) but for me, this would basically be a very close to a requirement to use this. I've got my quick tags on the website that help me out a whole lot... I used to have a grease monkey script that would... let me do this and while it's totally broken, I miss it a lot :) Clicking is way faster for me. :) without it, I'd probably still use it, but I'd end up editing the post on the website after to get all the tags I missed.

But. This is YOUR project. I'm just saying what I think :) You make it how you want to make it, and if it's not feasible or too much hassle, then it is what it is <3

That said, IO gotta blast off and make dinner--I'll keep thinking to see if there are any other featurerequests/whatever I can think of that might make this thing better for ya. :)

I would rather have an AI tag bot, but I don't see that happening this or next decade.

Eh, it might be sooner than one imagines.. but I don't see google helping out our little porn board ;) ... be neat to see what one of them nueral networks did over here though.

Here you go dude
https://streamable.com/xoxjk

DelurC said:
Here you go dude
https://streamable.com/xoxjk

ooooh yeeeeeeeeah! That is sexy and delicious. I am delighted by it's very existence. I want to make cookies and send them to you. I haven't made cookies in 3 years, but I'd do it for you.

That looks fabulous :D I love it!

Anything else?

DelurC said:
Anything else?

honestly, not that I can think of at the moment, but It's been another busy scatterbrained weekend. Bleh.

But, this still looks super exciting. :) can't wait to try it out :D

....and I'll inevitably find some feature requests then at that time, but for now... :)

Does anyone else that is viewing this thread have any input?

Like, what would make your life interacting with e6 easier.

DelurC said:
Like, what would make your life interacting with e6 easier.

Interacting inside (a browser) or only outside (a program)?

From within the browser:

[redacted, saved]

Outside the browser:

Improve this tool or make a better one? forum #172541
Source code linked in final post.

Not everything I mentioned in that thread was fixed/implemented, but what I need works, which is comparing the files in a local folder against e621 and fav'ing the missing files.

Other than that, you are competing against Hydrus.

abadbird said:
Outside the browser:

Improve this tool or make a better one? forum #172541

Mine is kind of different, you scrape artist pages then upload to e6 without downloading anything.
You basically have a list of things that artist made and work with that.

abadbird said:
Not everything I mentioned in that thread was fixed/implemented, but what I need works, which is comparing the files in a local folder against e621 and fav'ing the missing files.

Well, there is a compare thingy, you can see it at 1:05 @ https://streamable.com/iqcs6

It's taking a while, I haven't had that much free time and I'm also rewritting the scrapers, so it should also be faster and more efficent when done.

looking forward to things~

So far so good : https://streamable.com/2xmcw

DelurC said:
So far so good : https://streamable.com/2xmcw

Delicious! and Sexy!

What websites are/are planned to be supported, out of curiosity?

I'm so hyped for this tool :D

This Plurk is so odd, like you scroll from left to right, though I could get all pots spent the afternoon on fixing JSON format just to find out it's useless in the end.

SnowWolf said:
What websites are/are planned to be supported, out of curiosity?

Twitter,FurAfinitty,Pixiv,Inkbunny,HicceArs and Tumblr already are.

DelurC said:
This Plurk is so odd, like you scroll from left to right, though I could get all pots spent the afternoon on fixing JSON format just to find out it's useless in the end.

Yikes!

Twitter,FurAfinitty,Pixiv,Inkbunny,HicceArs and Tumblr already are.

Sexy! What about Deviant art? :)

SnowWolf said:
Yikes!
Sexy! What about Deviant art? :)

I thought porn was not allowed on DA.

DelurC said:
I thought porn was not allowed on DA.

I think 95% of what I upload is not porn. :) (*checks... 1753 uploads... 125 rating:e so.. 7.1% is rating:e ... and at least a dozen of those are from realistically drawn animals with sheaths. so... ;)

Also, I believe their rules say that sexy pinup type stuff is okay, and nudity and stuff as long as there's not signs of aroused genitalia, or whatever.

I mean, this person's pretty popular: Link (warning, 3D porn focusing on humans and humanoid aliens) (I really should do her gallery one day...)

Ok, but I don't really come to e6 to look at safe stuff.

DelurC said:
Ok, but I don't really come to e6 to look at safe stuff.

Well, I mean...

I guess it depends on what your goal with your program is? If it's just to provide more porn, then you should probably skip Deviantart, haha.

If it's to provide a tool that can be used for porn and not-porn, then deviant art's a pretty good idea. A lot of artists use it, it provides easy links to full sized images and it's my prefered go to to get the best quality of an image.

But it all comes down to your own goals. it's your program, after all :)

rating:e source:deviantart.com gives me 49 pages.

The more sexual results seem to have crappy or plain wrong sourcing though, like a sex post sourced to its artist's DA home page and their other galleries. The legitimate DA sources look like female presenting posts, which basically supports what SnowWolf said. Further, the erection posts that I checked all had similarly poor DA sourcing. Gotta love gender inequality ;).

post #1499783 on DA

Anyway, all that's beside the point IMO. DA hosts full resolution art or they're at least better than FA's initial downscale, which alone makes DA deserving of attention. Perhaps more importantly, DA is regarded as a primary gallery site by many (terrific, clean, nonhuman) artists, analogous to Pixiv, and many of those artists only post to DA, although I believe Tumblr and Twitter have cut into that greatly.

abadbird said:
rating:e source:deviantart.com gives me 49 pages.

The more sexual results seem to have crappy or plain wrong sourcing though, like a sex post sourced to its artist's DA home page and their other galleries. The legitimate DA sources look like female presenting posts, which basically supports what SnowWolf said. Further, the erection posts that I checked all had similarly poor DA sourcing. Gotta love gender inequality ;).

post #1499783 on DA

Anyway, all that's beside the point IMO. DA hosts full resolution art or they're at least better than FA's initial downscale, which alone makes DA deserving of attention. Perhaps more importantly, DA is regarded as a primary gallery site by many (terrific, clean, nonhuman) artists, analogous to Pixiv, and many of those artists only post to DA, although I believe Tumblr and Twitter have cut into that greatly.

feature ideas to help uploaders and site maintainers

Does your tool help with bvas detection? Portable IQDB / harry.lu / SauceNAO? From IQBD's readme : "the image will be resized to 128x128 and added to the database." I wonder how big harry.lu's database is, if it can be downloaded, and if its updates can be sync'd.

Perhaps your tool can help automate or at least streamline the reupload process. If so, it makes sense to support the other major hosts of full resolution/better-than-FA quality art, like DA, SoFurry, (non-upscaled, publicly viewable) Patreon, Furry Network. Also, grab the tags from our lower resolution version and fill out the Upload page. Can your program upload posts + tags directly?

Speaking of bvas detection and Patreon, here's a wonderfully unpopular idea: paid content (and upscale) detection. Can e621's version of a post be checked against high percent similarity art from official sources? If e621's version has a higher resolution than official sources, then mark it as a potential upscale/non-public version.

Also, can you grab artist gallery links from their e621 artist page?

scraping for galleries and alt versions

Running with that thought...

Second level is checking all of our posts for a given artist for galleries not listed on their e621 artist page. Third level is doing the same for proper gallery sites that let artists list "Other Sites" or "My Sites". Fourth level is checking other major boorus for their known galleries of a given artist. Fifth level is checking artist home pages on their various galleries for other gallery links because not everyone uses the "Other Sites" feature, but a lot of galleries found that way are artists' friends and groups. Sixth level is checking art descriptions on official galleries for links to art sites for potential full resolution, alternative, and uncensored versions. I can't think of anything else.

Finding these alternative galleries is important for discovering higher quality sources and unknown art.

In addition to checking e621 for a specific piece of art, any matches or likely matches can be checked for unknown_artist, [none] artist, or unexpected artist tags (i.e., split, unaliased galleries). So, for example, check an artist's Tumblr (whatever your program does) and compare against all posts on e621. I'll just say that Sankaku Channel seems to better source and ID eastern artists than us.

What say you to that?<sup>1
1. Jack Sparrow quote. Can't make noise to find and link appropriate YouTube video.</sup>

I'm not making google crawlers lol.
As for the part about checking post I already posted a video, but here is new one with tumblr test (interesting that tumblr is slower than others).
https://streamable.com/8hrp9

DelurC said:
As for the part about checking post I already posted a video, but here is new one with tumblr test (interesting that tumblr is slower than others).
https://streamable.com/8hrp9

Looks pretty sexy! :D

https://streamable.com/57nks

At my current skillz, I can't get it any faster.

DelurC said:
https://streamable.com/57nks

At my current skillz, I can't get it any faster.

Still faster then doing it by hand :D

Since I haven't done much satisfactory work in the last 3 days, decided I would add DA support today, turns out it's not straight forward as I though.

SOMEHOW, when you navigate from gallery to post, it does not contain post html even I'm looking at it, it works fine when you go directly to it like hyperlink, so idk wtf is going on there + they block the http requests (at least I haven't managed to get an OK) and put random strings in direct image links and you have to navigate to them over some token crap, you can just replace part like on tumblr for example.

https://streamable.com/dgmyz plz kill me

DelurC said:
Since I haven't done much satisfactory work in the last 3 days, decided I would add DA support today, turns out it's not straight forward as I though.

SOMEHOW, when you navigate from gallery to post, it does not contain post html even I'm looking at it, it works fine when you go directly to it like hyperlink, so idk wtf is going on there + they block the http requests (at least I haven't managed to get an OK) and put random strings in direct image links and you have to navigate to them over some token crap, you can just replace part like on tumblr for example.

https://streamable.com/dgmyz plz kill me

Oh dear... I'm sorry ^^;

Geeze, that's.... crazy. Uh, let me see...

Uh.... *head scratch*

... It works for me...

I mean, I"m trying it in several different ways, and every time, I can see it.

I admit, I might not understand what you're saying (I'm better then the average bear, technically speaking, but not the sharpest alligator in the swamp)... from what I can gather though... I can see it, I'm not sure why you can't, except that the tool's involved.

I know that the 'download' link off on the right will stop working after a few minutes of sitting on a page, but that really shouldn't affect the description of the image o_o

I"m really sorry for suggesting something that's causing you extra headache :(

Uh.. have you tried turning it off and back on again? is it properly plugged in? :)

(also, the mouse wiggle of frustration never fails to make me giggle)

SnowWolf said:
Oh dear... I'm sorry ^^;

Geeze, that's.... crazy. Uh, let me see...

Uh.... *head scratch*

... It works for me...

I mean, I"m trying it in several different ways, and every time, I can see it.

I admit, I might not understand what you're saying (I'm better then the average bear, technically speaking, but not the sharpest alligator in the swamp)... from what I can gather though... I can see it, I'm not sure why you can't, except that the tool's involved.

I know that the 'download' link off on the right will stop working after a few minutes of sitting on a page, but that really shouldn't affect the description of the image o_o

I"m really sorry for suggesting something that's causing you extra headache :(

Uh.. have you tried turning it off and back on again? is it properly plugged in? :)

(also, the mouse wiggle of frustration never fails to make me giggle)

" SOMEHOW, when you navigate from gallery to post, it does not contain post html " that part.
(and I know, it works fine on FF)

DelurC said:
" SOMEHOW, when you navigate from gallery to post, it does not contain post html " that part.
(and I know, it works fine on FF)

(chrome too~)

Man that is... bizarre, to say the least. o.O

None of your streamable.com links actually play anything, for me, so there may be some information I'm missing, but is it possible that DA has identified your program as abusive? Modifying it to send a completely different User-Agent header would be one way to test that.

Ripme has a deviantart ripper that, last time I checked, worked fine.
One thing that is immediately notable from the source is that it waits 3000 ms between loading pages, and 2000 ms between downloading images. Presumably loading things faster than that triggers anti-site-abuse measures.

savageorange said:
None of your streamable.com links actually play anything, for me, so there may be some information I'm missing, but is it possible that DA has identified your program as abusive? Modifying it to send a completely different User-Agent header would be one way to test that.

Ripme has a deviantart ripper that, last time I checked, worked fine.
One thing that is immediately notable from the source is that it waits 3000 ms between loading pages, and 2000 ms between downloading images. Presumably loading things faster than that triggers anti-site-abuse measures.

Disable script blocker and click.
No anti abuse can be triggered by navigating to pages.
Site wouldn't work for 99% user then.

DelurC said:
Disable script blocker and click.

Yeah, I did that. No effect. I can click, the thing acts like it's playing (bar updates), but the display otherwise remains the same.

EDIT: trying again, even the bar doesn't update now. I can adjust the nonexistent volume, that's basically it.

No anti abuse can be triggered by navigating to pages.
Site wouldn't work for 99% user then.

Am I to assume that you are using some kind of browser automation toolkit, rather than making direct requests to the webserver, then?

savageorange said:
Yeah, I did that. No effect. I can click, the thing acts like it's playing (bar updates), but the display otherwise remains the same.

EDIT: trying again, even the bar doesn't update now. I can adjust the nonexistent volume, that's basically it.

Am I to assume that you are using some kind of browser automation toolkit, rather than making direct requests to the webserver, then?

No, it's simply a browser with which you normally navigate.

Ok, reading back over the thread, I see you are using IE (in some .. sorta-embedded way?). So I guess when you use IE without the involvement of your program, it works?

If not, it seems reasonably likely that the DA server is tailoring its output -- badly -- to try to cope with some problematic behaviour of IE. If you could persuade IE to falsify its User-Agent header (to say, pretend to be Firefox; or to pretend to be a different version of IE), and the result was different than without the falsification, that would be a reasonable indicator.

Tried to log in into e6 with http request, failed.
Now I have a headache, that's enough of nothing for today.

I don't know if this is the right thread but, I feel a bit pissed and discouraged here, because my upload limit is diminishing due to some posts that I have uploaded are being deleted due to being a duplicate/inferior version. (read: Some are uploading a RAW format on Tumblr and Twitter, deleting the one that I had posted prior.)

AlexYorim said:
I don't know if this is the right thread but, I feel a bit pissed and discouraged here, because my upload limit is diminishing due to some posts that I have uploaded are being deleted due to being a duplicate/inferior version. (read: Some are uploading a RAW format on Tumblr and Twitter, deleting the one that I had posted prior.)

This is basically completely the wrong place for this, haha.

Your upload limit is fine though--12 deletions is not so many :) Don't worry... just keep uploading and doing the best you can.

We want the best versions available posted here, so try to upload the biggest you can. (excluding, of course, pay content <3)

You can read how to get the best copies here ... or if you use a thumbnail viewer like imagus, it can be even easier.

Just learn and keep uploading. <3

Thanks. ☺️

AlexYorim said:
Thanks. ☺️

No problem :) I know it's not much, but seriously, don't let it get you down. It happens to all of us who upload sometimes--Heck, I uploaded about a dozen pictures the other day, then realized that the artist posted WAY higher quality pictures on a different gallery. Had to reupload and get them all deleted, haha. It happens-- As long as you don't keep making the same mistake over and over again, your upload will keep going up <3

We hit the magic breakpoint girls.
Never give up, never surrender!

It's a login for something we don't need, but it's (just) a start.