New to posting and API stuff so sorry if this is an obvious question.

I am building a personal program that is essentially a local version of e621 via python, though it is only certain pictures that I have selected.

I have been using this API to successfully grab the tags from the pictures via their md5 hash up until tonight. But know, for some reason, I cannot seem to get past a 503 error and I am unsure if this is cloudflare throwing the error or e621.

A couple odd things are that I can repeat the same request in my browser (chromium) and receive the data that I want. As well as I have a timer in my code (500 ms) that ensures that I shouldn't get a 503

Again, this was working up until tonight, so I am unsure if there was a change to the site or something is not working.

code for reverence:

def readTags(fn):
    hashName = fn.split('.')[0] # get hash from downloaded file name
    url = urllib2.Request("https://e621.net/post/show.json?md5=" + hashName) # create url request
    url.add_header("User-Agent", "Mozilla/5.0 (Windows NT 6.0; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0")
    # I have login information here,but have removed this for obvious reasons
    tempFile = open("TagFile.txt", 'a')
    time.sleep(0.5) # Wait to avoid a timeout (503)
    try:
        data = json.loads(urllib2.urlopen(url).read())['tags'] # throws 503 anyway (-_-)
        # file stuff for building a bst later
        tempFile.write(fn + "@")
        for s in data.split(' '):
            tempFile.write(s+"#")
        tempFile.write("*\n")
    except Exception as inst:
        print inst
        tempFile.write(fn +"@noTag#*\n") # incase I grab a pic from another site
    tempFile.close()

EDIT: Python 2, Ubuntu on an intel chip, spectrum internet

From the Wiki's API page:

You should make a best effort not to make more than one request per second over a sustained period.

So really your timer should be at least 1000ms.

Secondly, have you checked that your rate limit code works? Maybe print the time, rate limit, print the time, and loop it a few times, just to make sure it's working as you expect it to.

Increased timer to 2 seconds, same error.

It also happens with the first of any request that I make, regardless of what it is. so it appears that it does not have anything to do with rate limiting.

import urllib2, json, time

millis = int(round(time.time() * 1000))

while True:
    millis2 = millis
    url = urllib2.Request("https://e621.net/post/index.json?limit=1") # create
    url.add_header("User-Agent", "Mozilla/5.0 (Windows NT 6.0; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0")
    time.sleep(2) # Wait to avoid a timeout (503)

    data = json.loads(urllib2.urlopen(url).read())['tags'] # throws 503

    print millis2 - millis

    millis = int(round(time.time() * 1000))

KamerErrik said:
Increased timer to 2 seconds, same error.

Your user agent should be set properly:

DO NOT impersonate a browser user agent, as this will get you blocked.

It should be something like:
MyProject/1.0 (by username on e621)

Edit:
I just tried my metadata downloader and got 503 as well, I forgot that when E6 gets DDoSed the protection increases and you can't use the api. It works through the browser because, I presume, you'll have had a cloudflare message saying they were checking your browser, so the only thing you can do with that is wait and try again in half an hour or so.

Pupslut said:
Your user agent should be set properly:
It should be something like:
MyProject/1.0 (by username on e621)

Edit:
I just tried my metadata downloader and got 503 as well, I forgot that when E6 gets DDoSed the protection increases and you can't use the api. It works through the browser because, I presume, you'll have had a cloudflare message saying they were checking your browser. So the only thing you can do with that is wait and try again in half an hour or so.

ahh, I'll change the tip about the user agent.
So something like "project/1.0 (by KamerErrik)" will work as the user agent? I wouldn't need to add anything else?

And I did get the cloudflare message.

KamerErrik said:
ahh, I'll change the tip about the user agent.
So something like "project/1.0 (by KamerErrik)" will work as the user agent? I wouldn't need to add anything else?

Yep, that'd work, though I'd keep the "on e621" part, just to be sure the admins know they can contact you here if it starts doing anything bad.
So it'd be "project/1.0 (by KamerErrik on e621)"

Maybe give it a better name, like:
post_metadata_downloader/1.0 (by KamerErrik on e621)

Just to give a better idea of what it's doing in the site logs.

KamerErrik said:
And I did get the cloudflare message.

Yeah, that'd probably explain that then.

Yeah, the DDoS protection is mucking up submission requests from Hydrus clients as well recently. Multiple reports about it in their support thread via discord.

However, I think Hydrus scrapes from the webpages, not via the API (which it should). So that might be the issue.

I have 2 bots using the API, one made one request every 15-20 minutes, another made 2-5 requests every 3 hours. The first bot had the correct user agent with my nickname, in the second, unfortunately, I forgot to set it up. Now I corrected it all, as well as increased the pause between requests and pause between requests for errors. As long as I still receive an error 503, I suppose this is IP blocking.

I've always been using cURL to access the API which was working fine, but since today CloudFlare is messing up those requests. Have been using them since sending direct AJAX requests gives 503 errors, not because of the request rate but rather because of Cross-Origin requesting (I'm running the requests from my own domain). I hope this will get sorted out soon, none of my API requests is working anymore...

Now, stopped getting error 503. It was a good reason to fix the code in accordance with the API rules :D

Ecstatis said:
I've always been using cURL to access the API which was working fine, but since today CloudFlare is messing up those requests. Have been using them since sending direct AJAX requests gives 503 errors, not because of the request rate but rather because of Cross-Origin requesting (I'm running the requests from my own domain). I hope this will get sorted out soon, none of my API requests is working anymore...

My script stopped working before I went to sleep (roughly 8 hours ago) too.

Since I am updating tags under Member limitations anyway, I make about 2 API calls per 20+ seconds (so nowhere near the API call limit), I figured the problem was somewhere else.

It also started working again, so I guess that was it.

Again, we have Cloudflare protection active. If a DDOS is detected and Cloudflare goes into paranoia mode all non-webbrowser traffic will be blocked until the attack subsides. AFAIK this is not something we have a large amount of control over, but we'd rather lose API access than all access to the page.

NotMeNotYou said:
Again, we have Cloudflare protection active. If a DDOS is detected and Cloudflare goes into paranoia mode all non-webbrowser traffic will be blocked until the attack subsides. AFAIK this is not something we have a large amount of control over, but we'd rather lose API access than all access to the page.

Ah, good to know. I guess we'll have to sit it out then. Thanks for the insight.