Good news everyone!

I've backed up all of the popular posts on e621.net between 2007 and 2023!

Now I'm sharing them with you!

post #2294957

The archives contain all of the files from https://e621.net/popular between 2007-02-10 and 2023-12-31 at the time the archives were created. Popular items change frequently as older items accumulate upvotes. The archives may not represent complete 1:1 accuracy with what is currently shown on e621.net.

The individual archives are broken out by year, into months and days with the following folder structure:

e621_popular_<year>_raw/<month>/<day>/<posts>

The files are raw pulled directly from http://static1.e621proxy.ru/data/, no modifications or tags have been added to the files.

Note: The 2007 archive starts on Feb 10th when the first post (that wasn't deleted) was made and has days missing when nothing was posted.

You can download the torrents using a Bittorrent application like Transmission, qBittorrent, Deluge or rTorrent.

Character Archives

I've also created a perl script to embed e621.net tags into image EXIF metadata so you can search images directly in windows file explorer search:

https://e621.net/forum_topics/37801

Interesting.

Any chance of posting just the metadata of the posts? Meaning either their ids or the list of files? Or are the popular posts just the top scoring 90 posts of each day (and thus we can calculate it from the daily stats dump)?

mrox said:
Interesting.

Any chance of posting just the metadata of the posts? Meaning either their ids or the list of files? Or are the popular posts just the top scoring 90 posts of each day (and thus we can calculate it from the daily stats dump)?

Each section now contains a file list metadata link for the archive. :)
Metadata not needed anymore now that the popular archives are raw directories.

kora_viridian said:
For critters in the USA: If you're on Scumcast Comcast or Xfinity (same company), you most likely have a data cap of somewhere between 1,024 GB and 1,200 GB a month; the exact amount depends on where you live. This includes everything you do online from your home Internet connection - web browsing, email, Youtube, Netflix, Hulu, Steam, Twitch, torrents, whatever.

If you did nothing else on your home Internet all month, and downloaded all of the 2017 through 2022 torrents linked here, you'd end up going over the cap.

I think you can hit the cap once, without charge, and after that it costs $10 per 50 GB. If you've ever hit the cap before, that means it would cost somewhere around $110 to $150 to download all of these torrents within one month.

Other Internet providers, including AT&T, Cox, and Mediacom, also have data caps on at least some of their services. Research your provider to make sure. If there is a data cap, there's usually a way to see how much data you've used on the provider's website somewhere - like where you log in to pay your bill.

You can usually get unlimited addons for $25-$50 extra depending on your ISP. But yeah, definitely watch your limit if you have one.

I suggest getting a seedbox if you want to get more into torrenting. I use whatbox.ca and really like it. I have a 100Gbps network and 100TB upload limit lol.

Dang, each torrent is a single zip file. I have a full rip of e621 and was going to help you seed without having to download anything, but chances are I won't be able to recreate identical zip files. Thanks for the file lists.

mrox said:
Dang, each torrent is a single zip file. I have a full rip of e621 and was going to help you seed without having to download anything, but chances are I won't be able to recreate identical zip files. Thanks for the file lists.

This might be my next project. Do you structure your rip identical to static1.e621.net? Where abcdef.jpg is at ./data/ab/cd/abcdef.jpg? Would probably take me a couple weeks to do a full rip up to beginning of 2023.

I structured it by "year/week/<postId>-<filename>" and a mirror folder structure for the json (not really needed now that we have metadata dumps). If I were to redo it I'd do it by day instead of week since there's now 15k files per week and a lot of file managers don't handle that well. If you structure it with hash based folders it's a lot harder to view new content. Using a time based structure you can start at one end and view everything sequentially. My script hits e621 daily which provides enough lag to not worry about illegal content. I download at a rate of 1 post per 6 seconds.

Originally I was going to make a tagging bot but that project has moved to the bottom of my list of things to do. It's amazing how much free time it feels like you have when you're in school and all that disappears when you're a working adult. The hardest part of doing computer vision projects is getting good source material and the tags here are pretty good for that.

mrox said:
I structured it by "year/week/<postId>-<filename>" and a mirror folder structure for the json (not really needed now that we have metadata dumps). If I were to redo it I'd do it by day instead of week since there's now 15k files per week and a lot of file managers don't handle that well. If you structure it with hash based folders it's a lot harder to view new content. Using a time based structure you can start at one end and view everything sequentially. My script hits e621 daily which provides enough lag to not worry about illegal content. I download at a rate of 1 post per 6 seconds.

Originally I was going to make a tagging bot but that project has moved to the bottom of my list of things to do. It's amazing how much free time it feels like you have when you're in school and all that disappears when you're a working adult. The hardest part of doing computer vision projects is getting good source material and the tags here are pretty good for that.

Wouldn't you normally structure it like this? Date and time/post_id, hash, filesize? Say, 202303230340_0123456789abcdef0123456789abcdef_1000000? You can obviously use an index to convert post ID to date and time, but same for the hashes and sizes. PostID_Date&Time_MD5Hash_FilesizeInBytes would naturally sort large numbers of posts quite nicely, and be manageable. Added bonus is that if organized this way, it's a lot easier to mirror torrents in the future )! One feature I miss of E-hentai here is the ability to show 'strong' tags. I think you used to be able to see actual tag votes that you could sort by. E621 is binary. It either is tagged, or not.

Thing is, I think I know of people that might already have all the older content. I left a link to this topic for someone on IRC who may know. Data hoarders might already have these torrents. What would be clever is to just count up the number of posts in a torrent, and group them by months or weeks for really old posts, then gradually speed it up and go to daily. Heaven help us when it has to go to hourly archives! (LOL)

:EDIT: OK, got a reply and they said they only have E-H archives. Oh wells. I wonder if I can pay for a HDD shipped, because of this horrible 5Mb/s bandwidth, LOL.

mrox said:
Originally I was going to make a tagging bot but that project has moved to the bottom of my list of things to do. It's amazing how much free time it feels like you have when you're in school and all that disappears when you're a working adult. The hardest part of doing computer vision projects is getting good source material and the tags here are pretty good for that.

I just wrote a small bash script to tag images for anyone who's interested: https://e621.net/forum_topics/37801

post #2771992
lol

One thing that changes those calculations: The size of the files are not the only consideration. The amount greatly matters, as well.

Assuming the person is using Windows, and they most likely would be in that situation, then you're going to have a bad time with throughput. Windows chokes on very large amounts of files, regardless of their size. I've seen it dragged to its knees by hundreds of thousands of files at 1KB or LESS each. So you're either going to have to copy them in relatively small batches or wait for a week while it chokes and slows to a crawl as it gets them transferred.

Would the now-deleted infamous cheese grater image still be included in the 2008 archives?

alexyorim said:
Would the now-deleted infamous cheese grater image still be included in the 2008 archives?

These archives were created within the past month, so if it was deleted before then it won't be in the archive.

alphamule said:
Wouldn't you normally structure it like this? Date and time/post_id, hash, filesize? Say, 202303230340_0123456789abcdef0123456789abcdef_1000000?

e621 uses the hash as the filenames so when I mentioned file name that means hash. I see no reason to include file size in the name since the actual file easily tells you that. The thing about your folder structure is you need to ensure those folders don't get excessively large. 10k is a good limit. Folders that large tend to stall the system when you open them up and it tries to generate thumbnails for everything. If you only use a command line it's fine, but if you use a GUI you really want to stay under 10k entries in a folder.

Windows is slow when dealing with tons of little files. However there are alternative file transfer tools you can install on Windows to replace its default copy/move functions. If you're routinely dealing with lots of tiny files I'd recommend looking into using one.

I don't remember the company, but someone is selling a square, silicone cheese grater. You can finally experience that image for yourself without the need for a trip to the emergency room.

mrox said:
e621 uses the hash as the filenames so when I mentioned file name that means hash. I see no reason to include file size in the name since the actual file easily tells you that. The thing about your folder structure is you need to ensure those folders don't get excessively large. 10k is a good limit. Folders that large tend to stall the system when you open them up and it tries to generate thumbnails for everything. If you only use a command line it's fine, but if you use a GUI you really want to stay under 10k entries in a folder.

Windows is slow when dealing with tons of little files. However there are alternative file transfer tools you can install on Windows to replace its default copy/move functions. If you're routinely dealing with lots of tiny files I'd recommend looking into using one.

I don't remember the company, but someone is selling a square, silicone cheese grater. You can finally experience that image for yourself without the need for a trip to the emergency room.

Right... But the filesize was for the indexing. I guess it's redundant for the actual files but hypothetically, MD5s could match but have different filesizes. <--- Highly unlikely, so not worth bothering? ;) If you've seen programs like H@H/Hath(Hentai@Home), they group files into ranges like /01/0102030405060708090a0b0c0d0e0f and limit allocations of ranges (i.e. You only get 12 or something) based on size and speed of node. It might make sense to use a different division than /256 or to group by year?

Windows file operations are totally broken and everyone knows this? Haha, slow software is slow. There's things like filename sorting that can be almost instant, or incredibly slow when you scale to 10000<n<100000 number of files. 100000 1-KB files are going to be slow to copy even on a SSD (but not as bad as HDD!), so I keep large numbers of such files archived. However, this is not too practical when someone does something crazy like makes a 2TB RAR file. XD

That's a lot of furry porn.

azero said:
That's a lot of furry porn.

https://youtu.be/WZLg1ARnaxE

alphamule said:
MD5s could match but have different filesizes. <--- Highly unlikely, so not worth bothering? ;)

The basics of hashing on a large site like this is to spread the content out as evenly as possible so no part of the system (assuming a multi-server system with load balancing) is handling more load than any other. So you're correct in binning files by their hash's first few characters is good since that evenly spreads them out. If you don't know how many files you're going to get in any given category, that's the best way to do things. I didn't need to do that since I only have one 'server'. The only thing I need to worry about is having too many files per folder, so I did need something more advanced than everything in the same folder. Splitting by Year and then by Day works well enough. Originally I used to view every image on this site but that's no longer feasible. When I was doing that, browsing the weekly folders in thumbnail view was more efficient than visiting the site. Binning things by week/day made it easy to skim over the content only once. If I binned by hashes the new files would be scattered all over the place and I'd be constantly reviewing files I'd already seen. That's good for web server, but not for individual users. I didn't include the size because I don't believe e621 has any hash collisions due to how it formats its content urls. Though not why I did it and I hadn't considered the issue until now, including the post id in the file name resolves any collisions on my side of things.

Hash collisions are highly unlikely, but a site with a type of porn a bunch of people hate is more likely to be targeted by such files. I hope something checks for that. I assume the site just says the image is already here and blocks the upload then expects the user to post about it if the collision is accidental. I've never uploaded content so I really can't say.

Added 2023 popular torrent to OP.

Updated all torrent's from zip archives to raw directories. Due to the number of files in later archives your torrent client may lag for a minute if you look at the torrent contents. The benefit is you no longer have to extract the files and use twice the amount of disk space. I also audited all past archives and found missing days in Feb 2022 and Nov 2022, these have been fixed.

👀

uwubanana said:
Updated all torrent's from zip archives to raw directories. Due to the number of files in later archives your torrent client may lag for a minute if you look at the torrent contents. The benefit is you no longer have to extract the files and use twice the amount of disk space. I also audited all past archives and found missing days in Feb 2022 and Nov 2022, these have been fixed.

A workaround for this would have been segmenting by smaller ranges of post IDs?

uwubanana said:
👀

Heh

I ran a script verifying each file's md5 against it's md5 from the file name and I found 5 corrupted files in the torrents:

file: e621_popular_2022_raw\01\03\fce888d059408a0dc1ecf5d7e4f3c116.png; post id: 3103439. MD5 from torrent is B605E14E24A051137378934CDFA94EE8
file: e621_popular_2022_raw\11\03\cbdfdd6f8524ec441a28edf34b3b039e.jpg; post id: 3667865. MD5 from torrent is 9AA7A0030F2A52F05919852C2921B3F3
file: e621_popular_2022_raw\12\07\d51fdff22270f5d53c9b86f1e4f6f1fe.gif; post id: 3736870; MD5 from torrent is 750F242EE8C8716359334990C9AD053D
file: e621_popular_2022_raw\12\25\4e853e80d720b13d6d5834c67853cc58.webm; post id: 3770756; MD5 from torrent is 119D22593623EB0FBA5055588782F8AA
file: e621_popular_2023_raw\01\02\331791a77feb69f846350bf33ac216f6.webm; post id: 3787554; MD5 from torrent is 43E344551595DE957D1350C62ADEDE13

All of these files except from the second one are actually corrupted, they won't open or playback is broken. I made sure that it's not just me by redownloading these files from the torrent. The same files on e621 are fine, the have the right hash and load up with no issues, would be nice to have them fixed in the torrent

note: I checked all yearly torrents but not character ones

itsobviousenough said:
I ran a script verifying each file's md5 against it's md5 from the file name and I found 5 corrupted files in the torrents:

file: e621_popular_2022_raw\01\03\fce888d059408a0dc1ecf5d7e4f3c116.png; post id: 3103439. MD5 from torrent is B605E14E24A051137378934CDFA94EE8
file: e621_popular_2022_raw\11\03\cbdfdd6f8524ec441a28edf34b3b039e.jpg; post id: 3667865. MD5 from torrent is 9AA7A0030F2A52F05919852C2921B3F3
file: e621_popular_2022_raw\12\07\d51fdff22270f5d53c9b86f1e4f6f1fe.gif; post id: 3736870; MD5 from torrent is 750F242EE8C8716359334990C9AD053D
file: e621_popular_2022_raw\12\25\4e853e80d720b13d6d5834c67853cc58.webm; post id: 3770756; MD5 from torrent is 119D22593623EB0FBA5055588782F8AA
file: e621_popular_2023_raw\01\02\331791a77feb69f846350bf33ac216f6.webm; post id: 3787554; MD5 from torrent is 43E344551595DE957D1350C62ADEDE13

All of these files except from the second one are actually corrupted, they won't open or playback is broken. I made sure that it's not just me by redownloading these files from the torrent. The same files on e621 are fine, the have the right hash and load up with no issues, would be nice to have them fixed in the torrent

note: I checked all yearly torrents but not character ones

Do you have a copy of the actual torrent files and not the magnet links? I couldn't get any past around 2012 to even show the filenames.

it just says unable to find anytime i try to use the magnet links

itsobviousenough said:
I ran a script verifying each file's md5 against it's md5 from the file name and I found 5 corrupted files in the torrents:

file: e621_popular_2022_raw\01\03\fce888d059408a0dc1ecf5d7e4f3c116.png; post id: 3103439. MD5 from torrent is B605E14E24A051137378934CDFA94EE8
file: e621_popular_2022_raw\11\03\cbdfdd6f8524ec441a28edf34b3b039e.jpg; post id: 3667865. MD5 from torrent is 9AA7A0030F2A52F05919852C2921B3F3
file: e621_popular_2022_raw\12\07\d51fdff22270f5d53c9b86f1e4f6f1fe.gif; post id: 3736870; MD5 from torrent is 750F242EE8C8716359334990C9AD053D
file: e621_popular_2022_raw\12\25\4e853e80d720b13d6d5834c67853cc58.webm; post id: 3770756; MD5 from torrent is 119D22593623EB0FBA5055588782F8AA
file: e621_popular_2023_raw\01\02\331791a77feb69f846350bf33ac216f6.webm; post id: 3787554; MD5 from torrent is 43E344551595DE957D1350C62ADEDE13

All of these files except from the second one are actually corrupted, they won't open or playback is broken. I made sure that it's not just me by redownloading these files from the torrent. The same files on e621 are fine, the have the right hash and load up with no issues, would be nice to have them fixed in the torrent

note: I checked all yearly torrents but not character ones

Hmmm there shouldnt be any corrupted files since I did a verification pass but maybe I missed a couple.. When I get some time over the next couple weeks I'll try to take a look at these.

milkyrossini said:
it just says unable to find anytime i try to use the magnet links

I've added a magnet link for all of the .torrent files in the OP. Hopefully this helps!

magnet:?xt=urn:btih:39c2fba02b52c06c979c8bfeac2da6dfd5f8408d&dn=e621_torrents.zip&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.torrent.eu.org%3A451%2Fannounce&tr=udp%3A%2F%2Fopentracker.i2p.rocks%3A6969%2Fannounce&tr=udp%3A%2F%2Fopen.stealth.si%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker2.dler.org%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker1.bt.moack.co.kr%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker.tiny-vps.com%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.moeking.me%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.dler.org%3A6969%2Fannounce