Topic: [Feature Request] Dynamic WEBM file size limits

Currently, there is a flat 100MB cap on video sizes.
This is rather restrictive. There's a rather large amount of videos that have had to be compressed to fit here and have significantly better off-site versions.

This limit exists partially to prevent upload of videos that are absurdly large for what they contain. For instance, a lossless conversion from MP4 to WEBM will be absolutely huge for the content. However, it does not do this well - short but extremely badly compressed videos slip through, while preventing proper archival of longer or higher resolution videos.

This proposal is to instead limit WEBM file size by contents, in particular the number of frames and the resolution. This would allow for preventing uploading video far too large for what it is, while allowing uploading high-quality videos that otherwise could not be archived without destroying the quality.

This is doable - the site already gets upload statistics to determine duration and resolution, which could be reused for such a system.

There would still have to be an upper limit of course, and the actual details would be up for discussion. My first thought is simply a linear relation between the total number of displayed pixels and allowed file size, along with lower and upper bounds.

I'd like to mention with e6 using cloudflare, each tier has a hard limit on upload sizes
The free/pro tiers (which I believe is likely used by e6) only allow up to 100MB (pro is $25/month or $240/year
The business tier, which is $250/month or $2400/year allows up to 200MB
The enterprise tier which requires getting a quote allows 500MB by default

I in no way believe e6 is paying $2400 a year or more for cloudflare

donovan_dmc said:
I'd like to mention with e6 using cloudflare, each tier has a hard limit on upload sizes
The free/pro tiers (which I believe is likely used by e6) only allow up to 100MB (pro is $25/month or $240/year
The business tier, which is $250/month or $2400/year allows up to 200MB
The enterprise tier which requires getting a quote allows 500MB by default

I in no way believe e6 is paying $2400 a year or more for cloudflare

Maybe, but this can definitely be worked around. If it would be accepted other than actually coding it, I can do that.

scth said:
Maybe, but this can definitely be worked around. If it would be accepted other than actually coding it, I can do that.

The only "workaround" is bypassing cloudflare, which requires either infrastructure changes or very specific routing of an unproxied domain, and also in both circumstances opens up an attack surface for unmitigated DDoS attacks

donovan_dmc said:
The only "workaround" is bypassing cloudflare, which requires either infrastructure changes or very specific routing of an unproxied domain, and also in both circumstances opens up an attack surface for unmitigated DDoS attacks

My first idea would be splitting the file into multiple parts then combining server side.

scth said:
My first idea would be splitting the file into multiple parts then combining server side.

Ignoring that this all has to be done in the browser in memory, and presuming the tab doesn't get OOM killed for using hundreds of megabytes of memory while looking like it's doing nothing, this just opens the doors for more timing/network issues, and is definitely not worth the hassle

donovan_dmc said:
Ignoring that this all has to be done in the browser in memory, and presuming the tab doesn't get OOM killed for using hundreds of megabytes of memory while looking like it's doing nothing, this just opens the doors for more timing/network issues, and is definitrly.not worth the hassle

Not worth the hassle? Archiving the best available version is a pretty important part of e621's goal.

It also doesn't necessarily have to be done in memory, since there's an API for slicing files.

I'm confident that it is a solvable problem, and that I could code it. Perhaps not for all browsers necessarily, but uploading such large files would be a relatively niche use and requiring a proper desktop browser might be fine.

parasprite said:
We'd still like to increase it (at least for videos) since we have plenty of disk space and bandwidth to spare, it just hasn't been done yet. I don't have a timeframe to give you but the figure I keep hearing is around 300 MB (note that the exact number is still subject to change ;) ).

^I came across this post from over 10! years ago and was curious why the increase never happened.

After reading through this thread I get it, cloud flare's restrictions are unfortunate, and its unfortunate that bot attacks have gotten so out of hand that they are so needed.

I can see why SCTH's suggestion might not be ideal, It could work technically, but it introduces a lot of complexity, might break for some users, and would require long-term maintenance from volunteers who have their hands full.

My next animation project is likely to end up being around 20 minutes which is why I was looking into this. I was stretching the limits of 100mb compression with my previous 8.5 min animation at 720p, so I will be making some prayers to the video compression gods this time around. That or I may have to throw in the towel and split it into parts as a special e621 specific version, fortunately pools exist so it will be kinda like a sequential comic ig.

I've done the implementing of multipart uploads myself in another codebase and still confidently say that it is not worth it for how complex it is and how little it would be actually used for large videos, and how often it would instead encourage badly optimized videos which would have previously been rejected for size limitations

donovan_dmc said:
I've done the implementing of multipart uploads myself in another codebase and still confidently say that it is not worth it for how complex it is and how little it would be actually used for large videos, and how often it would instead encourage badly optimized videos which would have previously been rejected for size limitations

I suppose it depends on how the Cloudflare limit is implemented. If it allows multiple files totaling over 100MB in a single request (multipart form upload), that would be trivial to stitch server side. I'm guessing it doesn't, though. Would still be feasible, but then you'd have to have some state maintained across requests which would be way more bug prone.

I've seen many animations which would have been much higher quality with a dynamic limit - many of the top posts on the site. A dynamic limit also wouldn't give too much room for badly optimized videos, and in fact would've prevented some of the existing ones if it can go below 100MB.

scth said:
I suppose it depends on how the Cloudflare limit is implemented. If it allows multiple files totaling over 100MB in a single request (multipart form upload), that would be trivial to stitch server side. I'm guessing it doesn't, though. Would still be feasible, but then you'd have to have some state maintained across requests which would be way more bug prone.

It's a per request limit, nothing related to files
It needs to be multiple independent requests which makes it a significant challenge

I implemented it on my own site here (though that commit contains more than just simply allowing multi-request uploads, and of course that commit isn't the final/current code)

Despite extensive testing and a month worth of development time, I still found several bugs in it weeks to months later, and I'm sure there are still more to find

E6 can't really afford that amount of jank nor does it have the resources to test enough to avoid that amount of jank

donovan_dmc said:
It's a per request limit, nothing related to files
It needs to be multiple independent requests which makes it a significant challenge

What about using a WebSocket rather than splitting into multiple requests?

scth said:
What about using a WebSocket rather than splitting into multiple requests?

Websockets are hell in rails, and cloudflare's docs are unclear on how limits are applied
https://developers.cloudflare.com/network/websockets/
They say only the initial upgrade request is treated as an HTTP request, and that the whole connection is considered one long lived HTTP request, but it doesn't say if that means the entire thing is limited to 100MB (though I really doubt it is)

donovan_dmc said:
Websockets are hell in rails, and cloudflare's docs are unclear on how limits are applied
https://developers.cloudflare.com/network/websockets/
They say only the initial upgrade request is treated as an HTTP request, and that the whole connection is considered one long lived HTTP request, but it doesn't say if that means the entire thing is limited to 100MB (though I really doubt it is)

I suppose it's something easy enough to test. I highly doubt it would be cut off, since that would make it unusable for more standard use cases. As for it being difficult in Rails, yeah I could see that being an issue (though personally I think everything is difficult in rails).

On top of technical issues, there are also the factor of practical stuff.
Site has actually decreased filesize limit to stuff like GIF and APNG files, to reduce on literally insane uploads of people uploading ten seconds of video in 4K but as GIF so now it's 100MB with major color banding and dithering artifacts both at once.

Between the starting post and now, we have AV1 support. VP9 was already better for bitrate compared to h264, but especially super low bitrates, AV1 is even to me sort of magic with some material as there's no visible artifacts, but also noisy detail is surprisingly well maintained.

Obviously it would be nicer to have some sort of rolling limit as I have hit 1 hour limit for uploads with one post already and thus had to cut it to couple posts to be able to upload it, which isn't viable in some cases.