Page MenuHomePhabricator

Increase the limit of 50 uploads at once (maxUploads)
Closed, ResolvedPublic

Description

After the performance improvements from T121901 earlier this year, we can probably increase the number of uploads allowed at once (maxUploads), currently set to 50.

I'm not sure how far we can bump it, but we could probably do 500 easily and then maybe try 5000, which should be enough even for mass uploads.

We'll need to test stuff a bit, verify that things are not taking unbearably long, verify that we're not using unreasonable amounts of memory with this many files, and verify that we're not doing something accidentally quadratic on the number of uploads.

Event Timeline

Restricted Application added subscribers: Zppix, Steinsplitter, Aklapper. · View Herald Transcript

(When I do mass-uploads it is for streams on Flickr beteen 800-7 000 files. Would also be great if UW could check if already uploaded etc (based on link-IDs/SHA1).)

@Josve05a (or other community Commoners), do you think that this higher limit should be available to all or should it be restricted in some way?

@Josve05a (or other community Commoners), do you think that this higher limit should be available to all or should it be restricted in some way?

Most uploaders that uploads copyvios either only upload 1-10 files, or upload a lot using cross-wiki-upload, so upping the maximum number for all on UW does not seem to be a high rigsk target for issues, imo...

@Josve05a (or other community Commoners), do you think that this higher limit should be available to all or should it be restricted in some way?

Most uploaders that uploads copyvios eithe ronly upload 1-10 files, or upload a lot using cross-wiki-upload, so upping the maximum number for all on UW does not seem to be a high rigsk target for issues, imo...

Thanks! Let's get this done, then.

Hmm… It actually doesn't look that rosy. I tried with 200 uploads, testing with Firefox 48. While I was able to upload them just fine, proceeding to the "Release rights" step locked up the browser for a few seconds. Proceeding to "Describe" locked it up for a bit longer, and I had to click through two "Unresponsive script" dialogs before I got there. Everything worked fine apart from that, just slowly, but I'm not sure if we want to permit this experience for all users. (If you're a power user, remember this trick: run mw.UploadWizard.config.maxUploads = 500; in the browser console after UW loads, but before you start uploading.)

MarkTraceur subscribed.

I assume we still want this? @matmarex are the issues you mentioned deal-breakers? @Jdforrester-WMF what do you think?

I think they are deal-breakers. We should find out why it's slow, and make it faster. Or if that's impossible, we should make things be lazy-loaded.

I was just experimenting with uploads in batches of 500 (thanks to matmarex mw.UploadWizard.config.maxUploads = 500; trick) and did not run into any performance issues. The only concern is that it is impractical to be expecting people to manually add categories or descriptions with batches that size. In my case I am using descriptions from flickr and do some after upload cleaning with AutoWikiBrouwser, which is faster in my case. May be we can add mw.UploadWizard.config.maxUploads to UploadWizard Campaigns customization so the maximum number of uploads can be changed through that interface.

MarkTraceur raised the priority of this task from Low to Medium.Sep 7 2017, 4:13 PM
MarkTraceur moved this task from Next up to Triaged on the Multimedia board.

@Jarekt: What browser are you using?

Chrome. I explained the process I was using at Village_pump

Ramsey-WMF moved this task from Triaged to Next up on the Multimedia board.
Ramsey-WMF subscribed.

With community consultation and performance testing complete, we have enough input to move forward with increasing the limit for admins and license reviewers (with perhaps other types of users to be considered later).

Change 394309 had a related patch set uploaded (by Matthias Mullie; owner: Matthias Mullie):
[mediawiki/extensions/UploadWizard@master] Increase limit of amount of uploads to 500 for certain user groups

https://gerrit.wikimedia.org/r/394309

Change 394309 merged by jenkins-bot:
[mediawiki/extensions/UploadWizard@master] Increase limit of amount of uploads to 500 for certain user groups

https://gerrit.wikimedia.org/r/394309

His upload power is over 500!!!
Macro over9000:

Seems to be working for me so far. I have not gone as far as adding descriptions and dates for all 500 items yet.

I noticed a couple of things with thumbnails, but with that many files it is expected I think:

Screen Shot 2018-02-27 at 9.09.20 AM.png (1×2 px, 991 KB)

Screen Shot 2018-02-27 at 9.15.46 AM.png (1×2 px, 995 KB)

Screen Shot 2018-02-27 at 9.10.02 AM.png (1×2 px, 1 MB)

Screen Shot 2018-02-27 at 9.15.37 AM.png (1×2 px, 1015 KB)

Screen Shot 2018-02-27 at 9.15.25 AM.png (1×2 px, 1 MB)

Thumbnails for images should be displayed immediately (the browser just needs to re-use the upload file).
Thumbnails for STL files need to be generated and take a little longer - especially with a batch this large & on beta :)

I assume we can resolve this - please re-open if not!