Commons:Requests for comment/Technical needs survey/File upload stability

Previous proposal

Overview page

Next proposal

File upload stability

Description of the Problem

• Problem description: When uploading files using the UploadWizard or the API users experience very frequent problems resulting in aborted uploads or broken files. When the error is not recognized broken files or file description page info might be lost for Commons. If they are recognized they are very inconvenient to the uploads resulting in long term term contributors leaving or scaring new contributors.

• Proposal type: bugfix

• Proposed solution: Define the goal that only 1:10000 uploads using the API should fail because of server side problems. Only 1:1000 uploads should fail when uploading in the web browser because of server or website errors.

• Phabricator ticket: There are multiple tickets on the various problems: Commons: UploadChunkFileException: Error storing file: backend-fail-internal; local-swift-codfw Files occasionally getting uploaded to Commons without file pages

• Further remarks: Feel free to add other relevant tickets. GPSLeo (talk) 14:05, 9 December 2023 (UTC)

Discussion

• Diesem Vorschlag schließe ich mich aus tiefstem Herzen an. Insbesondere der UploadWizard könnte die Server-Fehlermeldungen viel verständlicher darstellen und viele auch besser abfangen. Ich möchte auch nochmals auf das Android-Tool Offroader hinweisen, das zeigt, wie stabil Uploads auf Commons mit der vorhandenen Server-Implementierung selbst unter widrigsten Bedingungen sein können, dass ein abgebrochener Upload ohne weiteres - auch auf einem anderen Gerät und mit einem anderen Internetzugang fortgesetzt werden kann, dass Uploads auf Fehlerfreiheit verifiziert werden können, dass Duplikate bereits vor Beginn eines Uploads erkannt und verhindert werden können und das - als Hilfe fürs Entwickeln, die Server-Meldungen während eines Uploads mitschneiden kann für ein PostMortem. --C.Suthorn (@Life_is@no-pony.farm - p7.ee/p) (talk) 18:41, 9 December 2023 (UTC)

I agree with this suggestion from the bottom of my heart. The UploadWizard in particular could display the server error messages much more comprehensibly and intercept many of them better. I would also like to point out again the Android tool Offroader, which shows how stable uploads to Commons can be with the existing server implementation, even under the most adverse conditions, that a canceled upload can easily happen - even on a different device and with a different Internet access can be continued, that uploads can be verified to be free of errors, that duplicates can be detected and prevented before an upload begins and that - as an aid to development, the server messages can be recorded during an upload for a postmortem.

translator: Google Translate via   — 🇺🇦Jeff G. ツ _{please ping or talk to me}🇺🇦 00:27, 10 December 2023 (UTC)

• @C.Suthorn: I wish all tools were so resilient.   — 🇺🇦Jeff G. ツ _{please ping or talk to me}🇺🇦 01:03, 22 January 2024 (UTC)

Yes, this is sorely needed! I like the idea of having target metrics especially. Nosferattus (talk) 17:40, 21 December 2023 (UTC)

• +avoid file upload interruption: infamous 5 MB interruption and newer (and still unresolved) 10 MB interruption (mainly linked to cross-wiki upload). — Draceane ^talk_contrib. 08:10, 23 January 2024 (UTC)

I tried to gather some stats (These might not be fully robust, take with a bit of salt)

Event	Feb 23	Feb 24	Feb 25	Feb 26	Feb 27	Feb 28 [new code deployed]	Feb 29
Chunked upload recombine success	11,164	10,796	11,161	10,575	10,420	11,766	10,457
Recieved last chunk	11,191	10,839	11,211	10,609	10,470	11,830	10,495
Recieved first chunk	11,231	10,880	11,284	10,691	10,504	11,892	10,555
Stash error (excluding logged out, may include duplicates from multiple attempts)	869	306	138	27	677	508	232

The stash error row might be misleading, as most of them are all for the same user and might represent a bot with a programming mistake. Bawolff (talk)

Votes

1. yes.--RZuo (talk) 21:11, 21 January 2024 (UTC)
2.  Support.   — 🇺🇦Jeff G. ツ _{please ping or talk to me}🇺🇦 21:26, 21 January 2024 (UTC)
3. qualified yes. Of course I would welcome improvements in this area and think it would be a good use of resources, but I think the target metric of 1 server-side failure in 10,000 uploads is probably too ambitions. - Jmabel ! talk 23:47, 21 January 2024 (UTC)
4.  Support — Draceane ^talk_contrib. 08:10, 23 January 2024 (UTC)
5.  Support Jklamo (talk) 08:37, 23 January 2024 (UTC)
6.  Support I thought only I had this issue. It happens to me every few uploads that I have to remove (some) files because they don't load.Paradise Chronicle (talk) 16:50, 24 January 2024 (UTC)
7.  Support --Fl.schmitt (talk) 06:42, 26 January 2024 (UTC)
8.  Support Daniel Case (talk) 02:37, 2 February 2024 (UTC)
9.  Support Chunked upload code should be torn out and rewritten. Bawolff (talk) 18:46, 3 February 2024 (UTC)
10.  Support Pleeeeaaase! I also have problems uploading files above 2 GiB right now... --PantheraLeo1359531 😺 (talk) 18:50, 3 February 2024 (UTC)

    @PantheraLeo1359531: Please see phab:T355433.   — 🇺🇦Jeff G. ツ _{please ping or talk to me}🇺🇦 10:59, 8 February 2024 (UTC)

    Ah, thanks! --PantheraLeo1359531 😺 (talk) 14:50, 9 February 2024 (UTC)

11.  Support Yesterday, I encountered this issue where I had to upload this image again. It can be found in Image history --iMahesh (talk) 10:29, 8 February 2024 (UTC)
12.  Support --ToprakM ^✉ 15:04, 12 February 2024 (UTC)
13.  Support Jo-Jo Eumerus (talk) 12:13, 13 February 2024 (UTC)
14.  Support as proposer --GPSLeo (talk) 18:28, 13 February 2024 (UTC)