Table of Contents
Large File Uploads
Requirement
One should be able to create content by uploading large files. In our case a large file is anything that takes more than 5 mins to upload. The connection of the client is kept alive by apache only for 5 mins as the default. If the response is not received in 5 mins the connection is lost and the client cannot receive the response.
One fix to this is to increase the apache time out but this is not recommended as the server thread serving a request waits for longer in case of orphned requests and will not be available to serve other requests.
Solution
There are some ways to keep the connection persistent till the file upload is complete(HTML5, flash and Ajax). The challenge for us is to have a crossbrowser solution.
- HTML5 solution does not work on HTML4 only browsers and especially on IE.
- Flash is not the universal choice as i devices do not support it and also on Linux flash players the upload response timesout if not recevied within 30 sec. So, this is a good solution only on windows.
- IE does not support asych uploads with Ajax. In other browsers this is a good solution.
Implementation
We decided to go with flash based approach (http://swfupload.org/) on IE and Ajax file upload(https://github.com/valums/file-uploader version 2.0) on other browsers.
This allows file uploads of size upto 2GB on older browsers and 4GB on latest browsers.
File processing timeout
One problem with our content creation is we process files after they are uploaded and preview them in the response. In cases where the file processing is taking more than 5 mins the connection to server times out and it creates problems (#0002837). This is seen mostly with large videos.
Possible Fixes
a) Background the processing (Not recommended) - We can push the file processing to a background job and show the proper preview once the file is processed.
We may not prefer this as this is not the best user experience.
b) Ajax requests to keep the connection alive - We can send ajax requests after the file upload is complete and keep the connection alive so that the file preview is shown once it is processed.
There is a simpler change in terms of coding and we do not need to change the architecture. The down side of this approach is more load on the server due to Ajax requests. We can try to minimize this load by using Rails metal or Juggernaut.
c) Nodejs based uploads - We can have a nodejs app which allows all the file uploading and processing. This is a better user experience as we can do live transcoding with ffmpeg and nodejs uploads (https://transloadit.com/blog/2010/12/realtime-encoding-over-150x-faster). So, the file conversion is faster.
The downsides of these are -
1. We need to change the architecture to have nodejs do the upload and processing. Time taking implementation.
2. Nodejs's scalability on production is unknown to us yet. (transloadit.com uses this on produciton but still we do not have the experience.)
References -
