One should be able to create content by uploading large files. In our case a large file is anything that takes more than 5 mins to upload. The connection of the client is kept alive by apache only for 5 mins as the default. If the response is not received in 5 mins the connection is lost and the client cannot receive the response.
One fix to this is to increase the apache time out but this is not recommended as the server thread serving a request waits for longer in case of orphned requests and will not be available to serve other requests.
There are some ways to keep the connection persistent till the file upload is complete(HTML5, flash and Ajax). The challenge for us is to have a crossbrowser solution.
We decided to go with flash based approach (http://swfupload.org/) on IE and Ajax file upload(https://github.com/valums/file-uploader version 2.0) on other browsers.
This allows file uploads of size upto 2GB on older browsers and 4GB on latest browsers.
One problem with our content creation is we process files after they are uploaded and preview them in the response. In cases where the file processing is taking more than 5 mins the connection to server times out and it creates problems (#0002837). This is seen mostly with large videos.
a) Background the processing (Not recommended) - We can push the file processing to a background job and show the proper preview once the file is processed.
We may not prefer this as this is not the best user experience.
b) Ajax requests to keep the connection alive - We can send ajax requests after the file upload is complete and keep the connection alive so that the file preview is shown once it is processed.
There is a simpler change in terms of coding and we do not need to change the architecture. The down side of this approach is more load on the server due to Ajax requests. We can try to minimize this load by using Rails metal or Juggernaut.
c) Nodejs based uploads - We can have a nodejs app which allows all the file uploading and processing. This is a better user experience as we can do live transcoding with ffmpeg and nodejs uploads (https://transloadit.com/blog/2010/12/realtime-encoding-over-150x-faster). So, the file conversion is faster.
The downsides of these are -
1. We need to change the architecture to have nodejs do the upload and processing. Time taking implementation.
2. Nodejs's scalability on production is unknown to us yet. (transloadit.com uses this on produciton but still we do not have the experience.)
References -