Right now the content conversion process (i.e. document conversion to PDF, SWF and png OR video conversion to various formats) is done by queuing the conversion process in delayed job. The delayed job instance runs on the application server, which is responsible for serving web requests. When larger files are uploaded, the conversion process uses up bulk of memory and cpu because of which the application server is not able to process the web requests.

To device a scalable solution to distribute the processing over different servers

1) To make sure that the longer running OR memory consuming jobs do not affect the web request/response process.

2) To make sure that conversion process does not require a single server with very large memory and cpu requirement. Rather the processing load should be distributed amongst multiple servers.

Refer to the diagram below

• Unlike what we have today, where the delayed job instance is running in the application server itself, we will have seperate machines which will host only the delayed job instance (call it worker server).

• The shared filesystem will be available to application/web server as well as the the worker servers.

• Delayed Job instances will not run on application/web server. This will mean that the web server machine's responsibility will be only to server the web requests and nothing else.

• All the worker servers should have to have synced clock.

• Since the job queue is maintained in the database (which is central), even though we have multiple worker instances running, only one worker will lock the job and process it. In the mean time, if another job is added to the queue, the another worker instance will pick it.

• Note that the worker servers need not have the web server (Apache, mongrel etc…) running. It simply needs the rails application and required dependencies (i.e. ruby and rails framework). Hence the job of worker servers will be to only process the queued jobs.

Following are the things be considered to scale up this setup -

1. Most of the large files could be videos and video conversion is CPU heavy. So, we need at least 8 Cpu or 12 CPU core machines.

2. We need to add one more slave database and scale the master if needed so that replication is smooth.

3. The distributed file system should be really scaling well to handle all the reads and writes as there will be about 5 machines hitting it at all the time. We are considering hadoop dfs as of now. This seems good and needs to be fine tuned to the best possible extent.