Hi Amit, we can sync up on Monday regarding this.

These are the current open issues/points we need to consider related to background processing and document processing.

1) https://tracker.exphosted.com/view.php?id=4719

This is more of a ruby garbage collection issue. God always sees that delayed job is taking too much memory during document conversion and restarts it. This needs to be fixed. As ruby process spawns delayed job worker, we may need to explicitly garbage collect in the code that is run by delayed job. It is observed that delayed job worker is more memory heavy than the other options like resque/sidekique.

2) To have multiple workers of delayed job we need to upgrade delayed job plugin and also make sure it supports our REE/rails versions(as DJ depends on active record). Also, make sure that our code is thread safe.

3) https://tracker.exphosted.com/view.php?id=5965

Observed that as soon as a record is queued in delayed jobs table , it is removed and mail is sent that files are processed. But actually the conversion did not happen.

4) We have issues with office 2007+ document conversions due to current openoffice version. To achieve better results, we need to use libreoffice on Ubuntu servers. So, not only background processing , synchronous document conversion also needs to be moved to separate machine.

5) As of now, we use both workling and delayed job in our code base for background tasking. We need to standardize them and decide on which one can be used across the system.

1) Fix delayed job : Running multiple delayed job processes on same machine and also have delayed job run on multiple machines.

http://stackoverflow.com/questions/4621817/rails-can-i-run-backgrounds-jobs-in-a-different-server

2) Delayed job alternatives:

- http://www.celeryproject.org/

- http://sidekiq.org/

- http://resquework.org/

Initial evaluation of delayed jobs alternatives -

https://wiki.exphosted.com/doku.php/background_jobs_crossbow

Below are few changes made in how the background jobs are performed

1. Moved all the background processing to Delayed Job. Earlier at few places we were using Workling to do background processing. Now have changed those instances to use DJ.
2. Earlier were were using the DJ code from https://github.com/tobi/delayed_job/. This is an older version and no longer being maintained. Now using the DJ plugin from https://github.com/collectiveidea/delayed_job/tree/v2.0 . There is no change in the way we push process to DJ or the way to run the DJ process.
3. The generic_monitoring.rb file has been modified to remove monitoring for Workling and starling processes (as we no longer use it). Also have updated the monitoring section for DJ to create proper pid file.
4. workling.yml has been removed and all the references to Workling has been removed from the code.
5. We can run instance of DJ on several servers (call it worker servers) simultaneously to ensure that the memory load is distributed (a single server is not overloaded). The process and setup for doing this is simple. Each server should have the copy of our entire rails app and required softwares (ruby, rails etc…) to ensure that our application runs there. The server machine should be able to access the Mysql database and the shared drive where all the files that are uploaded via our application is stored. Note that since the Worker servers are meant only to handle background jobs, it does not require any Webserver setup (Apache & Passenger).

1. Stop god on both app servers. (Ignore unable to stop god messages during deployment as we are stopping god here.)

2. Changes to /deploy/systasks/god.sh .

• Remove workling from WATCHES (around line 51).

• Remove the symlinks to workling.pid and job_runner.pid (line 8 and 9).

3. Changes to /deploy/crossbow/shared/config/generic_monitoring.god

• Remove starling and workling blocks.

• Change he pid file name of delayed job to delayed_job.pid from job_runner.pid (location remains same)

4. Remove workling reference from /deploy/crossbow/shared/config/<environment>.rb

(remove the line - Workling::Remote.dispatcher = Workling::Remote::Runners::StarlingRunner.new)

1) By default delayed job failed jobs are attempted 25 times. This count can be reduced to 10 or even 5 to make sure delayed job won't waste time reprocessing the failing jobs again.