To better handle the errors thrown by solr server when the load is high.

At present we run solr 1.4 server on tomcat 6. We use sunspot and sunspot_rails gems (v 1.2.1) as the ruby client for solr. They are dependent on rsolr gem(v 0.12.1)

As per the notes here - http://help.websolr.com/kb/common-problems/503-service-unavailable

The best fixes for this would be -

a) Server - Update to the latest version of solr server(4.6 as of now) solr 3+ has scalability improvements. (- Not implemented)

b) Client - Change the client to retry once when service unavailable exception is thrown by solr server. After one retry if still the same error is occurring then just raise the error. (- Implemented)

The following errors will be tried again once -

• Service Unavailable
• Internal Server Error
• Client not connect to server

• Solution (b) is implemented by the sunspot and related gems starting from version 1.3.1 but these versions do not support Rails 2. Only rails 3+ is supported. Hence, we implemented this retry ourselves with some code inspiration from sunspot 1.3.1 gem.
• Solution (a) is not possible as from solr 3+ the schema.xml file is different and not compatible with our current schema and the current version of gem rsolr(0.12.1). So, we need to stay with solr version 1.4 at this point.
• One more limitation is the exception handling by rsolr 0.12.1 does not return status code but just the error message. So, we are doing a regexp match for 'Service Unavilable' and 'Internal Server Error' errors and we retry once.

When we move to ruby 2 and rails 4 , we can also try to update solr server and sunspot gems to the latest supported versions.