Mantis issue - https://tracker.exphosted.com/view.php?id=4535
This is happening on every environment including local. The partials which seemed to have taken time to load, did not have any loops or queries so this is not a direct code issue.
This seems to be ruby garbage collector issue. On dev01, did some log statements to print the garbage collector time after enabling garbage collection stats. The extra time while rendering seems to be taken by the garbage collector. Observed this by enabling the garbage collection stats in REE and printing the value of GC.time(http://www.rubyenterpriseedition.com/documentation.html#_garbage_collector_statistics). Every time when this extra amount of time is taken, CPU usage spikes up as garbage collector runs.
We should figure out optimal GC related settings for the app. Tried this by following instructions here - http://technology.customink.com/blog/2012/03/16/simple-garbage-collection-tuning-for-rails/ . The frequency of the late responses is reduced but still seeing the late responses. Will need to spend more time.
* Use new relic to do a better benchmarking. (https://docs.newrelic.com/docs/ruby/garbage-collection)
* Figure out optimal Gc settings for app on all environments.
Some References :
http://stackoverflow.com/questions/8139040/rendering-haml-partials-takes-a-long-time-why
http://www.rubyenterpriseedition.com/documentation.html#_garbage_collector_and_object_space
http://technology.customink.com/blog/2012/03/16/simple-garbage-collection-tuning-for-rails
http://blog.railsupgrade.com/2011/02/garbage-collector-performance-tuning.html
They seem to have faster GC.
Ref - https://blog.twitter.com/2011/building-faster-ruby-garbage-collector