DokuWiki

We currently issue a hard commit after every individual change operation with waitSearcher on(default). This results in both Searcher threads getting preoccupied in the “reload + warmup” process occasionally.

This is a common issue.
Suggested fixes are either to reduce the commits or increase number of searchers.

• Increase the number of searchers (by 1) + system optimizations.
  

    <maxWarmingSearchers>3</maxWarmingSearchers>

Current memory footprint is around 30-90Mb (for on disk index of ~4 mb). We have room for growth.
Allocate about 256 - 1024 Mb of RAM to JVM at start (xms xmx).

#catalina.sh
-Xms256m -Xms1024m

File Descriptors' limit is set to max 1024. This has to be increased.

sysctl -w fs.file-max=700000
echo "fs.file-max = 700000" >> /etc/sysctl.conf
sysctl -p

issue an “optimize” statement nightly.

• Replication (Horizontal Scaling)

Its best if we enable replication after upgrading Solr. If not, we will need this phase in place to handle high connection clients. Switch the session proxy abstraction in sunspot. Setup Solr Master Slave.

• Upgrade to Solr 4 - SolrCloud.

Discuss system and code changes.
Code Changes: Discuss SunSpot Issues (Ruby 1.8.7), Switch from 5xx retry router in sunspot to router that support master-slave architecture, Application does soft commits while Solr does automatic hard commits. System Changes: Index Upgrade, IOPS and separate storage, Replication and nodes.

• Batch commits.

This will boost commit performance and help us control the commit rate and plan better.

• Autosoft commit

Make Sunspot (from crossbow) do soft commits instead of hard commits.
Automatic hard commits via Solr @ every 10 documents, with waitSearcher property set to True (default). Soft commits are volatile but cheap and performant.
Hard commits are persistent but expensive. We will be optimizing the IO subsystem so flushing to disk process is faster to improve hard commit performance.

• NRTCachingDirectory

Migrate from NIOFS to MMAP http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html

• Merge Policy

#system level starts

• Dedicated storage

Move Solr data folder to a different storage unit.

• IOPS

Opsource does not support higher IOPS on the US EAST 1 location. US EAST 2 supports higher IOPS.
If intra-site latency is acceptable, it would be great to carve out the Solr role.

• Autowarm up on reboot

Read through the data directory on reboot, before starting Solr to load the index in OS FS RAM cache.

vmtouch -dl /opt/solr/data #or simply a parallel cat

• Replication.

We can use ruby benchmarker to get the whole time taken to send and recieve http request to solr server from the application. As ruby benchmarking is not always needed and it might add a slight over head, this code is not checked in.

1. rsolr.zip needs to be downloaded, extracted and kept in rails app's lib directory.
2. Restart the server and rails logs will now log the solr request-response times.

rsolr.zip

#tests on PROD on JUNE 30.

  ***************TIME*******************
user     system      total        real
HomeController#show 
0.000000   0.000000   0.000000 (  0.010064)
0.000000   0.000000   0.000000 (  0.003728)
0.000000   0.000000   0.000000 (  0.007079)
PostsController#video_preview (2 each id)
0.000000   0.000000   0.000000 (  0.006521)
0.000000   0.000000   0.000000 (  0.034731)
0.000000   0.000000   0.000000 (  0.002753)
0.000000   0.000000   0.000000 (  0.023866)
0.000000   0.000000   0.000000 (  0.002840)
0.000000   0.000000   0.000000 (  0.006822)
0.000000   0.010000   0.010000 (  0.002675)
0.000000   0.000000   0.000000 (  0.005755)
AssetsController#download
0.000000   0.000000   0.000000 (  0.006731)
0.010000   0.000000   0.010000 (  0.072848)
0.000000   0.000000   0.000000 (  0.006939)
0.000000   0.000000   0.000000 (  0.064942)
---- homepage----
 MyLearningsController#show
0.000000   0.000000   0.000000 (  0.007098)
0.000000   0.000000   0.000000 (  0.063049)
 MyLearnings::AvailableSoonItemsController#index
0.000000   0.000000   0.000000 (  0.003361)
0.000000   0.000000   0.000000 (  0.037070)
MyLearnings::CompletedItemsController#index
0.000000   0.000000   0.000000 (  0.003368)
0.000000   0.000000   0.000000 (  0.008719)
HomeController#about
0.000000   0.000000   0.000000 (  0.007007)
0.000000   0.000000   0.000000 (  0.007065)
MyLearnings::CompletedItemsController#index
0.010000   0.010000   0.020000 (  0.008936)
0.000000   0.000000   0.000000 (  0.074535)
MyLearnings::AvailableSoonItemsController#index
0.000000   0.000000   0.000000 (  0.002148)
0.000000   0.000000   0.000000 (  0.068811)
MyLearnings::AvailableSoonItemsController#index
0.000000   0.000000   0.000000 (  0.007524)
0.000000   0.000000   0.000000 (  0.034764)
MyLearnings::CompletedItemsController#index
0.000000   0.000000   0.000000 (  0.003059)
0.000000   0.000000   0.000000 (  0.005746)
ChatsController#mark_as_read
0.010000   0.000000   0.010000 (  0.003409)
0.010000   0.000000   0.010000 (  0.033103)
AssetsController#download 
0.010000   0.000000   0.010000 (  0.006219)
0.000000   0.000000   0.000000 (  0.006716)
ChatsController#mark_as_read 
0.000000   0.000000   0.000000 (  0.006078)
0.000000   0.000000   0.000000 (  0.032725)
ChatsController#typing_msg 
0.010000   0.000000   0.010000 (  0.006259)
0.000000   0.000000   0.000000 (  0.043800)
ChatsController#typing_msg
0.000000   0.000000   0.000000 (  0.007180)
0.440000   0.000000   0.440000 (  0.440934)
ChatsController#typing_msg
0.000000   0.000000   0.000000 (  0.003182)
0.000000   0.000000   0.000000 (  0.038253)
ChatsController#send_message
0.000000   0.000000   0.000000 (  0.002215)
0.000000   0.000000   0.000000 (  0.051115)
ChatsController#send_message
0.000000   0.000000   0.000000 (  0.022058)
0.000000   0.000000   0.000000 (  0.085306)
ChatsController#typing_msg
0.000000   0.000000   0.000000 (  0.004146)
0.000000   0.000000   0.000000 (  0.019743)
ContentsController#index 
0.000000   0.000000   0.000000 (  0.014343)
0.440000   0.010000   0.450000 (  0.448826)
ContentsController#index
0.000000   0.000000   0.000000 (  0.004163)
0.000000   0.000000   0.000000 (  0.014175)
0.010000   0.000000   0.010000 (  0.004789)
GroupsController#index 
0.010000   0.000000   0.010000 (  0.008813)
0.000000   0.000000   0.000000 (  0.032253)
GroupsController#index 
0.000000   0.000000   0.000000 (  0.006650)
0.000000   0.000000   0.000000 (  0.037412)