====== Prodapp04 service check list ======
Most of the servers listed were updated or outdated. This is just a reference of past. Latest RoR servers detail are maintained at [[ror_hardwares_feb2020|RoR Servers]] since 2020.
login as expprodl
[expprodl@prodapp04 ~]$ /deploy/systasks/god.sh status
Showing status for memcached ...
expprodl 2457 1 0 Aug02 ? 00:00:11 memcached -d -m 3500
Showing status for office ...
expprodl 2455 1 0 Aug02 ? 00:00:05 /opt/openoffice4/program/soffice.bin -headless -accept=socket,host=127.0.0.1,port=8100;urp; -nofirststartwizard -norestore -nodefault
Showing status for xvfb ...
expprodl 2451 1 0 Aug02 ? 00:00:00 Xvfb :1 -screen 0 800x600x24
Showing status for starling ...
Showing status for rufus ...
expprodl 2449 1 1 Aug02 ? 01:23:23 ruby /deploy/crossbow/current/script/rufus
Showing status for delayed_job ...
expprodl 2612 1 0 Aug02 ? 00:06:30 delayed_job
if not running execute below cmd as expprodl
/deploy/systasks/god.sh start
[expprodl@prodapp04 ~]$ ps -ef | grep sendmail
root 2213 1 0 Aug02 ? 00:00:09 sendmail: accepting connections
smmsp 2222 1 0 Aug02 ? 00:00:00 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
if not running execute below cmd as root
/etc/init.d/sendmail start
[expprodl@prodapp04 ~]$ ps -ef | grep opendkim
opendkim 2111 1 0 Aug02 ? 00:00:02 /usr/sbin/opendkim -x /etc/opendkim.conf -P /var/run/opendkim/opendkim.pid
if not running execute below cmd as root
/etc/init.d/opendkim start
[expprodl@prodapp04 ~]$ ps -ef | grep http
root 2291 1 0 Aug02 ? 00:00:11 /opt/apache2/bin/httpd -k start
root 2293 2291 0 Aug02 ? 00:00:14 /opt/apache2/bin/rotatelogs /opt/apache2/logs/http-crossbow_access_log.%Y-%m-%d-%H_%M 86400 -240
daemon 15783 2291 0 Aug05 ? 00:00:08 /opt/apache2/bin/httpd -k start
daemon 15785 2291 0 Aug05 ? 00:00:08 /opt/apache2/bin/httpd -k start
daemon 15814 2291 0 Aug05 ? 00:00:08 /opt/apache2/bin/httpd -k start
daemon 15822 2291 0 Aug05 ? 00:00:08 /opt/apache2/bin/httpd -k start
daemon 15830 2291 0 Aug05 ? 00:00:08 /opt/apache2/bin/httpd -k start
daemon 15838 2291 0 Aug05 ? 00:00:08 /opt/apache2/bin/httpd -k start
daemon 15852 2291 0 Aug05 ? 00:00:08 /opt/apache2/bin/httpd -k start
if not running execute below cmd as root
/opt/apache2/bin/httpd -k start
[expprodl@prodapp04 ~]$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
[expprodl@prodapp04 ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/volGroup00-rootVol00
49G 15G 33G 31% /
tmpfs 7.8G 0 7.8G 0% /dev/shm
/dev/sda1 190M 125M 55M 70% /boot
prodcollab01.learninguniver.se:/streams
99G 59G 35G 64% /deploy/crossbow/shared/streams
10-166-152-13:/asset-volume
493G 374G 94G 81% /deploy/crossbow/shared/assets
10-166-152-13:/data-volume
493G 374G 94G 81% /deploy/crossbow/shared/system/uploaded_datas
10-166-152-13:/images-volume
493G 374G 94G 81% /deploy/crossbow/shared/system/images
10-166-152-13:/site_backgrounds
493G 374G 94G 81% /deploy/crossbow/shared/system/site_backgrounds
10-166-152-13:/site_logos
493G 374G 94G 81% /deploy/crossbow/shared/system/site_logos
10-166-152-13:/styles
493G 374G 94G 81% /deploy/crossbow/shared/system/styles
10-166-152-13:/sheets
493G 374G 94G 81% /deploy/crossbow/shared/system/sheets
10-166-152-13:/templates
493G 374G 94G 81% /deploy/crossbow/shared/system/templates
===== Prodapp03 service check list =====
[expprodl@prodapp03 ~]$ /deploy/systasks/god.sh status
Showing status for office ...
expprodl 2490 1 0 Aug02 ? 00:00:04 /opt/openoffice4/program/soffice.bin -headless -accept=socket,host=127.0.0.1,port=8100;urp; -nofirststartwizard -norestore -nodefault
Showing status for xvfb ...
expprodl 2487 1 0 Aug02 ? 00:00:00 Xvfb :1 -screen 0 800x600x24
Showing status for starling ...
Showing status for rufus ...
expprodl 2486 1 0 Aug02 ? 00:53:19 ruby /deploy/crossbow/current/script/rufus
Showing status for delayed_job ...
expprodl 2573 1 0 Aug02 ? 00:03:56 delayed_job
expprodl 20644 2573 0 Aug03 ? 00:00:00 [sh]
expprodl 21048 2573 0 Aug03 ? 00:00:00 [sh]
Showing status for roster ...
expprodl 2483 1 0 Aug02 ? 00:00:02 ruby /deploy/crossbow/current/script/roster
if process not running execute below cmd as expprodl
/deploy/systasks/god.sh start
[expprodl@prodapp03 ~]$ ps -ef | grep sendmail
root 2202 1 0 Aug02 ? 00:00:05 sendmail: accepting connections
smmsp 2211 1 0 Aug02 ? 00:00:00 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
if not running execute below cmd as root
/etc/init.d/sendmail start
[expprodl@prodapp03 ~]$ ps -ef | grep opendkim
opendkim 2638 1 0 Aug02 ? 00:00:01 /usr/sbin/opendkim -x /etc/opendkim.conf -P /var/run/opendkim/opendkim.pid
if not running execute below cmd as root
/etc/init.d/opendkim start
[expprodl@prodapp03 ~]$ ps -ef | grep http
root 2300 1 0 Aug02 ? 00:00:07 /opt/apache2/bin/httpd -k start
root 2302 2300 0 Aug02 ? 00:00:07 /opt/apache2/bin/rotatelogs /opt/apache2/logs/http-crossbow_access_log.%Y-%m-%d-%H_%M 86400 -240
daemon 5164 2300 0 Aug06 ? 00:00:02 /opt/apache2/bin/httpd -k start
daemon 9125 2300 0 Aug06 ? 00:00:01 /opt/apache2/bin/httpd -k start
daemon 14944 2300 0 Aug06 ? 00:00:00 /opt/apache2/bin/httpd -k start
daemon 15053 2300 0 Aug06 ? 00:00:00 /opt/apache2/bin/httpd -k start
daemon 15327 2300 0 00:16 ? 00:00:00 /opt/apache2/bin/httpd -k start
daemon 15372 2300 0 00:19 ? 00:00:00 /opt/apache2/bin/httpd -k start
if not running execute below cmd as root
/opt/apache2/bin/httpd -k start
[expprodl@prodapp03 ~]$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
[expprodl@prodapp03 ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/volGroup00-rootVol00
49G 21G 26G 45% /
tmpfs 7.8G 0 7.8G 0% /dev/shm
/dev/sda1 190M 125M 55M 70% /boot
prodcollab01.learninguniver.se:/streams
99G 59G 35G 64% /deploy/crossbow/shared/streams
10.166.152.13:/asset-volume
493G 374G 94G 81% /deploy/crossbow/shared/assets
10.166.152.13:/data-volume
493G 374G 94G 81% /deploy/crossbow/shared/system/uploaded_datas
10.166.152.13:/images-volume
493G 374G 94G 81% /deploy/crossbow/shared/system/images
10.166.152.13:/site_backgrounds
493G 374G 94G 81% /deploy/crossbow/shared/system/site_backgrounds
10.166.152.13:/site_logos
493G 374G 94G 81% /deploy/crossbow/shared/system/site_logos
10.166.152.13:/styles
493G 374G 94G 81% /deploy/crossbow/shared/system/styles
10.166.152.13:/sheets
493G 374G 94G 81% /deploy/crossbow/shared/system/sheets
10.166.152.13:/templates
493G 374G 94G 81% /deploy/crossbow/shared/system/templates
==== proddb mysql slave01 ====
Mysql passwords available in google sheet
https://docs.google.com/a/expertus.com/spreadsheets/d/1Bgd3ZrD7-MlF5j0O20j06AOPXVlzjB_13cC2ZYu5BiM/edit?usp=sharing
[expprodl@proddb-slav01 ~]$ mysql -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 197
Server version: 5.5.13 MySQL Community Server (GPL)
Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show slave status \G;
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 10.166.152.12
Master_User: replcbusr
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: cbprd.002853
Read_Master_Log_Pos: 6225791
Relay_Log_File: proddb-slav01-relay-bin.425989
Relay_Log_Pos: 6225933
Relay_Master_Log_File: cbprd.002853
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 6225791
Relay_Log_Space: 6226136
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 1
1 row in set (0.00 sec)
ERROR:
No query specified
mysql> Bye
[expprodl@proddb-slav01 ~]$ ps -ef | grep tomcat
expprodl 16893 1 0 Apr02 ? 03:52:45 /usr/bin/java -Djava.util.logging.config.file=/opt/tomcat6/conf/logging.properties -Dsolr.solr.home=/opt/solr -server -Xms256m -Xmx1024m -javaagent:/home/expprodl/newrelic//newrelic.jar -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/opt/tomcat6/endorsed -classpath /opt/tomcat6/bin/bootstrap.jar -Dcatalina.base=/opt/tomcat6 -Dcatalina.home=/opt/tomcat6 -Djava.io.tmpdir=/opt/tomcat6/temp org.apache.catalina.startup.Bootstrap start
if not running execute below cmd as expprodl
cd /home/expprodl/bin
./tomcat_sss.sh start
[expprodl@proddb-slav01 ~]$ ps -ef | grep gluster
expprodl 20451 20404 0 05:10 pts/0 00:00:00 grep gluster
root 22132 1 0 2015 ? 11:05:23 /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid
root 27394 1 0 2015 ? 05:34:58 /usr/sbin/glusterfsd -s 10.166.152.13 --volfile-id sheets.10.166.152.13.data00-sheets -p /var/lib/glusterd/vols/sheets/run/10.166.152.13-data00-sheets.pid -S /var/run/9064c4a2a03845f64b496e46daaf8f26.socket --brick-name /data00/sheets -l /var/log/glusterfs/bricks/data00-sheets.log --xlator-option *-posix.glusterd-uuid=4e5d8979-1d07-43b2-ab10-aaf65fde6e1c --brick-port 49152 --xlator-option sheets-server.listen-port=49152
root 27399 1 0 2015 ? 00:06:30 /usr/sbin/glusterfsd -s 10.166.152.13 --volfile-id site_backgrounds.10.166.152.13.data00-site_backgrounds -p /var/lib/glusterd/vols/site_backgrounds/run/10.166.152.13-data00-site_backgrounds.pid -S /var/run/a65bd4874ee31ec2ae56a8126955515a.socket --brick-name /data00/site_backgrounds -l /var/log/glusterfs/bricks/data00-site_backgrounds.log --xlator-option *-posix.glusterd-uuid=4e5d8979-1d07-43b2-ab10-aaf65fde6e1c --brick-port 49153 --xlator-option site_backgrounds-server.listen-port=49153
root 27405 1 0 2015 ? 00:10:14 /usr/sbin/glusterfsd -s 10.166.152.13 --volfile-id styles.10.166.152.13.data00-styles -p /var/lib/glusterd/vols/styles/run/10.166.152.13-data00-styles.pid -S /var/run/90bd16b1705fa8425e2c7efbea04217a.socket --brick-name /data00/styles -l /var/log/glusterfs/bricks/data00-styles.log --xlator-option *-posix.glusterd-uuid=4e5d8979-1d07-43b2-ab10-aaf65fde6e1c --brick-port 49154 --xlator-option styles-server.listen-port=49154
root 27411 1 0 2015 ? 1-02:27:06 /usr/sbin/glusterfsd -s 10.166.152.13 --volfile-id asset-volume.10.166.152.13.data00-assets -p /var/lib/glusterd/vols/asset-volume/run/10.166.152.13-data00-assets.pid -S /var/run/312cd3ada9ce9963ef25c05fe652aba3.socket --brick-name /data00/assets -l /var/log/glusterfs/bricks/data00-assets.log --xlator-option *-posix.glusterd-uuid=4e5d8979-1d07-43b2-ab10-aaf65fde6e1c --brick-port 49155 --xlator-option asset-volume-server.listen-port=49155
root 27419 1 0 2015 ? 03:27:41 /usr/sbin/glusterfsd -s 10-166-152-13 --volfile-id images-volume.10-166-152-13.data00-images -p /var/lib/glusterd/vols/images-volume/run/10-166-152-13-data00-images.pid -S /var/run/c8a2501be7c1f6d34543af6c2ea25239.socket --brick-name /data00/images -l /var/log/glusterfs/bricks/data00-images.log --xlator-option *-posix.glusterd-uuid=4e5d8979-1d07-43b2-ab10-aaf65fde6e1c --brick-port 49156 --xlator-option images-volume-server.listen-port=49156
root 27424 1 0 2015 ? 00:06:25 /usr/sbin/glusterfsd -s 10.166.152.13 --volfile-id site_logos.10.166.152.13.data00-site_logos -p /var/lib/glusterd/vols/site_logos/run/10.166.152.13-data00-site_logos.pid -S /var/run/359c68c467c3175497488f31a4375fd2.socket --brick-name /data00/site_logos -l /var/log/glusterfs/bricks/data00-site_logos.log --xlator-option *-posix.glusterd-uuid=4e5d8979-1d07-43b2-ab10-aaf65fde6e1c --brick-port 49157 --xlator-option site_logos-server.listen-port=49157
root 27430 1 0 2015 ? 00:23:38 /usr/sbin/glusterfsd -s 10.166.152.13 --volfile-id templates.10.166.152.13.data00-templates -p /var/lib/glusterd/vols/templates/run/10.166.152.13-data00-templates.pid -S /var/run/1cf76f18f254abe54fd0977d2e55eb00.socket --brick-name /data00/templates -l /var/log/glusterfs/bricks/data00-templates.log --xlator-option *-posix.glusterd-uuid=4e5d8979-1d07-43b2-ab10-aaf65fde6e1c --brick-port 49158 --xlator-option templates-server.listen-port=49158
root 27434 1 0 2015 ? 10:07:21 /usr/sbin/glusterfsd -s 10.166.152.13 --volfile-id data-volume.10.166.152.13.data00-sharedpub -p /var/lib/glusterd/vols/data-volume/run/10.166.152.13-data00-sharedpub.pid -S /var/run/83f6e4c6ed8eadfc0c038935cc1be369.socket --brick-name /data00/sharedpub -l /var/log/glusterfs/bricks/data00-sharedpub.log --xlator-option *-posix.glusterd-uuid=4e5d8979-1d07-43b2-ab10-aaf65fde6e1c --brick-port 49159 --xlator-option data-volume-server.listen-port=49159
Login as root
[root@proddb-slav01 ~]# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
=== Prod Mysql Master 01 ===
[expprodl@proddb-mast01 ~]$ ps -ef | grep mysql
root 16491 1 0 2016 ? 00:00:00 /bin/sh /usr/bin/mysqld_safe --datadir=/data00/mysql/datadir --pid-file=/data00/mysql/datadir/proddb-mast01.pid
mysql 16970 16491 0 2016 ? 1-14:55:02 /usr/sbin/mysqld --basedir=/usr --datadir=/data00/mysql/datadir --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/data00/mysql/datadir/proddb-mast01.err --pid-file=/data00/mysql/datadir/proddb-mast01.pid --socket=/var/lib/mysql/mysql.sock --port=3306
login as root
[root@proddb-mast01 ~]# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
== prodcollab01 ==
expbbbp@prodcollab01:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/volGroup00-rootvol00
48G 11G 35G 24% /
none 7.9G 204K 7.9G 1% /dev
none 7.9G 0 7.9G 0% /dev/shm
none 7.9G 144K 7.9G 1% /var/run
none 7.9G 0 7.9G 0% /var/lock
none 7.9G 0 7.9G 0% /lib/init/rw
none 48G 11G 35G 24% /var/lib/ureadahead/debugfs
/dev/mapper/streamsGroup00-streamsvol00
99G 59G 35G 64% /opt/streams
/dev/sda1 184M 57M 119M 33% /boot
/dev/mapper/streamsGroup00-streamsvol00
99G 59G 35G 64% /usr/share/red5/webapps/video/streams
expbbbp@prodcollab01:~$ ps -ef | grep gluster
root 18853 1 0 2016 ? 00:33:27 /usr/local/sbin/glusterfsd -s prodcollab01.learninguniver.se --volfile-id streams.prodcollab01.learninguniver.se.opt-streams -p /var/lib/glusterd/vols/streams/run/prodcollab01.learninguniver.se-opt-streams.pid -S /var/run/033a80ca8021852866d620b0be1153d3.socket --brick-name /opt/streams -l /usr/local/var/log/glusterfs/bricks/opt-streams.log --xlator-option *-posix.glusterd-uuid=811b8797-a3d7-4a7b-a881-7e80fb821895 --brick-port 49153 --xlator-option streams-server.listen-port=49153
expbbbp 26007 25956 0 05:24 pts/0 00:00:00 grep --color=auto gluster
root 28365 1 0 Apr02 ? 00:04:57 /usr/local/sbin/glusterd -p /var/run/glusterd.pid
root 28664 1 0 Apr02 ? 00:03:10 /usr/local/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /usr/local/var/log/glusterfs/nfs.log -S /var/run/98738654fa83300a661af7775c4d4df9.socket
expbbbp@prodcollab01:~$ sudo su -
[sudo] password for expbbbp:
root@prodcollab01:~# bbb-conf --check
BigBlueButton Server 0.81 (1554)
Kernel version: 2.6.32-58-preempt
Distribution: Ubuntu 10.04.4 LTS (64-bit)
Memory: 16084 MB
/var/www/bigbluebutton/client/conf/config.xml (bbb-client)
Port test (tunnel): prodcollab01.learnexa.com
Red5: prodcollab01.learnexa.com
/etc/nginx/sites-available/bigbluebutton (nginx)
server name: prodcollab01.learnexa.com
port: 80
bbb-client dir: /home/expbbbp/dev/bigbluebutton/bigbluebutton-client
/var/lib/tomcat6/webapps/bigbluebutton/WEB-INF/classes/bigbluebutton.properties (bbb-web)
bbb-web host: prodcollab01.learnexa.com
/var/lib/tomcat6/webapps/demo/bbb_api_conf.jsp (API demos)
api url: prodcollab01.learnexa.com
/usr/share/red5/webapps/bigbluebutton/WEB-INF/red5-web.xml (red5)
voice conference: FreeSWITCH
capture video: true
capture desktop: true
/usr/local/bigbluebutton/core/scripts/bigbluebutton.yml (record and playback)
playback host: prodcollab01.learnexa.com
** Potential problems described below **
# The following properties in /var/lib/tomcat6/webapps/bigbluebutton/WEB-INF/classes/bigbluebutton.properties have no value:
# defaultWelcomeMessage
defaultWelcomeMessageFooter
# IP does not match:
# IP from ifconfig: 10.166.152.17
# /etc/nginx/sites-available/bigbluebutton: prodcollab01.learnexa.com
# Warning: API URL IPs do not match host:
#
# IP from ifconfig: 10.166.152.17
# /var/lib/tomcat6/webapps/demo/bbb_api_conf.jsp: prodcollab01.learnexa.com
# Error: The voice application failed to register with the sip server.
# Try running:
#
# sudo bbb-conf --clean
#
# Warning: The API demos are installed and accessible from:
#
# http://prodcollab01.learnexa.com/
#
# These API demos allow anyone to access your server without authentication
# to create/manage meetings and recordings. They are for testing purposes only.
# If you are running a production system, remove them by running:
#
# sudo apt-get purge bbb-demo
# Error: Unable to reach default URL for presentation:
#
# http://prodcollab01.learnexa.com/DefaultMockPresentationXYZ.pdf
#
# Check value for beans.presentationService.defaultUploadedPresentation in
# /var/lib/tomcat6/webapps/bigbluebutton/WEB-INF/classes/bigbluebutton.properties
root@prodcollab01:~# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
== loadbalancer ==
login as expprodl
cd bin
[expprodl@10-166-152-16 bin]$ ./chat_sss.sh status
expprodl 3569 1 0 Aug02 ? 00:01:41 node server.js --port 8010
expprodl 3570 1 0 Aug02 ? 00:00:26 ./redis-server redis.conf
if not running execute below cmd as expprodl
./chat_sss.sh start
[expprodl@10-166-152-16 ~]$ ps -ef | grep haproxy
haproxy 7781 1 0 Aug04 ? 00:02:14 /usr/local/sbin/haproxy -D -f /opt/haproxy/haproxy.cfg -p /var/run/haproxy.pid
if not running execute below cmd as root
/etc/init.d/haproxy start
make sure iptables service on