DokuWiki

To monitor application stack (Ex: Apache/Passenger/Haproxy/MySQL etc) logs as well as applications logs (Ex: /deploy/crossbow/shared/log).
We do have Zabbix monitoring resources and state.

A system wide Ruby gem.
It is light weight and its main job is to “tail” configured files and deliver data across to the remote endpoint.
Initial network overhead can be expected when it syncs to the current state but then its differential.

servolux-0.10.0
tins-0.13.2
file-tail-1.0.12
eventmachine-1.0.3
eventmachine-tail-0.6.4
syslog_protocol-0.9.2
em-resolv-replace-1.1.3
remote_syslog-1.6.14

Associated demon process binary as well as log tailing and protocol helper binaries are installed.

http://help.papertrailapp.com/kb/configuration/configuring-centralized-logging-from-text-log-files-in-unix#remote_syslog

Exact checklist to be prepared but highlevel:

1) Update rsyslog (if not on v7/8 stable)
2) Configure rsyslog for TLS delivery to paper tail
3) Install remote_syslog
4) configure remote_syslog to monitor interesting app log files
5) install remote_syslog to init.d
6) configure rsyslog to deliver sys logs to papertail
7) restart rsyslog and start remote_syslog
8) check if events appear on paper tail

Checklist: https://docs.google.com/a/expertus.com/spreadsheets/d/1b3HeKZZH3B5phIXVEBKO44ZnkIDLu0QIE6JCgF-LUX4/edit#gid=0

Q- where does he client program reside?
A- On the monitored server itself.
   A system wide gem is installed which in turn installs about 3 helper binaries along with the above mentioned dependency gems on the system. \\
Q- How is the program invoked?
A- The program is invoked and demonized at boot using servolux.
Q- How often are is the FS polled?
A- Existing files are monitored for new content via EventMachineReader which is extended from file-tail. \\ FT sets up blocking call to deadline and uses EventMachine to schedule that across multiple files.
New files are monitored at "interval" set. We should not set it too short (like in example).
new_file_check_interval:  # Check every 5 seconds
Q- What is a typical execution workflow?
A- BOOT-> init.d -> remote_syslog -> read & parse /etc/log_files.yml -> hooks to the files via EventMachine / EventMachineTail -> Assemble TCP/UDP packets for the new data and transport to paper trail. 
Q- Security of data (PII) during transport?
A- Transported over TLS.
Q- Security of data (PII) at rest?
A- Papertrail does not encrypt any data. However, they have the following lockdowns in place:
   There is no way to access any customer data via the interface; not even for admin interface.
   Only people with direct access to the data store have access to this data.
   In rare situations, where they might need to access customer data; permission is requested from the customer first. If the customer approves, 
   they are required to add "support@papertrail.com" as an authorized user with read permission.
   They will not release any data to any third party.
<del>http://help.papertrailapp.com/kb/how-it-works/why-should-i-trust-you. I reached out to them to get that cryptogram link.</del>