Log aggregation within CPM can be accomplished using the cpm-efk container.

The cpm-efk container includes:

elasticsearch - https://www.elastic.co/products/elasticsearch
fluentd - http://www.fluentd.org/
kibana - https://www.elastic.co/products/kibana

These tools when combined provide a capability similar to that of tools like splunk for log analysis and aggregation.

1.1. Starting cpm-efk

Some customization to the run-cpm-efk.sh script and rsyslog.conf files are required:

modify LOCAL_IP environment variable to use your local system’s IP address
modify EFKDATA environment variable to use a local directory where the cpm-efk container will persist it’s data
modify /var/cpm/conf/rsyslog.conf to specify your local IP address that cpm-efk will be listening on

Start up cpm-efk as follows:

cd images/cpm-efk
sudo ./run-cpm-efk.sh

This will start the cpm-efk container. When running, the cpm-efk container will listen to the following ports:

fluentd - 24224
fluentd-syslog - 5140
kibana-http - 5601

You can start using the kibana web interface at http://cpm-efk:5601

1.2. CPM Internal Logs

Log messages produced by the CPM administration containers (cpm-admin, cpm-task, etc.) are logged to stdout (except cpm-web) by default. We can view the stdout logs using the normal docker log command or, as of Docker 1.8, we can specify a Docker log driver to route the container’s stdout log output to a fluentd server (running inside the cpm-efk container).

In the startup scripts for CPM, run-cpm.sh, you will see that the Docker log driver is specified to use the fluentd driver by default:

--log-driver=fluentd \
--log-opt fluentd-address=192.168.0.107:24224 \
--log-opt fluentd-tag=docker.cpm-admin \

This will send the docker log output for the CPM containers to the fluentd server running at the specified address (e.g. 192.168.0.107:24224).

The exception to this is the cpm-web (nginx) container. Nginx has a configuration that requires you to send stdout and stderr to a file. There is a bug in Docker (1.8.2 and earlier versions), that prevents a non-root user write access to /dev/stdout and /dev/stderr. See https://github.com/docker/docker/issues/6880 for details on the bug.

So, until the next release of Docker, we will continue to send cpm-web output to log files mounted from the local host (e.g. /var/cpm/logs).

If you want to have the CPM product containers send their logs to the Docker log instead of using the cpm-efk and fluentd logging, just remove the log driver lines above from the run-cpm.sh startup script when you start the CPM product containers.

1.3. Postgres Container Logs

On each CPM provisioned Postgresql container, we have enabled Postgresql to send log output to both stdout and syslog.

We use syslog within each container a the mechanism to aggregate all the Postgresql logs into the cpm-efk container.

The use of syslog within each container is triggered by the presense of /syslogconfig/rsyslog.conf within each container. There is a volume mount that is used by the container to find the syslog configuration files:

-v /var/cpm/config:/syslogconfig

On each CPM server, to use the syslog feature, you will need a /var/cpm/config directory that contains both rsyslog.conf and listen.conf configuration files. If the configuration files are not present, the cpm-node containers will not use configure syslog logging to cpm-efk.

1.3.1. rsyslog Configuration

To support syslog logging within each Postgres container (cpm-node), the rsyslogd binary is installed within the cpm-node container.

The configuration of rsyslog within a container is tricky and is documented by Dan Walsh here:

http://www.projectatomic.io/blog/2014/09/running-syslog-within-a-docker-container/

The sample rsyslog.conf file causes the container’s rsyslog to forward syslog messages to a remote syslog server (e.g. 192.168.0.107:5140) which is served by the cpm-efk fluentd-syslog component:

$WorkDirectory /var/lib/rsyslog
$ActionQueueFileName fwdRule1
$ActionQueueMaxDiskSpace 2g
$ActionQueueSaveOnShutdown on
$ActionQueueType LinkedList
$ActionResumeRetryCount -1
*.* @@192.168.0.107:5140

Also, in /etc/rsyslog.d/listen.conf the line $SystemLogSocketName is commented out.

Sample rsyslog.conf and listen.conf files are stored in the github CPM_ROOT/images/cpm-efk/conf directory. These config files are copied to your local /var/cpm/config directory when you start up cpm-efk.

1.3.2. Postgres Configuration Changes

The Postgresql running inside each cpm-node container is configured to log to both standard log files and syslog.

This dual logging is accomplished in postgresql.conf as follows:

log_destination = 'stderr,syslog'
syslog_facility = 'LOCAL0'
syslog_ident = 'postgres'

1.4. Kibana

Kibana is the web console where you can view all the log messages, create dashboards, create queries, and create graphs. Here is a sample screenshot: