Blog about things

blogging experiment about hacking

Sending Logs to Centralized Logging Services

Sending log entries to hosted logging services (from a linux)

Below are some instructions on forwarding log entries from a server to a centralized logging service.

There are many advantages to such an arrangement, including

  • access to logs from multiple related services in a single location.
  • access control where different groups can see different views.
  • search apis - query log records by indexed fields, time ranges, etc.
  • built-in reporting, filtering, etc on web pages and queries via API.
  • in virtualized and cloud environments where servers may come and go on-demand the log entries are captured even if the server and it’s volumes have gone away.

Centralized logging services, or Logging as a Service (LaaS), accept streams of log entries over the network, index them, and make them available to you.

We are going to utilize two LaaS providers in these examples: Loggly and Logsene.

Loggly

Create a (free) loggly account from https://www.loggly.com/ if you don’t have one yet.

rsyslog

Loggly has a script to automate the initial process: https://okcra.loggly.com/sources/setup/linux. There are also links on manually configuring rsyslog or syslog-ng.

Run the following commands and then verify you’ve got some new test log entries on loggly’s web site. Replace the parameters values with those for your loggly account.

1
2

curl -O https://www.loggly.com/install/configure-linux.sh
sudo bash configure-linux.sh -a <loggly domain name> -t <loggly token> -u <loggly user id>
apache httpd server

Now, we set up apache httpd server log file watches in rsyslog to forward web access and error logs to loggly.

Loggly has a script and manual instructions at https://okcra.loggly.com/sources/setup/apache We’re going to use manual instructions since our log files have names other than the defaults.

The instance I’m using is amazon linux which already has /var/spool/rsyslog. We’ll change ownership of the directory from root:root to syslog:adm as the instructions suggest. As you will see in the configuration we instruct the rsyslog daemon to use /var/spool/rsyslog as it’s work directory and drop to privileges of group adm so we need to give that group write permissions to the work directory.

The install script create we ran earlier created /etc/rsyslog.d/22-loggly.conf which configures rsyslog to forward all messages to logs-01.loggly.com via TCP port 514 using the defined format LogglyFormat.

1
2
3
4
5
6
7
8
9
10
11
12
13

#          -------------------------------------------------------
#          Syslog Logging Directives for Loggly (okcra.loggly.com)
#          -------------------------------------------------------

# Define the template used for sending logs to Loggly. Do not change this format.
$template LogglyFormat,"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [92aff88c-05a2-480e-b2f4-4d2e4fb395e7@41058] %msg%"

# Send messages to Loggly over TCP using the template.
*.*             @@logs-01.loggly.com:514;LogglyFormat

#          -------------------------------------------------------
#          End of Syslog Logging Directives for Loggly
#          -------------------------------------------------------

We’ll create a new file named /etc/rsyslog.d/23-apache-loggly.conf for our apache-related loggly config.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

$ModLoad imfile
$InputFilePollInterval 10 
$PrivDropToGroup adm
$WorkDirectory /var/spool/rsyslog

# Apache access file:
$InputFileName /var/log/httpd/api_okcra_access_log
$InputFileTag apache-access:
$InputFileStateFile stat-apache-access
$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

# Apache SSL access file:
$InputFileName /var/log/httpd/api_okcra_ssl_access_log
$InputFileTag apache-ssl-access:
$InputFileStateFile stat-apache-ssl-access
$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

#Apache Error file: 
$InputFileName /var/log/httpd/error.log
$InputFileTag apache-error:
$InputFileStateFile stat-apache-error
$InputFileSeverity error
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

#Add a tag for apache events
$template LogglyFormatApache,"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [92aff88c-05a2-480e-b2f4-4d2e4fb395e7@41058 tag=\"apache\"] %msg%\n"

if $programname == 'apache-access' then @@logs-01.loggly.com:514;LogglyFormatApache
if $programname == 'apache-access' then ~
if $programname == 'apache-ssl-access' then @@logs-01.loggly.com:514;LogglyFormatApache
if $programname == 'apache-ssl-access' then ~
if $programname == 'apache-error' then @@logs-01.loggly.com:514;LogglyFormatApache
if $programname == 'apache-error' then ~
also send logs to logsene from rsyslog

Create a (free) account from https://www.sematext.com/logsene if you don’t have one yet.

Create another rsyslog config file, say /etc/rsyslog.d/25-logsene.conf. You’ll need to add your logsene app token in the config file.

1
2

$template LogseneFormat,"<%PRI%>%TIMEREPORTED:::date-rfc3339% %HOSTNAME% %syslogtag%@cee: {\"logsene-app-token\": \"YOUR-LOGSENE-APP-TOKEN-GOES-HERE\", \"message\": \"%msg:::json%\"}\n"
*.* @@(o)logsene-receiver-syslog.sematext.com;LogseneFormat

Next, recycle the rsyslog daemon to get config changes:

1

sudo service rsyslog restart

Logstash - log parser, filter, shipper

sample install on redhat-style server

1
2

wget https://download.elasticsearch.org/logstash/logstash/packages/centos/logstash-1.4.2-1_2c0f5a1.noarch.rpm
sudo rpm -ivh logstash-1.4.2-1_2c0f5a1.noarch.rpm

create config file(s) e.g. /etc/logstash/conf.d/logstash.conf

/etc/logstash/conf.d/logstash.conf 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

input {
  file {
    path => "/var/log/httpd/api_okcra_ssl_access_log"
    type => "apache-access"  # a type to identify those logs (will need this later)
    start_position => "beginning"
  }
}

filter {
  if [type] == "apache-access" {   # this is where we use the type from the input section
    grok {
      # ensure the matching logs use LogFormat combined for this parser
      match => [ "message", "%{COMBINEDAPACHELOG}" ]
    }
  }
}

output {
  # this is for logsene elasticsearch format
  elasticsearch {
    host => "logsene-receiver.sematext.com"
    port => 80
    index => "YOUR_LOGSENE_APP_TOKEN_GOES_HERE"
    protocol => "http"
    manage_template => false
  }
}

Troubleshooting

Debugging rsyslog

Stop your syslog daemon and run it in the foreground with debugging turned on, printing to stdout. These commands worked on my particular system - ymmv.

1
2

sudo system rsyslog stop
RSYSLOG_DEBUG="Debug" sudo /sbin/rsyslogd -d -n -c5

Verify network connections to logger endpoints

1
2
3

sudo netstat -taupn | grep syslog
tcp        0      0 10.65.29.118:56116          54.235.102.246:514          ESTABLISHED 5197/rsyslogd
tcp        0      0 10.65.29.118:35319          54.236.68.122:514           ESTABLISHED 5197/rsyslogd

Trace IP packets to and from an endpoint

1

sudo tcpdump -nnvvXS dst 54.235.102.246