Introduction

Exchange 2013 maintains a detailed record of messages sent between the transport services within an Exchange organization via message tracking logs.

The default location for these logs is; C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\MessageTracking.

Exchange generates 3 main log files (there is a 4th, but its use is negligible):

MSGTRKMSyyyymmddhh-nnnn.log; traffic events (sent messages)
MSGTRKMDyyyymmddhh-nnnn.log; traffic events (received messgaes)
MSGTRKyyyymmddhh-nnnn.log; Transport service events (message flow)

These files are in CSV (comma-separated value) format, making for easy parsing by Logstash. Your ELK server can be used to analyse message activity trends, such as which users are sending the most emails, to whom they're sending them and with what frequency. You can also determine the total volume of messages received and sent over a period, average size of message, total message volume over a period by user, and much more.

More detailed information regarding Exchange Server 2013 message tracking logs can be found here

Setup

Enable Message Tracking in Exchange 2013

Message tracking in Exchange 2013 should be enabled by default. If it's not, you can use either the Exchange Admin Centre (EAC) or the Exchange Management Shell (EMS) to enable/configure it.

EAC

In the EAC, navigate to Servers > Servers.

Select the Mailbox server you want to configure, and then click Edit

On the server properties page, click Transport Logs.

Make sure the Enable message tracking log checkbox is checked.

Click Save.

EMS

Start the EMS and run the following command:

Set-TransportService <ServerIdentity> -MessageTrackingLogEnabled <$true | $false> -MessageTrackingLogMaxAge <dd.hh:mm:ss> -MessageTrackingLogMaxDirectorySize <Size> -MessageTrackingLogMaxFileSize <Size> -MessageTrackingLogPath <LocalFilePath> -MessageTrackingLogSubjectLoggingEnabled <$true|$false>

e.g.

Set-TransportService CASMBX01 -MessageTrackingLogPath "C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\MessageTracking" -MessageTrackingLogMaxFileSize 20MB -MessageTrackingLogMaxDirectorySize 1.5GB -MessageTrackingLogMaxAge 45.00:00:00

Sets the location of the message tracking log files to C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\MessageTracking. Note that if the folder doesn't exist, it will be created for you.

Sets the maximum size of a message tracking log file to 20 MB.

Sets the maximum size of the message tracking log directory to 1.5 GB.

Sets the maximum age of a message tracking log file to 45 days.

Configure Logstash to parse Exchange 2013 message tracking logs

On your ELK server, add the following input & filter to your logstash.conf file in the /etc/logstash/conf.d/ configuration directory, or in separate config files (depending on your setup) e.g. 01-inputs.conf & 12-exchange_msg_trk.conf.

input

#udp syslogs stream via 5141
input {
  udp {
    type => "Exchange"
    port => 5141
  }
}

filter

filter {
  if [type] == "Exchange" {
	csv {
            add_tag => [ 'exh_msg_trk' ]
            columns => ['logdate', 'client_ip', 'client_hostname',  'server_ip', 'server_hostname', 'source_context', 'connector_id', 'source', 'event_id', 'internal_message_id', 'message_id', 'network_message_id', 'recipient_address', 'recipient_status', 'total_bytes', 'recipient_count', 'related_recipient_address', 'reference', 'message_subject', 'sender_address', 'return_path', 'message_info', 'directionality', 'tenant_id', 'original_client_ip', 'original_server_ip', 'custom_data']
	    remove_field => [ "logdate" ]
	    }
	grok {      
        match => [ "message", "%{TIMESTAMP_ISO8601:timestamp}" ]
	    }
	mutate {
    	convert => [ "total_bytes", "integer" ]
	    convert => [ "recipient_count", "integer" ]
	    split => ["recipient_address", ";"]
	    split => [ "source_context", ";" ]
	    split => [ "custom_data", ";" ]
  	    }
	date {
        match => [ "timestamp", "ISO8601" ]
        timezone => "Europe/London"
	    remove_field => [ "timestamp" ]
	    }
	if "_grokparsefailure" in [tags] {
	      drop { }
	    }
	}
}

output

output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
}

Run the following command to test the validity of your configuration:

# /opt/logstash/bin/logstash --configtest -f /etc/logstash/conf.d/logstash.conf

Once you get a 'Configuration OK' message. Restart the Logstash service for the configuration to take effect.

Install & Configure NXLog on your Exchange Server

NXLog is a popular open source log management tool for collecting and forwarding logs from Windows (as well as GNU/Linux) platforms. It also happens to be dead simple to install and configure.

Logon to your Exchange server as Administrator. Next, download and run the latest version of the NXLog installer. Follow through the on screen prompts.

Open the configuration file C:\Program Files (x86)\nxlog\conf\nxlog.conf, (or on 64bit installs C:\Program Files\nxlog\conf\nxlog.conf).

Edit your nxlog.conf to look like the following:

## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html

## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension syslog>
    Module      xm_syslog
</Extension>

define BASEDIR C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\MessageTracking

<Input in_exchange>
   Module     im_file
   File       '%BASEDIR%\MSGTRK????????*-*.LOG' # Exports all logs in Directory
   SavePos    TRUE
   Exec       if $raw_event =~ /HealthMailbox/ drop();
   Exec       if $raw_event =~ /^#/ drop();
</Input>

<Output out_exchange>
    Module    om_udp
    Host      192.168.0.2 # Replace with your Logstash hostname/IP
    Port      5141        # Replace with your desired port
    Exec      $SyslogFacilityValue = 2;
    Exec      $SourceName = 'exchange_msgtrk_log';
    Exec      to_syslog_bsd();
</Output>

<Route exchange>
    Path      in_exchange => out_exchange
</Route>

N.B. Replace the Host IP and Port value with those you configured on your Logstash server earlier.

Exchange uses the health mailboxes to establish that email connectivity exists to the various databases in the system by sending artificial messages to and from the mailboxes every five minutes or so. For my use case, I had no need to export the message tracking logs associated with these health messages. The line Exec if $raw_event =~ /HealthMailbox/ drop(); drops these log entries from those being exported to Logstash. You can of course comment out or remove this line if you require these entries for your own analysis.

In Powershell or a Command Prompt run net start nxlog to start NXLog.

Kibana

Once your logs are successfully flowing to your logstash server, you can use queries and filters in Kibana to create panels like these:

Message Volumes

Top Senders & Message Percentage Breakdown

Link to Exchange message tracking Dashboard; Gist: 4b9cd98715c0ba2a75de

If using my dashboard;

1. In the Gist, replace my (Outbound) Send Connector name ('Outbound Internet Mail') with that of your own (or at least with that of the Send Connector you wish to analyse).

You can find the name of your Send Connector via the EAC; Click mail flow > send connectors and you'll see your Send Connectors name listed in the table. Alternatively, you can run Get-SendConnector in the EMS. Your send connector name(s) will be listed under the Identity column.

2. You may have to adjust the default pinned queries depending on how you differentiate between internal and external communication in your Exchange organization setup.

Sources:

Configure Message Tracking: http://technet.microsoft.com/en-us/library/aa997984(v=exchg.150).aspx

Spot any mistakes? Or have any suggestions? Please make a comment below.

Share this post