Introduction

Exchange 2013 maintains a detailed record of messages sent between the transport services within an Exchange organization via message tracking logs.

The default location for these logs is; C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\MessageTracking.

Exchange generates 3 main log files (there is a 4th, but its use is negligible):

• MSGTRKMSyyyymmddhh-nnnn.log; traffic events (sent messages)
• MSGTRKMDyyyymmddhh-nnnn.log; traffic events (received messgaes)
• MSGTRKyyyymmddhh-nnnn.log; Transport service events (message flow)

More detailed information regarding Exchange Server 2013 message tracking logs can be found here

Setup

Enable Message Tracking in Exchange 2013

Message tracking in Exchange 2013 should be enabled by default. If it's not, you can use either the Exchange Admin Centre (EAC) or the Exchange Management Shell (EMS) to enable/configure it.

EAC

In the EAC, navigate to Servers > Servers.

Select the Mailbox server you want to configure, and then click Edit

On the server properties page, click Transport Logs.

Make sure the Enable message tracking log checkbox is checked.

Click Save.

EMS

Start the EMS and run the following command:

Set-TransportService <ServerIdentity> -MessageTrackingLogEnabled <$true | $false> -MessageTrackingLogMaxAge <dd.hh:mm:ss> -MessageTrackingLogMaxDirectorySize <Size> -MessageTrackingLogMaxFileSize <Size> -MessageTrackingLogPath <LocalFilePath> -MessageTrackingLogSubjectLoggingEnabled <$true|$false>

e.g.

Set-TransportService CASMBX01 -MessageTrackingLogPath "C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\MessageTracking" -MessageTrackingLogMaxFileSize 20MB -MessageTrackingLogMaxDirectorySize 1.5GB -MessageTrackingLogMaxAge 45.00:00:00

Sets the location of the message tracking log files to C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\MessageTracking. Note that if the folder doesn't exist, it will be created for you.

Sets the maximum size of a message tracking log file to 20 MB.

Sets the maximum size of the message tracking log directory to 1.5 GB.

Sets the maximum age of a message tracking log file to 45 days.

Configure Logstash to parse Exchange 2013 message tracking logs

On your ELK server, add the following input & filter to your logstash.conf file in the /etc/logstash/conf.d/ configuration directory, or in separate config files (depending on your setup) e.g. 01-inputs.conf & 12-exchange_msg_trk.conf.

input

#udp syslogs stream via 5141
input {
  udp {
    type => "Exchange"
    port => 5141
  }
}

filter

filter {
  if [type] == "Exchange" {
	csv {
            add_tag => [ 'exh_msg_trk' ]
            columns => ['logdate', 'client_ip', 'client_hostname',  'server_ip', 'server_hostname', 'source_context', 'connector_id', 'source', 'event_id', 'internal_message_id', 'message_id', 'network_message_id', 'recipient_address', 'recipient_status', 'total_bytes', 'recipient_count', 'related_recipient_address', 'reference', 'message_subject', 'sender_address', 'return_path', 'message_info', 'directionality', 'tenant_id', 'original_client_ip', 'original_server_ip', 'custom_data']
	    remove_field => [ "logdate" ]
	    }
	grok {      
        match => [ "message", "%{TIMESTAMP_ISO8601:timestamp}" ]
	    }
	mutate {
    	convert => [ "total_bytes", "integer" ]
	    convert => [ "recipient_count", "integer" ]
	    split => ["recipient_address", ";"]
	    split => [ "source_context", ";" ]
	    split => [ "custom_data", ";" ]
  	    }
	date {
        match => [ "timestamp", "ISO8601" ]
        timezone => "Europe/London"
	    remove_field => [ "timestamp" ]
	    }
	if "_grokparsefailure" in [tags] {
	      drop { }
	    }
	}
}

output

output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
}

Run the following command to test the validity of your configuration:

# /opt/logstash/bin/logstash --configtest -f /etc/logstash/conf.d/logstash.conf

Once you get a 'Configuration OK' message. Restart the Logstash service for the configuration to take effect.

Install & Configure NXLog on your Exchange Server

NXLog is a popular open source log management tool for collecting and forwarding logs from Windows (as well as GNU/Linux) platforms. It also happens to be dead simple to install and configure.

1. Logon to your Exchange server as Administrator. Next, download and run the latest version of the NXLog installer. Follow through the on screen prompts.


2. Open the configuration file C:\Program Files (x86)\nxlog\conf\nxlog.conf, (or on 64bit installs C:\Program Files\nxlog\conf\nxlog.conf).


3. Edit your nxlog.conf to look like the following:

   ## This is a sample configuration file. See the nxlog reference manual about the
   ## configuration options. It should be installed locally and is also available
   ## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html
   
   ## Please set the ROOT to the folder your nxlog was installed into,
   ## otherwise it will not start.
   
   #define ROOT C:\Program Files\nxlog
   define ROOT C:\Program Files (x86)\nxlog
   
   Moduledir %ROOT%\modules
   CacheDir %ROOT%\data
   Pidfile %ROOT%\data\nxlog.pid
   SpoolDir %ROOT%\data
   LogFile %ROOT%\data\nxlog.log
   
   <Extension syslog>
       Module      xm_syslog
   </Extension>
   
   define BASEDIR C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\MessageTracking
   
   <Input in_exchange>
      Module     im_file
      File       '%BASEDIR%\MSGTRK????????*-*.LOG' # Exports all logs in Directory
      SavePos    TRUE
      Exec       if $raw_event =~ /HealthMailbox/ drop();
      Exec       if $raw_event =~ /^#/ drop();
   </Input>
   
   <Output out_exchange>
       Module    om_udp
       Host      192.168.0.2 # Replace with your Logstash hostname/IP
       Port      5141        # Replace with your desired port
       Exec      $SyslogFacilityValue = 2;
       Exec      $SourceName = 'exchange_msgtrk_log';
       Exec      to_syslog_bsd();
   </Output>
   
   <Route exchange>
       Path      in_exchange => out_exchange
   </Route>
   

   N.B. Replace the Host IP and Port value with those you configured on your Logstash server earlier.
   
   

   Exchange uses the health mailboxes to establish that email connectivity exists to the various databases in the system by sending artificial messages to and from the mailboxes every five minutes or so. For my use case, I had no need to export the message tracking logs associated with these health messages. The line Exec if $raw_event =~ /HealthMailbox/ drop(); drops these log entries from those being exported to Logstash. You can of course comment out or remove this line if you require these entries for your own analysis.



4. In Powershell or a Command Prompt run net start nxlog to start NXLog.

Kibana

Once your logs are successfully flowing to your logstash server, you can use queries and filters in Kibana to create panels like these:

Message Volumes


Top Senders & Message Percentage Breakdown

Link to Exchange message tracking Dashboard; Gist: 4b9cd98715c0ba2a75de

If using my dashboard;

1. In the Gist, replace my (Outbound) Send Connector name ('Outbound Internet Mail') with that of your own (or at least with that of the Send Connector you wish to analyse).

You can find the name of your Send Connector via the EAC; Click mail flow > send connectors and you'll see your Send Connectors name listed in the table. Alternatively, you can run Get-SendConnector in the EMS. Your send connector name(s) will be listed under the Identity column.

2. You may have to adjust the default pinned queries depending on how you differentiate between internal and external communication in your Exchange organization setup.

Sources:

Configure Message Tracking: http://technet.microsoft.com/en-us/library/aa997984(v=exchg.150).aspx

Spot any mistakes? Or have any suggestions? Please make a comment below.