Microsoft Exchange servers store different types of logs. These log types include message tracking, Exchange Web Services (EWS), Internet Information Services (IIS), and application/system event logs. With Exchange servers deployed on a global scale, logs are often scattered in multiple directories that are local to these servers. This requires Exchange administrators to log into each server to monitor status, health, and events. Centralizing these logs and converting them into useful metrics allows Exchange administrators to identify a majority of issues, like high load or service/application errors without logging into each server.

This blog post discusses an efficient architecture to stream, analyze, and store Microsoft Exchange Server logs. For frequent queries and operational analytics, we use Amazon Elasticsearch Service (Amazon ES) and Kibana for real-time visualization. For example, you can provide various types of reports. These reports can be top email senders and recipients, top HTTP status codes in IIS logs, top error codes in EWS logs, and narrow down spikes in load/errors. For infrequent queries such as audit, legal and compliance requirements, we use Amazon S3 as the final destination. It provides low-cost storage options and high durability and Amazon Athena for simple queries using standard SQL.

Amazon Kinesis Agent for Microsoft windows (Kinesis Agent for Windows) is a highly configurable and extensible agent. Kinesis Agent for Windows gathers, parses, transforms, and streams logs, events, and metrics to various AWS services, including Amazon Kinesis Data Streams , Amazon Kinesis Data Firehose , and Amazon CloudWatch . It helps to make it more efficient and reliable to centralize logs from Windows-based services. This lets you see the extent of an issue, monitor those issues, and generate an alarm when errors or loads breach certain thresholds. For more information about Kinesis Agent for Windows, see What is Amazon Kinesis Agent for Microsoft Windows?

Parsing logs

Amazon ES requires JSON-formatted data. Kinesis Agent for Windows efficiently parses the Exchange logs lines in CSV format and converts them to JSON. You can enrich the data by using Kinesis Agent for Windows by adding details, such as the hostname, EC2 instance ID, and custom date and time formats to help pinpoint the exact issue reported in the logs. Kinesis Agent for Windows dynamically computes the log header. It does this even if the header names are changed, or if there are multiple header lines in a single log file because of a service restart.It streams the right data even when logs are rotated.

The log flow

In this use case, we send the same log to both Amazon ES for real-time analytics and to Amazon S3 for offline analytics with Amazon Athena. Instead of streaming the data twice from the host to each destination, you can configure Kinesis Agent for Windows to stream it once to a Kinesis data stream. From the stream, Amazon Kinesis Data Firehose gathers logs and delivers them to Amazon ES. Another Kinesis Data Firehose gathers the same logs and delivers them to an Amazon S3 bucket for Amazon Athena. If there is a need to send logs to another destination, we can use another Kinesis Data Firehose instance.

Manage centralized Microsoft Exchange Server logs using Amazon Kinesis Agent for ...

AWS Lambda periodically analyzes the logs in Amazon ES and post statistics to CloudWatch metrics. CloudWatch alarms are used to trigger on the anomalies detected in the posted metrics.

Kibana visualizes the log data. By looking at spikes and anomalies in the graphs, we can drill down into specific log data. That helps us diagnose specific problems with the Exchange service. Several authentication features protect access to Kibana. For information about using Amazon Cognito with an identity provider, see Amazon Cognito Authentication for Kibana .

Agent configuration

Kinesis Agent for Windows configurations are described in the appsettings.json located at %PROGRAMFILES%\Amazon\AWSKinesisTap\ path. It is here where we define the sources (log location), sinks (Kinesis Data Stream information), and pipes, which connect the source and sinks.

The following is an example source configuration that queries all files with a .log extension under the specified directory. When ExchangeLogSource is set as the type, it dynamically parses the log lines for the header. It then automatically picks up the column that is needed for ‘Time Stamp’.

"Sources": [ { "Id": "MessageTracking-LogsSource", "SourceType": "ExchangeLogSource", "Directory": "C:\\Program Files\\Microsoft\\Exchange Server\\V15\\TransportRoles\\Logs\\MessageTracking", "FileNameFilter": "*.log", "TimeZoneKind": "UTC", "TimeStampField": "date-time" //Optional. ExchangeLogSource can automatically detect if the TimestampField name is "date-time" or "DateTime". For other names, please specify } ]

Message tracking logs are similar to the following sample.

2018-10-22T10:53:13.404Z,, ExchangeServer01,,ExchangeServer01,;250 2.0.0 OK;ClientSubmitTime:2018-10-22T10:53:10.680Z,Intra-Organization SMTP Send Connector,SMTP,SEND,157882997807893,<[emailprotected]>,9b3f4489-a158-4126-0d41-08d6380c8f0f,[emailprotected],250 2.1.5 Recipient OK,[emailprotected], [emailprotected] ,…

// Sinks (Destinations) define where the logs go

Next, we define the sinks or destination where the logs go. We can also stream logs to a Kinesis Data Stream in another AWS account by assuming the role that has access to the stream. For information about how to set up access, see Sink Security Configuration . Logs are converted to JSON when the Format is specified.

"Sinks": [ { "Id": "MessageTracking-Kinesis-Sink", "SinkType": "KinesisStream", "Region": "us-west-2", "RoleARN": "arn:aws:iam::<another aws account>::role/exch-kinesis-log", // only if logs are sent to Kinesis Data Stream in another account. "StreamName": "ex-messagetracking", "Format": "json" } ] // Pipes, connects sources a

本文系统(windows)相关术语:三级网络技术 计算机三级网络技术 网络技术基础 计算机网络技术

本文标题:Manage centralized Microsoft Exchange Server logs using Amazon Kinesis Agent for ...

技术大类 技术大类 | 系统(windows) | 评论(0) | 阅读(127)