Audit Collection Services
I’m now working on designing the role of Audit Collection Services for our company. This blog post will be to show what Audit Collection Services is and what the best practices for implemting of this SCOM role are. So let’s get started
Audit Collection Services (ACS) is a high-performance, secure solution that collects and stores events from the Security Event Log on monitored computers. Events are stored in a separate database, the ACS database , in Microsoft SQL Server 2005 SP1 or later and Microsoft SQL Server 2008 SP1. ACS collects all events written to the Security Event Log on computers that the ACS Forwarder is enabled on. Events are forwarded from monitored computers to the ACS Collector, which runs on a management server, which then processes them and writes them to the ACS database. The events are transmitted in an encrypted, near real-time fashion from the forwarders to the collector. A separate component, ACS Reporting, is then used to generate reports from the stored ACS data. A key to using ACS effectively is the development of a sound Windows Audit Group Policy that is implemented as a domain Group Policy.
ACS Forwarder
The ACS Forwarder is embedded in the Operations Manager 2007 agent, so no separate deployment or configuration is required. The ACS Forwarder appears as the Audit Forwarder service and is disabled by default. The ACS Forwarder on an individual computer or on groups of computers is enabled via a task in the Operations console.
ACS Collector Server
The main purpose of the ACS Collector server is to collect, filter, and pre-process all the Windows security log events for insertion into the database. Because the ACS collects all security events in near real-time, vast amounts of data enters the system from the forwarders. Not all of this information will be of interest to your company, as defined in your company’s Windows Audit Group Policy. The filtering mechanism at the collector allows you to specify which events you want written to the ACS database for long-term storage.
The ACS Collector server has a separate installation program from the Operations Manager servers, agents, or reporting components. It can be installed only on an existing management server or RMS if you have not installed any additional management servers. One ACS Collector server can support hundreds to thousands of servers, depending on the server role and Windows Audit Group Policy, and tens of thousands of workstations. However, there is a one-to-one relationship between the ACS Collector server and the ACS Database (which is discussed in the next section). If for scalability or control reasons your company requires additional ACS Collectors, you will need one ACS Database per ACS Collector.
ACS Database
After the data has been pre-processed by the ACS Collector server, it is written to its ACS Database, which is just a database created on a Microsoft SQL Server 2005 SP1 or SP2 instance. Because it is a standard SQL database, it can be clustered for high-availability. To accommodate the one-to-one relationship between collectors and databases, you can create, via named instances, multiple ACS Databases on a single SQL Server 2005 server as long as it can support the additional load. For more information about sizing and capacity planning for ACS, see that section later in this guide.
ACS Reporting
The ACS Reporting server is also a separately installed component. A number of preconfigured reports are available. Installation of ACS Reporting requires an existing instance of SQL Server 2005 SP1 or later Reporting Services or Microsoft SQL Server 2008 Reporting Services. This can be a stand-alone instance, or you can install ACS Reporting along with Operations Manager 2007 Reporting with one tradeoff.
If you install ACS Reporting into the same Reporting Services instance as Operations Manager 2007 Reporting, ACS Reporting is fully integrated with Operations Manager Reporting. This results in reduced administrative overhead, because anyone who has been assigned to the Operations Manager Reporting role will have access to the ACS reports. Some companies might not find this to be a desirable configuration, and they might elect to install ACS Reporting into its own instance of SQL Server Reporting. In this case, you must define your own security groups and roles, resulting in higher administrative overhead but extremely tight control over access to ACS data.
It might be beneficial to implement a management group that exclusively supports the ACS function if your company’s security requirements mandate that the ACS function be controlled and administered by a separate administrative group other than that which administers the rest of the production environment.
Designing Audit Collection Services
ACS is not a stand-alone solution. ACS can be hosted only in an existing management group because its agent is integrated and installed with the Operations Manager agent, and the ACS Collector can be installed only on a management server or gateway server. The remaining components, the ACS database and ACS Reporting, can be installed on the same SQL Server 2005 server or instance as the rest of the OperationsManager database and reporting components as well. However, for performance, capacity, and security reasons, you will probably choose to install these on dedicated hardware.
Design Decisions
There are four fundamental design decisions to make when planning your ACS implementation. As you make these decisions, keep in mind that there is a one-to-one relationship between the ACS Collector server and its ACS database. An ACS database can have only one ACS Collector feeding data to it at a time, and every ACS Collector needs its own ACS database. It is possible to have multiple ACS Collector/Database pairs in a management group; however, there are no procedures available out of the box for integrating the data from multiple ACS databases into a single database.
The first decision that must be made is whether or not to deploy a management group that is exclusively used to support ACS or to deploy ACS into a management group that also provides health monitoring and alerting services. Here are the characteristics of these two ACS deployment scenarios.
- ACS hosted in a production management group scenario:
- Scaled usage of ACS—Given that ACS collects every security event from the systems that ACS Forwarders are enabled on, the use of ACS can generate a huge amount of data. Unless you are using dedicated hardware for the ACS Collector and Database roles, processing this data might negatively affect the performance of the hosting management group, particularly in the database layer.
- Separate administration and security is not required—Because ACS is hosted in a management group, people with administrative control in the management group will have administrative rights in ACS. If the business, regulator/audit, and IT requirements mandate that ACS be under nonproduction IT control, deploying ACS into a production management group scenario is not an option.
- ACS hosted on a dedicated management group scenario:
- Separate administration and security is required—If there is a separate administrative group that is responsible for audit and security controls at your company, hosting ACS on a dedicated management group administered by the audit/security group is recommended.
The second decision that must be made is whether or not to deploy ACS Reporting into the same SQL Server 2005 Reporting Services instance as the Operations Manager 2007 Reporting component. Here are the characteristics of these two scenarios.
- ACS reporting integrated with Operations Manager Reporting:
- Single console for all reports—When ACS Reporting is installed with Operations Manager Reporting, the ACS reports are accessed via the Operations Manager Operations console.
- Common security model—When Operations Manager 2007 Reporting is installed into SQL Server 2005 Reporting Services, it overwrites the default security model, replacing it with the Operations Manager role-based security model. ACS Reporting is compatible with this model. All users who have been assigned the Report Operator role will have access to the ACS Reports as long as they also have the necessary permissions on the ACS database.
| Note |
If Operations Manager Reporting is later uninstalled, the original SRS security model must be restored manually using the ResetSRS.exe utility found on the installation media in the SupportTools directory.
- ACS reporting installed on a dedicated SQL Server Reporting Services instance:
- Separate console for ACS and Operations Manager reports—When installed on a dedicated SRS instance, the ACS Reports are accessed via the SRS Web site that is created for it at installation. This provides greater flexibility in configuring the folder structure and in using SRS Report designer.
- Separate security model—A consequence of using a dedicated SRS instance is that you can create security roles as needed to meet the business and IT requirements to control access to the ACS reports. Note that the necessary permissions must still be granted on the ACS database.
The third design decision that must be made is how many ACS Collector/Database pairs to deploy to support your environment. The rate that a single ACS Collector/Database pair can support an ongoing event collection and insertion is not an absolute number. This rate is dependent upon the performance of the storage subsystem that the database server is attached to. For example a low-end SAN solution can typically support up to 2,500 to 3,000 security events per second. Independent of this the ACS Collector has been observed supporting bursts of 20,000 security events per second. Following are factors that affect the number of security events generated per second:
- Audit Policy Configuration—The more aggressive the audit policy, the greater the number of Security events that are generated from audited machines
- The role of the machine that the ACS forwarder is enabled on, given the default Audit Policy, Domain Controller will generate the most security events. Member servers will generate the next highest amount, and workstations will generate the least.
| Machine Role | Approximate Number of Unfiltered Security Events per Second generated under high load |
| Windows Server 2003 Domain Controller | 40 events per second |
| Windows Server 2003 Member Server | 2 events per second |
| Workstation | 0.2 events per second |
- Using the numbers in the preceding table, a single, high-end ACS Collector/Database pair can support up to 150 Domain Controllers, 3,000 Member Servers, or 20,000 Workstations (with the appropriate ACS Collector filter applied).
- The amount of user activity on the network—If your network is used by high-end users conducting a large number of transactions, as is experienced, for example, at Microsoft, more events will be generated. If your network users conduct relatively few transactions, such as might be the case at a retail kiosk or in a warehouse scenario, you should expect fewer security events.
- The ACS Collector Filter configuration—ACS collects all security events from a monitored machine’s security event log. Out of all the events collected, you might be interested in only a smaller subset. ACS provides the ability to filter out the undesired events, allowing only the desired ones to be processed by the Collector and then inserted into the ACS database. As the amount of filtering increases, fewer events will be processed and inserted into the ACS database.
The last design decision that must be made is the version of SQL Server 2005 or SQL Server 2008 to use for the ACS database. ACS supports the use of SQL Server 2005 Standard edition and SQL Server 2005 Enterprise edition or SQL Server 2008 Standard or Enterprise editions. Which version is used has an impact on how the system will behave during the daily database maintenance window. During the maintenance window, database partitions whose time stamps lie outside the data retention schedule (with 14 days being a typical configuration for data retention) are dropped from the database. If SQL Server Standard edition is used, Security event insertion halts and events queue up on the ACS Collector until maintenance is completed. If SQL Server Enterprise edition is used, insertion of processed Security events continues, but at only 30 percent to 40 percent of the regular rate. This is one reason why you should carefully pick the timeframe for daily database maintenance, selecting a time when there is the least amount of user and application activity on the network.
Audit Collection Service Guidelines and Best Practices
The overall performance of the ACS system is most affected by the performance of the ACS database and its disk subsystem. Given that many thousands of events per second will be inserted continuously, with potential peaks of tens of thousands per second, this is easy to see. It is not uncommon with a large number of monitored devices, including domain controllers, to accumulate more than a terabyte of data in a 14-day time span in the ACS database. Following are some best practices for ACS:
- Use 64-bit hardware and operating system for the Collector and SQL Server, along with a high-performance SAN solution.
- Separate the database files from the transaction logs.
- Use dedicated hardware to host ACS if warranted.
- Use tight filters to reduce the number of noise Security events that get inserted into the database.
- Plan your Windows Audit policy carefully so that only relevant events are logged on monitored systems.
- Enable the ACS Forwarder only on necessary systems.
Configure Security Event logs with sufficient space so that if communication is lost with the ACS Collector, the Security Event log file will not wrap on itself and overwrite previous events, resulting in a loss of event data.
In a later blogpost I will explain howto deploy Audit Collection Services