Common Event Format Inside
One by one, manufacturers of software and hardware and software solutions claim to receive a certificate confirming support for the HP ArcSight Common Event Format (CEF) format: Stonesoft , Tripwire , Citrix , Imperva , NetScout and several dozen vendors ...
According to Gartner's annual reports, HP ArcSight (until September 2010 - ArcSight) is confidently among the leaders in developing SIEM class solutions . In short, the essence of such decisions is to collect in one place
Each manufacturer of SIEM solutions in their advertising brochures usually indicates the number of supported event sources. For example, it is stated that ArcSight ESM supports 300+ devices and applications, while QRadar SIEM has “only” 200+. Who is there and how do the sources think - now it doesn’t matter, because within the framework of this article we will be interested in the format of information security events (magazines).
There are no general requirements for the structure of IS logs and events yet. Therefore, each developer draws up magazines in a way that is more convenient for him.
Someone simply writes them to a text file, someone provides the ability to send data to the Syslog server. One only needs SNMP, and the other wants to put all the logs in a relational database. There is Microsoft with its .evt format, there is CheckPoint with its OPSEC. And do not forget about SDEE.
About the problem
Both the structure of the logs and the protocols used to transmit them are different everywhere. From the point of view of integration with SIEM solutions (and not only with them), this is bad. Because for the unified processing of logs they need to be normalized, i.e. lead to a single format. Events are easier to store in a single format. Easier to search for data and generate reports.
Normalization involves parsing and mapping fields. Therefore, each connector between the information security system and the SIEM solution is an application that parses and maps in accordance with its configuration settings.
If the SIEM manufacturer does not have support for the product you need, then you will have to perform configuration settings yourself. Design and test regular expressions. To study the structure of the database, master SQL * Plus. Write matching rules. Sad, in general ...
“It would be nice if all manufacturers of information security systems alienated their logs in a way that is understandable to everyone,” thought ArcSight and, in 2006, submitted for general
Requirement No. 1 - We use Syslog as a transport.
Here, it seems, it’s clear.
I want - I use the UDP protocol, I want - TCP.
Requirement No. 2 - We fill in eight required fields.
Of course, the traditional syslog header should be present at the beginning of the message.
Jan 18 11:07:53 host
And already after it - a prefix
CEF:and a set of required fields, separated by the symbol "|"
CEF:Version|Device Vendor|Device Product|Device Version|Signature ID|Name|Severity|Extension
- Version - CEF format version
- Device Vendor, Device Product and Device Version - these lines uniquely identify the event source. There are no products that have the same set of these three values
- Signature ID - a unique identifier for the type of event
- Name - human-readable description of the event
- Severity - event severity (0 to 10)
- Extension - see below
Requirement No. 3 - The Extension field is filled in according to the CEF dictionary.
The Extension field is a set of key-value pairs. Here are a few keys from the dictionary:
dmac is the Destination Mac Address (for example, 00: 0D: 60: AF: 1B: 65)
spt is the Source Port (port 0 to 65535)
request is Request URL (in the case of HTTP request indicates the URL)
In the description of the CEF format, the dictionary is given in full, indicating the type of data for each key and the maximum permissible size.
At the end of the document are some more general requirements. In particular, the entire message must use UTF-8 encoding. The rules for the design of some special characters and multiline entries are also indicated there.
As a result, the message formatted in exact accordance with the CEF format should look something like this:
May 29 15:26:33 host CEF:0|McAfee|Antivirus|5.2|100|worm successfully stopped|10|src=10.0.0.1 dst=220.127.116.11 spt=1232
What is the benefit?
1. Connecting a new event source turns into a “plug and play”
2. The load on the SIEM system itself is reduced
3. SIEM solution manufacturers no longer measure by the number of supported sources and pay more attention to other aspects (product usability, fault tolerance, etc. .)
4. And in general, this format was invented not only for SIEM. It is already used in other solutions related to log processing.