OSSEC: Keeping your system safe from nasty scanners and attackers

01 Mar 2012

OSSEC is a host based intrusion detection system. It goes looking for rootkits, analyses logs and checks file integrity. It helps increasing the security of your server. OSSEC is being actively developed and they have commercial support.

What is this tool ?

As said before it is a host based intrusion detection system ( Wikipedia: An intrusion detection system (IDS) is a device or software application that monitors network and/or system activities for malicious activities or policy violations and produces reports to a Management Station.). OSSEC provides many functionalities which will be explained further in this post.   It has two working models:
  • Local: for one system
  • client/server: for more than one system
OSSEC has different processes that are chrooted as a different user to provide extra security. It has different daemons:
  • analysisd: main process
  • remoted: receives input from remote agents
  • logcollector: reads log files
  • agentd: forwards data to the server
  • maild: sends out email alerts
  • execd: executes active responses (do command on triggered event)
  • monitord: monitors the agent status
  • ossec-control: manages the start and stop of all above processes
After OSSEC collects logs, it will decode them, analyze them and decide if it needs to send out an alert to the administartors. The decoders are stored in XML format and can be addapted or written yourself. However this is beyond the scope of this article, it might be a good topic for future posts.

What does it try to battle?

OSSEC tries to tackle many security issues, I've made a summary with some explanation.


It goes hunting for rootkits. What is a rootkit ? A rootkit is a stealthy piece of malware, it is designed to hide processes from normal detection methods and gives the intruder access to root priviliges. For example, a hacker has hacked your system and wants to host files on it (e.g. pirated movies,games,pr0n). The problem is these files can get rather large, so an administrator might already notice this by just running a df -h command on a linux-server. The hacker will make sure his rootkit supplies df -h wrong values about diskusage. The administrator might only start noticing when his system starts failing because it ran out of disk space. To track these rootkits down it stores a list of hashes of binaries. If the binaries are changed it will allert you. If you upgraded the binary, you will expect this message. But when you haven't upgraded anything, you might want to have a look at that binary.

Log Analysis

Log Analysis for intrusion detection is the process or techniques used to detect attacks on a specific environment using logs as the primary source of information. It will decode the logs and decide what to do when he found a possible attack. OSSEC also comes together with a large database of known rootkit signatures.

File Integrity

There is one thing in common to any attack to your networks and computers: they change your systems in some way. The goal of file integrity checking (or FIM - file integrity monitoring) is to detect these changes and alert you when they happen. It can be an attack, or a misuse by an employee or even a typo by an admin, any file, directory or registry change will be alerted to you.

Installing OSSEC

To install this tool first we need to make sure we have a compiler on our server and all our agents (run this command on every machine):

apt-get install gcc make

Then we need to get ossec from here. When you have the package check its MD5 and SHA1 sum.

md5 ossec-hids-latest.tar.gz
sha1 ossec-hids-latest.tar.gz

The checksum should be the same as the one given on the website. Next untar it:

tar -zxf ossec-hids-latest.tar.gz

Go into the folder and run install.sh:


When installing you will be presented with several questions:

  • 1. On our server we pick server
  • 2. pick a directory or enter for default
  • 3.1 enter an email address where notifications should be sent to
  • 3.2 this is for file integrity checking, alerts you to changes to files on your system
  • 3.3 rootkit detection
  • 3.4 trivial questions
  • 3.5 if you want remote syslog
  • 3.6 trivial
start the service:

/var/ossec/bin/ossec-control start

Now we need to install the same software package on our clients/agents but use agent instead of server at question 3.1. After this run on the server:


We will generate add an agent first with A. Then after you have provided that information we will extract a key for the Agent with E. Copy the key of the agent. Now run the following on your agent:


Select I to import a key and paste the key generated by the server. Now start the service on your agent with:

/var/ossec/bin/ossec-control start

Restart the service of your server

/var/ossec/bin/ossec-control restart

If you want a windows client, there is a windows binary on the download page as well. I do not know how this works as I did not have any windows servers available to test it on. I find the webui for OSSEC quite handy as it gives an overview of all the possible threats. Providing you have a working Apache2 with php 4.1 or later.: Download the package from the download page. Check the hashes:

md5 ossec-wui-0.3.tar.gz
sha1 ossec-wui-0.3.tar.gz

Providing you are using the 0.3 package, it should be:

  • c79fa486e9a20fb06a517541033af304
  • e00bff680721982ee55295a5292eb4e2a638b820
Now untar the package:

tar -zxf ossec-wui-0.3.tar.gz

Move it into a webfolder, like /var/www/ossec

mkdir /var/www/ossec
mv ossec-wui-0.3 /var/www/ossec

Run the setup:

cd /var/www/ossec

Now we need to change group permissions and the permissions of the files:

 vim /etc/group

Now change




if your apache2 user is www-data. Fix the permissions for the tmp folder in /var/ossec:

    chmod 770 /var/ossec/tmp/
    chgrp www-data /var/ossec/tmp/
    apachectl restart

You can access your webserver here at http://yourip/ossec. I suggest using a htaccess to secure that page. OSSEC will always block attackers and put their IP in hosts.deny in /etc and in iptables. It's not blocked permanently but just temporary. But sometimes it may occure that it was a false positive and you need to unban an ip manually. This can be done by first looking at the /var/ossec/logs/active-responses.log. You will find entries similar to:

 /var/ossec/active-response/bin/host-deny.sh add - 1328614852.61546 5712
 /var/ossec/active-response/bin/firewall-drop.sh add - 1328614852.61546 5712

To manually unblock them you need to change the 'add' to 'delete', so to the delete the previous rules it would be:

 /var/ossec/active-response/bin/host-deny.sh delete - 1328614852.61546 5712
 /var/ossec/active-response/bin/firewall-drop.sh delete - 1328614852.61546 5712


In your rules directory you can find many rules:


Sometimes rules are to strict or not strict enough. You might want to change something or add something yourself. This can be done in localrules.xml file. Suggest we want to increase the tresshold of failed login on http auth for apache2. If we look at the apacherules.xml we see a number of rules. The interesting one is:

    Multiple attempts blocked by Mod Security.
To change the frequency from 6 to 10, we need to copy the rule and paste it in localrules.xml. Then we add a parameter overwrite="yes" to tell OSSEC it needs to overwrite the rule defined in apacherules.xml and instead use the one defined in localrules.xml. The rule would look like this:
    Multiple attempts blocked by Mod Security.
If we want to completely ignore this rule as it is not relevant for us, we just change the level to 0:
    <samesourceip />
    Multiple attempts blocked by Mod Security.
You might notice that your file, even as root, seems unwritable. What we need to do is change the rights of the file to make it writable.

chmod +w /var/ossec/rules/local_rules.xml

After you have edited the rule, take away the write capability again.

chmod -w /var/ossec/rules/local_rules.xml

Now restart OSSEC and you're set:

/var/ossec/bin/ossec-control restart

I hope his small introduction was useful to you and as always, have nice day.