Introduction, disclaimer and other informations
There's no radical and ultimate method for testing HIPS.
For evident reasons, we can't submit each product to all available malwares and try all possible attacks.
Finally, we choose to submit the HIPS to various examples of malwares, malwares behaviours, infection vectors, and possible attacks.
This method is quite restrictive and could not circumscribe the potential and limitations of each product.
But on the other hand, this methodology demonstrates the capability or not of each HIPS in front of malwares and attacks examples.
Moreover, we need to choose a "real and in the wild" approach, and this is not very simple.
We also consider how malwares infect the system: in many of our scenarios, malwares and attacks occur via client (browser, Outlook, Instant Messaging etc) and P2P files (a package with a renamed malware for instance).
Most of the time, it's a mail or a link which uses social engineering: by prevailing upon the target (the user) to click in a faked link (which downloads and runs malicious content) or to execute an attached fille, the system can be infected.
This is the easiest method to penetrate any system and it does not require to exploit a vulnerability and to have an advanced know-how: there's unfortunately no patch for naivety, credulity, curiousness or vice.
Example 1 (via Messenger for instance): "have you heard the latest single of 50cent? it's here" (faked link).
Example 2 (via Webmail or Outlook): "You are the winner of the International Lottery.
Just click here to see what is your present".
Example 3 via instant messenging: "hé i know an interesting toolbar which will provide you free porn movies.Just click on the link and run it. you won't believe it!"
We have also introduce specif tests to show that an HIPS is not the ultimate defense agains all threats:
As said Bruce Schneier "Security is a process, not a product".
These products are very efficient and this is an evidence that they're able to block most malwares easily.
An exemple of HIPS tests (in this case Buffer Zone) can be found on Trustware's Web site.
But a paper which demonstrates that a security software has passed a test with 100% is not satisfied for me.
"OK. This product blocks these malwares; so what?"
This is the same for the Virus Bulletin test: many antivirus pass the database with 100%; and the result only proove that:
-they have a good database,
-they are good products.
But that does not mean they're unbypassed: many malwares are still undetectable.
And we must consider some scenario which can be really used.
That's why this methodology is more difficult to pass: as said someone, "software is not security" (each security software has its own limits).
In our case (HIPS for home users), the methodology has been elaborated for a result which never reach 100% : as far as i know, there's no security software which provides 100% security!
Malwares and attacks are more and more sophisticated, and unfortunately, it's difficult to reproduce "real-life" examples of sophisticated attacks via Web servers as pointed out by the report of Websensesecuritylabs.
Some products have abilities that other don't have: therefore, we also need to add some tests which will allow the potential customer to distinguish different features between available choices.
Anti-Executable for instance can prevent downloads and works on DOS mode; Zorro PC Protector monitors system's area and has the ability to detect and put into a quarantine any new file (dll, .sys and so on), DefenseWall protects Windows accounts etc...
-In the first part, we'll test some kind of malware behaviour like dll injection, startup registry entry, new dll, message hooks, process killing etc;
-in the second part, we'll illustrate some of these behaviours with real malwares;
-in the third part, we'll focus on client/server side attacks.
I've planned Hardware keylogger, Buffer Overflow and Cross Site Scripting tests; but there is no reason to be more vicious than we should be: the goal here is not to demonstrate the limits of these products.
And if corporate HIPS are designed to prevent many attacks (like Buffer Overflow), this is not the case of personal HIPS.
Unfortunately, we can't be as exhaustive as possible by covering all possible security issues: these tests only demonstrate and illustrate the abilities and the potential of each product.
This methodology is easily criticized, then each publisher or vendor is free to publish his own tests; and in the same way, each user is free to do his own test.
Moreover, sample of test files have been sent to editors to allow them test verifications; and they're also free to post their own commnents.
Each time it's possible, the tests files are run from external drives (as unknown/untrusted): we can also consider that the're run from a mail via Outlook for instance: the most important is that they musn't be known from the HIPS.
And the files are executed (logically) unziped.
When a malware is not detected by antivirus (scan online on VirusTotal/Jotti Online Scan), it is often renamed (marked with an "R").
This methodology is elaborated on Windows XP2 with the antivirus disabled (BitDefender in the test machine) and connections under control via the firewall (here Jetico).
We can also consider that results in a test environment will not be exactly the same in classical using: it depends often of many criteria, especially the configuration and the human factor.
These tests are provided for free by volunteers not involved in the HIPS industry, and there is no kind of financial participation from publishers.
The goal of these tests is to show HIPS abilities and estimate their efficiency: this is a good way for potential consumers to make the right choice and to avoid any kind of excessive and pretentious marketing.
And in the same way, this is for HIPS programmers, as pointed out by some HIPS editors (System Safety Monitor, BufferZone, OnlineArmor etc), an interesting method for improving the effectiveness of their product.
This methodology is designed for "HIPS based White List", but i've included some screenshots of Desktop IPS based "anomaly detection": i still believe that the combination of two kind of HIPS increases the line defense: the first ones can detect an unusual event in real time, and the second ones limit the impact of this event on the system by their integrity protection.
System Safety Monitor and Neoava Guard have been used to represent HIPS based anomaly detection.
I've also included 2 screenshots of Online Armor to illustrate its antiphishing abilities (thanks to Mike Nash who sent to me these images via mail).
For more information about HIPS for home users and the classification of products, it can be suited to take a look a my article (recently updated) here.
NB (june of this year). as i'm releasing the first test publication (DefenseWall), i've noticed since may that a team of firewall testers has been very influenced by this methdology.
This affirmation can be easily prooved with a copy of their methodology at three different dates (HTTrack) and with HIPS editors testimonials.
Taken advantage of what other have done is not scandalous, since it's done with a minimum of deontolgy.
Unfortunately, their tests are for sale: the web is really a world wide web, and then each one his deontolgy.
Personally, i dont't consider firewall as HIPS, and as most security softwares editors (AV, firewall, HIPS) integrates teams of talented and experienced programmers, i really doubt that one of them needs to pay for testing his product.
But as usual, best wishes of success.
NEXT: Presentation part 2