Jul 24th

What Can The Security Industry And Its Customers Learn From Prevx’s Home Page?

Posted by: Mel Morris

Bookmark Now

Metrics Are Key To The Survival Of The Security Industry

I am a very strong advocate of metrics. They are fundamental to understanding, measurement and improvement. My decision to showcase metrics about how many new malicious programs we see each day and to show the results of their detection by products from Symantec, McAfee, Microsoft, Computer Associates and Trend Micro on our home page should help our customers, prospective customers, observers and our competitors understand the value of Prevx2.0 and our back end technology at automatically, identifying, determining and protecting against new malware on a daily basis.

The Security Industry IS struggling to scale

We believe that scale is probably the biggest challenge facing the security industry today. Managing the exponential growth in malware volume is critical to its survival and to our ability to use the Internet safely. To give you some idea of the volumes involved we see the emergence of around 250,000 new software programs each and every day. Of these around 1%-4% will be determined malicious, that’s 2,500 to 10,000 new malicious programs identified and determined by Prevx each and every day. There is a symbiotic relationship between malware and benign software (BENWARE) which is part of the solution but is now becoming part of the problem. Software evolves, bugs are found, they are fixed this is the natural force that drives the growth in the new programs we see each day. Every time a new piece of software is launched the bad guys, we call them Malware Inc., test it literally to destruction looking for signs of weakness that can be exploited for malicious intent, hence a new family of malware is born. In response the BENWARE is modified to plug the holes, new security products, patches, signatures or behavior patterns are formulated and the cycle begins all over again. Our stats indicate that security applications are those that show the most rapid evolution, followed by Operating System components and Browsers. To give you some idea of the volumes, in the last three years we have seen more than 7,000 different executable versions of Internet Explorer with the name IExplore.exe. I know of no other industry which is stimulated by a 3rd party antagonist like the security industry.

It’s A Malware Research Problem

Prevx has justifiably earned a credible standing in the security industry on the back of a new and still somewhat unique model. That of centralised or so called ‘herd’ intelligence. In practise it is the strength of our back end technology that sets us apart. It is responsible for managing scale, for checking the behavior of millions and millions of programs run by a diverse user base, for monitoring and analysing the emergence, propagation and behavior of these programs, and ultimately for concluding if they should be allowed to run or not. A lot believe we are only concerned with identifying malicious programs but our goal is to assign an accurate determination of good or bad to every new program Prevx2.0 encounters. No mean feat. Our back end architecture has been designed to provide linear scalability to handle 1,000 times more than we are handling today. And we still question if that will be enough by 2012, just five years away. Today, we believe our processes are as good, or better than those of even the largest security firms. The difference is that our system improves geometrically with scale. Sure Microsoft could choose to add thousands of researchers to its lab in an attempt to cope with the volumes, but the reality is that security will soon be a cyber war fought by hyper-computers on both sides; Malware Inc versus the Security Industry. Now back to metrics.

Prevx’s Statistics Are Available To Our Competitors

We are not asking anyone to draw conclusions in isolation from the statistics on our home page. We have made the detail behind the home page stats available to our competitors, analysts and CSOs of major enterprises. If we missed you out you can register by clicking the arrow on the bottom right of the chart and signing up for Detection Analysis service. This let’s you see the detail of each sample used in that day’s test and who stopped it as well as who didn’t.

Prevx's Home Page Tells A Powerful Story, But The Trend Will Be The Key Factor

Over time our home page will tell its own story. Viewers can see the rising volume of malware and the relative performance of each vendor’s products at detecting it. The trend is probably the most important factor. We think the chart already matches popular perceptions of those in the know, of the relative strengths and weaknesses of the major products at catching malware in its first hours of distribution. We also think the chart is a clear evidence of the balance of scale and resources. Symantec is the largest vendor, it has the most conventional honeypots and it detects more early life malware than anyone else, Prevx excepted here. Microsoft, who it is understood recently added 50 or 60 malware researchers has now overtaken McAfee for the number two slot. However, it is a critical message that 50 to 60 additional researchers has only moved them to second place. We suspect they would need to add a further 100 to 150 to close the gap on Symantec, who only score 50-60% on our home page stats. Forget, what people conclude about Prevx from this chart, these stats will be a revelation to some of these vendor’s customers. It is also interesting to note that the chart reflects the proportion of these vendor’s customers who turn to Prevx when they realise that their existing security products have failed them.

Last Week’s Targeted Attack Is A Wake Up Call For The Security Industry And Its Customers

Last week we caught a Trojan NTOS.EXE that had sailed straight through all of the top five security products to infect a US state department, an airline, two computer manufacturers and a US defense contractor. Forget that Prevx found it and ask how could this happen that one single fairly unsophisticated malware program bypassed all top five products and infected 494 separate computer systems. The answer is scale. None of the security vendors have the resources and processes in place to investigate low volume targeted attacks like this. Otherwise they would all have stopped it. Even after we supplied samples it took between 18 hours and 3 days for these vendors to make protection available, the average was 35 hours! What did this Trojan do? It only stole the addresses, logons and passwords for numerous secure web sites and email accounts, bank details, details of airline reservations, IM messages and tons personal information. As the malware volumes continue their exponential growth we can expect to see more successful targeted attacks slipping under the radar. If the limit is 500 infections before the large vendors will see it we are already in BIG trouble.

We Hope Prevx’s Competitors Will Return The Favor

We encourage the industry to be forthcoming with similar statistics to those we have chosen to make available. The industry has a lot to learn. What we have done is to take a bold step towards measurement, understanding and improvement. Sure we are not perfect either. But our model is built with the industry’s key challenge firmly in focus. And we would welcome knowing details of any malware that other vendors stop that Prevx2.0 does not. That information could only improve our service to our customers and we would thank any source that provided it.

By the way, as a matter of record we currently have just three malware researchers. Why? Because automation is the only way to cope.

Here’s to an open attitude towards Internet Security, and please.......... MORE Metrics.

Mel Morris
CEO
Prevx Limited

6 comments so far

  1. Roger Neame on Jul 23 23:04, 2007
  2. Your metrics are a good start, but what about false positives? Clearly, some files that Prevx blocks and that other programs failed to detect turn out to be valid programs.

    Since false positives take time to discover, the metric might have to be a few weeks old and might have to be over a period longer than a single day. E.g., "Our false positive rate (as a percent of programs blocked) is currently averaging x%"

  3. SIRIUS on Jul 24 21:01, 2007
  4. Great overall mission statement yielding on target and in control!

  5. Gary Mickelson on Jul 28 1:07, 2007
  6. Your false positives are driving me nuts! Maybe you should hire another researcher or two to log all the programs that have been around for YEARS that are being blocked. And there is no over ride if the customer knows better. I had to uninstall Prevx 2 to get my web browser back. It called Maxthon an infection after two weeks of no problems. Maxthon had no update. It has been around forever and has over 100 million downloads to date-- check their site. Maxthon is no more malware than Prevx. It pulled the same thing the other day on Clipmate-- the most popular clipboard extender out there.

    Before you start crowing about your superiority you had better KNOW that you are not perfect as you say, and in fact have a long way to go. Latest bug is the C++ Runtime error at shutdown in XP systems. The forum is working on that one still after a month.

    Rely a little less on technology and a little more on your people and your customers. THEY are telling you what you need to hear.

    I am just two weeks into a test of Prevx and am already seeing major problems. I returned two copies of Spyware Sweeper and was planning on buying Prevx. I'm glad I decided to wait out the test and see what happened. Today was the final straw when my browser was labeled an infection and blocked. I had to uninstall Prevx (a chore in its own right) to get Internet Connectivity back. Kind of hard to get ahold of customer service when your browser has had it's exe file taken out, isn't it.

    Maybe your service people will answer my request for service and enlighten me. I liked the product until the last couple days when all hell is breaking loose. At the very least you need a database infusion of safe programs. Spyware Sweeper ans my Zone Alarm Security Suite don't have this problem.

    So far as your false positive rate, give me a break. Two well knownprograms diab;ed in three days.

  7. AndyM on Jul 28 6:26, 2007
  8. "Here’s to an open attitude towards Internet Security"

    Does this mean your currently submitting the samples you discover to other security vendors ?

    You clearly take the time to scan them against competing programs so it would make sense to also distribute the samples to those companies that you scan the files with so that they can protect their users,

  9. Mel Morris on Jul 29 10:13, 2007
  10. Gary

    Clearly you had some pretty significant issues. I asked our researchers to check on the state of both Maxthon and Clipmate and to ensure that these were appropriately determined. We apologise for any inconvenience caused.

    There are a number of issues that you raise in your comments and I'd like to address these in turn:

    False Positives

    False Positives occur with every security product and we are prone to this like anyone else. However, our community view should make us more aware and therefore less prone over time. However, as a direct result of your comments I am going to have our development team provide details of false positives prominently on our web site along with the detection metrics. This will be a great addition. We will add a chronological log of these so everyone can track what was affected, when and how long it took to resolve.

    Reporting False Positives or Negatives

    We have tried to make it easy for our users to report false positive and false negative determinations simply by clicking the name of the blocked application in the Prevx2.0 console and then selecting the "I Disagree" link on the web page displayed. This immediately alerts our researchers to investigate and hels resolves issues faster than a comment on one of our blog pages.

    Overriding A Bad Determination With Prevx2.0

    You should have been able to override Prevx2.0 from blocking an application by simply dragging and dropping the app into the Probation area using the console.

    Prevx Striving For Perfection

    We are the first to admit that we are not perfect. We hope that by having an open and honest approach to what we do and by providing metrics that chart our progress that this will lead to on-going improvements in our technology. Like the industry as a whole we have some way to go.

    More Reliance On People

    The Prevx2.0 community learns about 2-3 new programs every second of every day. Thats around 5-6 million every month. The only way the industry can cope is through automation. Our false positive rate is very low indeed. We determine about 2,000 to 10,000 new programs as bad every day. Should a false positive occur and be reported via the appropriate method this will be resolved as a priority by our researchers.

    Finally, thank you for bringing this to my attention. We welcome your feedback, good or bad.

    Mel Morris

    CEO

    Prevx

  11. Tom Owen on Sep 20 12:23, 2007
  12. I'm fascinated by what you're doing.

    1) Can I suggest providing a malware file lookup for all us poor saps trying to interpret dodgy-looking "hjt" or "autoruns" entries on servers we're not allowed to touch.....

    2) Aren't you vulnerable to Malware Inc buying prevx licences and running their stuff past the agent in a "behave yourself" mode first?

Leave a reply








Monthly Archives

Yearly Archives

Stay Updated

YouTube Channel

Find us on Facebook