NSS Labs Hits Back at FireEye 'Untruths'


A war of words has erupted between FireEye Inc. and an independent testing house that questioned the vendor’s ability to catch malware.

NSS Labs handed FireEye (NASDAQ:FEYE) a “caution rating” in its first ever comparative analysis of breach detection systems (BDS) after its box scored a “below average” security effectiveness rating of 94.5.

All but one of the other vendors tested scored more than 98, with Fortinet Inc., Sourcefire Inc. and Trend Micro Inc. all scoring 99 or more.

NSS said its findings helps end users cut through the vendor marketing fluff in what is a “rapidly evolving” market.

In a forceful blog, FireEye product boss Manish Gupta moved immediately to discredit NSS Labs’ findings, claiming its methodology was “severely flawed.” The FireEye product the NSS used wasn’t even fully functional, used an old version of its software and did not have access to its threat intelligence, Gupta said.

But this was like a red rag to a bull for NSS Labs, whose founder Bob Walder last night responded with an equally robust blog designed to address “a number of untruths and misdirections” in Gupta’s missive.

Walder responded to Gupta’s claims one by one, saying many were untrue, including that the product it tested was indeed a fully functional product installed and configured by FireEye engineers.

According to Gupta, FireEye insisted the only way to properly test was to run in a real environment, but that the NSS declined to change its testing methodology.

Walder, however, rejected this version of events. He claimed NSS Labs does use a live environment “with real PCs going to real, live malicious URLs.” Walder confirmed FireEye did ask for it to change its methodology, but claimed this happened only after it saw the results, something the NSS “clearly cannot do.”

“In the grand scheme of things, FireEye’s results were not that bad,” Walder said. “The real issue here is that FireEye now has credible competition in the BDS marketplace, and the data from this NSS test shows it.”

Walder said it was rare for NSS to respond to criticism from vendors that have performed poorly in its tests, although a similar spat with WatchGuard Technologies Inc. erupted last year over its next-generation firewall report.

For more channel coverage from CRN UK, visit

Related Articles:

5 Responses to “NSS Labs Hits Back at FireEye ‘Untruths’”

  • John Sikes:

    FireEye is a sore loser! Their product simplify does not perform as well as the competition. Instead of claiming nobody understands malware and their product can not be tested humans they should be working to make their product better. Do they really believe everyone should just take their word for it their product is the best. Its the most expensive breach protection system on the market they need to prove why.

  • Mr. Rand:

    Actually, FireEye does have an issue with detection rates for a couple of reasons. Joe Giron reported on one of these already:

    Additionally, they have started locking their customers and their own staff out of the product and they have major support issues. FireEye (not mandiant) has more helpful information on their public blog than they provide in their walled customer knowledge base.

    While they found a few zero-days last year, it is a drop in the bucket compared to an organization like HP’s ZDI which published as many zero-days in a month as FireEye provided in a year.

    I personally can’t wait for an open source version to be released to really shake the high prices of these point solutions.

  • Rally Queen:

    NSS tests appliances in a way that would never reflect the real world.
    FireEye should have known this and tweaked their box to work like an AV gateway detecting everything that VT says is malware, Manish Gupta is ex-McAfee and should have known this would happen from his previous experience with NSS.

    Better luck next time FireEye, I still think you are the #1 product with no real competition, at least not yet.

  • craig kensek:

    If you don’t like the results, attack the test group and their methodology, something both Trend Micro and Symantec have done with Virus Bulletin in the past. The public also has varying degrees of test group’s methodologies. Go to a recent survey performed by AV-Cdmparatives. NSS finished in an okay position, but was not in the top position. Meanwhile, reports at the value of the stock when FireEye execs sold some of their shares would be interesting to look at.

  • Nik Anderson:

    Palo Alto Networks CTO Nir Zuk criticized the legitimacy of NSS Labs’ testing methodology. The security networking vendor has declined to participate in NSS Labs testing in the past because of what Zuk calls a “flawed sales model.”

    I have a question to NSS Lab, if what they claim is true, then how come FireEye has detected all those zero-days vulnerabilities??

    On the other hand, if the claims of NSS Lab are true, why then no one yet were able to breach FireEye?

    I totally agree with Zuk ( NSS Lab “flawed sales model.” ), who pays more gets higher..

    Good luck believers of NSS Lab.

Leave a Reply