FireEye: An helpful product that needs to grow up

When I first heard of FireEye I thought this was an ideal product to help combat zero-day malware infections.  It takes a snapshot of user downloads such as applications, PDFs, and other things that might be capable of “running” on a system.  It runs them in an unpatched virtual machine to see what malicious activities take place, and reports and blocks any malware trying to call back to command-and-control.   You can simply hang it off your network in monitor mode, or you can put it inline so it can block traffic when needed.  It will catch things that no other products will catch.

It can also bring down your network at random times of the day.  This isn’t an issue of misconfiguration, or coming under attack by a DDoS.  This is a known flaw in the hardware that the vendor has recently discovered.  The way it was described to me, it sounds almost like a quality control issue.  The FireEye network card that is used to tap into the network traffic has a problem.  From my experience, this problem is not immediately evident, but can become worse over time.  It’s almost like a circuit on the network card is slowly degrading over time, gradually causing the problem to occur more frequently.

This hardware issue results in the FireEye randomly dropping the network link.  If you saw enough potential in the FireEye to put it inline of your network traffic, this means your network link will drop for a brief moment.  If you put the FireEye in the recommended location directly between your firewall’s internal interfaces and the network switches, this means your firewall will lose its network connection.  You have a firewall cluster, so you’re okay with it handling the failover?  Hopefully you don’t have a Cisco ASA, with version 8.2 or less, and running a dynamic routing protocol.  If this is the case, your firewall may not failover so gracefully and you’ll be left with a longer outage until you get the Cisco’s routing protocol stabilized.  This is a known caveat in these versions of Cisco ASA, best to upgrade to the later versions soon.

The most unfortunate part is FireEye’s reactions:  Rather than reaching out to all their customers to mention there is a problem, very likely with all their equipment, they have been very quiet.  To compound matters, this is not an easy situation to detect.  FireEye does not send operational error messages to syslog, like network links going up or down.  It has to be dug up by FireEye’s support staff using specialized license keys that allow them deeper access into the machine.  Since the customer can’t see the FireEye issues this can lead to a lot of frustration because you may suspect other devices like firewalls and switches that are putting out link alerts and showing signs of some sort of problem.   This inability to send basic operational messages shows a lack of maturity in the product.  Yes, it is a very new product, but these sort of operational features are around in any enterprise products.  It shouldn’t have been left out of the FireEye, especially with its critical positioning in the network right behind the firewalls.

I was worried that this was the end of a wonderful product.  Word from FireEye was that there was  no workaround for this.  FireEye's best feature, preventing malware callbacks, would becomes useless.  You can continue to use it in a monitor mode, but that means your workstations will be getting infected more often.  At least you’ll known it has a zero-day infection, and that will  be enough to save the FireEye by some measure.  

Thankfully word has come down from the FireEye management that they do have replacement hardware to solve the problem.  FireEye has made a really useful product that stands alone in its capabilities.  If you suspect the FireEye might be causing network issues, I highly recommend contacting their support team immediately.  There are still far too many users willing to click on things they really shouldn’t, and you'll need FireEye's protection.