McAfee False Positive

You may or may not of heard about the massive mistake McAfee made last week.

On Wednesday 25th April they released a virus definition file (5958 – April 21st) that incorrectly identified svchost.exe as a threat and deleted it on systems running Windows XP SP3.

Svchost is used for launching services (full description here) and any individual instance can run a group of services. This means its a pretty critical process!

Unfortunately for us a large chunk of our client base is running McAfee anti-virus software, the others run Trend Micro.

We knew something wasn’t quite right when we received several calls all around the same time with similar symptoms. However, while the symptoms were similar they weren’t identical so initially we didn’t quite know what was going on. Unfortunately the one thing they did have in common was a loss of network connectivity which meant we couldn’t fully diagnose the issue.

Later that day McAfee issued a notice, an updated definition file and details of how to fix the issue.

Basically we had to,

Boot into safe mode

Add an EXTRA.dat to the c:\program files\commonfiles\mcafee\engine folder (or just run the 5959 Super DAT which is quicker)

Recover a copy of svchost from the service pack cache c:\windows\ServicePackFiles\i386\ or if not present, C:\WINDOWS\system32\dllcache\

Restart the computer

McAfee released an automated tool for this the following day (It’s in this KB article)

A simple enough fix to but as I said earlier every PC we’d seen with this issue had no network connectivity.

This meant we potentially had to physically visit ever single PC we look after.

I say potentially because this only impacts running Windows XP SP3, we do have some clients running Vista or Windows 7. But most of our clients still currently run Windows XP. Also VirusScan 8.7 systems were harder hit. Some of the PCs were still running 8.5.

Still, for some people it would be every PC they own

Now regardless of the size of your company ask yourself some questions.

How long would it take you to spend 5-10 minutes on every PC you look after?

Did you factor travel time into that?

Who do you make a priority when everyone is offline?

Fortunately we got a little lucky

We configure the McAfee products to fetch updates from the global McAfee update site every hour. Any servers on site will then check for and get updated every hour

PCs check every 2-3 hours but we also put a random delay on this. The main reason is so that on larger sites we don’t want lots of PCs all generating network traffic at the same time. By putting in the random offset it’s staggered through the day. So this is in combination with the fact McAfee actually got the DAT update out the same day meant that lots of PCs never actually received the faulty update.

That said. We still had a LOT of work to do.

We visited as many sites as we physically could over a two day period and some other sites that had some tech savvy people on site we managed to go through it on the phone with them.

I also had to cancel other appointments which I hate doing and some other promises I made were a little strained.

I’m sure we’ll still be dealing with issues at the start of next week

Obviously for our contract customers this was all at our expense.

I can’t even begin to think what this will cost McAfee as customers start to move away at their next renewal period.

McAfee have an FAQ here as well as a couple of blog post apologies.

As you can imagine there has been a lot of commentary on this and other vendors are jumping in to take advantage. 

http://www.pcmag.com/article2/0,2817,2363018,00.asp

http://www.betanews.com/article/One-very-false-positive-McAfee-in-full-damage-control-mode/1272040662

http://blogs.zdnet.com/Bott/?p=2031

Especially since it turned out this down to poor quality testing.

As the IT world always seems to throw odd coincidences, on Friday i got an email inviting me to the McAfee stand at the InfoSec exhibition next week – I imagine that stand is either going to be very busy……or very empty

This scenario is truly a management nightmare – an automated update that renders a PC unusable that can only be repair by hand. On top of this we’re going to have our own PR exercise to sort out.

All our end-users see is a broken PC. It’s our responsibility to keep them up and running and while we still fixed the problem. They’ll still be asking US questions as to why it

For new installations we moved away from McAfee long ago (there are other McAfee posts on this blog)

Our existing customers have been using McAfee for a variety of reasons but when the renewals come up we’ll be making a concerted effort to get them away.

The Case of the Vanishing E-mails

I just wanted to relay a support call I’ve just dealt with as it really had me stumped. For a while I was concerned there was something really bad happening but now that I’ve gotten to the bottom of it I can breathe easy as it wasn’t actually all that serious

Our client called to say that she was expecting an e-mail from their German contractor and it hadn’t arrived. He’d resent it a couple of times to confirm he’d got the email address, etc correct

I logged onto the server and used Exchange message tracking to see if the e-mail had arrived

The message did arrive at the server as expected

My next port of call was to logon to the PC and check Outlook, sometimes I’ve seen similar cases where a filter has been applied by mistake that means the user has “lost” the message. Unfortunately this drew a blank as well

I used the advanced find feature to display all emails that had been delivered today and I even created a search folder to do the same thing but the message was nowhere to be seen

I logged in via Outlook Web Access to see if the message was visible there as I’ve also seen cases where the message is corrupt and Outlook can’t display the message but OWA seems to do ok with it. Again no luck here

So I checked to see if she hadn’t deleted the message by mistake. The message wasn’t in there but when I used the “Recover Deleted Items” feature in Outlook there is was! I thought this would be a simple matter of clicking the message and pressing restore……unfortunately not

After I clicked restore I expected the message to reappear in the Inbox but after waiting a few minutes, nothing

Confused I went back into the dialog in case I’d selected the wrong message. The message was still available for restore so I clicked it again. Once again the message failed to reappear so I went back in again

One of the columns is “Deleted On”, I noticed that time was only a minute ago

I recovered the item again, noted the time, waited a minute and went back in and lo and behold the deleted on time had changed to the time just after I’d recovered the item

Confused I turned to the server, this particular server uses McAfee GroupShield for checking inbound messages for malware so I trawled through the logs to see if it was picking this up as a dodgy message. I wasn’t holding my breath though as this isn’t how GroupShield works (the message is usually quarantined or a part of it replaced and the user alerted). As expected this was a dead end

I was starting to get a little paranoid now so checked the PC over for malware and my initial checks didn’t turn anything up. I even used SysInternals Process Monitor to track what was going on during the recovery to see if there was an outside process doing something but again this didn’t show anything of use

I fired up Outlook in safe mode but didn’t get very far as the recover items feature is an add-in itself!

I decided to keep my attentions on Outlook and created a new profile for the user. This time I disabled Outlook cache mode and recovered the item again

This time the message reappeared as expected!!

As soon as this happened a huge light bulb came on somewhere in the back of my head

There is a feature in Outlook that will ONLY work when cache mode is enabled

Outlook Junk Filter

I went into the options for this it hit me straight in the face!

This is how the junk filter was configured on the PC

Can you see the problem?

Only allow from safe lists and permanently delete suspect messages!!!

I’m surprised she gets any email at all! But if she only receives messages from a group of people that she always emails then they would be on her safe list

The junk filter checks for messages as soon as they arrive in the inbox, irrespective of whether that is because the message is an incoming e-mail or has just appeared there after being restored

I have no idea why this has changed. It must have only have changed recently because as I said earlier the configuration would surely mean she would notice lots of emails going missing.

Once I turned off the junk mail filter I was able to recover the message ok

Ordinarily when a client doesn’t require the Outlook junk filter because they are getting their spam filtered elsewhere I use the Office ADM files and setup group policy to explicitly disable the Outlook filter (Office 2007 version here) so I need to go and set this up I think!

Minor panic over!

Unable to release items from quarantine when using Outlook 2007 with McAfee Groupshield

Following on from my last post i mentioned that getting an email out of quarantine wasn’t fun

Basically the procedure should go like this

• End user receives a message telling them an email to be delivered to them has been quarantined
• End user contacts administrator with ticket ID from quarantine notice
• Administrator locates email and releases message
• End user receives an email telling them their message has been released. In this message is a save button they have to press to save a copy of the email

It’s not quite that simple though for a couple of reasons

1) When the email arrives Outlook blocks the scripts from running in the email (quite rightly) which means there is an extra step to save the message to the hard disk as a HTML file

2) Once saved as a HTML file the save functionality isn’t very dynamic. You have to specify the save location manually as there is no browse button

3) The email is supposed to be saved in .eml format but the webpage adds an extra extension on for some reason which isn’t a great experience for the end user

4) EML files aren’t readable by Outlook so you have to open in them in Outlook Express / Windows Mail and then import them into Outlook

Lengthy and irritating

To make matters worse if you using Outlook 2007 when you do save the quarantine release notice as a HTML file the save button doesn’t work

It turns out this is because Outlook 2007 butchers the HTML and adds it’s own tags

To get around this

McAfee Groupshield Incorrectly Identifying E-mail as Corrupt

Just a quick one as a reminder in case i come across this again!

If you are using McAfee Groupshield 7 and are noticing users are receiving the following on emails that are seemingly ok

CORRUPT CONTENT ALERT
The content this replaces was found to be corrupt.
Cause of corruption: Invalid character set or encoding.
Ticket Number: <ticket number>
See your system administrator for further information.

Upgrade to Service Pack 1 and this will fix the problem

We had two different sites that were experiencing this problem and the procedure you have to go through to release a quarantined message isn’t fun!

Full details in McAfee article 53009

I’ve been critical of McAfee support in this past but it appears as their knowledgebase as undergone a revamp as it’s helped me out quite a lot recently

kc.mcafee.com

Internet Explorer Crashes on Opening – EntAPI.dll

Busy, busy busy on the first week back!

Got a call today from someone who was having problems opening Internet Explorer

Each time he’d get an application fault

My first thought down to previous experience was that it was an add-on trying to load I so ran IE in “no add-ons” mode. IE opened fine so I spent a little while disabling lots off add-ins with little success so I went back to the beginning and looked in the event log for the actual error (should have just done this first!)

Faulting application iexplore.exe, version 7.0.6000.16762, faulting module entapi.dll, version 8.0.0.240, fault address 0x0000368a.

I found the entapi.dll file in the windows\system32 folder and the file properties suggested it was a McAfee component

A quick search turned up lots of people with similar problems who “resolved” the issue by simply renaming the file

I renamed the file and as reported this stopped Internet Explorer crashing

Not a complete fix in my book though as this means all I’d done was stop a component of the anti-virus software from loading!

Since it was a McAfee component I went to their knowledge base and did I search on entapi.dll hoping to find out a little more about what it does and the very first entry was this

Internet Explorer crashes in a Buffer Overflow module (KB57756)

entapi.dll is part of the buffer overflow module? Glad I wanted to dig a bit deeper

The article says that it’s a know issue with version 8.0.0.240 and updating to one of the following will fix it;

VirusScan Enterprise 8.0i cumulative Patch 14. or upgrade to VirusScan Enterprise 8.5i/8.7

This particular PC had 8.0i on so I loaded the latest patch (which was 16) and this fixed the problem

I checked the properties of the old and new entapi.dll files and we’d jumped from version 8.0.0.240 to 8.0.0.455

Sorted!

McAfee updating – Unable to find a valid repository

I posted a couple of days ago about a new client who is using McAfee. I went back on Tuesday just to finish off and came across a bit of a weird problem

This particular PC didn’t have any anti-virus loads so i installed VirusScan 8

I configured it to point to a share to check for definition updates and fired it up and got this

Starting VirusScan task: AutoUpdate
Starting update session.
Unable to find a valid repository.
Closing the update session.

My initial thought was that i’d enter the UNC details wrong so i checked that and it was fine. I then realised that if i’d entered the UNC wrong it would have fallen back to the HTTP or FTP repository….something else was going on here!

So first i checked the updater log

3/25/2008 9:24:57 AM Starting VirusScan task: AutoUpdate
3/25/2008 9:24:59 AM Starting update session.
3/25/2008 9:25:00 AM Unable to find a valid repository.
3/25/2008 9:25:00 AM Closing the update session.

Not helpful!

Onto the event log

VirusScan Enterprise: The update failed; see event log.(from PCName IP 192.168.x.x user X running VirusScan Ent. 8.0.0 UPD)

Even less helpful!

I look all around the program files directory to see if there were any extra log files and i couldn’t find anything

I did some searching and came across this article 

https://knowledge.mcafee.com/article/299/5170943_f.SAL_Public.html

It talks about troubleshooting the auto-update. What helped was this

A verbose log file is available to assist troubleshooting. The default location is:
C:\Documents and Settings\All Users\Application Data\Network Associates\Common Framework\Db

The file is agent_<machine_name>.log, where <machine_name> is the hostname of the machine where the update is being initiated.

This log was much more helpful

20080325095031 I #1676 Updater OnDemand update started.
20080325095036 I #808 FrmSvc User SID is S-1-5-18 and SessionID is 0
20080325095037 I #3980 Script Searching for first available site.
20080325095039 I #3980 InetMgr Sitelist validation failed because it is missing SPIPE site information.
20080325095039 I #3980 InetMgr Retaining the existing sitemaplist.
20080325095039 i #3980 Script Unable to find a valid repository.
20080325095039 i #808 Script Closing the update session.

The updater was having a problem reading the settings I’d put in so was defaulting to the sitelist it previously had. Since this was clean installation that meant it had nowhere to go!

I also found this

http://forums.mcafeehelp.com/showthread.php?t=141537&page=3

It’s a forum post where someone suggests manually creating a sitelist.xml file. The scenario was slightly different as they were talking about ePos but the theory must be the same

So I copied the sitelist.xml file from another PC onto this PC in this folder

C:\Documents and Settings\All Users\Application Data\Network Associates\Common Framework

This sorted the problem!

Not sure why the sitelist.xml file didn’t get created in the first place

I’ve talked plenty of times on here about how fed up with McAfee i am. I had a meeting with Trend a couple of weeks ago and I’m about to evaluate their stuff. The “worry-free” management console looks pretty cool