Last february, I’ve ordered and received a new laptop, to act as my main personal PC. Quite sophisticated, with lots of RAM, CPU, GPU, and SSD, so I could use it also as my infosec lab (running VMs, calculating hashes, doing forensics and more). As my close friends know, I always buy licenses of software I use, and don’t mess with pirated software for a few ethical reasons. So I also bought a Windows 8 Pro set of DVDs from my local store, and installed it a few days later. I also spent several weeks reinstalling all my software from scratch, reactivating licenses and configuring the whole so I could find my old environment back on the new laptop. So this was a laptop which wasn’t supposed to be crashing, using legit software, and admit I really enjoy using it.
But the issues that I experience since the acquisition are quite annoying, forcing me to stay « offline » for several weeks since the begining of the year, and spending days trying to recover backups, understand issues and more… Some of my frequent email correspondants know that I had those issues, since I was quite slow in answering emails during those « shutdown » periods…
A. Hardware issues
Everything went fine since then, regarding to software, but I had a few occasional hardware glitches that worries me more and more :
1/ First, I had a considerable amount of bluescreen (BSOD) crashes in random cases more or less relating to standby/hibernate/wakeup cycles, forcing me to perform a total shutdown each time I wanted to move my laptop around. Not an issue, but strange. As seen below, using NirSoft Bluescreenview, I experience a lot of different processes crashing… Doesn’t seem related to a single software :
Below is an extract. Click to enlarge the screen capture…
2/ Hoping to solve this and considering my BIOS wasn’t the latest, I decided to upgrade it. After downloading the upgrade from Dell website, and applying it using their tool, my PC rebooted once to reflash, then went totally black… No leds, nothing… A Dell technician had to come to change the whole motherboard to solve this.
3/ After this incident, everything went fine for 1 or 2 months, but BSOD crashes started again from time to time, when leaving hibernation or suspend state. Not an issue any more for me since as a workaround, I now have the habit to do a full shutdown each time. But this has to be investigated.
4/ A few weeks ago, « my » Dell technician had to come for the second time, to replace my motherboard again. Same symptom, but this time I didn’t even dare to upgrade the BIOS, the PC refused to start one morning. As a weird side effect, after changing the motherboard for the second time, my internal keyboard didn’t work any more, so the technician came a third time to replace it… (Luckily I paid for a 5-year insurance covering hardware issues with on-site repair, but that is really weird).
5/ And I also have to mention my 256 Gb SSD drive that totally crashed twice. The first time, I got it replaced. The second time, 2 weeks ago, I used a « magic » procedure documented on the manufacturer’s website. All I have to say even if useful, I hate such « magic procedure » consisting at powering the drive during 30mn then powering-it off, and back again using an USB charger to recover my partitions, because there is nothing rational.
That’s all from the hardware perspective. I may think my PC or my SSD have got hardware issues, but after changing the motherboard and SSD several times, I just tend to believe the root issue is elsewhere. I admit some forums say BSOD can occur with Crucial SSDs, but I did follow the procedures, and kept upgrading the SSD firmware too, as it’s told it’s supposed to fix the issue.
B. Software issues
Luckily, I currently don’t have the same issues as @Dragosr has described while dealing with the #BadBIOS sophisticated malware. My PC stills boots on CDs, using Legacy or UEFI bios, I can boot on the Windows 8 x64 original DVD, and more. But I have another software issue that drives me crazy :
Since a few months, Firefox has a weird behavior, when accessing several precise sites (my own sites, only, which is even more strange). I have a « Corrupted content » error when trying to display any page of my sites, even a simple blank page, on any of my domains.
– This only occurs on Firefox : No issue with IE, cygwin lynx, curl or wget for example.
– This occurs with a brand new install of Firefox, new profile and no plugins/extensions.
– This only occurs on this specific PC : the corrupted traffic issue doesn’t appear on my other laptops, even on the same subnet, even with the same Firefox version
As you can see above, the issue only occurs on Firefox, not IE… And the following screen capture shows the issue isn’t there when I use HTTPS :
Up to now, I didn’t notice this issue when accessing other sites than mine… So it may be the website, but I don’t catch why only Firefox on my PC would be affected (my hoster uses a Varnish proxy to cache the requests, but I already forced a cleaning of the cache several times without any improvement). I also first believed that it was caused by a bad WordPress installation of my blog, but since it occurs on *every* wordpress installation I have even on other domains I own, and also on very simple HTML pages such as on 9.éé.net, it’s not a WordPress issue…
Obviously, I deleted Firefox local cache, desactivated all extensions, started Firefox in safe mode, uninstalled Firefox, reinstalled a fresh new version (in another language just in case) in another location, ran it with or without a sandbox… but the behavior is always the same : « Corrupted content » !
I also though it could be caused by my firewall, which inspects traffic. So I ran the tests after desactivation, and lot of time spent trying several configurations and diagnostics, but the issue is still the same. And nothing special in the logs.
I am now in the process of investigating more of this, hoping I’ll be able to understand the issue. Wireshark on my PC *and* on a network bridged device may help, I hope. Maybe it’s a simple misconfiguration somewhere in between Firefox and the network stack…
During the last days, I experienced a bad crash of Windows 8, most probably caused by a driver issue, that kept me away from rebooting during several days… It could boot but always ended with a black screen, mouse cursor activated as well as the « Windows-P » shortcut to change screen from LCD to external monitor… Seems quite common according to blog posts, but the only way to restore the previous configuration is to boot into safe mode, which I never occured to make happen… Even with the original Windows 8 DVD and several evenings spent trying and reading support forums, I couldn’t make my way to safe mode, the option not being present, and the restore mode saying that my « drive was locked ». I was only able to Shift-F10 to access a Command-prompt and figure out that bcdedit couldn’t even be started (but Diskpart ran well). During those days, I learned a lot about UEFI, GPT, MSR, Secure boot and more, although it didn’t help.
Finally, I « solved » the issue with my backups, overwriting the C:\Windows directory with a recent backup archive, and was able to boot again (dirty solution, I admit, but the only one I could use, since even Windows 8 repair mode refused to reinstall a fresh OS, claiming the drive was locked…)
Now, I am investigating this too, along with my « Firefox » issue. Hoping I will find a solution and even most, the cause of all this mess… I have done some forensic images of the drive that may help locate corrupted system binaries, that will take some time…
Any advice would be appreciated !
Comment by Bruno Kerouanton on 2013-11-01 11:52:06 +0200
UPDATE 1 nov, 11h52: I’ve setup a brand new VM, and installed Windows XP from scratch using genuine install CD, then downloaded Firefox 25 US and installed it. The VM is running on my laptop, and configured to use NAT. Here is the result !
Getting weird to investigate… As you may have guessed, I’m now using another laptop (a mac, to be precise) to update my blog, since I’m unable to do it with my main PC… Time to get my old PC laptop out of the closet to start forensics…
Comment by Bruno Kerouanton on 2013-11-01 13:34:32 +0200
Based on some suggestions from @cynicalsecurity and @vickyjo (many thanks, to follow my adventures), I tried several things :
– (BSOD) Check SoDIMM : Already done several times, notably after changing the motherboards. I also did the « intensive hardware diagnostics » available from the Dell BIOS, which took several hours, and ended concluding my PC was fully operational, including all RAM banks.
– (Corrupted traffic) Check if the issue is linked to the NIC hardware checksum option : The issue remains whatever the network card I use : Embedded WiFi, Embedded wired Gbit, VPN.
– (Corrupted traffic) Check if the issue occurs at home, office or on different networks : Yes, in all cases ! I even tried setting a 3G hotspot (using my iPhone) and connecting directly to Internet bypassing my home network infrastructure, but the traffic from my websites is still corrupted on Firefox.
– (Corrupted traffic) Check my websites from somewhere else : @vickyjo confirms that there is no issue to get the pages on her side.
Next step for me : restore some old and recent system backups on several spare disks and boot on them, to see if there is changes…
Comment by Bruno Kerouanton on 2013-11-01 15:00:39 +0200
Insteresting : Using the same VM configured this time with its network interface in Bridge mode and not NAT as previously, the problem is no longer happening. That seems to mean as long as packets are flowing through Windows 8 network stack, the issue appears. If I bypass the stack, it’s solved.
Also, @cynicalsecurity asked me if I had gone into the windows driver to turn off the hardware checksum offloading. After checking, it seems no such option is available. I doubt the issue is at this level, anyway here are the advanced configuration options I found for both interfaces :
- Intel® 82579LM Gigabit Network Connexion – Driver date : 31.10.2012, version 126.96.36.19931 (up-to-date)
Flow control: RX and TX activated
Delestage ARP Protocol : Enabled
Delestage NS Protocol : Enabled
Low-power saving ethernet : On if system is awake
Intel System Idle Power : Disabled
Extended packet : Disabled
Packet priority and VLAN : Packet priority and vlan enabled
Link speed to save battery power : Disabled
Speed and duplex : Autoneg
Wake-on magic packet : Enabled
Wake-on pattern match : Enabled
Intel® Centrino® Ultimate-N 6300 AGN – Driver date : 22.08.2013, version 188.8.131.52 (up-to-date)
- Agressivité de l’itinérance (Roaming agressivity) : 3. Medium
Priviliged band : 1. No preference
Bluetooth® AMP : Enabled
Ad-hoc 802.11b/g channel: 1
Matching to WOL model : 1
ARP delestage for WoWLAN : enabled
NS delestage for WoWLAN :
Incompatible with 40MHz channels : Disabled
802.11n – 2.4 band channel Bandwidth : 20MHz
802.11n – 5.2 band channel Bandwidth : Auto
802.11n mode : Enabled
Ad-Hoc QoS mode : WMM disabled
Wireless mode : 6. 802.11a/b/g
Mixed mode protection : Auto-CTS enabled
Transmit power : 5. Max
GTK Keying regeneration for WoWLAN : enabled
Sleep mode at WoWLAN disconnect : Disabled
Wake on Magic Packet : Enabled
- Agressivité de l’itinérance (Roaming agressivity) : 3. Medium
(approximate translation from french)
Comment by Bruno Kerouanton on 2013-11-01 16:05:40 +0200
Another point that makes me doubting : If the issue is caused by the Windows network driver,
why is the data corruption occuring only on my own websites, as it seems I don’t have any issue with other sites ?
why is the trafic only altered on HTTP, as I don’t have any issue with other protocols such as SSH/SMB/RDP/VNC/FTP and more ? Encrypted traffic such as ssh for example should be less tolerant in such cases.
why is it only occuring with (anx version) of Firefox ?
This makes me think there is some kind of hook on port 80. That’s normally not what a network driver is supposed to do.
Comment by Bruno Kerouanton on 2013-11-01 18:14:42 +0200
Back to the checksum issue… I ran Wireshark and simply found a lot of « checksum=0 » errors.
the Intel 82579LM chip that is supposed to do the checksum calculation in hardware and send it to the OS seems not to do it. The specs below say it can, but I don’t find any option to enable the feature (normally it’s on by default)
Comment by jm on 2013-11-01 20:19:15 +0200
As-tu essayé avec firefox mais depuis linux (en bootant sur une clef usb). Ca devrait permettre de determiner si ca vient de la pile tcp/ip de windows ou pas…
Comment by Bruno Kerouanton on 2013-11-01 20:52:08 +0200
TCP Checksum issue solved : I finally managed to get rid of the original network drivers from Microsoft and Dell, that were shipped with Windows Update (and stated as « up to date » by Microsoft when checking), and went to the Intel website to download the « real » drivers, updated a few weeks ago: what a change ! Lots more options available in the hardware settings, and ability to enable/disable checksum calculation.
Now, Wireshark no longer complains about checksum=0 errors, but… I still have the same issue of data being seem as « corrupted » while in Firefox (still, only my websites, even a 100byte page), and okay elsewhere…
Comment by Bruno Kerouanton on 2013-11-02 08:24:09 +0200
@jm : Yes, I’ve tried with several distros on CD : no issue. I guess the problem is located between the Win8/Intel network driver and Firefox. But where ?
And so strange : everything is fine *except* my own sites. How can could it be so selective ?
Comment by Bruno Kerouanton on 2013-11-02 08:30:44 +0200
UPDATE 2 nov, 8:24
After sleeping, correcting some issues on my blog (I didn’t notice that the pages weren’t displaying correctly, and slowed-down so much it wasn’t possible to enter comments, sorry) and disabling bad extensions, back to the #BadPC issue :
Last action was that I had hardware checksum issues on my Intel drivers. Fixed by installing good drivers and setting the right parameters. But problem not solved.
This morning, I started Wireshark to listen to trafic generated by Firefox while accessing my « corrupted webpage » (http://9.éé.net).
On Wireshark, *everything* is normal, the request is basic, and the answer is totally clean and complete :
The pcap is here:
But Firefox still says my data is corrupted… so there *must* be something in-between the network stack and Firefox that corrupts the data…
Comment by Denis on 2013-11-02 10:33:29 +0200
Have you tried to host your site locally ? Have you tried to check Firefox Network Monitor ?
Comment by Bruno Kerouanton on 2013-11-05 18:32:33 +0200
I didn’t take the time yet to host the sites (or any webpage) locally, I should have a try, right.
On the other hand, I tried to proxify the traffic through Burp Suite… and like with Wireshark I can see the trafic is intact and correct :
– http trafic concerning my own sites are seen as « corrupt » by Firefox *and IE* (that’s something I didn’t notice until now, but it seems that IE doesn’t display my websites correctly, the pages are also partly broken
– I have no problem accessing any other sites, whether with IE or Firefox
– Wireshark sniffer and Burp suite proxy both see the traffic as correct.
– reinstalling Firefox, using another instance in a sandbox or *even* in a VM with NAT don’t correct the issue.
Based on this, I tend to believe that the traffic is being corrupted somewhere between the NIC driver and the browser, but I still really don’t see where it could be, since a sniffer or proxy sees good traffic…
any idea ?! I’m a bit lost…
Comment by Bruno Kerouanton on 2013-11-05 19:38:45 +0200
I’ve started looking at the network filters. Usually, on Windows, a network card has filters that can be enabled or disabled, such as IPv4, IPv6, Microsoft network, QoS, File and print sharing, and more. There are also Firewalls that come here, and add their packet filters, as well as VMware that needs to intercept traffic from/to VMs.
(sorry, it’s in french…)
I checked at my interface parameters, and disabled *all* network filters and protocols (except IPv4), and rebooted, hoping I would have an improvement. But, again, with all disabled including VMware packet filter and even the firewall packet filter, nothing changes…
So ??? What next… probably run a local version of my website and retry.
Comment by Bruno Kerouanton on 2013-11-05 19:50:41 +0200
Local test done…
I’ve taken a daily backup of my websites, fired a small HTTP server (included in my Utools suite), and tried to browse 127.0.0.1 : it works.
I also tried by replacing 127.0.0.1 by my local IP address (192.168.115.1), and it also works. But as soon as the trafic is coming from outside, it continues to be corrupted if it’s my website.
Comment by Bruno Kerouanton on 2013-11-05 21:10:34 +0200
Next step : I’ll setup another laptop with « fake » DNS + webserver, to answer with my real domain names, but locally, too see if something changes. But for now, I’m making my daily (!) full system backup, using a bootable cd. Will be interesting to study offline, but I’ve not started that yet.
Comment by Kinsk on 2013-11-12 10:36:35 +0200
L’hypothèse est une corruption des paquets entre l’interface réseau et le navigateur au niveau Win 8.
Ce phénomène semble propagé dans une VM XP en NAT.
As-tu fait un wireshark depuis la VM XP?
Tu devrais y trouver des paquets déjà altérés et par comparaison avec le pcap de l’hôte il y aura peut être un début de piste.
Comment by Bruno Kerouanton on 2013-11-12 13:27:31 +0200
Thanks for the feedback & tip. For the moment, I’m stuck again with my PC : motherboard had to be changed for the 4th time… and I’m now struggling since several days to restore my backups : of the 4 backups I have, none are able to start correctly, they all crash and Microsoft repair tools (automatic or manual) won’t fix them. I’m still spending hours waiting for my PC to restore different backups, setting each time different options… Very painful.
Comment by hik3 on 2013-11-12 13:31:57 +0200
Salut Bruno, as-tu fait la mise à jour à Windows 8.1 ? J’ai eu énormément de soucis avec la couche TCP/IP de Windows 8 (release RTM), notamment à cause du tunnel Teredo que Windows crée et ses diverses « interfaces » réseau virtuelles.
Comment by Bruno Kerouanton on 2013-11-17 12:14:23 +0200
Believe it or not…
Based on all the issues I had with my laptop, I decided to reinstall from scratch on a new hard-drive my Windows 8 OS and applications.
I installed Windows 8 from the Original DVD
I installed Mozilla Firefox 25.0.1 from Internet
I tried to go to « bruno.kerouanton.net », « 9.éé.net » and more : Corrupted content….
This is insane and nonsense !
Comment by Bruno Kerouanton on 2013-11-18 23:01:55 +0200
Tonight, I understood part of the problem, or more precisely what component caused the issue.
I feel a bit guilty on this… I wrote too fast yesterday, when I said I had reinstalled a fresh Windows 8 Pro, downloaded Firefox and tried to see that the data was corrupted…. It was, indeed, but I had started to install additional software such as Outpost Firewall.
Tonight, I reinstalled again another fresh Windows 8 pro (on a new hard drive, I have really many), tested with IE that my sites were displayed correctly, they were. Then I downloaded and installed Firefox 25, and retried : no issue, all was still displayed correctly. Then, I downloaded Agnitum Outpost Security Suite, installed it and retried : Data went corrupted !
So, the guilty component seems to be identified : The packet filter that is enabled when Outpost Firewall is installed, seems to add corruption on data coming from web sites that are hosted at Gandi.net and use Varnish Cache.
That explains why it was the only PC that causes issues (since it’s the only one with Outpost), and why it was caused either using wired or wifi connection.
That doesn’t explain why the Outpost network filter driver corrupts data coming from Gandi Simple Hosting, and why it was still active even after I desactivated all filter drivers in the network configuration, including Outpost Firewall driver.
Next step for me, will to install a fresh Windows+Outpost (on a VM?), and try to debug, first with the embedded logs, and then I hope not to have to debug the driver itself…. I know how to use IDA and Ollydbg, but in user-mode… I don’t want to mess with Windbg in kernel/driver mode ;(
And I guess I’ll start talking about this in Outpost forums, maybe it will help.
Sorry guys for keeping you waiting for some kind of #BadBIOS issue on my PC. But I hope I’ll find the answer for my issue, as I cannot go on any of my sites from my PC (5-6 months ago it worked, but I had a few previous versions of Outpost firewall installed. I guess new versions broke something somewhere.
Comment by Bruno Kerouanton on 2013-11-19 00:31:28 +0200
Okay, I’ve created a new discussion on Agnitum Outpost support forum… let’s see what happens.
Comment by Bruno Kerouanton on 2013-11-20 12:01:41 +0200
Follow-up of my LONG investigation…
1. Agnitum Outpost support
Support team on Agnitum user forum has started asking me for logs and more, still doing tests. You can follow the discussion here, I’ve posted some logs there too.
They also don’t understand the reason of this issue, yet.
2. « Bare Install » tests
I’ve started to do several tests using bare-install configurations, running on VMware on my Dell M4700 laptop (this detail is important, as it has Outpost Securitysuite installed).
Here is the basic procedure for each test :
create a new VM
set network to « NAT, Disable »
boot on the chosen Windows installation DVD
install Windows, reboot
copy the following setup binaries to VM desktop using drag-n-drop from host :
– OutpostFirewall v8.1.2 x32 4313.670.1936 (on XpPro VM only)
– OutpostSuitePro v8.1.2 x32 4313.670.1936 (on XpPro VM only)
– OutpostFirewall v8.1.2 x64 4313.670.1936 (on Win7 and 8 VMs)
– OutpostSuitePro v8.1.2 x64 4313.670.1936 (on Win7 and 8 VMs)
– Microsoft Installer 3.1 (on XpPro VM only)
– Microsoft C++ 2010 Redistribuable
set VM network to « NAT, Enabled »
test « 9.éé.net » on IE
test « 9.éé.net » on Firefox
make a VM snapshot
install Microsoft Installer 3.1 (on the XpPro VM only)
install Microsoft C++ 2010 Redist
install OutpostSuitePro v8.1.2
test « 9.éé.net » on IE
test « 9.éé.net » on Firefox
set VM network to « Bridge, Enabled »
test « 9.éé.net » on IE
test « 9.éé.net » on Firefox
revert to last snapshot
repeat steps 11 to 19 with OutpostFirewall instead of SecuritySuite (step 13)
I did those 23 steps using :
– Windows XP Pro (32 bit)
– Windows 7 Pro x64
– Windows 8 Pro x64
Here are the results of the steps 7, 9, [15, 16, 18, 19] x2
So, in short : As soon as packets flow through Outpost (Firewall or SecuritySuite), either installed on the VM, or on the host via NAT, they introduce a « Corrupted Content Error » when viewed on Firefox. And it is not OS-dependant.
In addition, it may have beeen possible that Firefox is less tolerant to errors than IE, which could explain why IE doesn’t complain and displays the page. But again, when doing « View source » on IE, the displayed page is complete from the first « » tag to the last « » tag, and doesn’t contain any noticeable error.
So this is just nonsense why there is an interaction with Firefox only… even if Outpost is located on the host machine and Firefox on the VM !
Obviously, next step will to start sniffing between the Host and the VM, to see differences when accessing the same page from Firefox and IE with and without Outpost Firewall installed…
Comment by Bruno Kerouanton on 2013-11-20 13:45:55 +0200
3. Firefox tests
Before starting sniffing, I did another test, using my bare Win8Pro+NAT (traffic flowing trough the host Outpost SecuritySuite)
– Downloaded and installed the following versions of Firefox in dedicated directories : 3.6.28, 10.0, 15.0, 20.0, 25.0, all en-us, from http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/
– Tried to access my pages. All Firefox versions display « corrupted content », except the old 3.6.28 version, which is simply not able to display anything, as if I didn’t ask for any URL. Interestingly enough, when I type « http://xn--9caa.net » which is redirected several times until it reaches « bruno.kerouanton.net/blog », I see the version 3.6.28 resolve the links, but again doesn’t display the page and stays on the original Firefox page.
So it seems all Firefox versions are affected.
Comment by Bruno Kerouanton on 2013-11-20 15:18:00 +0200
4. Other browsers
Clearly, I should have started here… I downloaded Chrome and Opera, which gives the following results :
Okay, those browsers are more verbose than Firefox. And, as often, Internet Explorer seems blind !
Let’s do the sniffing…