Question Immediate and consistent BSOD after enabling Driver Verifier

[Lando6090]

I keep getting constant BSOD, typically 2-3 back to back, which forces a restore to before enabling driver verifier. Driver at fault is amdgpio3.sys. This is a new build, I have already RMA'd the MOBO, Processor, and RAM. While playing games or doing other tasks I get intermittent BSOD for various issues (can't remeber the 30+ I've had and no longer have the dump files). Attached is the only Minidump I have as of now. System specs are as follows

CPU: Ryzen 5 7600x
Mobo: Asus X670E-i
Ram: Corsair32gb 5600Mhz (CMK32GX5M2B5600C36)
GPU: EVGA 3080ti XC3
SSD: Crucial P5 2TB
PSU: Corsair SF750

https://www.dropbox.com/sh/0e3vv5dpz6ns03v/AAAQYzXT8qienmpqihuh4k4Za?dl=0

 

[Bob.B]

I keep getting constant BSOD, typically 2-3 back to back, which forces a restore to before enabling driver verifier. Driver at fault is amdgpio3.sys. This is a new build, I have already RMA'd the MOBO, Processor, and RAM. While playing games or doing other tasks I get intermittent BSOD for various issues (can't remeber the 30+ I've had and no longer have the dump files). Attached is the only Minidump I have as of now. System specs are as follows

CPU: Ryzen 5 7600x
Mobo: Asus X670E-i
Ram: Corsair32gb 5600Mhz (CMK32GX5M2B5600C36)
GPU: EVGA 3080ti XC3
SSD: Crucial P5 2TB
PSU: Corsair SF750

https://www.dropbox.com/sh/0e3vv5dpz6ns03v/AAAQYzXT8qienmpqihuh4k4Za?dl=0

You might want to start with a quality 850w or larger psu.

 

[ubuysa]

In addition to the above, the dump highlights amdgpio3.sys as the driver which caused the BSOD. The driver caused a code integrity failure by specifying the wrong pool type (executable instead of non-executable). The amdgpio3.sys driver is part of the AMD chipset, the install date of the version you have is from January 2022...

2: kd> lmDva 0xfffff809`72131463
Browse full module list
start             end                 module name
fffff809`72130000 fffff809`7213b000   amdgpio3 T (no symbols)           
    Loaded symbol image file: amdgpio3.sys
    Image path: \SystemRoot\System32\drivers\amdgpio3.sys
    Image name: amdgpio3.sys
    Browse all global symbols  functions  data
    Timestamp:        Fri Jan 14 03:54:53 2022 (61E0D7ED)
    CheckSum:         000134E0
    ImageSize:        0000B000
    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4
    Information from resource tables:

It may be worth looking for a chipset driver update....

 

[Lando6090]

Yes, as odd as it is that is the most current driver for the chipset. The driver for gpio3 is old.

Edit: In device manager it's displaying that the driver is from September 15 2022.

 

[SkyNetRising]

I keep getting constant BSOD, typically 2-3 back to back, which forces a restore to before enabling driver verifier.

Driver Verifier is for driver development purposes only.
Normally you have it disabled.

Who told you to enable it?

 

[Lando6090]

Driver Verifier is for driver development purposes only.
Normally you have it disabled.

Who told you to enable it?

No one told me to, I wish I had the dump files from the plethora of crashes but all of them pointed to either RAM or drivers, I ran memtest for a total of 8 passes, which passed so that left drivers. After doing some research it seemed I could use driver verifier to locate which driver/hardware was faulty, thus landing on gpio3.

 

[johnbl]

driver verifier just flagged the driver
amdgpio3.sys
as doing something that is not recommended anymore.
Code Integrity Issue: The caller specified an executable pool type. (Expected: NonPagedPoolNx)

NonPagedPool (memory allocated for kernel drivers where code will run)
NonPagedPoolNx= means memory allocated but for data only ( no code execution)

Pool is just space allocated in RAM for kernel drivers. they allocated a space that could be used for code or data. This could be used for malware to attack the system. ie data is loaded into the memory and then later the cpu can be told to execute the data as code.

looks like this was done while the system plug and play was setting up the driver.
Normally you would now need to exclude this driver from the verifier.exe testing and continue testing.
use /driver.exclude DriverList
so add
/driver.exclude amdgpio3.sys
to your verifier command line
something like this:
run:
verifier.exe /standard /all /driver.exclude amdgpio3.sys

and amdgpio3.sys will not be tested.

since the bugcheck happens in 5 seconds you will have to boot into safe mode and run the command

or run verifier.exe /reset
to turn verifier off (while in safe mode) so you can boot.

This error is not likely the error that you are looking to find. it would be considered a warning to the programmer.

note: it did look like the stack was corrupted coming out of a call
Wdf01000!FxPnpDevicePrepareHardware
(raw stack looks reasonable so I would ignore and just look at the raw stack)

and when I look at the driver:
Wdf01000.sys (looks ok)
but
\SystemRoot\system32\drivers\wd\WdFilter.sys * Invalid
(microsoft antimalwar driver, maybe just need a windows update?)


was not able to read the bios info from the memory dump. Sometimes this means the the bios is really outdated. You should look at updating the bios or resetting it to defaults. BIOS makes a database of hardware that is sent to windows, this database is the starting point for the windows plug and play for configuration of the hardware.
even with a new motherboard you should update to the most current bios. Lots of bugs get fixed in bios after the product ships.

bios update:
ROG STRIX X670E-I GAMING WIFI - Support (asus.com)
4 updates shown

I would not install Armoury Crate
and make sure it is turned off in BIOS.
it installs files directly from bios onto your drive during boot up. Just like a rootkit
I tried it before an just got bugcheck after bugcheck and had to turn it off in bios and manually delete the files from the drive, and remove the installer from the system task scheduler. problem keeps coming back until you get all of the install vectors cut off. (guess it might be related to the problem you are looking for)

note: asus did not put the file dates for drivers on their website, just the driver version number. This makes it hard for me to quickly see which 3rd party drivers you have installed that are out of date.
for example:
the debugger shows:
\SystemRoot\System32\drivers\mtkwl6ex.sys Fri Nov 18 00:41:05 2022

but the website shows Version 3.3.0.713 14.14 MB
without a date. they might be the same version you will not know until you download and look at the driver. (looks like the download has a driver that is two days newer than the date of the driver you have installed, looks like some newer wifi firmware patches also)

asus is not putting the dates on the websited that the patch was last updated is really bad news.
People should complain to ASUS so they change it back to make it easier to see that a update is needed.

(fyi)
Most oem drivers still have driver dates, Microsoft stopped using the dates and now use
build hash: this is what a microsoft timestamp looks like in the debugger:
Timestamp: 11447CC9 (This is a reproducible build file hash, not a timestamp)
this is what the oem timestamp looks like in the debugger:
Timestamp: Tue Jul 12 05:22:17 2022 (62CD6779)
modified microsoft driver often show invalid in the debugger
some modified (hacked) drivers show a zero out date and time stamp

 

[Lando6090]

driver verifier just flagged the driver
amdgpio3.sys
as doing something that is not recommended anymore.
Code Integrity Issue: The caller specified an executable pool type. (Expected: NonPagedPoolNx)

NonPagedPool (memory allocated for kernel drivers where code will run)
NonPagedPoolNx= means memory allocated but for data only ( no code execution)

Pool is just space allocated in RAM for kernel drivers. they allocated a space that could be used for code or data. This could be used for malware to attack the system. ie data is loaded into the memory and then later the cpu can be told to execute the data as code.

looks like this was done while the system plug and play was setting up the driver.
Normally you would now need to exclude this driver from the verifier.exe testing and continue testing.
use /driver.exclude DriverList
so add
/driver.exclude amdgpio3.sys
to your verifier command line
something like this:
run:
verifier.exe /standard /all /driver.exclude amdgpio3.sys

and amdgpio3.sys will not be tested.

since the bugcheck happens in 5 seconds you will have to boot into safe mode and run the command

or run verifier.exe /reset
to turn verifier off (while in safe mode) so you can boot.

This error is not likely the error that you are looking to find. it would be considered a warning to the programmer.

note: it did look like the stack was corrupted coming out of a call
Wdf01000!FxPnpDevicePrepareHardware
(raw stack looks reasonable so I would ignore and just look at the raw stack)

and when I look at the driver:
Wdf01000.sys (looks ok)
but
\SystemRoot\system32\drivers\wd\WdFilter.sys * Invalid
(microsoft antimalwar driver, maybe just need a windows update?)


was not able to read the bios info from the memory dump. Sometimes this means the the bios is really outdated. You should look at updating the bios or resetting it to defaults. BIOS makes a database of hardware that is sent to windows, this database is the starting point for the windows plug and play for configuration of the hardware.
even with a new motherboard you should update to the most current bios. Lots of bugs get fixed in bios after the product ships.

bios update:
ROG STRIX X670E-I GAMING WIFI - Support (asus.com)
4 updates shown

I would not install Armoury Crate
and make sure it is turned off in BIOS.
it installs files directly from bios onto your drive during boot up. Just like a rootkit
I tried it before an just got bugcheck after bugcheck and had to turn it off in bios and manually delete the files from the drive, and remove the installer from the system task scheduler. problem keeps coming back until you get all of the install vectors cut off. (guess it might be related to the problem you are looking for)

note: asus did not put the file dates for drivers on their website, just the driver version number. This makes it hard for me to quickly see which 3rd party drivers you have installed that are out of date.
for example:
the debugger shows:
\SystemRoot\System32\drivers\mtkwl6ex.sys Fri Nov 18 00:41:05 2022

but the website shows Version 3.3.0.713 14.14 MB
without a date. they might be the same version you will not know until you download and look at the driver. (looks like the download has a driver that is two days newer than the date of the driver you have installed, looks like some newer wifi firmware patches also)

asus is not putting the dates on the websited that the patch was last updated is really bad news.
People should complain to ASUS so they change it back to make it easier to see that a update is needed.

(fyi)
Most oem drivers still have driver dates, Microsoft stopped using the dates and now use
build hash: this is what a microsoft timestamp looks like in the debugger:
Timestamp: 11447CC9 (This is a reproducible build file hash, not a timestamp)
this is what the oem timestamp looks like in the debugger:
Timestamp: Tue Jul 12 05:22:17 2022 (62CD6779)
modified microsoft driver often show invalid in the debugger
some modified (hacked) drivers show a zero out date and time stamp

Went ahead and excluded amdgpio3.sys, disabled armoury crate as well. Was able to boot properly this time without it throwing the bug check 5 seconds in.

As far as BIOS I already updated bios to version 0822, drivers have been installed directly from Asus, and I made sure they were up to date.

Going forward should I now just use my PC normally and see if it throws another check?

Edit: Thanks so much for the details, things are starting to make sense now.

 

[johnbl]

Went ahead and excluded amdgpio3.sys, disabled armoury crate as well. Was able to boot properly this time without it throwing the bug check 5 seconds in.

As far as BIOS I already updated bios to version 0822, drivers have been installed directly from Asus, and I made sure they were up to date.

Going forward should I now just use my PC normally and see if it throws another check?

Edit: Thanks so much for the details, things are starting to make sense now.

yes, run normally. delete the old .dmp files. on the next bugcheck provide the new one.
and double check the asus files, they like to come back since the armory crate puts a task in the tasks scheduler.
that downloads files from asus.

just see if you still get the bugchecks. if you do the next step would be to get a kernel dump to figure out the problem. Not many people look at kernel dumps but they contain the debug logs and more debug info about what is running on the computer. minidumps just show what was running on the core that called the bugcheck.

running verifier.exe is painful if you do not know how to get into safe mode and turn it off
via verifier.exe /reset
sometimes you even have to turn off fastboot or the timing window is too short to get into safe mode.
often you end up having boot, find a error, exclude the driver, try again then exclude the next driver until you get the system thru the first 1 minute of booting. lots of people do not know about getting into safe mode and doing the reset to enable booting.

there are switches for verifier so that it only turns it on for the next boot up but most people do not use it. it is not listed in the old verifier docs but is listed here:
verifier | Microsoft Learn
from above doc:
verifier /bootmode [persistent | disableafterfail | oneboot]

you would add /bootmode disableafterfail
or /bootmode oneboot
to your verifier line.

(I have never actually used it myself, option was not there before I retired)

 

[Lando6090]

yes, run normally. delete the old .dmp files. on the next bugcheck provide the new one.
and double check the asus files, they like to come back since the armory crate puts a task in the tasks scheduler.
that downloads files from asus.

just see if you still get the bugchecks. if you do the next step would be to get a kernel dump to figure out the problem. Not many people look at kernel dumps but they contain the debug logs and more debug info about what is running on the computer. minidumps just show what was running on the core that called the bugcheck.

Just to verify, should I run the PC normally with verifier enabled or disabled? With it enabled PC is very unresponsive (not a huge issue I can leave a game running in the background tonight)

 

[SkyNetRising]

Just to verify, should I run the PC normally with verifier enabled or disabled? With it enabled PC is very unresponsive

With Driver Verifier enabled you'll have BSODs all the time.
It is meant for Driver development purposes. Not for normal use.

 

[johnbl]

Just to verify, should I run the PC normally with verifier enabled or disabled? With it enabled PC is very unresponsive (not a huge issue I can leave a game running in the background tonight)

run normally without having verifer testing running.
a normal working system should be able to run verifier without bugchecking But often that is not the case if you have old drivers. Windows puts new rules into verifier as causes to problems are discovered.
if you do have verifier running your system will run slowly until you turn it off. Verifier checks every memory allocation for a bunch of potential coding errors and this really will slow down the system and take up a bunch of your limited memory. if the system is too slow then you have verifier only check certain drivers. Generally, just the 3rd party drivers that windows can not update. Microsoft drivers that have errors will automatically be reported to microsoft and microsoft will push out driver updates when it gets a fix. Third party drivers you have to install the updates manually. (microsoft still gets the error report but will not change out the 3rd party driver or update the bios for you (except for surface devices and certain oem machines)

 

[johnbl]

you can also just tell verifier to test a specific suspected driver.
I might test this one:
\SystemRoot\system32\DRIVERS\LifeCamTrueColor.sys Sun May 15 22:27:46 2016
just based on the date and I do not see this driver very often.

or just look to see if you can find a update for this driver since update or removal is your only real fix anyway. if this is a usb device any you have removed it : it turns out the driver still runs and you have to go into windows control panel device manager and find the menu option to show hidden devices, turn it on then find the greyed out device and delete the entry. Otherwise the driver still runs and can cause corruption even after the physical device has been removed. happens when people plug in devices and they don't work and they just unplug them and continue to have problems. most people end up reinstalling to get rid of the problem.