Question LatencyMon issue ?

[trenchy]

I've been experiencing some random pops in my recordings on my apollo, I ran into a bunch of performance guides but I am yet to get a good score on latencymon.

I'm not extremely knowledgeable on what I am looking at and would appreciate any help.

_________________________________________________________________________________________________________
CONCLUSION
_________________________________________________________________________________________________________
Your system appears to be having trouble handling real-time audio and other tasks. You are likely to experience buffer underruns appearing as drop outs, clicks or pops. One problem may be related to power management, disable CPU throttling settings in Control Panel and BIOS setup. Check for BIOS updates.
LatencyMon has been analyzing your system for 0:09:11 (h:mm:ss) on all processors.


_________________________________________________________________________________________________________
SYSTEM INFORMATION
_________________________________________________________________________________________________________
Computer name: MUSICBOX
OS version: Windows 11, 10.0, version 2009, build: 22631 (x64)
Hardware: MS-7D25, Micro-Star International Co., Ltd.
BIOS: 1.90
CPU: GenuineIntel 12th Gen Intel(R) Core(TM) i7-12700K
Logical processors: 12
Processor groups: 1
Processor group size: 12
RAM: 32555 MB total


_________________________________________________________________________________________________________
CPU SPEED
_________________________________________________________________________________________________________
Reported CPU speed (WMI): 3610 MHz
Reported CPU speed (registry): 3610 MHz

Note: reported execution times may be calculated based on a fixed reported CPU speed. Disable variable speed settings like Intel Speed Step and AMD Cool N Quiet in the BIOS setup for more accurate results.


_________________________________________________________________________________________________________
MEASURED INTERRUPT TO USER PROCESS LATENCIES
_________________________________________________________________________________________________________
The interrupt to process latency reflects the measured interval that a usermode process needed to respond to a hardware request from the moment the interrupt service routine started execution. This includes the scheduling and execution of a DPC routine, the signaling of an event and the waking up of a usermode thread from an idle wait state in response to that event.

Highest measured interrupt to process latency (µs): 14992.30
Average measured interrupt to process latency (µs): 4.522634

Highest measured interrupt to DPC latency (µs): 168.40
Average measured interrupt to DPC latency (µs): 2.066979


_________________________________________________________________________________________________________
REPORTED ISRs
_________________________________________________________________________________________________________
Interrupt service routines are routines installed by the OS and device drivers that execute in response to a hardware interrupt signal.

Highest ISR routine execution time (µs): 52.000554
Driver with highest ISR routine execution time: ndis.sys - Network Driver Interface Specification (NDIS), Microsoft Corporation

Highest reported total ISR routine time (%): 0.000903
Driver with highest ISR total time: ndis.sys - Network Driver Interface Specification (NDIS), Microsoft Corporation

Total time spent in ISRs (%) 0.001518

ISR count (execution time <250 µs): 84300
ISR count (execution time 250-500 µs): 0
ISR count (execution time 500-1000 µs): 0
ISR count (execution time 1000-2000 µs): 0
ISR count (execution time 2000-4000 µs): 0
ISR count (execution time >=4000 µs): 0


_________________________________________________________________________________________________________
REPORTED DPCs
_________________________________________________________________________________________________________
DPC routines are part of the interrupt servicing dispatch mechanism and disable the possibility for a process to utilize the CPU while it is interrupted until the DPC has finished execution.

Highest DPC routine execution time (µs): 273.358449
Driver with highest DPC routine execution time: nvlddmkm.sys - NVIDIA Windows Kernel Mode Driver, Version 546.33 , NVIDIA Corporation

Highest reported total DPC routine time (%): 0.018044
Driver with highest DPC total execution time: dxgkrnl.sys - DirectX Graphics Kernel, Microsoft Corporation

Total time spent in DPCs (%) 0.055974

DPC count (execution time <250 µs): 764246
DPC count (execution time 250-500 µs): 0
DPC count (execution time 500-10000 µs): 2
DPC count (execution time 1000-2000 µs): 0
DPC count (execution time 2000-4000 µs): 0
DPC count (execution time >=4000 µs): 0


_________________________________________________________________________________________________________
REPORTED HARD PAGEFAULTS
_________________________________________________________________________________________________________
Hard pagefaults are events that get triggered by making use of virtual memory that is not resident in RAM but backed by a memory mapped file on disk. The process of resolving the hard pagefault requires reading in the memory from disk while the process is interrupted and blocked from execution.

NOTE: some processes were hit by hard pagefaults. If these were programs producing audio, they are likely to interrupt the audio stream resulting in dropouts, clicks and pops. Check the Processes tab to see which programs were hit.

Process with highest pagefault count: system

Total number of hard pagefaults 5551
Hard pagefault count of hardest hit process: 3845
Number of processes hit: 44


_________________________________________________________________________________________________________
PER CPU DATA
_________________________________________________________________________________________________________
CPU 0 Interrupt cycle time (s): 4.686338
CPU 0 ISR highest execution time (µs): 52.000554
CPU 0 ISR total execution time (s): 0.026861
CPU 0 ISR count: 19116
CPU 0 DPC highest execution time (µs): 273.358449
CPU 0 DPC total execution time (s): 1.004371
CPU 0 DPC count: 336948
_________________________________________________________________________________________________________
CPU 1 Interrupt cycle time (s): 2.20380
CPU 1 ISR highest execution time (µs): 31.436565
CPU 1 ISR total execution time (s): 0.013984
CPU 1 ISR count: 13714
CPU 1 DPC highest execution time (µs): 99.303047
CPU 1 DPC total execution time (s): 0.565943
CPU 1 DPC count: 86755
_________________________________________________________________________________________________________
CPU 2 Interrupt cycle time (s): 1.839657
CPU 2 ISR highest execution time (µs): 34.270360
CPU 2 ISR total execution time (s): 0.010193
CPU 2 ISR count: 9863
CPU 2 DPC highest execution time (µs): 263.864266
CPU 2 DPC total execution time (s): 0.390764
CPU 2 DPC count: 59585
_________________________________________________________________________________________________________
CPU 3 Interrupt cycle time (s): 1.645355
CPU 3 ISR highest execution time (µs): 34.501385
CPU 3 ISR total execution time (s): 0.007483
CPU 3 ISR count: 8014
CPU 3 DPC highest execution time (µs): 91.239335
CPU 3 DPC total execution time (s): 0.357476
CPU 3 DPC count: 54686
_________________________________________________________________________________________________________
CPU 4 Interrupt cycle time (s): 1.636308
CPU 4 ISR highest execution time (µs): 26.031579
CPU 4 ISR total execution time (s): 0.009014
CPU 4 ISR count: 8193
CPU 4 DPC highest execution time (µs): 88.017729
CPU 4 DPC total execution time (s): 0.349644
CPU 4 DPC count: 53774
_________________________________________________________________________________________________________
CPU 5 Interrupt cycle time (s): 1.653419
CPU 5 ISR highest execution time (µs): 50.131856
CPU 5 ISR total execution time (s): 0.014679
CPU 5 ISR count: 10368
CPU 5 DPC highest execution time (µs): 92.160665
CPU 5 DPC total execution time (s): 0.382545
CPU 5 DPC count: 63259
_________________________________________________________________________________________________________
CPU 6 Interrupt cycle time (s): 1.640633
CPU 6 ISR highest execution time (µs): 32.165651
CPU 6 ISR total execution time (s): 0.012071
CPU 6 ISR count: 8999
CPU 6 DPC highest execution time (µs): 92.933518
CPU 6 DPC total execution time (s): 0.354719
CPU 6 DPC count: 59485
_________________________________________________________________________________________________________
CPU 7 Interrupt cycle time (s): 1.560002
CPU 7 ISR highest execution time (µs): 44.0
CPU 7 ISR total execution time (s): 0.006123
CPU 7 ISR count: 6033
CPU 7 DPC highest execution time (µs): 98.316898
CPU 7 DPC total execution time (s): 0.278446
CPU 7 DPC count: 45412
_________________________________________________________________________________________________________
CPU 8 Interrupt cycle time (s): 0.667016
CPU 8 ISR highest execution time (µs): 0.0
CPU 8 ISR total execution time (s): 0.0
CPU 8 ISR count: 0
CPU 8 DPC highest execution time (µs): 31.423269
CPU 8 DPC total execution time (s): 0.004522
CPU 8 DPC count: 1120
_________________________________________________________________________________________________________
CPU 9 Interrupt cycle time (s): 0.710008
CPU 9 ISR highest execution time (µs): 0.0
CPU 9 ISR total execution time (s): 0.0
CPU 9 ISR count: 0
CPU 9 DPC highest execution time (µs): 28.123546
CPU 9 DPC total execution time (s): 0.004713
CPU 9 DPC count: 1031
_________________________________________________________________________________________________________
CPU 10 Interrupt cycle time (s): 0.733357
CPU 10 ISR highest execution time (µs): 0.0
CPU 10 ISR total execution time (s): 0.0
CPU 10 ISR count: 0
CPU 10 DPC highest execution time (µs): 35.285319
CPU 10 DPC total execution time (s): 0.004890
CPU 10 DPC count: 1386
_________________________________________________________________________________________________________
CPU 11 Interrupt cycle time (s): 0.746052
CPU 11 ISR highest execution time (µs): 0.0
CPU 11 ISR total execution time (s): 0.0
CPU 11 ISR count: 0
CPU 11 DPC highest execution time (µs): 48.548476
CPU 11 DPC total execution time (s): 0.003684
CPU 11 DPC count: 807
_________________________________________________________________________________________________________

 

[Colif]

Driver with highest ISR routine execution time: ndis.sys - Network Driver Interface Specification (NDIS), Microsoft Corporation

try updating your Ethernet or Wifi drivers, which ever you use.

In my experience, Nvidia always at top so that isn't necessarily a sign they to blame
some of the answers in this could help

What are specs of the PC?

 

[trenchy]

try updating your Ethernet or Wifi drivers, which ever you use.

In my experience, Nvidia always at top so that isn't necessarily a sign they to blame
some of the answers in this could help

What are specs of the PC?

32gb ddr4 3600
12700k
3080 12gb

weirdly after a few restarts and a ddu reinstall of nVidia minus the geforce experience app, I seemingly have stabilized the machine.

~2 hour latencymon run with the largest interrupt at ~350 microseconds, in comparison to 15000 microseconds in the previous runs.

I did follow a ~80 page optimization guide that went over quite a few methods including processor power state (which I already applied long ago but important to do if you face this issue), core unparking, etc etc.

I'm not at all sure what fixed the issue, or if it will be stable forever. but regardless this guide helped me a lot with understanding realtime audio on windows or any operating system/

Glitch Free - Cantabile - Software for Performing Musicians

 

[Colif]

I did follow a ~80 page optimization guide that went over quite a few methods including processor power state (which I already applied long ago but important to do if you face this issue), core unparking, etc etc.

do you have a link to that?

 

[ubuysa]

weirdly after a few restarts and a ddu reinstall of nVidia minus the geforce experience app, I seemingly have stabilized the machine.

~2 hour latencymon run with the largest interrupt at ~350 microseconds, in comparison to 15000 microseconds in the previous runs.

According to that LatencyMon output the biggest contributor ro the latency was the dxgkrnl.sys DPC run time. The DPC code is part of the device driver, so using DDU and manually reinstalling the graphics driver may well have fixed the problem.

Note that dxgkrnl.sys is the Windows DirectX driver, but it will call nvlddmkm.sys, the Nvidia graphics card driver - that's where any problems will have been.

You might also be concerned about the number of hard page faults being reported. You ran LatencyMon for just over 9 minutes and had 5551 hard page faults in that time - that's quite high, especially in a 32GB system. The System process (which is running everything else) was the hardest hit with 3845 hard page faults in that time.

A consistently high hard page fault count is a strong indicator of RAM exhaustion and if you run out of RAM then you will see high latency, because the page-in process that results from a hard page fault also introduces latency. You might want to open Task Manager, click the Performance tab, then click the Memory icon on the left, and post a screenshot of that display - especially the numbers at the bottom.

 

[trenchy]

do you have a link to that?

its at the bottom of the reply, but here:

Glitch Free - Cantabile - Software for Performing Musicians

 

[trenchy]

According to that LatencyMon output the biggest contributor ro the latency was the dxgkrnl.sys DPC run time. The DPC code is part of the device driver, so using DDU and manually reinstalling the graphics driver may well have fixed the problem.

Note that dxgkrnl.sys is the Windows DirectX driver, but it will call nvlddmkm.sys, the Nvidia graphics card driver - that's where any problems will have been.

You might also be concerned about the number of hard page faults being reported. You ran LatencyMon for just over 9 minutes and had 5551 hard page faults in that time - that's quite high, especially in a 32GB system. The System process (which is running everything else) was the hardest hit with 3845 hard page faults in that time.

A consistently high hard page fault count is a strong indicator of RAM exhaustion and if you run out of RAM then you will see high latency, because the page-in process that results from a hard page fault also introduces latency. You might want to open Task Manager, click the Performance tab, then click the Memory icon on the left, and post a screenshot of that display - especially the numbers at the bottom.

i have this problem consistently indeed. My latency issue is solved but the page faults remain stupidly high - the sources changing usually from chrome or whatever game I play during tests.

https://drive.google.com/file/d/1G1svoR-SpqymWm1st8_qUBXbiwS1iGvq/view?usp=sharing

heres that screenshot.

 

[ubuysa]

There's no issue there, you might want to look at that memory data when you're seeing lagging.

 

[trenchy]

There's no issue there, you might want to look at that memory data when you're seeing lagging.

So I should check the bottom two numbers if I experience stutters in game/processing audio? I don't really experience any lagging other than the audio pops I specified before - but these have seemed to be fixed by whichever step in the plethora of changes I made recently.

 

[ubuysa]

What you want to look at in those numbers are the Committed values*.

The left Commit number is the amount of virtual storage that Windows has committed at the moment. The right number is the maximum virtual storage that Windows can ever commit.

If the left-hand commit number (the virtual storage committed now) is consistently larger than your installed RAM (at top right of this display) then you are short of RAM and processes are paging - this causes latency. This is going to be unlikely on a 32GB system, but it's not impossible.

It's also important to check that Windows reports all 32GB of RAM as being available. It's possible for one RAM stick to be bad and not used.

I think the solution was using DDU and reinstalling the graphics driver. That was where the long running DPC was being seen.

*Some detailed explanations if you're interested
A definition first; threads talk about memory, if they need more memory then they allocate additional memory. Windows talks about virtual storage (which is RAM and the paging file). For our purposes memory = virtual storage.

When any thread wants to allocate more memory it must ask permission from the Windows Memory Manager. As long as the current commit value, plus the size of memory requested, does not exceed the max commit value then Windows will allow the request and add the size of memory requested to the current commit value. The current commit value is thus the total amount of virtual storage that Windows has given permission to be used.

Note that committed memory is just virtual storage that has been allocated, it doesn't mean that any of it has actually been used yet. None of it may end up being used, for example. However, virtual storage must exist somewhere, and that's either in RAM or in the paging file on disk. (The max Committed value is just the sum of the installed RAM and the current size of the paging file (the paging file can grow in size in some circumstances)).

When a thread wants to use any of this allocated memory Windows has to find free pages in RAM in which to store the data. If RAM is being fully used then some other, less regularly referenced, pages will be copies out to the paging file and those RAM pages used instead for this thread's data.

If this thread then doesn't reference that memory for some time, and if RAM is being fully used, then eventually the pages holding that thread's data will be copied out to the paging file to make room for another thread's data. When the original thread then later wants to read from that memory the Windows Memory Manager has to first put the thread in a wait state. Then it must locate the slots in the paging file where the data was copied out, find some free pages in RAM (which might mean paging out another thread's unused pages), and then read in those pages into RAM. Only then can the Memory Manager remove the wait and allow the thread to continue running. All of that wait time is latency - time your thread wasn't running when it wanted to.

In a 32GB system I would not expect to see this kind of demand paging because 32GB of RAM shouldn't be fully utilised - unless you're running lots of VMs of course.

It is important to realise that paging happens for other reasons too. Threads can voluntarily page out unused parts of their code and data, and Windows can voluntarily page out parts of its code and data too. This means that not all of the paging you're seeing in that LatencyMon report is necessarily demand paging.