Dual Xeon, Turbo Boost problem

[CZmiho]

Hi, i have 2x Xeon E5 2687W v2 on Supermicro X9DAi running Windows 8.1.

Intel Turbo Boost is acting very funny on this computer:
On no load CPU runs at 1.2 GHz - this is ok.
On full load (32 threads) both CPUs runs at 3.6 GHz - this is ok.
On 1-2 thread load CPUs runs at 3.4 GHz (expected 4 GHz or 3.9 GHz)
On 3-4 thread load CPU runs also at 3.4. When i manually set affinity to force all threads to run on single CPU, CPU runs at 3.6-3.7

So - massively muti-threaded apps runs super fast (2495 points Cinebench R15). Workloads with low number of threads (this includes also even most demanding games ;-) ) runs rather slow. Single-threaded Cinebench R15 - 120 points.

Switching off HT helps a little bit. It is easier to "force" computer to go into TB.

I use Argus Monitor, ThrottleStop, and Intel's own Tubo Boost monitoring app to watch current frequency.

There is ugly workaround to this problem. When i set in power management minimal cpu speed to 100%, problem disappears (single-threaded Cinebench R15 - 138 points). Computer power usage is 30W higher at idle though.

I have latest BIOS, latest drivers, latest Intel INF. EIST, Turbo are enabled in BIOS. Fiddling with NUMA, C1E, enabled C states, core parking makes no difference.

Any help would be appreciated.

 

[Karsten75]

I can almost make a case that this is intended behavior. However, I can see why you might not expect that. I suggest you open a direct inquiry with Intel Tech support.

It might be an aspect of power management on the motherboard though. I recall that on the consumer-grade CPUs, some motherboard manufacturers "tweaked" things in power management so that the CPUs would always run in Turbo boost mode, thus effectively overclocking them by 10%.

SO you could also explore this with your motherboard manufacturer. I'd start with Intel to get the straight dope on exactly how the CPU should know to kick one or more cores into Turbo boost mode.

 

[CZmiho]

Friend suggested me to try Linux. In live Ubuntu 14.04 Turbo Boost works perfectly. So it is some Windows/drivers issue.

BTW this motherboard isn't consumer grade... i hope. Price tag is 500$ so it shouldn't be anyway ;-) And there is exactly 0 overclocking options, this is usual in Xeon/workstation motherboards.

 

[bambiboom]

CZmiho,

Yours is an odd situation. According to Intel, the E5-2687w V2 should run on the first two cores at 4.0GHz and on all cores at 3.4GHz. I think there can be a slight speed penalty when there are dual CPU's, but this should be minimal.

You mentioned looking into power management, but you might check more carefully again into all aspects of hibernation and power saving. I had a situation in which my HP z420 (E5-1620, Win 7 Ultimate) suddenly dropped in performance. In Passmark Performance Test, the system rating dropped overnight from 3815 to 2593. Importantly, while the 2D and 3D graphics scores were similar to the original, the CPU score was significantly reduced from 8858 to 5008 and I at first believed that two cores had failed. However, the IT expert at my local wind tunnel suggested to check the power saving / sleep modes as Windows updates can sometimes reset these. And, there it was, set for extreme power saving. Reset to "performance" the system rating actually was a bit higher than previously at 3923 and the CPU score went from 8858 to 9251. This was such a simple solution with such profound effect, it seemed impossible.

It's unlikely the power saving settings are the cause as you appear to be comprehensively methodical- more knowledgeable than I, but is there a chance there is thermal throttling? My memory of the 150W E5-2687w V2 is that it is one of the very few CPU's that Intel recommends for liquid cooling. A constant diet of something like MATLAB might cause CPU indigestion,..

Sorry I can't be of more definite help. I'd certainly be interested to know the eventual solution.

Cheers,

BambiBoom

HP z420 (2014) > Xeon E5-1620 quad core @ 3.6 / 3.8GHz > 24GB ECC 1600 RAM > Quadro 4000 (2GB)> Samsung 840 SSD 250GB /Western Digital Black WD1003FZEX 1TB> M-Audio 192 sound card > AE3000 USB WiFi > HP 2711X, 27" 1920 X 1080 > Windows 7 Ultimate 64 >[Passmark system rating = 3923, 2D= 839 / 3D=2048]

Dell Precision T5400 (2008) > 2X Xeon X5460 quad core @3.16GHz > 16GB ECC 667> Quadro FX 4800 (1.5GB) > WD RE4 500GB / Seagate Barracuda 500GB > M-Audio 2496 Sound Card / Linksys 600N WiFi > Windows 7 Ultimate 64-bit >[Passmark system rating = 1859, 2D= 512 / 3D=1097]

2D, 3D CAD, Image Processing, Rendering, Text > Architecture, industrial design, graphic design, written projects

 

[CZmiho]

So far I haven't found a solution. Supermicro didn't bother to answer the question. So I made a simple C# program that changes the settings according to the current CPU utilization. This solution is still ugly but not as ugly as the permanent setting minimal cpu state to 100%.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace Supermicro_sucks
{
    class Program
    {

        static System.Diagnostics.Process process;
        static int current = 0;
        static float lastusage = 0;
        static void setstate(int i)
        {
            if (current == i) return;

            //System.Console.WriteLine("Transition to " + i.ToString());

            process = new System.Diagnostics.Process();


            process.StartInfo.FileName = "Powercfg.exe";
            process.StartInfo.Arguments = "-setacvalueindex SCHEME_CURRENT SUB_PROCESSOR PROCTHROTTLEMIN " + i.ToString();
            process.StartInfo.RedirectStandardInput = true;
            process.StartInfo.RedirectStandardOutput = false;
            process.StartInfo.UseShellExecute = false;
            process.StartInfo.WindowStyle = System.Diagnostics.ProcessWindowStyle.Hidden;            
            process.StartInfo.CreateNoWindow = true;
            process.Start();

            System.Threading.Thread.Sleep(100);

            process = new System.Diagnostics.Process();
            process.StartInfo.FileName = "Powercfg.exe";
            process.StartInfo.Arguments = "-setactive SCHEME_CURRENT";
            process.StartInfo.RedirectStandardInput = true;
            process.StartInfo.RedirectStandardOutput = false;
            process.StartInfo.UseShellExecute = false;
            process.StartInfo.WindowStyle = System.Diagnostics.ProcessWindowStyle.Hidden;
            process.StartInfo.CreateNoWindow = true;
            process.Start();

            current = i;

        }

        static void Main(string[] args)
        {

            setstate(5);
            System.Diagnostics.PerformanceCounter pc = new System.Diagnostics.PerformanceCounter();
            pc.CategoryName = "Processor";
            pc.CounterName = "% Processor Time";
            pc.InstanceName = "_Total";


            while (true)
            {

                System.Threading.Thread.Sleep(400);

                
                float usage = pc.NextValue();
                //System.Console.WriteLine(usage);
                if (lastusage >= 3 && usage >= 3 && current < 100) setstate(100);
                else if (lastusage < 3 && usage < 3 && current == 100) setstate(5);
                lastusage = usage;

            }



        }
    }
}

 

[bambiboom]

CZmiho,

An interesting approach to the problem with an understandable namespace entry.

More naive conjectures:

1. Thinking about this, it occurred to me to ask the version number of the X9DAi motherboard. There must be 6 or 7 different /X versions. Is the BIOS on the Supermicro site the same for all versions? As far as I know there are differences in the BIOS for MB's designed for the original or v2 CPU's and there

2. There may be still some Supermicro X9DAi versions that are limited to the 135W Xeon series 26XX CPU's. Could these potentially throttle by power limitations to the socket? You might consider trying the system with only one CPU.

Again, naive ideas on my part, but when someone is not having full performance from $4,400 of CPU's by the maker of the best performing dual Xeon boards, this problem is an important one. This is the MB I would choose for a dual Xeon system. In Passmark Performance Test, the highest performing Xeons use it..

If you would care to, I'd enjoy knowing what happens.

Cheers,

BambiBoom

 

[CZmiho]

Bambiboom, thanks for you answer. There are only two SKUs of this motherboard (MBD-X9DAi -O = retail package, MBD-X9DAi -B = bulk package) both of them are 150W(!) TDP ready from the begining of this product. Both "versions" share the same BIOS.

I think that this is Windows/driver issue. As I said before- there is no problem in Linux at all.

There is "power accounting policy" in windows server that could possibly cause this problem, see http://blogs.technet.com/b/winserverperformance/archive/2008/12/04/configuring-windows-server-2008-power-parameters-for-increased-power-efficiency.aspx

There is no such thing in windows 8 though.