performance – When should we use prefetch?

On recent Intel chips one reason you apparently might want to use prefetching is to avoid CPU power-saving features artificially limiting your achieved memory bandwidth. In this scenario, simple prefetching can as much as double your performance versus the same code without prefetching, but it depends entirely on the selected power management plan.

I ran a simplified version (code here) of the test in Leeor’s answer, which stresses the memory subsystem a bit more (since that’s where prefetch will help, hurt or do nothing). The original test stressed the CPU in parallel with the memory subsystem since it added together every int on each cache line. Since typical memory read bandwidth is in the region of 15 GB/s, that’s 3.75 billion integers per second, putting a pretty hard cap on the maximum speed (code that isn’t vectorized will usually process 1 int or less per cycle, so a 3.75 GHz CPU will be about equally CPU and memory bount).

First, I got results that seemed to show prefetching kicking butt on my i7-6700HQ (Skylake):

Prefetching off: SIZE=256 MiB, sum=1407374589952000, time=221, MiB/s=11583
Prefetching  on: SIZE=256 MiB, sum=1407374589952000, time=153, MiB/s=16732
Prefetching off: SIZE=256 MiB, sum=1407374589952000, time=221, MiB/s=11583
Prefetching  on: SIZE=256 MiB, sum=1407374589952000, time=160, MiB/s=16000
Prefetching off: SIZE=256 MiB, sum=1407374589952000, time=204, MiB/s=12549
Prefetching  on: SIZE=256 MiB, sum=1407374589952000, time=160, MiB/s=16000
Prefetching off: SIZE=256 MiB, sum=1407374589952000, time=200, MiB/s=12800
Prefetching  on: SIZE=256 MiB, sum=1407374589952000, time=160, MiB/s=16000
Prefetching off: SIZE=256 MiB, sum=1407374589952000, time=201, MiB/s=12736
Prefetching  on: SIZE=256 MiB, sum=1407374589952000, time=157, MiB/s=16305
Prefetching off: SIZE=256 MiB, sum=1407374589952000, time=197, MiB/s=12994
Prefetching  on: SIZE=256 MiB, sum=1407374589952000, time=157, MiB/s=16305

Eyeballing the numbers, prefetch achieves something a bit above 16 GiB/s and without only about 12.5, so prefetch is increasing speed by about 30%. Right?

Not so fast. Remembering that the powersaving mode has all sorts of wonderful interactions on modern chips, I changed my Linux CPU governor to performance from the default of powersave1. Now I get:

Prefetching off: SIZE=256 MiB, sum=1407374589952000, time=155, MiB/s=16516
Prefetching  on: SIZE=256 MiB, sum=1407374589952000, time=157, MiB/s=16305
Prefetching off: SIZE=256 MiB, sum=1407374589952000, time=153, MiB/s=16732
Prefetching  on: SIZE=256 MiB, sum=1407374589952000, time=144, MiB/s=17777
Prefetching off: SIZE=256 MiB, sum=1407374589952000, time=144, MiB/s=17777
Prefetching  on: SIZE=256 MiB, sum=1407374589952000, time=153, MiB/s=16732
Prefetching off: SIZE=256 MiB, sum=1407374589952000, time=152, MiB/s=16842
Prefetching  on: SIZE=256 MiB, sum=1407374589952000, time=153, MiB/s=16732
Prefetching off: SIZE=256 MiB, sum=1407374589952000, time=153, MiB/s=16732
Prefetching  on: SIZE=256 MiB, sum=1407374589952000, time=159, MiB/s=16100
Prefetching off: SIZE=256 MiB, sum=1407374589952000, time=163, MiB/s=15705
Prefetching  on: SIZE=256 MiB, sum=1407374589952000, time=161, MiB/s=15900

It’s a total toss-up. Both with and without prefetching seem to perform identically. So either hardware prefetching is less aggressive in the high powerssaving modes, or there is some other interaction with power saving that behaves differently with the explicit software prefetches.

Investigation

In fact, the difference between prefetching and not is even more extreme if you change the benchark. The existing benchmark alternates between runs with prefetching on and off, and it turns out that this helped the “off” variant because the speed increase which occurs in the “on” test partly carries over to the subsequent off test2. If you run only the “off” test you get results around 9 GiB/s:

Prefetching off: SIZE=256 MiB, sum=1407374589952000, time=280, MiB/s=9142
Prefetching off: SIZE=256 MiB, sum=1407374589952000, time=277, MiB/s=9241
Prefetching off: SIZE=256 MiB, sum=1407374589952000, time=285, MiB/s=8982

… versus about 17 GiB/s for the prefetching version:

Prefetching  on: SIZE=256 MiB, sum=1407374589952000, time=149, MiB/s=17181
Prefetching  on: SIZE=256 MiB, sum=1407374589952000, time=148, MiB/s=17297
Prefetching  on: SIZE=256 MiB, sum=1407374589952000, time=148, MiB/s=17297

So the prefetching version is almost twice as fast.

Let’s take a look at what’s going on with perf statfor both the **off* version:

Performance counter stats for ‘./prefetch-test off’:

   2907.485684      task-clock (msec)         #    1.000 CPUs utilized                                          
 3,197,503,204      cycles                    #    1.100 GHz                    
 2,158,244,139      instructions              #    0.67  insns per cycle        
   429,993,704      branches                  #  147.892 M/sec                  
        10,956      branch-misses             #    0.00% of all branches     

… and the on version:

   1502.321989      task-clock (msec)         #    1.000 CPUs utilized                          
 3,896,143,464      cycles                    #    2.593 GHz                    
 2,576,880,294      instructions              #    0.66  insns per cycle        
   429,853,720      branches                  #  286.126 M/sec                  
        11,444      branch-misses             #    0.00% of all branches

The difference is that the version with prefetching on consistently runs at the max non-turbo frequency of ~2.6 GHz (I have disabled turbo via an MSR). The version without prefetching, however, has decided to run at a much lower speed of 1.1 GHz. Such large CPU differences often also reflect a large difference in uncore frequency, which can explain the worse bandwdith.

Now we’ve seen this before, and it is probably an outcome of the Energy Efficient Turbo feature on recent Intel chips, which try to ramp down the CPU frequency when they determine a process is mostly memory bound, presumably since increased CPU core speed doesn’t provide much benefit in those cases. As we can see here, this assumption isn’t always true, but it isn’t clear to me if the tradeoff is a bad one in general, or perhaps the heuristic only occasionally gets it wrong.


1 I’m running the intel_pstate driver, which is the default for Intel chips on recent kernels which implements “hardware p-states”, also known as “HWP”. Command used: sudo cpupower -c 0,1,2,3 frequency-set -g performance.

2 Conversely, the slowdown from the “off” test partly carries over into the “on” test, although the effect is less extreme, possibly because the powersaving “ramp up” behavior is faster than “ramp down”.

Leave a Comment