We have recently purchased some new servers and are experiencing poor memcpy performance. The memcpy performance is 3x slower on the servers compared to our laptops.

**Server Specs**

 - Chassis and Mobo: SUPER MICRO 1027GR-TRF
 - CPU: 2x Intel Xeon E5-2680 @ 2.70 Ghz
 - Memory: 8x 16GB DDR3 1600MHz

Edit: I am also testing on another server with slightly higher specs and seeing the same results as the above server

**Server 2 Specs**

 - Chassis and Mobo: SUPER MICRO 10227GR-TRFT
 - CPU: 2x Intel Xeon E5-2650 v2 @ 2.6 Ghz
 - Memory: 8x 16GB DDR3 1866MHz

**Laptop Specs**

 - Chassis: Lenovo W530
 - CPU: 1x Intel Core i7 i7-3720QM @ 2.6Ghz
 - Memory: 4x 4GB DDR3 1600MHz

**Operating System**

    $ cat /etc/redhat-release
    Scientific Linux release 6.5 (Carbon) 
    $ uname -a                      
    Linux r113 2.6.32-431.1.2.el6.x86_64 #1 SMP Thu Dec 12 13:59:19 CST 2013 x86_64 x86_64 x86_64 GNU/Linux

**Compiler (on all systems)**

    $ gcc --version
    gcc (GCC) 4.6.1

Also tested with gcc 4.8.2 based on a suggestion from @stefan. There was no performance difference between compilers.


**Test Code**
The test code below is a canned test to duplicate the problem i am seeing in our production code. I know this benchmark is simplistic but it was able to exploit and identify our problem. The code creates two 1GB buffers and memcpys between them, timing the memcpy call. You can specify alternate buffer sizes on the command line using: ./big_memcpy_test [SIZE_BYTES]

    #include &lt;chrono&gt;
    #include &lt;cstring&gt;
    #include &lt;iostream&gt;
    #include &lt;cstdint&gt;
    
    class Timer
    {
     public:
      Timer()
          : mStart(),
            mStop()
      {
        update();
      }
    
      void update()
      {
        mStart = std::chrono::high_resolution_clock::now();
        mStop  = mStart;
      }
    
      double elapsedMs()
      {
        mStop = std::chrono::high_resolution_clock::now();
        std::chrono::milliseconds elapsed_ms =
            std::chrono::duration_cast&lt;std::chrono::milliseconds&gt;(mStop - mStart);
        return elapsed_ms.count();
      }
    
     private:
      std::chrono::high_resolution_clock::time_point mStart;
      std::chrono::high_resolution_clock::time_point mStop;
    };
    
    std::string formatBytes(std::uint64_t bytes)
    {
      static const int num_suffix = 5;
      static const char* suffix[num_suffix] = { &quot;B&quot;, &quot;KB&quot;, &quot;MB&quot;, &quot;GB&quot;, &quot;TB&quot; };
      double dbl_s_byte = bytes;
      int i = 0;
      for (; (int)(bytes / 1024.) &gt; 0 &amp;&amp; i &lt; num_suffix;
           ++i, bytes /= 1024.)
      {
        dbl_s_byte = bytes / 1024.0;
      }
    
      const int buf_len = 64;
      char buf[buf_len];
    
      // use snprintf so there is no buffer overrun
      int res = snprintf(buf, buf_len,&quot;%0.2f%s&quot;, dbl_s_byte, suffix[i]);
      
      // snprintf returns number of characters that would have been written if n had
      //       been sufficiently large, not counting the terminating null character.
      //       if an encoding error occurs, a negative number is returned.
      if (res &gt;= 0)
      {
        return std::string(buf);
      }
      return std::string();
    }
    
    void doMemmove(void* pDest, const void* pSource, std::size_t sizeBytes)
    {
      memmove(pDest, pSource, sizeBytes);
    }
    
    int main(int argc, char* argv[])
    {
      std::uint64_t SIZE_BYTES = 1073741824; // 1GB
      
      if (argc &gt; 1)
      {
        SIZE_BYTES = std::stoull(argv[1]);
        std::cout &lt;&lt; &quot;Using buffer size from command line: &quot; &lt;&lt; formatBytes(SIZE_BYTES)
                  &lt;&lt; std::endl;
      }
      else
      {
        std::cout &lt;&lt; &quot;To specify a custom buffer size: big_memcpy_test [SIZE_BYTES] \n&quot;
                  &lt;&lt; &quot;Using built in buffer size: &quot; &lt;&lt; formatBytes(SIZE_BYTES)
                  &lt;&lt; std::endl;
      }
      
    
      // big array to use for testing
      char* p_big_array = NULL;
    
      /////////////
      // malloc 
      {
        Timer timer;
      
        p_big_array = (char*)malloc(SIZE_BYTES * sizeof(char));
        if (p_big_array == NULL)
        {
          std::cerr &lt;&lt; &quot;ERROR: malloc of &quot; &lt;&lt; SIZE_BYTES &lt;&lt; &quot; returned NULL!&quot;
                    &lt;&lt; std::endl;
          return 1;
        }
        
        std::cout &lt;&lt; &quot;malloc for &quot; &lt;&lt; formatBytes(SIZE_BYTES) &lt;&lt; &quot; took &quot;
                  &lt;&lt; timer.elapsedMs() &lt;&lt; &quot;ms&quot;
                  &lt;&lt; std::endl;
      }
      
      /////////////
      // memset
      {
        Timer timer;
    
        // set all data in p_big_array to 0
        memset(p_big_array, 0xF, SIZE_BYTES * sizeof(char));
    
        double elapsed_ms = timer.elapsedMs();
        std::cout &lt;&lt; &quot;memset for &quot; &lt;&lt; formatBytes(SIZE_BYTES) &lt;&lt; &quot; took &quot;
                  &lt;&lt; elapsed_ms &lt;&lt; &quot;ms &quot;
                  &lt;&lt; &quot;(&quot; &lt;&lt; formatBytes(SIZE_BYTES / (elapsed_ms / 1.0e3)) &lt;&lt; &quot; bytes/sec)&quot;
                  &lt;&lt; std::endl;
      }
    
      /////////////
      // memcpy 
      {
        char* p_dest_array = (char*)malloc(SIZE_BYTES);
        if (p_dest_array == NULL)
        {
          std::cerr &lt;&lt; &quot;ERROR: malloc of &quot; &lt;&lt; SIZE_BYTES &lt;&lt; &quot; for memcpy test&quot;
                    &lt;&lt; &quot; returned NULL!&quot;
                    &lt;&lt; std::endl;
          return 1;
        }
        memset(p_dest_array, 0xF, SIZE_BYTES * sizeof(char));
    
        // time only the memcpy FROM p_big_array TO p_dest_array
        Timer timer;
    
        memcpy(p_dest_array, p_big_array, SIZE_BYTES * sizeof(char));
        
        double elapsed_ms = timer.elapsedMs();
        std::cout &lt;&lt; &quot;memcpy for &quot; &lt;&lt; formatBytes(SIZE_BYTES) &lt;&lt; &quot; took &quot;
                  &lt;&lt; elapsed_ms &lt;&lt; &quot;ms &quot;
                  &lt;&lt; &quot;(&quot; &lt;&lt; formatBytes(SIZE_BYTES / (elapsed_ms / 1.0e3)) &lt;&lt; &quot; bytes/sec)&quot;
                  &lt;&lt; std::endl;
    
        // cleanup p_dest_array
        free(p_dest_array);
        p_dest_array = NULL;
      }
    
      /////////////
      // memmove
      {
        char* p_dest_array = (char*)malloc(SIZE_BYTES);
        if (p_dest_array == NULL)
        {
          std::cerr &lt;&lt; &quot;ERROR: malloc of &quot; &lt;&lt; SIZE_BYTES &lt;&lt; &quot; for memmove test&quot;
                    &lt;&lt; &quot; returned NULL!&quot;
                    &lt;&lt; std::endl;
          return 1;
        }
        memset(p_dest_array, 0xF, SIZE_BYTES * sizeof(char));
    
        // time only the memmove FROM p_big_array TO p_dest_array
        Timer timer;
    
        // memmove(p_dest_array, p_big_array, SIZE_BYTES * sizeof(char));
        doMemmove(p_dest_array, p_big_array, SIZE_BYTES * sizeof(char));
        
        double elapsed_ms = timer.elapsedMs();
        std::cout &lt;&lt; &quot;memmove for &quot; &lt;&lt; formatBytes(SIZE_BYTES) &lt;&lt; &quot; took &quot;
                  &lt;&lt; elapsed_ms &lt;&lt; &quot;ms &quot;
                  &lt;&lt; &quot;(&quot; &lt;&lt; formatBytes(SIZE_BYTES / (elapsed_ms / 1.0e3)) &lt;&lt; &quot; bytes/sec)&quot;
                  &lt;&lt; std::endl;
    
        // cleanup p_dest_array
        free(p_dest_array);
        p_dest_array = NULL;
      }
    
    
      // cleanup
      free(p_big_array);
      p_big_array = NULL;
      
      return 0;
    }


**CMake File to Build**

    project(big_memcpy_test)
    cmake_minimum_required(VERSION 2.4.0)
    
    include_directories(${CMAKE_CURRENT_SOURCE_DIR})
    
    # create verbose makefiles that show each command line as it is issued
    set( CMAKE_VERBOSE_MAKEFILE ON CACHE BOOL &quot;Verbose&quot; FORCE )
    # release mode
    set( CMAKE_BUILD_TYPE Release )
    # grab in CXXFLAGS environment variable and append C++11 and -Wall options
    set( CMAKE_CXX_FLAGS &quot;${CMAKE_CXX_FLAGS} -std=c++0x -Wall -march=native -mtune=native&quot; )
    message( INFO &quot;CMAKE_CXX_FLAGS = ${CMAKE_CXX_FLAGS}&quot; )
    
    # sources to build
    set(big_memcpy_test_SRCS
      main.cpp
    )
    
    # create an executable file named &quot;big_memcpy_test&quot; from
    # the source files in the variable &quot;big_memcpy_test_SRCS&quot;.
    add_executable(big_memcpy_test ${big_memcpy_test_SRCS})


**Test Results**

    Buffer Size: 1GB | malloc (ms) | memset (ms) | memcpy (ms) | NUMA nodes (numactl --hardware)
    ---------------------------------------------------------------------------------------------
    Laptop 1         | 0           | 127         | 113         | 1
    Laptop 2         | 0           | 180         | 120         | 1
    Server 1         | 0           | 306         | 301         | 2
    Server 2         | 0           | 352         | 325         | 2

As you can see the memcpys and memsets on our servers is much slower than the memcpys and memsets on our laptops.

**Varying buffer sizes**

I have tried buffers from 100MB to 5GB all with similar results (servers slower than laptop)


**NUMA Affinity**

I read about people having performance issues with NUMA so i tried setting CPU and memory affinity using numactl but the results remained the same.

Server NUMA Hardware

    $ numactl --hardware                                                            
    available: 2 nodes (0-1)                                                                     
    node 0 cpus: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23                                         
    node 0 size: 65501 MB                                                                        
    node 0 free: 62608 MB                                                                        
    node 1 cpus: 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31                                   
    node 1 size: 65536 MB                                                                        
    node 1 free: 63837 MB                                                                        
    node distances:                                                                              
    node   0   1                                                                                 
      0:  10  21                                                                                 
      1:  21  10 


Laptop NUMA Hardware

    $ numactl --hardware
    available: 1 nodes (0)
    node 0 cpus: 0 1 2 3 4 5 6 7
    node 0 size: 16018 MB
    node 0 free: 6622 MB
    node distances:
    node   0 
      0:  10

Setting NUMA Affinity

    $ numactl --cpunodebind=0 --membind=0 ./big_memcpy_test


Any help resolving this is greatly appreciated.

**Edit: GCC Options**

Based on comments i have tried compiling with different GCC options:

Compiling with -march and -mtune set to native

    g++ -std=c++0x -Wall -march=native -mtune=native -O3 -DNDEBUG -o big_memcpy_test main.cpp 

Result: Exact same performance (no improvement)

Compiling with -O2 instead of -O3

    g++ -std=c++0x -Wall -march=native -mtune=native -O2 -DNDEBUG -o big_memcpy_test main.cpp

Result: Exact same performance (no improvement)

**Edit: Changed memset to write 0xF instead of 0 to avoid NULL page (@SteveCox)**

No improvement when memsetting with a value other than 0 (used 0xF in this case).

**Edit: Cachebench results**

In order to rule out that my test program is too simplistic i downloaded a real benchmarking program LLCacheBench (http://icl.cs.utk.edu/projects/llcbench/cachebench.html)

I built the benchmark on each machine separately to avoid architecture issues. Below are my results.

![laptop vs server memcpy performance][1]

Notice the VERY large difference is performance on the larger buffer sizes. The last size tested (16777216) performed at 18849.29 MB/sec on the laptop and 6710.40 on the server. That&#39;s about a 3x difference in performance. You can also notice that the performance dropoff of the server is much steeper than on the laptop.

**Edit: memmove() is 2x FASTER than memcpy() on the server**

Based on some experimentation i have tried using memmove() instead of memcpy() in my test case and have found a 2x improvement on the server. Memmove() on the laptop runs slower than memcpy() but oddly enough runs at the same speed as the memmove() on the server. This begs the question, why is memcpy so slow?

Updated Code to test memmove along with memcpy. I had to wrap the memmove() inside a function because if i left it inline GCC optimized it and performed the exact same as memcpy() (i assume gcc optimized it to memcpy because it knew the locations didn&#39;t overlap).

Updated Results

    Buffer Size: 1GB | malloc (ms) | memset (ms) | memcpy (ms) | memmove() | NUMA nodes (numactl --hardware)
    ---------------------------------------------------------------------------------------------------------
    Laptop 1         | 0           | 127         | 113         | 161       | 1
    Laptop 2         | 0           | 180         | 120         | 160       | 1
    Server 1         | 0           | 306         | 301         | 159       | 2
    Server 2         | 0           | 352         | 325         | 159       | 2

**Edit: Naive Memcpy**

Based on suggestion from @Salgar i have implemented my own naive memcpy function and tested it.

Naive Memcpy Source

    void naiveMemcpy(void* pDest, const void* pSource, std::size_t sizeBytes)
    {
      char* p_dest = (char*)pDest;
      const char* p_source = (const char*)pSource;
      for (std::size_t i = 0; i &lt; sizeBytes; ++i)
      {
        *p_dest++ = *p_source++;
      }
    }

Naive Memcpy Results Compared to memcpy()

    Buffer Size: 1GB | memcpy (ms) | memmove(ms) | naiveMemcpy()
    ------------------------------------------------------------
    Laptop 1         | 113         | 161         | 160
    Server 1         | 301         | 159         | 159
    Server 2         | 325         | 159         | 159

**Edit: Assembly Output**

Simple memcpy source

    #include &lt;cstring&gt;
    #include &lt;cstdlib&gt;
    
    int main(int argc, char* argv[])
    {
      size_t SIZE_BYTES = 1073741824; // 1GB
      
      char* p_big_array  = (char*)malloc(SIZE_BYTES * sizeof(char));
      char* p_dest_array = (char*)malloc(SIZE_BYTES * sizeof(char));
      
      memset(p_big_array,  0xA, SIZE_BYTES * sizeof(char));
      memset(p_dest_array, 0xF, SIZE_BYTES * sizeof(char));
        
      memcpy(p_dest_array, p_big_array, SIZE_BYTES * sizeof(char));
        
      free(p_dest_array);
      free(p_big_array);
      
      return 0;
    }

Assembly Output: This is the exact same on both the server and the laptop. I&#39;m saving space and not pasting both.

            .file   &quot;main_memcpy.cpp&quot;
            .section        .text.startup,&quot;ax&quot;,@progbits
            .p2align 4,,15
            .globl  main
            .type   main, @function
    main:
    .LFB25:
            .cfi_startproc
            pushq   %rbp
            .cfi_def_cfa_offset 16
            .cfi_offset 6, -16
            movl    $1073741824, %edi
            pushq   %rbx
            .cfi_def_cfa_offset 24
            .cfi_offset 3, -24
            subq    $8, %rsp
            .cfi_def_cfa_offset 32
            call    malloc
            movl    $1073741824, %edi
            movq    %rax, %rbx
            call    malloc
            movl    $1073741824, %edx
            movq    %rax, %rbp
            movl    $10, %esi
            movq    %rbx, %rdi
            call    memset
            movl    $1073741824, %edx
            movl    $15, %esi
            movq    %rbp, %rdi
            call    memset
            movl    $1073741824, %edx
            movq    %rbx, %rsi
            movq    %rbp, %rdi
            call    memcpy
            movq    %rbp, %rdi
            call    free
            movq    %rbx, %rdi
            call    free
            addq    $8, %rsp
            .cfi_def_cfa_offset 24
            xorl    %eax, %eax
            popq    %rbx
            .cfi_def_cfa_offset 16
            popq    %rbp
            .cfi_def_cfa_offset 8
            ret
            .cfi_endproc
    .LFE25:
            .size   main, .-main
            .ident  &quot;GCC: (GNU) 4.6.1&quot;
            .section        .note.GNU-stack,&quot;&quot;,@progbits


**PROGRESS!!!! asmlib**

Based on suggestion from @tbenson  i tried running with the [asmlib][2] version of memcpy. My results initially were poor but after changing SetMemcpyCacheLimit() to 1GB (size of my buffer) i was running at speed on par with my naive for loop!

Bad news is that the asmlib version of memmove is slower than the glibc version, it is now running at the 300ms mark (on par with the glibc version of memcpy). Weird thing is that on the laptop when i SetMemcpyCacheLimit() to a large number it hurts performance...

In the results below the lines marked with SetCache have SetMemcpyCacheLimit set to 1073741824. The results without SetCache do not call SetMemcpyCacheLimit()

Results using functions from asmlib:

    Buffer Size: 1GB  | memcpy (ms) | memmove(ms) | naiveMemcpy()
    ------------------------------------------------------------
    Laptop            | 136         | 132         | 161
    Laptop SetCache   | 182         | 137         | 161
    Server 1          | 305         | 302         | 164
    Server 1 SetCache | 162         | 303         | 164
    Server 2          | 300         | 299         | 166
    Server 2 SetCache | 166         | 301         | 166

Starting to lean towards cache issue, but what would cause this?

  [1]: http://i.stack.imgur.com/PTqZJ.png
  [2]: http://www.agner.org/optimize/asmlib.zip

All Numa Solutions on Gang of Coders

Total of 1 Numa Solutions

Poor memcpy Performance on Linux