pcie sdp

Socket Direct Protocol (SDP) over PCI Express Project Page

In this project we designed and implemented the Socket Direct Protocol (SDP) for PCI Express based fabric. SDP is an efficient data transfer protocol alternative to the TCP that utilizes the advanced features available in high performance interconnects such as direct memory access. Our SDP implementation is designed for PLX PCIe switches and supports only two nodes for now. SDP is compatible with Socket/TCP applications yet provides much better performance. It is developed as a Linux kernel module and has been tested under Redhat Linux 5.7.

Installing PCIe Drivers

1. Download the drivers from http://www.plxtech.com/download/file/899/217277 after registrations
2. Unzip the package

    # Unzip PLX_SDK_Linux_v6_50_Final_2011-09-30.zip

3. Make sure that you are at the same level as PlxSdk and apply the patch:

    #ls
    Documentation  PlxSdk  PlxSdk.tar 

    # patch -p1 < ../patch.txt 
    patching file PlxSdk/Include/PlxIoctl.h 
    patching file PlxSdk/Include/Plx_sysdep.h 
    patching file PlxSdk/Linux/Driver/Source.Plx8000/ApiFunc.c 
    patching file PlxSdk/Linux/Driver/Source.Plx8000/ApiFunc.h 
    patching file PlxSdk/Linux/Driver/Source.Plx8000/Dispatch.c 
    patching file PlxSdk/Linux/Driver/Source.Plx8000/Dispatch.c_back 
    patching file PlxSdk/Linux/Driver/Source.Plx8000/DrvDefs.h 
    patching file PlxSdk/Linux/Driver/Source.Plx8000/DrvDefs.h_back 
    patching file PlxSdk/Linux/Driver/Source.Plx8000/PlxInterrupt.c 
    patching file PlxSdk/Linux/Driver/Source.Plx8000/SuppFunc.c 
    patching file PlxSdk/Linux/Driver/Source.PlxDma/Dispatch.c 
    patching file PlxSdk/Linux/Driver/Source.PlxDma/Dispatch.c_back 
    patching file PlxSdk/Linux/Driver/Source.PlxDma/DrvDefs.h 
    patching file PlxSdk/Linux/Driver/Source.PlxDma/DrvDefs.h_back 
    patching file PlxSdk/Linux/Driver/.tmp_versions/PlxDma_dbg.mod 
    patching file PlxSdk/Linux/Samples/ApiTest/ApiTest.c 
    patching file PlxSdk/Linux/Samples/NT_DmaTest/NT_DmaTest.c 
    patching file PlxSdk/Linux/Samples/NT_myTest/Makefile 
    patching file PlxSdk/Linux/Samples/NT_myTest/NT_DmaTest.c 
    patching file PlxSdk/Linux/Samples/NT_myTest/PlxDmaPerf.c 
    patching file PlxSdk/Linux/Samples/PlxDma/PlxDma.c 
    patching file PlxSdk/Linux/Samples/PlxNotification/PlxNotification.c

4. Make sure that Plx main directory is defined in you environment:

    # export PLX_SDK_DIR=/home/ambu50/trunk/Plx_new_download/PlxSdk/

5. Change directory to the drivers folder:

    cd Linux/Driver/ 

6. Run build builddriver:

    # ./builddriver 8000 
    # ./builddriver Dma

7. Load the drivers:

    # ../../Bin/Plx_load 8000 
    # ../../Bin/Plx_load Dma

Installing libsdp Library

1. Download the library libsdp-1.1.108.tar.gz
2. Extract and install using "./configure", "make", "make install"

Installing SDP Kernel Module

1. Download the source code of SDP kernel module pcie_sdp.tar.gz
2. Extract and install

    # tar zxvf Pcie_sdp.tar.gz
    # cd Pcie_sdp
    # make

3. Load the module to the kernel

    insmod ib_sdp.ko

Testing the SDP over PCIe

To test the module you can run any TCP based client server application in one of two ways.

• Specifying the location of libsdp

   LD_LIBRARY_PATH=/usr/local/lib LD_PRELOAD=libsdp.so YOUR_SERVER_EXECUTABLE

• By configuring the process name and port number in libsdp.conf that can be found under /usr/local/etc/libsdp.conf or /etc/libsdp.conf