Skip to content

H
hpdos
Commit 122aaf3e authored by Shah Rinku's avatar Shah Rinku
Browse files
Options

Update README.md

parent 3f3b0c10
Hide whitespace changes
Inline Side-by-side
Showing with 69 additions and 99 deletions
code/DPDK_server_MDcache/README.md
View file @ 122aaf3e
## DPDK for Mellanox Bluefield-1 ARM cores
### Install and configure DPDK
1. Download the required DPDK compressed version & decompress it.
https://core.dpdk.org/download/
......@@ -12,117 +14,85 @@
a. Check if the Poll Mode Driver (PMD) for your Mellanox NIC is set; in our case it is MLX5
b. Add “CONFIG_RTE_LIBRTE_MLX5_PMD=y”, if it does not exist
3. DPDK configuration alternatives
3. DPDK configuration & alternatives
- Alternative 1: For newer DPDK versions (>= version 18)
<path-to-dpdk-folder>$ meson build
<path-to-dpdk-folder>$ meson build
<path-to-dpdk-folder>$ cd build
<path-to-dpdk-folder>$ ninja
<path-to-dpdk-folder>$ ninja install
- Alternative 2: For older DPDK versions & for our mTCP setup
- Run:
<path-to-dpdk-folder>/usertools$ sudo ./dpdk-setup.sh
- Choose
7: arm64-bluefield-linuxapp-gcc (NIC installation)
41: x86_64-native-linuxapp-gcc (Host installation)
Corresponding folder is created at DPDK root folder
- For both alternatives:
- Mellanox driver, mlx5 supports kernel and poll mode
- No need to unbind the interface from kernel
- Allocate huge pages for DPDK operation
$ sudo sysctl -w vm.nr_hugepages=4096
- Test huge page allocation
$ cat /proc/meminfo | grep -i huge
**Note: In case of numa.h not found error**
$ sudo apt-get install libnuma-dev
<path-to-dpdk-folder>$ cd build
### Run DPDK application
<path-to-dpdk-folder>$ ninja
1. Identify PCI address of the NIC port which you want to bind to DPDK
$ lspci
<path-to-dpdk-folder>$ ninja install
03:00.0 Ethernet controller: Mellanox Technologies MT416842 BlueField integrated ConnectX-5 network controller => p0
- Alternative 2: For older DPDK versions & for our mTCP setup
03:00.1 Ethernet controller: Mellanox Technologies MT416842 BlueField integrated ConnectX-5 network controller => p1
2. Run testpmd on the two smartNICs connected directly using a cable
<path-to-dpdk-folder>/arm64-bluefield-linuxapp-gcc/build/app/test-pmd$ sudo ./testpmd -w 03:00.1 -- --nb-cores=2 --txq=2 -i
**Parameters explained:**
w: whitelist device; nb-cores: #forwarding cores; txq: #transmit queues (requires RSS); i: interactive mode; a: automatic traffic start
You can check for more configuration parameters: testpmd> help
3. In the interactive mode, some configurations before testing network bandwidth
**I have considered our servers "ub-05" and "user" to demonstrate example configuration**
- “ub-05” server smartNIC
testpmd> set fwd txonly
testpmd> set eth-peer 0 0c:42:a1:df:ac:41
testpmd> set eth-peer 1 0c:42:a1:df:ac:40
testpmd> start
testpmd> show port stats all //repeat
- “user” server smartNIC
testpmd> set fwd rxonly
testpmd> set eth-peer 0 0c:42:a1:df:ac:49
testpmd> set eth-peer 1 0c:42:a1:df:ac:48
testpmd> start
testpmd> show port stats all //repeat
####
DPDK
#####
COMPILE
vim dpdk/config/common_linuxapp
#@rinku ADD FOLL
CONFIG_RTE_LIBRTE_MLX5_PMD=y
In case of numa.h not found error
sudo apt-get install libnuma-dev
RUN
ubuntu@linux:~/dpdk-stable-19.11.8/arm64-bluefield-linuxapp-gcc/build/app/test-pmd$
sudo ./testpmd -w 03:00.1 -- -i -a
OR
sudo ./testpmd -- -i -a
sudo ./testpmd -- --nb-cores=2 -i
testpmd> set fwd txonly
testpmd> port start 0
testpmd> port start 1
testpmd> set eth-peer 0 0c:42:a1:df:ac:41
testpmd> set eth-peer 1 0c:42:a1:df:ac:40
testpmd> start
testpmd> show port stats all
################
MTCP
IRON OUT CODE ISSUES
ubuntu@linux:~/dpdk-stable-19.11.8/arm64-bluefield-linuxapp-gcc/build/app/test-pmd$ sudo ./testpmd -w 03:00.1 -- -i -a
ubuntu@linux:~/mtcp-devel$ sudo ./setup_mtcp_dpdk_env.sh /home/ubuntu/dpdk-stable-19.11.8
export RTE_SDK=/home/ubuntu/dpdk-stable-19.11.8
export RTE_TARGET=arm64-bluefield-linuxapp-gcc
wget https://github.com/mtcp-stack/mtcp/archive/refs/heads/devel.zip
core.c: In function ‘mtcp_create_context’:
core.c:1332:4: error: ‘lcore_config’ undeclared (first use in this function)
1332 | lcore_config[master].ret = 0;
ubuntu@linux:~/mtcp-devel$ vim mtcp/src/core.c
if (master == whichCoreID(cpu)) {
//@rinku
//lcore_config[master].ret = 0;
//lcore_config[master].state = FINISHED;
/usr/include/aarch64-linux-gnu/bits/string_fortified.h:106:10: error: ‘__builtin_strncpy’ output may be truncated copying 1023 bytes from a string of length 1023 [-Werror=stringop-truncation]
106 | return __builtin___strncpy_chk (__dest, __src, __len, __bos (__dest));
ubuntu@linux:~/mtcp-devel$ vim mtcp/src/config.c
//@rinku
//strncpy(optstr, line, MAX_OPTLINE_LEN - 1);
memcpy(optstr, line, MAX_OPTLINE_LEN - 1);
ubuntu@linux:~/mtcp-devel$ vim mtcp/src/io_module.c
Comment the funct code for “probe_all_rte_devices” ; add return -1
COMPILE
export RTE_SDK=/home/ubuntu/dpdk-stable-19.11.8
export RTE_TARGET=arm64-bluefield-linuxapp-gcc
ubuntu@linux:~/mtcp-devel$ ./setup_mtcp_dpdk_env.sh [<path to $RTE_SDK>]
export RTE_SDK=/home/ubuntu/dpdk-stable-19.11.8
export RTE_TARGET=arm64-bluefield-linuxapp-gcc
ubuntu@linux:~/mtcp-devel$./configure --with-dpdk-lib=$RTE_SDK/$RTE_TARGET CFLAGS="-DMAX_CPUS=8" //Specify the max cpus which will run MTCP; use --disable-hwcsum when working in virtualized environment
ubuntu@linux:~/mtcp-devel$ make
configure: error: Could not find gmp.h
SOLUTION: sudo apt-get install libgmp-dev
Check the configurations in apps/example
epserver.conf for server-side configuration
epwget.conf for client-side configuration
Configure ~/mtcp-devel/apps/example/config/arp.conf
Configure ~/mtcp-devel/apps/example/config/route.conf
######
RUN THE MTCP APP
export RTE_SDK=/home/ubuntu/dpdk-stable-19.11.8
export RTE_TARGET=arm64-bluefield-linuxapp-gcc
sudo ./epserver -p /home/ubuntu/www -f epserver.conf
sudo ./epwget 192.168.220.35/example.txt 10000000 -N 8 -c 10000 -f epwget.conf
#######
REVERT BACK MTCP NIC CHANGES
./setup_linux_env.sh [<path to $RTE_SDK>]
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment