Sujal_Das_headshot.jpg

OVS Offload Models Used with NICs and SmartNICs: Pros and Cons

By Sujal Das | May 23, 2018

As SmartNICs become more popular, more decision makers at companies are being asked to look at the way SmartNICs work – specifically the different offload models related to the OVS datapath, and the pros and cons of each model. In this blog, I will go through and explain these various models. As we go through each model, you will realize that not all of them require offload of the OVS datapath to NIC hardware. In some cases the datapath is actually moved up to the user space as well (some call it on-loading). I will cover all of them using a consistent tabular format for easy readability and comparison.  Here we go…

Model 1: OVS datapath in the kernel

model1-900.width-800


Summary of features, pros (in green) and cons (in red):

Location of OVS datapath (kernel, user, NIC) Kernel

Most mature and proven model

Broadest vendor support

Easier to integrate and enhance the datapath with other kernel-based networking features such as Conntrack, BPF

Policy enforcement using OVS datapath (yes, no) Yes

Suitable for SDN and Network Virtualization

OpenStack Orchestration support (yes, no) Yes

Suitable for cloud-based deployments

Supported with most OpenStack distributions

VM hardware independence (yes, no) Yes

Uses Virtio driver in the VM making the VMs hardware independent and enabling support of broad array of guest operating systems and live VM migration

Performance, CPU core use for datapath processing (Mpps, No. of cores) 1-2 Mpps
Using 4 CPU Cores

Poor datapath performance

Performance degrades rapidly with more flows and more or complex policy rules

Consumes many CPU cores for datapath performance resulting in lowest server utilization and TCO

Model 2: SR-IOV is used where the OVS datapath is bypassed

model2-900.width-800


Summary of features, pros (in green) and cons (in red):

Location of OVS datapath (kernel, user, NIC) Kernel but not used (bypassed using SR-IOV)

Mature and proven model

Broadest vendor support

Policy enforcement using OVS datapath (yes, no) No
Relies on policy enforcement at TOR switch

Not suitable for SDN

Not suitable Network Virtualization at scale

OpenStack Orchestration support (yes, no) Yes

Suitable for cloud-based deployments where SDN-based policy enforcement is not required

Supported with most OpenStack distributions.

VM hardware independence (yes, no) No

Uses vendor-specific driver in the VM making the VMs hardware dependent

Live VM migration is not supported

Performance, CPU core use for datapath processing (Mpps, No. of cores) Close to 30 Mpps
No CPU cores as the OVS datapath is not used

Excellent performance delivering packets from network ports to VMs

Model 3: OVS datapath is moved to the user space using DPDK

model3-900.width-800


Summary of features, pros (in green) and cons (in red):

Location of OVS datapath (kernel, user, NIC) User space

Broad vendor support

Difficult to enhance features leveraging other kernel datapath implementations such as in Conntrack and BPF

Policy enforcement using OVS datapath (yes, no) Yes

Suitable for SDN and Network Virtualization

OpenStack Orchestration support (yes, no) Yes

Suitable for cloud-based deployments

Supported with many OpenStack distributions

VM hardware independence (yes, no) Yes

Uses Virtio driver in the VM making the VMs hardware independent and enabling support of broad array of guest operating systems and live VM migration

Performance, CPU core use for datapath processing (Mpps, No. of cores) 6-8 Mpps
Using 4 CPU Cores

Good datapath performance

Performance degrades rapidly with more flows and more or complex policy rules

Consumes many CPU cores for datapath performance resulting in low server utilization and TCO

Model 4: OVS datapath in the kernel is fully offloaded to the SmartNIC, hardware-dependent VMs


Summary of features, pros (in green) and cons (in red):

Location of OVS datapath (kernel, user, NIC) Kernel and SmartNIC
Fallback to kernel OVS for control traffic and new/first flow

Uses kernel-compliant and upstreamed TC flower based offload of the OVS datapath

Offload mechanism is included in the RHEL 7.5 distribution

Kernel-based offload makes it easier to enhance features leveraging other kernel datapath implementations such as in Conntrack and BPF

Available in the latest kernel releases only until backported to older versions

Policy enforcement using OVS datapath (yes, no) Yes

Suitable for SDN and Network Virtualization.

OpenStack Orchestration support (yes, no) Yes

Suitable for Cloud-based deployments

Supported with newer OpenStack distributions such as Queens and RHOSP 13

Available in the latest OpenStack releases only until backported to older versions

VM hardware independence (yes, no) Yes

Uses vendor-specific driver in the VM making the VMs hardware dependent

Live VM migration is not supported

Performance, CPU core use for datapath processing (Mpps, No. of cores) 25-28 Mpps
No CPU cores used for datapath processing

Excellent datapath performance

Performance is maintained well with more flows and more or complex policy rules

Frees all CPU cores from datapath processing resulting in high server utilization and TCO

Model 5: OVS datapath is fully offloaded to the SmartNIC, VMs are hardware-independent

model5-900.width-800


Summary of features, pros (in green) and cons (in red):

Location of OVS datapath (kernel, user, NIC) Kernel and SmartNIC
Fallback to kernel OVS for control traffic and new/first flow

Uses kernel-compliant and upstreamed TC flower-based offload of the OVS datapath

Offload mechanism is included in the RHEL 7.5 distribution

Kernel-based offload makes it easier to enhance features leveraging other kernel datapath implementations such as in Conntrack and BPF

Available in the latest kernel releases only until backported to older versions

Policy enforcement using OVS datapath (yes, no) Yes

Suitable for SDN and Network Virtualization

OpenStack Orchestration support (yes, no) Yes

Suitable for Cloud-based deployments

Supported with newer OpenStack distributions such as Queens and RHOSP 13

Available in the latest OpenStack releases only until backported to older versions

VM hardware independence (yes, no) Yes

Uses Virtio driver in the VM making the VMs hardware independent and enabling support of broad array of guest operating systems and live VM migration

Uses one of the two modes: Relay Agent with vhost/DPDK and Virtio 1.0, or vDPA (vhost datapath acceleration) with Virtio 1.1

Consumes 1-3 CPU cores for processing the Relay Agent in user space

Performance, CPU core use for datapath processing (Mpps, No. of cores) 25-28 Mpps
No CPU Cores used for datapath processing

Excellent datapath performance

Performance is maintained well with more flows and more or complex policy rules

Frees all CPU cores from datapath processing resulting in high server utilization and TCO

Model 6: OVS datapath is offloaded to the SmartNIC, VMs are hardware-independent

model6-900.width-800


Summary of features, pros (in green) and cons (in red):

Location of OVS datapath (kernel, user, NIC) Kernel and SmartNIC
Fallback to kernel OVS for control traffic and new/first flow

Uses kernel-compliant and upstreamed TC flower-based offload of the OVS datapath

Offload mechanism is included in the RHEL 7.5 distribution

Kernel-based offload makes it easier to enhance features leveraging other kernel datapath implementations such as in Conntrack and BPF

Available in the latest kernel releases only until backported to older versions

Policy enforcement using OVS datapath (yes, no) Yes

Suitable for SDN and Network Virtualization

OpenStack Orchestration support (yes, no) Yes

Suitable for cloud-based deployments

Supported with newer OpenStack distributions such as Queens and RHOSP 13

Available in the latest OpenStack releases only until backported to older versions

VM hardware independence (yes, no) Yes

Uses Virtio driver in the VM making the VMs hardware independent and enabling support of broad array of guest operating systems and live VM migration

Uses hardware implementation of Direct Virtio on the NIC

Performance, CPU core use for datapath processing (Mpps, No. of cores) 25-28 Mpps
No CPU cores used for datapath processing

Excellent datapath performance

Performance is maintained well with more flows and more or complex policy rules

Frees all CPU cores from datapath processing resulting in high server utilization and TCO

Model 7: OVS datapath is in the user space and some functions are offloaded to the SmartNIC

model7-900.width-800


Summary of features, pros (in green) and cons (in red):

Location of OVS datapath (kernel, user, NIC) User space
Partial offload of datapath functions to NIC or SmartNIC hardware using DPDK Flow API

Difficult to enhance features leveraging kernel datapath implementations and innovations such as in Conntrack and BPF

Policy enforcement using OVS datapath (yes, no) Yes

Suitable for SDN and Network Virtualization

OpenStack Orchestration support (yes, no) Yes

Suitable for cloud-based deployments

VM hardware independence (yes, no) Yes

Uses Virtio driver in the VM making the VMs hardware independent and enabling support of broad array of guest operating systems and live VM migration

Performance, CPU core use for datapath processing (Mpps, No. of cores) 10-15 Mpps
4-8 CPU cores used for datapath processing

Good datapath performance, full line rate for 10GbE networks

Performance degrades rapidly with more flows and more or complex policy rules

Consumes many CPU cores for datapath performance resulting in low server utilization and TCO

Model 8: Both the OVS datapath and control plane are offloaded to the SmartNIC

model8-900.width-800
Summary of features, pros (in green) and cons (in red):
Location of OVS datapath (kernel, user, NIC) SmartNIC

Control plane runs in the user space on an ARM or MIPS CPU on the SmartNIC

Datapath slow path runs in the kernel or user space on the ARM and fast path runs on accelerator chip available on some SmartNICs

Useful for bare metal cloud applications where the service provider has no control over what host operating system is used on the server

Best when SDN and cloud orchestration is implemented via the SmartNIC (not through the host)

Policy enforcement using OVS datapath (yes, no) Yes

Suitable for SDN and Network Virtualization

Non-Bare Metal Use Case: When the host runs OVS control plane and datapath, the version that runs on the SmartNIC can easily go out of sync with the version that runs on the host, causing anomalies and feature inconsistencies related to SDN deployments

OpenStack Orchestration support (yes, no) Yes

Suitable for cloud-based deployments

Non-Bare Metal Use Case: When the host runs OVS control plane and datapath, the version that runs on the SmartNIC can easily go out of sync with the version that runs on the host, causing anomalies and feature inconsistencies related to cloud orchestration.

VM hardware independence (yes, no) No

Uses vendor-specific driver in the VM making the VMs hardware dependent

Live VM migration is not supported

Performance, CPU core use for datapath processing (Mpps, No. of cores) 15-20 Mpps
No CPU cores used for datapath processing

OVS control plane processing on the host consumes less than a CPU core. This is eliminated by this approach

Good datapath performance, full line rate for 10GbE networks for small packets and 25GbE networks for mid-sized packets