In-band Network Telemetry – It’s Not Rocket Science

No matter what application you run or how your network infrastructure is architected, there is always the need to monitor what is actually happening inside your data center. Once upon a time, we would use physical tap ports and other monitoring solutions that assumed packets traversed from a physical switch to a server and back again. But in today’s virtualize data center environments, most traffic moves “east-west” originating at one VM, passing through a virtual switch to another VM. This means packets can move from source to destination without ever leaving the server. And when physical network elements are required to route packets, they compromise an “underlay” network that is transparent to the encapsulated packets running over it.

How do we monitor the packet flows that may never pass through a physical tap? And how do we debug issues in underlay networks when they are disaggregated from the overlay networks?

Using In-band Network Telemetry (INT) is one solution.

INT, which originated from The P4 Language Consortium (P4.org), is defined as a framework designed to allow the collection and reporting of network state, by the data plane, without requiring intervention or work by the control plane. The specification for INT can be found here. The key is to insert critical metadata, which can be extracted and interpreted later, in-band without affecting network performance. This also needs to be done using standard protocols.

INT leverages the encapsulation capabilities of the VXLAN& VXLAN GPE General Protocol Extension. GPE enables metadata to be inserted, in-band, into VXLAN packets by network devices. Let’s take a look at a VXLAN packet. The packet is the same as any other Ethernet packet with a VXLAN header added.

GPE adds the ability to insert metadata below the VXLAN header. That metadata can include:

• Ingress port ID
• Ingress timestamp
• Egress port ID
• Hop latency (the time taken for the INT packet to be switched within the device)
• Egress port TX Link utilization

The key question is, “What device can use GPE to insert the metadata into a packet?” Using a simple P4 program, a Netronome Agilio CX SmartNIC can insert GPE metadata on the fly without causing a network performance interruption. All processing is done on the SmartNIC and the server CPU is not involved in the process.

Let’s talk about a recent demo we created to show INT at work. A packet enters a cluster through a switch, which is directed to a server. On that server, an Agilio CX SmartNIC receives the packet and inserts an ingress time stamp. The Agilio CX then directs the packet to VM1 with the Agilio CX also inserting an egress time stamp as the packet leaves the SmartNIC. Once the packet is processed by VM1 it is directed to the Agilio CX for transport to its next location. Upon being received by the Agilio CX again, it receives a third timestamp, Ingress2. Because the packet’s destination is VM2 on the same server, it is sent to VM2 and receives another timestamp, Egress2. When VM2’s processing is complete, the packet is sent back to the Agilio CX (timestamp: Ingress3) where it is processed (without CPU involvement) and sent out to the network (timestamp: Egress3).

For many customers, latency is a crucial metric of performance. Normally considered solely as a network metric, latency can also be viewed as the amount of processing time required by each VM. Because the Agilio CX inserts timestamps at each step along the way, we can calculate latency for every single step. Note: this includes the steps that were entirely within the confines of the server:

Egress1-Ingress1 = latency of the Agilio CX SmartNIC

Ingress2-Egress1 = processing time of VM1

Egress2-Ingress2 = latency of Agilio CX SmartNIC (the hop after VM1)

Ingress2-Egress2= processing time of VM1

Egress3-Ingress3 = latency of Agilio CX SmartNIC (the hop after VM2)

By using the Agilio CX SmartNIC to insert timestamps via the VXLAN GPE, a user now has a complete picture of latency and processing time throughout the life of a packet, which can be visualized with a standard visualization tool as seen to the right.

By using standard networking tools, VXLAN and VXLAN GPE, and an Agilio SmartNIC, operators now have access to latency measurements that they could not get to before without extensive effort and intrusive tools. It’s not rocket science.