Ever Deeper with BPF – An Update on Hardware Offload Support

BPF Offload Framework Architecture

Helpers and Perf Events Support

bpf_map_lookup_elem() 
bpf_map_update_elem() 
bpf_map_delete_elem() 
bpf_get_prandom_u32() 
bpf_perf_event_output() 
bpf_xdp_adjust_head() 
bpf_xdp_adjust_tail()

An example usage of perf events is available with the “xdpdump” demonstration that we published not long ago.

BPF-to-BPF Function Calls

BPF-to-BPF function calls were added to the kernel last December. These are traditional function calls—by contrast with calls to kernel helpers, not implemented with BPF instructions but compiled as part of the kernel, or “tail calls” where you jump into a second program but never come back to the first. The user can define several non-inlined functions in their C source file, and compile them into BPF. Since we still do not have loops in BPF, this is especially useful to reuse a sequence of instructions without having to unroll the whole sequence each time: just repeatedly call your function instead. Support for BPF-to-BPF calls has just been upstreamed to the development branch of the kernel.

Under the Hood

Even if it does not translate into new features, a number of improvements have made their way to the internals of the driver. Think about it as “under the hood” optimizations or low-level improvements. Our JIT compiler now supports logic and arithmetic indirect shifts, one of the very few BPF instructions it did not have the ability to process thus far. Multiplication was improved to work on 8-bit or 16-bit operands only.

Many other technical optimizations have been delivered, often consisting in detecting a particular sequence of instructions and replacing it by something more efficient. The objective is:

• To reduce the number of instructions in the final program, either for better performance or for supporting a higher total number of instructions.
  
• To gain performance by favoring instructions that are faster to execute for the hardware.
  
• To optimize resources usage, such as the amount of memory used from the stack.
  

Offload-Specific Features

For most BPF programs, we try very hard to ensure that the offload is transparent for the user, and that the behavior is as close as possible as what would be obtained if the program was run on the host—save for the gain of performance, obviously. And yet, at times it is interesting to account for the difference of context, and to add something that would not make sense on the host, but that a BPF program can very easily do when running on the NIC. What I have in mind is programmable Receive-Side Scaling (RSS). This is a common feature on recent NICs (but not usually performed with BPF, of course), consisting in distributing the incoming traffic between host CPUs through the use of multiple queues. With BPF offload, this was trivial to implement: we simply allowed setting a RX queue metadata field which is read-only for the kernel so the program can indicate to which queue the processed packet should be sent. And this is it. If you feel interested, check out the programmable RSS demonstration application we recently published.

Improved Tooling and Resources

Along with the cards, Netronome’s NFP driver and the kernel, we continue to improve the tooling and the available resources so as to make offloading BPF programs the smoothest experience possible…even for people who are not familiar with the technology in the first place!

We keep maintaining and improving bpftool, the go-to utility for everything related to BPF introspection and simple management. The tool has received numerous contributions from the community, and getting information about the structure of keys and values in BPF maps has never been easier now that BTF is supported! This mechanism makes it possible to embed the structure of some BPF objects, so that they can be retrieved later for better introspection and debugging. For example, compare the following plain hexadecimal dump from a map entry:

# bpftool map dump id 1337
key:
0a 00 00 00 00 00 00 00
value:
07 00 00 00 00 00 00 00  01 00 00 00 00 00 00 00
03 02 01 4d 15 00 00 00

It can now be dumped with the actual structure that was used for keys and values when the map was created by the user, as in the following example:

# bpftool map dump id 1337
[{
       "key": 10,
       "value": {
           "": {
               "value": 7,
               "ifindex": 1,
               "mac": [0, 0, 21, 73, 1, 2, 3]
           }
       }
    }
]