Two SFs are created on the NIC, eth ports are created in them and moved into
separate netns. L3 connectivity has to be set up between them, one SF has the
prefix 2001:db8::1/56, the other one 2001:db8:2::/56. The rest of the
traffic (not matching by dst ip) must be dropped.
All modern clouds have an L3 Only network, VMs/containers have connectivity only over IPv4/IPv6, so they sit in an L2 segment that contains only the VM/container and the nexthop (a virtual router), which uses LPM to direct traffic where it needs to go. Within a single node it will just be a forward into another SF, and if the VM/container is already on another node, the traffic will be carried over an ipip overlay with an SRv6 controlplane.
Our VMs will be connected to the NIC through an SF vnet (vDPA) port, the traffic will go straight into the hardware. It is something like SR-IOV (VF), but a bit different.
First of all, minimal, safe, high-level wrappers over the low-level DOCA Flow primitives have to be implemented: basic/lpm pipes and entries for them, plus actions for swapping mac addresses.
On top of that, implement L3 Forwarding: the port a packet arrived on has to be determined, based on that the tenant is determined -> forward into the tenant's LPM pipe, where an entry forwards it into a port with a mac address swap action.
The mac addresses are static — 52:ed:ed:ed:ed:ed on the nexthop side,
52:ec:ec:ec:ec:ec on the SF side, ARP/ND is not needed.
Implementing ttl decrement is desirable, there is an example in doca-samples — flow_modify_header. The TTL=0/1 case can be ignored for now.
switch mode has to be used, in isolated mode, and fwd_miss set to drop
manually. And switching the NIC into esw_multiport (mpesw) mode right away is
desirable, since that is exactly how it will be in production.
Implementing dynamic adding/changing of the offload settings — creating/removing pipes, adding/removing entries in them — will also be a big plus.
vm/netns
˅
virtio-net nic
˅
vdpa
˅
hardware sf
˅
(doca flow) switch port
˅
(doca flow) basic decision (by port) pipe (determine tenant)
˅
(doca flow) lpm pipe (for tenant)
˅
(doca flow) forward to port with mac swap action
˅
hardware sf
˅
vdpa
˅
virtio-net nic
˅
another vm/netns
- Make sure you run
doca_flow_*_destroyfor everything that needs to have it.