From 57a15a4e963da51cc51f8ab002aa30794e055c82 Mon Sep 17 00:00:00 2001 From: Adam Harrison Date: Thu, 15 Oct 2015 17:30:45 +0100 Subject: [PATCH 1/2] Add fast datapath high level overview --- docs/fastdp.md | 168 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 168 insertions(+) create mode 100644 docs/fastdp.md diff --git a/docs/fastdp.md b/docs/fastdp.md new file mode 100644 index 00000000..231e9414 --- /dev/null +++ b/docs/fastdp.md @@ -0,0 +1,168 @@ +# Overlay Method Selection + +Weave automatically selects the best working overlay implementation +when connecting to a new peer. It does this by initiating both +forwarders in parallel, eventually settling on the most preferred +established forwarder at which point the other forwarder is shut down, +assuming it had not failed already. In the event that the remaining +forwarder fails at a later time (e.g. because of a heartbeat timeout) +the control plane TCP connection will recycle allowing the selection +algorithm to run again. By default fast datapath is preferred over +Sleeve. + +# Local Bridging + +Unlike the weave bridge netdev used by Sleeve, the OVS datapath has no +inherent bridging capability. Consequently fastdp implements an +ethernet bridge in addition to the vxlan overlay, maintaining its own +forwarding database by learning MAC addresses and dispatching +broadcast, unicast & multicast traffic accordingly. + +# Short Peer IDs + +The router relies on having access to the source and destination peer +when deciding how to forward packets. When the Sleeve overlay is in +use this information is conveyed directly within the encapsulation by +including the names of the peers in question, a solution not +accomodated directly by the vxlan wire format. Fortunately vxlan does +have a twenty four bit segment ID field in the header we can use to +encode this data - the challenge is to identify peers uniquely with a +twelve bit identifier instead of the seventeen bytes used by Sleeve. +In practice this is achieved by peers adopting a random 'short ID' and +resolving ownership collisions of same via the existing gossip +mechanism. + +# Heartbeats via vxlan + +Heartbeats are implemented by sending an encapsulated ethernet frame +with source and destination MAC addresses of `00:00:00:00:00:00` and a +payload consisting of the connection UID and total ethernet frame +length. The vxlan vport miss handler on the receiving side detects the +all-zero addresses and acknowledges the heartbeat via the TCP control +channel after validating the connection UID and frame length against +their expected values. + +# PMTU Discovery + +The length of the heartbeat ethernet frame is set to the MTU of the +datapath interface. In the event the vxlan packet is dropped or +truncated, the heartbeat will not be acknowledged; this lack of +acknowledgement will cause the peers to fall back to the Sleeve +overlay, which has a more sophisticated dynamic mechanism for coping +with low path MTUs. + +To avoid triggering this fallback in typical deployments, the datapath +interface is statically configured with an MTU of 1410 bytes allowing +it to work with most underlay network provider MTUs, including GCE at +1460 bytes (the fifty byte difference accomodates the vxlan overhead). + +# Virtual Ports + +There are three kinds of virtual port associated with the weave +datapath: + +* `internal` - one of these is created automatically, named after the + datapath. It corresponds to the network interface of the same name + that appears on the host when the datapath is created; it is the + ingress/egress port for `weave expose` +* `netdev` - one for each application container, corresponding to the + host end of the veth pair +* `vxlan` - one for each UDP port on which we are listening for vxlan + encapsulated packets. Typically there is only one of these, but + there may be more in a network in which peers do not all use the same + port configuration + +# Misses and Flow Creation + +As mentioned above, the datapath has no inherent behaviour - any +ingressing packet which does not match a flow is passed to a userspace +miss handler. The miss handler is then responsible for a) instructing +the datapath to take actions for the packet that caused the miss (for +example by copying it to one or more ports) and optionally b) +installing flow rules which will allow the datapath to act on similar +packets in future without invoking the miss handler. + +At a high level the fast datapath implementation can be viewed as a +set of miss handlers that determine what actions the OVS datapath +should take based on router topology information, together with some +additional machinery that manages the expiry of resulting flows when +that state changes. + +When a miss handler is invoked, it has two pieces of context: a byte +array containing the packet that triggered the miss, and a set of +'flow keys' that have been extracted from the packet by the kernel +module. The following flow keys are of interest to weave: + +* `InPort` - the identifier of the ingress virtual port +* `Ethernet` - the source and destination MAC addresses of the + ethernet frame +* `Tunnel` - tunnel identifier (see section on short peer IDs above), + and source/dest IPv4 addresses. Only present for ingress via a vxlan + vport + +Crucially, the router must use this information alone to determine +which actions to take - this allows the specification of a flow which +matches these keys and instructs the kernel to take action +automatically in future without further userspace involvement. + +Two actions are of interest: + +* `Output` - output the packet to the specified vport +* `SetTunnel` - update the effective vxlan tunnel parameters. Only + required prior to output from a vxlan vport + +It is possible to have multiple actions in a flow, so the router can +for example create a single rule that copies broadcast traffic to all +'local' vports (save the ingress vport obviously) as well as to vxlan +vports for onward peers as dictated by the routing topology. + +Every flow specified by the router has the following characteristics: + +* An `InPort` key matching the ingress vport +* A `Tunnel` key if the ingress vport is of type vxlan +* An `Ethernet` key matching the source and destination MAC +* A list of `Output` (for internal and netdev vports) and + `SetTunnel`+`Output` (for vxlan vports) actions + +Under certain circumstances the miss handler will instruct the +datapath to execute a set of actions for a packet without creating a +corresponding flow: + +* When broadcasting packets to discover the path to unknown + destination MACs. This condition is transient in the overwhelming + majority of cases and so the benefit of creating a flow is + outweighed by the need to invalidate it once the MAC is learned +* When the packet also needs to be forwarded to one or more peers via + Sleeve. In this case the router needs to handle all subsequent + matching packets as there is no OVS flow action to do it without + userspace involvement + +# Flow Invalidation + +Once a flow has been created for a particular combination of keys, the +miss handler will never again be invoked for matching packets. It is +therefore extremely important that we detect events which invalidate +existing flow actions: + +* Addition of netdev vports to the datapath +* Route invalidation (topology change) +* Short peer ID collision + +The response in each case is the same - delete all flows from the +datapath, allowing them to be recreated taking into account the +updated state. + +In addition to these event based invalidations there is an expiry +process that executes every five minutes. This process enumerates all +flows in the datapath, removing any which have not been used since the +last check; this cleans up: + +* Flows referring to netdev vports which have been removed +* Flows created by forwarders which have been stopped +* Flows related to MAC addresses which have not communicated recently + +Flows keyed on tunnel IPv4 address do not need to be cleared when a +peer appears to change IP address due to NAT; this will cause a miss +resulting in a new flow, and the old flow will expire naturally via +the timer mechanism. + From 2f69b4d190dca9dd512c684be09f7aa877b2f0b7 Mon Sep 17 00:00:00 2001 From: Adam Harrison Date: Thu, 26 Nov 2015 10:39:41 +0000 Subject: [PATCH 2/2] Mention WEAVE_MTU --- docs/fastdp.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/fastdp.md b/docs/fastdp.md index 231e9414..51aa211e 100644 --- a/docs/fastdp.md +++ b/docs/fastdp.md @@ -55,6 +55,8 @@ To avoid triggering this fallback in typical deployments, the datapath interface is statically configured with an MTU of 1410 bytes allowing it to work with most underlay network provider MTUs, including GCE at 1460 bytes (the fifty byte difference accomodates the vxlan overhead). +This value can be overriden by setting `WEAVE_MTU` at launch if +necessary. # Virtual Ports