weave/docs/fastdp.md

# Overlay Method Selection

Weave automatically selects the best working overlay implementation
when connecting to a new peer. It does this by initiating both
forwarders in parallel, eventually settling on the most preferred
established forwarder at which point the other forwarder is shut down,
assuming it had not failed already. In the event that the remaining
forwarder fails at a later time (e.g. because of a heartbeat timeout)
the control plane TCP connection will recycle allowing the selection
algorithm to run again. By default fast datapath is preferred over
Sleeve.

# Local Bridging

Unlike the weave bridge netdev used by Sleeve, the OVS datapath has no
inherent bridging capability. Consequently fastdp implements an
ethernet bridge in addition to the vxlan overlay, maintaining its own
forwarding database by learning MAC addresses and dispatching
broadcast, unicast & multicast traffic accordingly.

# Short Peer IDs

The router relies on having access to the source and destination peer
when deciding how to forward packets. When the Sleeve overlay is in
use this information is conveyed directly within the encapsulation by
including the names of the peers in question, a solution not
accommodated directly by the vxlan wire format. Fortunately vxlan does
have a twenty four bit segment ID field in the header we can use to
encode this data - the challenge is to identify peers uniquely with a
twelve bit identifier instead of the seventeen bytes used by Sleeve.
In practice this is achieved by peers adopting a random 'short ID' and
resolving ownership collisions of same via the existing gossip
mechanism.

# Heartbeats via vxlan

Heartbeats are implemented by sending an encapsulated ethernet frame
with source and destination MAC addresses of `00:00:00:00:00:00` and a
payload consisting of the connection UID and total ethernet frame
length. The vxlan vport miss handler on the receiving side detects the
all-zero addresses and acknowledges the heartbeat via the TCP control
channel after validating the connection UID and frame length against
their expected values.

# PMTU Discovery

The length of the heartbeat ethernet frame is set to the MTU of the
datapath interface. In the event the vxlan packet is dropped or
truncated, the heartbeat will not be acknowledged; this lack of
acknowledgement will cause the peers to fall back to the Sleeve
overlay, which has a more sophisticated dynamic mechanism for coping
with low path MTUs.

To avoid triggering this fallback in typical deployments, the datapath
interface is statically configured with an MTU of 1376 bytes allowing
it to work with most underlay network provider MTUs, including GCE at
1460 bytes (the eighty four byte difference accommodates the encrypted vxlan overhead).
This value can be overridden by setting `WEAVE_MTU` at launch if
necessary.

# Virtual Ports

There are three kinds of virtual port associated with the weave
datapath:

* `internal` - one of these is created automatically, named after the
  datapath. It corresponds to the network interface of the same name
  that appears on the host when the datapath is created; it is the
  ingress/egress port for `weave expose`
* `netdev` - one for each application container, corresponding to the
  host end of the veth pair
* `vxlan` - one for each UDP port on which we are listening for vxlan
  encapsulated packets. Typically there is only one of these, but
  there may be more in a network in which peers do not all use the same
  port configuration

# Misses and Flow Creation

As mentioned above, the datapath has no inherent behaviour - any
ingressing packet which does not match a flow is passed to a userspace
miss handler. The miss handler is then responsible for a) instructing
the datapath to take actions for the packet that caused the miss (for
example by copying it to one or more ports) and optionally b)
installing flow rules which will allow the datapath to act on similar
packets in future without invoking the miss handler.

At a high level the fast datapath implementation can be viewed as a
set of miss handlers that determine what actions the OVS datapath
should take based on router topology information, together with some
additional machinery that manages the expiry of resulting flows when
that state changes.

When a miss handler is invoked, it has two pieces of context: a byte
array containing the packet that triggered the miss, and a set of
'flow keys' that have been extracted from the packet by the kernel
module. The following flow keys are of interest to weave:

* `InPort` - the identifier of the ingress virtual port
* `Ethernet` - the source and destination MAC addresses of the
  ethernet frame
* `Tunnel` - tunnel identifier (see section on short peer IDs above),
  and source/dest IPv4 addresses. Only present for ingress via a vxlan
  vport

Crucially, the router must use this information alone to determine
which actions to take - this allows the specification of a flow which
matches these keys and instructs the kernel to take action
automatically in future without further userspace involvement.

Two actions are of interest:

* `Output` - output the packet to the specified vport
* `SetTunnel` - update the effective vxlan tunnel parameters. Only
  required prior to output from a vxlan vport

It is possible to have multiple actions in a flow, so the router can
for example create a single rule that copies broadcast traffic to all
'local' vports (save the ingress vport obviously) as well as to vxlan
vports for onward peers as dictated by the routing topology.

Every flow specified by the router has the following characteristics:

* An `InPort` key matching the ingress vport
* A `Tunnel` key if the ingress vport is of type vxlan
* An `Ethernet` key matching the source and destination MAC
* A list of `Output` (for internal and netdev vports) and
  `SetTunnel`+`Output` (for vxlan vports) actions

Under certain circumstances the miss handler will instruct the
datapath to execute a set of actions for a packet without creating a
corresponding flow:

* When broadcasting packets to discover the path to unknown
  destination MACs. This condition is transient in the overwhelming
  majority of cases and so the benefit of creating a flow is
  outweighed by the need to invalidate it once the MAC is learned
* When the packet also needs to be forwarded to one or more peers via
  Sleeve. In this case the router needs to handle all subsequent
  matching packets as there is no OVS flow action to do it without
  userspace involvement

# Flow Invalidation

Once a flow has been created for a particular combination of keys, the
miss handler will never again be invoked for matching packets. It is
therefore extremely important that we detect events which invalidate
existing flow actions:

* Addition of netdev vports to the datapath
* Route invalidation (topology change)
* Short peer ID collision

The response in each case is the same - delete all flows from the
datapath, allowing them to be recreated taking into account the
updated state.

In addition to these event based invalidations there is an expiry
process that executes every five minutes. This process enumerates all
flows in the datapath, removing any which have not been used since the
last check; this cleans up:

* Flows referring to netdev vports which have been removed
* Flows created by forwarders which have been stopped
* Flows related to MAC addresses which have not communicated recently

Flows keyed on tunnel IPv4 address do not need to be cleared when a
peer appears to change IP address due to NAT; this will cause a miss
resulting in a new flow, and the old flow will expire naturally via
the timer mechanism.