weave/site/how-it-works.md

title, layout

title	layout
How Weave works	default

How does it work?

A weave network consists of a number of 'peers' - weave routers residing on different hosts. Each peer has a name, which tends to remain the same over restarts, and a unique identifier (UID) which is different each time it is run. These are opaque identifiers as far as the router is concerned, although the name defaults to a MAC address.

Weave routers establish TCP connections to each other, over which they perform a protocol handshake and subsequently exchange topology information. These connections are encrypted if so configured. Peers also establish UDP "connections", possibly encrypted, which carry encapsulated network packets. These "connections" are duplex and can traverse firewalls.

Weave creates a network bridge on the host. Each container is connected to that bridge via a veth pair, the container side of which is given the IP address & netmask supplied in 'weave run'. Also connected to the bridge is the weave router container.

A weave router captures Ethernet packets from its bridge-connected interface in promiscuous mode, using 'pcap'. This typically excludes traffic between local containers, and between the host and local containers, all of which is routed straight over the bridge by the kernel. Captured packets are forwarded over UDP to weave router peers running on other hosts. On receipt of such a packet, a router injects the packet on its bridge interface using 'pcap' and/or forwards the packet to peers.

Weave routers learn which peer host a particular MAC address resides on. They combine this knowledge with topology information in order to make routing decisions and thus avoid forwarding every packet to every peer. Weave can route packets in partially connected networks with changing topology. For example, in this network, peer 1 is connected directly to 2 and 3, but if 1 needs to send a packet to 4 or 5 it must first send it to peer 3:

Encapsulation

When the weave router forwards packets, the encapsulation looks something like this:

+-----------------------------------+
| Name of sending peer              |
+-----------------------------------+
| Frame 1: Name of capturing peer   |
+-----------------------------------+
| Frame 1: Name of destination peer |
+-----------------------------------+
| Frame 1: Captured payload length  |
+-----------------------------------+
| Frame 1: Captured payload         |
+-----------------------------------+
| Frame 2: Name of capturing peer   |
+-----------------------------------+
| Frame 2: Name of destination peer |
+-----------------------------------+
| Frame 2: Captured payload length  |
+-----------------------------------+
| Frame 2: Captured payload         |
+-----------------------------------+
|                ...                |
+-----------------------------------+
| Frame N: Name of capturing peer   |
+-----------------------------------+
| Frame N: Name of destination peer |
+-----------------------------------+
| Frame N: Captured payload length  |
+-----------------------------------+
| Frame N: Captured payload         |
+-----------------------------------+

The name of the sending peer enables the receiving peer to identify who sent this UDP packet. This is followed by the meta data and payload for one or more captured frames. The router performs batching: if it captures several frames very quickly that all need forwarding to the same peer, it fits as many of them as possible into a single UDP packet.

The meta data for each frame contains the names of the capturing and destination peers. Since the name of the capturing peer name is associated with the source MAC of the captured payload, it allows receiving peers to build up their mappings of which client MAC addresses are local to which peers. The destination peer name enables the receiving peer to identify whether this frame is destined for itself or whether it should be forwarded on to some other peer, accommodating multi-hop routing. This works even when the receiving intermediate peer has no knowledge of the destination MAC: only the original capturing peer needs to determine the destination peer from the MAC. This way weave peers never need to exchange the MAC addresses of clients and need not take any special action for ARP traffic and MAC discovery.

Topology

The topology information captures which peers are connected to which other peers. Weave peers communicate their knowledge of the topology (and changes to it) to their neighbours, who then pass this information on to their own neighbours, and so on, until the entire network knows about any change.

Topology is communicated over the TCP links between peers, using a Gossip mechanism. Topology messages are sent by a peer...

• when a connection has been added; the entire topology is sent to the remote peer, and an incremental update, containing information on just the two peers at the ends of the connection, is sent to all neighbours,
• when a connection has been marked as 'established', indicating that the remote peer can receive UDP traffic from the peer; an update containing just information about the local peer is sent to all neighbours,
• when a connection has been torn down; an update containing just information about the local peer is sent to all neighbours,
• periodically, on a timer, in case someone has missed an update.

The receiver of a topology update merges that update with its own topology model, adding peers hitherto unknown to it, and updating peers for which the update contains a more recent version than known to it. If there were any such new/updated peers, then an improved update containing them is sent out on all connections.

If the update mentions a peer that the receiver does not know, then the entire update is ignored.

Message details

Every gossip message is structured as follows:

+-----------------------------------+
| 1-byte message type - Gossip      |
+-----------------------------------+
| 4-byte Gossip channel - Topology  |
+-----------------------------------+
| Peer Name of source               |
+-----------------------------------+
| Gossip payload (topology update)  |
+-----------------------------------+

The topology update payload is laid out like this:

+-----------------------------------+
| Peer 1: Name                      |
+-----------------------------------+
| Peer 1: UID                       |
+-----------------------------------+
| Peer 1: Version number            |
+-----------------------------------+
| Peer 1: List of connections       |
+-----------------------------------+
|                ...                |
+-----------------------------------+
| Peer N: Name                      |
+-----------------------------------+
| Peer N: UID                       |
+-----------------------------------+
| Peer N: Version number            |
+-----------------------------------+
| Peer N: List of connections       |
+-----------------------------------+

Each List of connections is encapsulated as a byte buffer, within which the structure is:

+-----------------------------------+
| Connection 1: Remote Peer Name    |
+-----------------------------------+
| Connection 1: Remote IP address   |
+-----------------------------------+
| Connection 2: Remote Peer Name    |
+-----------------------------------+
| Connection 2: Remote IP address   |
+-----------------------------------+
|                ...                |
+-----------------------------------+
| Connection N: Remote Peer Name    |
+-----------------------------------+
| Connection N: Remote IP address   |
+-----------------------------------+

Removal of peers

If a peer, after receiving a topology update, sees that another peer no longer has any connections within the network, it drops all knowledge of that second peer.

Out-of-date topology

The peer-to-peer gossiping of updates is not instantaneous, so it is very possible for a node elsewhere in the network to have an out-of-date view.

If the destination peer for a packet is still reachable, then out-of-date topology can result in it taking a less efficient route.

If the out-of-date topology makes it look as if the destination peer is not reachable, then the packet is dropped. For most protocols (e.g. TCP), the transmission will be retried a short time later, by which time the topology should have updated.

Crypto

Weave can be configured to encrypt both the data passing over the TCP connections and the payloads of UDP packets sent between peers. This is accomplished using the NaCl crypto libraries, employing Curve25519, XSalsa20 and Poly1305 to encrypt and authenticate messages. Weave protects against injection and replay attacks for traffic forwarded between peers.

NaCl was selected because of its good reputation both in terms of selection and implementation of ciphers, but equally importantly, its clear APIs, good documentation and high-quality go implementation. It is quite difficult to use NaCl incorrectly. Contrast this with libraries such as OpenSSL where the library and its APIs are vast in size, poorly documented, and easily used wrongly.

Establishing the Ephemeral Session Key

For every connection between peers, a fresh public/private key pair is created at both ends, using NaCl's GenerateKey function. The public key portion is sent to the other end as part of the initial handshake performed over TCP. Peers that were started with a password do not continue with connection establishment unless they receive a public key from the remote peer. Thus either all peers in a weave network must be supplied with a password, or none.

When a peer has received a public key from the remote peer, it uses this to form the ephemeral session key for this connection. The public key from the remote peer is combined with the private key for the local peer in the usual Diffie-Hellman way, resulting in both peers arriving at the same shared key. To this is appended the supplied password, and the result is hashed through SHA256, to form the final ephemeral session key. Thus the supplied password is never exchanged directly, and is thoroughly mixed into the shared secret. The shared key formed by Diffie-Hellman is 256 bits long, appending the password to this obviously makes it longer by an unknown amount, and the use of SHA256 reduces this back to 256 bits, to form the final ephemeral session key. This late combination with the password eliminates "Man In The Middle" attacks: sniffing the public key exchange between the two peers and faking their responses will not grant an attacker knowledge of the password, and so an attacker would not be able to form valid ephemeral session keys.

The same ephemeral session key is used for both TCP and UDP traffic between two peers.

TCP

TCP connection are only used to exchange topology information between peers, via a message-based protocol. The router generates a fresh 192-bit random nonce for every message to be sent, and prepends the nonce to the encrypted message, as is normal in NaCl, so that the receiver knows the nonce. Encryption of each message is carried out using NaCl's secretbox.Seal function using the ephemeral session key. Each TCP connection has a monotonically incrementing message counter, the current value of which is included in the encrypted part of the message. Given the assumption that TCP is reliable and ordered, a message received via TCP is only acted upon if the message counter in the received message is the expected message counter. This prevents replay attacks on the TCP connection.

As TCP connections do not carry captured traffic, minimising message size or latency is not a major concern, so the potentially substantial increase in length of messages by prepending the full nonce, or the cost of generating a fresh random nonce for each message is not considered likely to cause problems. The random nonces are created by the go crypto/rand package, which implements a cryptographically secure pseudorandom number generator.

UDP

UDP connections carry captured traffic between peers. For a UDP packet sent between peers that are using crypto, the encapsulation looks as follows:

+-----------------------------------+
| Name of sending peer              |
+-----------------------------------+
| Nonce offset and flags            |
+-----------------------------------+
| NaCl SecretBox overheads          |
+-----------------------------------+ -+
| Frame 1: Name of capturing peer   |  |
+-----------------------------------+  | This section is encrypted
| Frame 1: Name of destination peer |  | using the ephemeral session
+-----------------------------------+  | key between the weave peers
| Frame 1: Captured payload length  |  | sending and receiving this
+-----------------------------------+  | packet.
| Frame 1: Captured payload         |  |
+-----------------------------------+  |
| Frame 2: Name of capturing peer   |  |
+-----------------------------------+  |
| Frame 2: Name of destination peer |  |
+-----------------------------------+  |
| Frame 2: Captured payload length  |  |
+-----------------------------------+  |
| Frame 2: Captured payload         |  |
+-----------------------------------+  |
|                ...                |  |
+-----------------------------------+  |
| Frame N: Name of capturing peer   |  |
+-----------------------------------+  |
| Frame N: Name of destination peer |  |
+-----------------------------------+  |
| Frame N: Captured payload length  |  |
+-----------------------------------+  |
| Frame N: Captured payload         |  |
+-----------------------------------+ -+

This is very similar to the non-crypto encapsulation.

All of the frames are encrypted with the same ephemeral session key and all must be decrypted by the receiving peer. Frames which are to be forwarded on to some further peer will be re-encrypted with the relevant ephemeral session keys for the onward connections. Thus all traffic is fully decrypted on every peer it passes through. Encryption is again done with the NaCl secretbox.Seal function.

The name of the sending peer enables the receiver to identify the peer who sent this UDP packet, and in turn to determine which ephemeral session key was used and which nonce, and perform decryption. To avoid sending a fresh 192-bit nonce with every UDP packet, which would pose an unacceptable overhead, each UDP packet only carries the lowest 15 bits of the nonce, which is treated as an offset from the "established nonce". The lifecycle of a nonce for UDP is:

1. A fresh nonce is generated and the most significant 177 bits are sent to the receiving peer over the TCP connection. These upper most 177 bits are used for the next 2^15 (32768) UDP messages. This is the "established nonce".

2. Each UDP message carries the lowest 15 bits as a unique counter. Thus the lowest 15 bits are combined with (appended to) the uppermost 177 bits to form the unique nonce for that message.

3. Once the sending side has sent the 16384'th message on the current nonce (50% of the way through the available range), it generates a new nonce (upper 177 bits only) and sends that over the TCP connection, thus hopefully ensuring it arrives and is ready before it is needed. The sending side will switch to using the new nonce once the full 32768 messages of the current nonce have been used. At this point, the sending side resets the offset to 0.

4. The receiving side must deal with the fact that UDP is unreliable and unordered. The receiving side keeps track of the highest offset seen for each established nonce. The condition to switching to the new nonce it has received via the TCP connection is: the highest offset seen for the current nonce must be above 24576 (75% of the available range) and the current message just received must have an offset less than 8192 (25% of the available range) and the new offset must be less than 8192 messages ahead of the highest seen offset so far (assuming modulo 32768). If these conditions are met, the highest offset is set to the offset of the current message and the new nonce is used. If those conditions are not met, the receiving side continues with the current nonce (according to the rules below), updating the highest offset seen as appropriate. Thus in order for the nonce to not be updated correctly would require a loss of at least 8192 messages.

5. When the receiving side switches to the new nonce, it does not discard the old nonce. If the highest offset seen is below 8192 (25% of the range) and the current message offset is above 24576 (75% of the range), and the current message offset is less than 8192 behind the highest seen nonce (assuming modulo 32768), then the current message is decoded with the old nonce.

6. In the remaining cases, the current message is only decoded if its offset is within 8192 either side of the highest seen offset.

7. To avoid replay attacks, the receiving side keeps a set of which offsets have been used with the current and previous nonce. If the offset doesn't exist in the set, and the message can be correctly decoded, the offset is added to the relevant set (thus we avoid poisoning attacks). If the offset already exists in the set, or the message cannot be correctly decoded, the message is not processed further.

Further reading

More details on the inner workings of weave can be found in the architecture documentation.