Previously when a peer restarted, information about the new
incarnation (i.e. with a different UID) was not accepted by other
peers (and connections would be dropped) unless all knowledge of the
previous incarnation had been purged. This could result in a lot of
connection churn and hence connectivity disruption, and, in some
pathological cases, very slow convergence and hence acceptance of the
new incarnation into the network.
We now no longer drop connections when encountering different
incarnations of a peer. There are two situations when that can happen:
1) on connection establishment
we simply proceed
2) on receipt of gossip
to ensure convergence we
a) treat the UID as an additional discriminator when deciding whether
we should update our information about a peer with that which was
gossiped. Specifically, we update the information we hold when a) the
gossiped version is greater, or b) is the same and the UID is
greater.
b) include the UID in the information we update
c) move our own version number beyond any we receive for ourselves, if
the received UID differs from ours.
With (a) we establishes a total order of peer information across
several incarnations of the same peer. i.e. we consider information to
be fresher if it has a higher version, or the same version and higher
UID. This may seem somewhat counter intutive, since it will generally
treat information about new incarnations as older than old
incarnations, since incarnations always start life with version 1. But
to do better we'd need to establish a total order of incarnations that
matches their temporal occurrence. Which requires some sort of durable
state.
So instead we have (c). Through that we learn the highest version
number of any old incarnation of ourselves that other peers still
hold, and then make sure that our version is greater than
that. Essentially we continue where the old incarnations left
off. It's as if instead of restarting we had simply changed UIDs. And
due to (a) and (b) the information about the new incarnation of
ourselves, now with a higher version, will supersede that of the old
incarnations.
Fixes#1554.
If the local peer's short ID was updated, we broadcastPeerUpdate at
the end (in unlockAndNotify), so there is no need to include the local
peer in the current update.
Also, reading peers.ourself.Version is bad here - we don't take a lock
and it's written by other goroutines. Doesn't actually break anything
though since it's a uint64 and hence atomic.
These are needed to support vxlan encapsulation. In a vxlan packet,
the only place we can straightforwardly hold the source and
destination peer identifiers used by the weave router is in the
24-bit VNI (vxlan network identifier) field. This gives us 12 bits to
indicate a peer. The resulting limit of 4096 peers on a weave network
does not seem like a problem in the near future. But it does mean
that decentralized allocation makes collisions fairly likely even in
weave networks of sizes that are common today. Fortunately, the
problem is much more tractable than IPAM because we can detect when
collisions occur and take action to resolve them.
- get rid of individual Peer locks
- hold Peers read (or write) lock and LocalPeer read lock while
performing route calculations
This eliminates the deadlock resulting from the inconsistent lock
acquisition order in route calculcation.
Fixes#643.
This makes for a more symmetric API - Peers.FetchWithDefault does the
increment and Peers.Dereference does the decrement, whereas previously
the latter was done by Peer.DecrementLocalRefCount.
This does increase lock granularity, i.e. modifying any ref count now
requires a write lock on Peers.
When we want a string representation of a Peer, we nearly always just
want the full name. So make String() return that and introduce Info()
to return expanded info.
Introduce an intermediary - GossipSender - between GossipChannel and
Connection. This accumulates gossip data (now represented by the new
GossipData interface) from the former until it can be passed onto the
latter, allowing the former to proceed when the latter may be blocked
on i/o.
This
1) prevents deadlocks that arise from cycles in the communication
topology
2) improves performance by allowing GossipChannel and its calling code
to proceed when connection i/o is blocked
3) improves performance by accumulating GossipData - the accumulated
data often is considerably more compact than the sum of all the
accumulated bits, and it is transmitted in a single communication
event.
In order to support accumulation, the GossipData interface has a Merge
method. Furthermore, encoding is deferred until the data can actually
be sent, since accumulation would be hard/impossible to encoded
data. This required changes on some interfaces, most notably Gossiper.
The implementation of GossipData for topology gossip is
TopologyGossipData. It carries a reference to Peers and a set of
PeerNames, indexing into Peers and referencing the Peer entries which
have changed. Previously updates were represented as a list of Peer
entries. The indirection via PeerNames allows Encode to omit entries
which have been removed. The change in representation required changes
to the signature of some methods on Peers.
Note that all the above only applies to the "periodic gossip" portion
of the Gossiper API, which topology gossip in fact uses for *all*
gossip. GossipUnicast and GossipBroadcast are unchanged, but could
conceivably receive the same treatment in the future.
Fixes#445.