With the per-broadcast-source GossipSenders gone, we can get rid of
GossipSender gc...
Instead of maintaining a Connection->GossipSender map in the
GossipChannels - which we then need to gc when connections get closed
- we keep gossip_channel_name->GossipSender maps in LocalConnections.
Access to the map is guarded by a separate lock, rather than re-using
the LocalConnection lock. This reduces lock contention and risk of
deadlock.
In the GossipSenders, we replace the Stop() method with a signalling
channel, i.e. senders stop when that channel gets closed. Handily, the
LocalConnection.finished channel fits that description. So when a
LocalConnection's run loop terminates, any existing and future
associated GossipSenders will end up terminating too.
Previously broadcasts were being handled by one GossipSender per
broadcast source (and channel), which sends the broadcast to all next
hops, and hence can stall when a single destination is stalled.
Here we get rid of these per broadcast source GossipSenders. Instead
broadcasts are sent to per-connection GossipSenders for all next hops.
For this we use the existing GossipSenders we have set up for ordinary
gossip. In order to deal with broadcasts they need one GossipData cell
per broadcast source, since only broadcasts from the same source can
be Merge()ed. So we add a PeerName->GossipData map of cells, in
addition to the existing cell for ordinary gossip.
The GossipSender goroutine picks one of the cells at a time, Encode()s
the contents and sends it. It prefers the ordinary gossip cell over
the broadcast cells since typically ordinary gossip is more important.
To reduce coupling, the GossipSenders don't actually know how to
construct protocol messages. They just invoke a couple of functions
for that - one for ordinary gossip and one for broadcast - which are
supplied by the GossipChannel.
There are two downsides to this change:
1. broadcasts get encoded per connection, rather than just once
2. we can potentially end up with O(n_peers^2) cells, each containing
accumulated (via GossipData.Merge()) broadcasts. For this to happen,
the peer must
- deal with gossip from most nodes. This doesn't happen in
(near)complete connection topologies. Hypercube topologies are
probably the worst case uniform topology. And a star topology is the
worst case for a single (the centre) peer.
- have backlogged connections to most of its neighbours, without
those connections being completely stalled (since that would cause
heartbeat timeouts to terminate them).
Furthermore, for this to matter in practise, the accumulated broadcast
GossipData must be sizeable:
- For topology gossip, GossipData is just a set of PeerNames. Which
takes very little space and is bounded, since there is a finite
number of peers. And, if we could get rid of workaround for #1793
(cbaa92dcb), then topology broadcasts would only ever contain
information about the source peer, so the GossipData would contain
just one PeerName.
- IPAM only employs broadcast during initialisation and shutdown.
- DNS broadcasts contain DNS entries for containers on the source
peer. Each entry will typically be 100-200 bytes. The accumulated
broadcast GossipData from a peer will contain entries for all the
peer's DNS entries, worst case. This includes tombstones,
i.e. entries for containers that have died. If there is churn,
i.e. DNS entries being added and removed continuously, and the churn
rate exceeds the rate at which we can forward those entries, then
the accumulated broadcast GossipData can grow unbounded. Note that
this is the case on master too; the difference here is that we can
have up to n_peers copies of that GossipData.
In particular don't require Send() to be serialised by the caller.
We split up the triple(!) purpose 'cell' channel into:
1) a 'cell' member, holding pending GossipData or nil
2) a 'more' channel, for signalling that 'cell' has been filled
3) a 'stop' channel, for requesting termination
An error in SendProtocolMsg indicates that the connection is
permanently broken. So there is no point continuing to attempt to send
gossip on it.
The connection will eventually be removed from LocalPeer, and any
associated GossipSenders will be gc'ed, but that doesn't happen
straight away, so this change is about not doing unnecessary work in
the meantime.
This is a one-sided operation, so we shouldn't be touching the
remote's Peers.
Also fix the associated test... When both 1 and 2 have lost their
connection to 3, they no longer have a route to 3 and hence gc it.
And because they no longer have a route, gossip from them won't get
through to 3.
We were creating peers with NewPeerFrom, which copies the Version (but
not connections). That's not what happens for a real connection, where
remote peers are always created with an initial version of 0.
The mistake results in brokenness: when we subsequently receive a
topology update for that peer with the same version then we ignore it,
even though the update contains connection information which is new to
us.
Related, it is conceptually wrong to call FetchWithDefault on the
remote router. That is not what happens for a real connection - the
FetchWithDefault occurs in the symmetric counterpart.
If overlay encryption is required between peers, the fast datapath
forwarder returns an error rather than aborting the process, allowing
the overlay switch to fall back gracefully to an encrypted sleeve
overlay connection.
Add a 'Trusted' feature to the protocol exchange which indicates whether
the remote considers us to be reachable via a secure network; overlay
encryption will be disabled in the event that both peers agree. For
backwards compatibility purposes a default value of untrusted is assumed
if the feature is missing.
Instead of calculating all routes in one go, we calculate them when
needed.
Except for unicast routes, and broadcast routes from ourself. We still
calculate those eagerly because they are needed to route locally
captured/fdp-missed packets, and performing route calculcations in
that critical path increases the chances of dropping packets due to
not keeping up.
Well, technically only the established & symmetric routes are needed
for the data plane, but we might as well calculate the equivalent
routes for the control plane - it keeps the code simple and uniform.
Calculating all routes is O(n_peers^2), so this change represents a
huge saving in situations where not all of them are needed, which is
quite common, especially when the topology is evolving rapidly.
One downside of this new approach is that we can no longer suppress
OnChange callbacks when the data plane routes remain unchanged. This
means we will be purging the FDP flows more frequently than before,
e.g. when only the control plane topology changed, e.g. new
connections appeared that haven't been marked as 'established'
yet. It's a small price to pay, fixable if we really care, and in any
case making FDP flow invalidation more fine-grained would be a better
approach.
Fixes#1761.
real Connections do not process gossip until after AddConnection.
mockChannelConnection wasn't behaving in that way, which resulted in
some sporadic test failures since it would cause some gossip to get
processed too early, which in turn would trigger gc of some of the
just-received peers, which we would then find missing in the test
assertions.
To fix this we introduce a latch on mockChannelConnection which closes
when the remote end is ready to receive gossip. Effectively this
emulates the entering of LocalConnection.receiveTCP after the call to
AddConnection completes.
Unfortunately an open latch will cause sendPendingGossip to block. We
get around that by abandoning checks in "half-connected" connection
states.
Fixes#1765.
This is more efficient since broadcasting about other peers is
strictly redundant. Any change to the peers will be broadcast by
those peers anyway. And SendAllGossipDown takes care of informing new
peers about others on first connect, and periodic gossip take care of
missed broadcasts.
The only reason we did broadcast about the remote peer is to reduce
the liklihood of UnknownPeerError and the associated throwing away of
gossip. This was always a bit of kludge, and is no longer necessary
since UnknownPeerError is gone.
Now peers only ever broadcast information about themselves. Which,
unlike information about other peers, is always authorative and most
up-to-date. Which is nice.
when encountering an unknown peer in received gossip, rather than
ignoring the entire gossip we create a placeholder Peer. This has
version 0 and uid 0, which means it will be replaced by any complete
peer information subsequently received via gossip.
Matthias Radestock