With the per-broadcast-source GossipSenders gone, we can get rid of
GossipSender gc...
Instead of maintaining a Connection->GossipSender map in the
GossipChannels - which we then need to gc when connections get closed
- we keep gossip_channel_name->GossipSender maps in LocalConnections.
Access to the map is guarded by a separate lock, rather than re-using
the LocalConnection lock. This reduces lock contention and risk of
deadlock.
In the GossipSenders, we replace the Stop() method with a signalling
channel, i.e. senders stop when that channel gets closed. Handily, the
LocalConnection.finished channel fits that description. So when a
LocalConnection's run loop terminates, any existing and future
associated GossipSenders will end up terminating too.
Previously broadcasts were being handled by one GossipSender per
broadcast source (and channel), which sends the broadcast to all next
hops, and hence can stall when a single destination is stalled.
Here we get rid of these per broadcast source GossipSenders. Instead
broadcasts are sent to per-connection GossipSenders for all next hops.
For this we use the existing GossipSenders we have set up for ordinary
gossip. In order to deal with broadcasts they need one GossipData cell
per broadcast source, since only broadcasts from the same source can
be Merge()ed. So we add a PeerName->GossipData map of cells, in
addition to the existing cell for ordinary gossip.
The GossipSender goroutine picks one of the cells at a time, Encode()s
the contents and sends it. It prefers the ordinary gossip cell over
the broadcast cells since typically ordinary gossip is more important.
To reduce coupling, the GossipSenders don't actually know how to
construct protocol messages. They just invoke a couple of functions
for that - one for ordinary gossip and one for broadcast - which are
supplied by the GossipChannel.
There are two downsides to this change:
1. broadcasts get encoded per connection, rather than just once
2. we can potentially end up with O(n_peers^2) cells, each containing
accumulated (via GossipData.Merge()) broadcasts. For this to happen,
the peer must
- deal with gossip from most nodes. This doesn't happen in
(near)complete connection topologies. Hypercube topologies are
probably the worst case uniform topology. And a star topology is the
worst case for a single (the centre) peer.
- have backlogged connections to most of its neighbours, without
those connections being completely stalled (since that would cause
heartbeat timeouts to terminate them).
Furthermore, for this to matter in practise, the accumulated broadcast
GossipData must be sizeable:
- For topology gossip, GossipData is just a set of PeerNames. Which
takes very little space and is bounded, since there is a finite
number of peers. And, if we could get rid of workaround for #1793
(cbaa92dcb), then topology broadcasts would only ever contain
information about the source peer, so the GossipData would contain
just one PeerName.
- IPAM only employs broadcast during initialisation and shutdown.
- DNS broadcasts contain DNS entries for containers on the source
peer. Each entry will typically be 100-200 bytes. The accumulated
broadcast GossipData from a peer will contain entries for all the
peer's DNS entries, worst case. This includes tombstones,
i.e. entries for containers that have died. If there is churn,
i.e. DNS entries being added and removed continuously, and the churn
rate exceeds the rate at which we can forward those entries, then
the accumulated broadcast GossipData can grow unbounded. Note that
this is the case on master too; the difference here is that we can
have up to n_peers copies of that GossipData.
Matthias Radestock