Topology gossip can be received on a connection before that connection
has been marked as 'established'. If we do not include unestablished
connections in topology gossip, then parts of the topology can be
removed by gc, since they are deemed unreachable, which can cause
subsequent incremental topology gossips to be rejected due to them
containing references to peers that have been removed by the gc.
Including unestablished connections in topology gossip requires:
- removing the check for 'established' connections in the encoding of
peers for topology gossip,
- adding the 'established' flag to the internal representation of
remote connections,
- adding the 'established' flag to to encoding of connections in
topology gossip,
- treating the *addition* of a connection as an update to the local
peer.
The latter entails
- bumping the peer version on connection addition (as well as on
marking 'established' as previously),
- bumping the peer version *unconditionally* on connection deletion
(rather than only when the connection was 'established', as
previously),
- moving the initial sending of complete topology on the connection,
and the broadcasting of a topology update containing the two peers
at the ends of the connection, from the connection 'established'
logic to the connection addition logic,
- broadcasting just an topology update for the local peer when a
connection gets marked as 'established',
- broadcasting a topology update for the local peer *unconditionally*
when a connection is removed (rather than only doing this when the
connection was 'established', as previously).
Caused by a race between adding the connection to our map and
receiving a topology gossip on it.
We remove the race by making AddConnection synchronous and invoking it
prior to entering the tcp receive loop.
Fixes#367.
Prior to the introduction of gossip, a tcp connection would not see
regular communication until it had been marked as 'established',
because the topology communication scheme would periodically send
'FetchAll' only after that. So in the absence of udp connectivity we'd
time out. The timeout could of course occur for other reasons, namely
disrupted tcp connectivity, but the combination of a timeout and 'not
established' pointed to a lack of udp connectivity as a likely
cause. So that's what we checked for.
In the new gossip-based topology communication, however, gossip (of
anything, not just topology) occurs as soon as we have tcp
connectivity. Therefore we will no longer encounter a timeout when
just udp connectivity is disrupted. Which makes our check invalid.
So instead we introduce a one-shot timer, created on handshake
completion and cancelled when we mark the connection as
'established'. When the timer fires, and the connection still hasn't
been established, we shut down the connection with an error indicating
lack of udp connectivity.
Instead of separately calculating routes to every individual peer,
carry out one calculation for routes to all peers. Same as unicast
routes except taking into account asymmetric connections.
Gossip layer to communicate topology between peers
Note that I have squashed over 100 commits from @bboreham and @rade into the three for this PR; you can see the original history at https://github.com/bboreham/weave/compare/add_gossip_full
Adam Harrison