We were invoking AddConnection in a separate goroutine, and this could
result in RemoveConnection getting invoked first, resulting in bogus
entries in our connection map, which would effectively prevent
connectivity to the other peer until the stale entry got removed
through a lost tie break. And if, per chance, the remote end ran into
exactly the same race condition, then connectivity between that peer
and ourself was broken permanently.
Hitting this race condition required a combination of a) no UDP
heartbeat being received for HeartbeatTimeout(30s) - this causes the
actorLoop to terminate and RemoveConnection to be invoked, and b) the
AddConnection invocation getting delayed for at least that long.
...by obtaining our connection set just once. This also guards against
potential locking issues, since we no longer obtain the required
lock on our local peer while holding a lock on Peers.
Note that this change means the connection list can be out of date by
the time we enter Peers.ForEach. However, previously it was possible
for the list to change for each peer we are iterating over. So it's
just a different flavour of out-of-dateness.
Instead of following the usual actor pattern, we use a 1-place channel
with a non-blocking send. Essentially, if there is already a pending
recalculation request queued up, there is no point enqueuing another
one.
I can get 29 weaves running on my laptop, and once all connections are
established, the avg cpu load is ~10%. 30 runs into difficulty because
the concurrent connection establishment does create a very high load,
resulting in some timeouts. But that wouldn't happen when running each
weave in a separate machine (with more CPU resources overall than my
single laptop).
Stop gap solution for #426.
which allows launching of several weave routers on the same machine,
by invoking this multiple times with different WEAVE_CONTAINER_NAMEs.
The routers are not connected to the bridge and do not perform any
capture/injection. So this is mainly intended for testing of
administrative and control plane functionality, rather than routing.
Bryan Boreham