reweave/weave
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

initial commit

Browse Source
This commit is contained in:
Matthias Radestock committed 2014-08-18 17:55:26 +01:00
commit e6feefa1fc
26 files changed
+4476
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/architecture.txt
+209
View File
@@ -0,0 +1,209 @@
Router:
- Starts Topology
- Starts Connection Maker
- Starts TCP Listener
- Starts UDP Listener
- Sniffs traffic
- Starts the local peer
- UDP Listener
This is a process that reads off the UDP socket, decodes the frames
and does either relaying to a connected peer (or peers), or
injection to the local interface (or both) (router.listenUDP /
router.handleUDPPacketFunc)
- Traffic Sniffer
This is a process that captures packets with pcap, decodes the
frames and forwards them to a local peer (or peers) (router.Sniff /
router.handleCapturedPacket)
- Topology
Maintains unicast and broadcast routing tables which can be read
directly from any thread by just obtaining a readlock on topology.
- Unicast answers the question "If I want to send a packet to peer
X, which of the peers I'm connected to should I send the packet
to?"
- Broadcast answers the question "If peer X broadcasts a packet, by
the time it reaches me, which of my connected peers is X expecting
me to pass the packet to?"
Also spawns a thread which runs an actor loop, responding to
requests to encode all peers, or rebuild the routing tables.
- Connection Maker
Spawns a thread which runs an actor loop. This actor loop is passed
the known locations of peers as they're discovered and is informed
when connections we have made die. It proactively tries to create
connections to remote peers we're not connected to using a random
binary exponential backoff to regulate the period between connection
attempts.
- Local Peer
This is the representation of the local peer. The local peer is only
peer that is "active" in any running weave. Inactive peers simply
get given some state such their name, version, UID and
connections. These are set when a network update is received from a
remote peer. This state can be read directly whilst holding a read
lock regardless of whether the peer is local or not.
The local active peer spawns an actor thread which is mainly used by
the local connections. The actor thread manages state changes
relating to the local connections, identifying duplicate
connections, managing broadcasting network updates when its set of
connections (and hence version) changes.
The local peer is directly called by the router traffic sniffer and
router udp listener processes to send traffic to neighbouring
peers. This is done directly with read locks. In these methods, we
inspect the unicast and broadcast routing tables as generated by
Topology.
- Connections
The connection life cycle is as follows:
1. Either a TCP connection is created to, or received from, the remote peer
2. We spawn a thread which will eventually become the connection actor loop
3. This thread starts by doing the TCP handshake directly
4. Assuming this succeeds, a TCP receiver thread is spawned. This
simply responds to traffic received from the remote peer via TCP
5. We register this connection with the local peer
6a. If we initiated this connection then we now start sending fast
heartbeats (a new thread is spawned to do the heartbeating) to the
remote peer so that the remote peer can determine what
address/port it should use to send UDP back to us. To do this, we
also spawn off two further threads which are the forwarder
loops. These receive frames which are to be sent to the remote
peer. One of them sends frames without the DF flag set. To do this
it just sends the packets out of the UDP Listener socket. The
other needs to send frames with the DF flag set and needs its own
socket so that it can do PMTU discovery easily. To do this, it
uses a Raw IP socket (IP has no ports, so there's no collision
issue with the UDP Listener socket) and so it must add UDP headers
itself.
6b. If we did not initiate this connection then the UDP Listener
should start receiving fast heartbeats from the remote peer. From
those it should be able to identify the local connection via the
local peer. It will tell the local connection (communicating to
the actor thread) about the UDP address of the remote peer. The
local connection will then start its forwarder threads as
described in 6a, and start sending slow heartbeats (heartbeating
is done by another process). The local connection is marked as
established which means it is included in network updates
broadcast to our peers. We also send to the remote peer via TCP a
ConnectionEstablished message. The remote peer receives this (on
the TCP receiver process), tells the connection actor process,
which then replaces the fast heartbeater with a slow heartbeater
and similarly marks the connection as established.
7. Whenever a connection is established or terminated, the local
peer's version is incremented. Whenever this happens, the peer
generates a network update message which is broadcast to its
directly connected neighbours via TCP. This network update message
contains the relevant changes to the network topology due to the
connection change. When such a message is received by a TCP
receiver threads, they invoke the peers cache to apply the update
to the local model of the network topology. This may fail for a
number of reasons (for example the update may contain reference to
peers that we have no prior knowledge of. In this case, we ignore
the update and send back to the peer from whom we received the
update a request for the complete network topology), or it may
apply and elicit some changes to our model. If it does elicit some
changes then we send out an updated update message to all our
peers. In this way the changes are passed quickly from peer to
peer based on the established connections and stop being sent once
a received update causes no changes to the peer's topology
model. Changes are additive and care is taken to ensure that no
loops can occur.
Routing
In general, we build up knowledge by looking only at layer 2
(ethernet) MAC addresses. Packets that we sniff from the local
interface must have MAC address of local interfaces thus we associate
such MAC addresses with ourself. Packets that we receive via UDP from
other peers have the association embedded in the UDP traffic so that
we can add the association of the src MAC with the originating
peer. Even when a packet is relayed by an intermediate peer, we
maintain the information as to who originally sniffed the packet so
that all peers can build up the same set of associations.
If we were implementing a hub, it would be legal to just broadcast all
packets to everyone. That would work, but it would be wasteful. As
we're more implementing a switch, if we know the destination of a
packet then we will form a packet which includes the frame sniffed,
our own identity as the original sniffer of the packet and the
destination peer identity (so that we don't rely on intermediate peers
having the same knowledge as us as to which MACs are where). We then
consult the routing tables to determine which of our connections we
should use in order to try and get the packet to its ultimate
destination peer. The packet we form does not include any information
as to the route we expect the packet to take - we merely determine the
next hop and entrust that to do the onwards routing. Any intermediate
peer will receive the packet, should be able to identify the
destination peer and then similarly consult its own routing tables to
determine the next hop. The intermediate peer does not need to know in
advance the association between the destination MAC and the
destination peer. Intermediate peers do however decode the frame
sufficiently to add the association of the source peer and the source
MAC.
PMTU
PMTU is the lowest MTU of hop in the path between two different
nodes. In general, it is a benefit to know what the PMTU is so that
you can perform any necessary fragmentation of packets at the
endpoints of the path to avoid any refragmentation needing to
occur. If you allow refrafmentation to occur then you can end up with
many small packets and network performance can suffer. To discover
PMTU, there is an IP flag called "Don't Fragment" (DF). With this set,
if a node receives a packet that is bigger than the next MTU, it is
required to drop the packet and send back an ICMP 3,4 packet which
informs the sending node of the next hop MTU (RFC 1191). In theory,
these ICMP packets should have all the reverse NAT and so forth
applied, to make it all the way back to the sender. In practise this
does often work, but in some cases all ICMP are sometimes blocked,
often by idiots who don't understand that ICMP is really the error
channel of all network traffic.
Because of the encapsulation overheads, it is important that weave
respects PMTU. If it sniffs a packet of X bytes, the weave-to-weave
traffic will be some N bytes bigger than X. If the packet happens to
be an IP packet and it happens to have DF set then we should set DF on
the larger packet we send between weaves. If X+N > the PMTU between
weave peers then our send will error. We will then hopefully be able
to query what the actual PMTU is, and then can subtract N from this
and send an resultant ICMP3,4 packet back to the original sender.
It is frequently the case that large UDP packets that don't have DF
set (either the sending side chose not to set DF or the packet is
greater than the PMTU so it can't have DF set) get dropped. Ideally,
large packets without DF set should just get transparently fragmented
and reassembled and there should be no packet loss but in reality this
often doesn't occur. Therefore weave tests to see whether or not
fragmentation of large UDP packets between peers is reliable or not,
and retests from time to time.
If weave determines that fragmentation is reliable then when weave
sniffs large packets (i.e. packets larger than the MTU and thus don't
have DF set) then weave will encapsulate these packets as necessary
and similarly send them out as-is, without DF set, trusting to the
network to do all necessary fragementation and reassembly.
However, if weave determines that fragementation is not reliable
between any two peers then it will manually fragment larger packets
correctly according to the IP spec, and will then send them between
weave peers with the DF flag set (i.e. the fragmentation will ensure
that the encapsulatated traffic will not be greater than the PMTU
between weave peers). Because the fragmentation is done according to
the IP spec, we don't need to do reassembly ourselves - on the
receiving weave, we jus inject all the fragments and rely on the
stack to do reassembly.
Sometimes PMTU discovery doesn't work as ICMP packets sometimes get
dropped by firewalls. Whenever weave sends between peers with DF set
and gets an error informing it of a new PMTU, it will attempt to
verify that PMTU by sending packets of exactly that size to the remote
peer. If it gets no indication from the remote weave that these
packets have been received within a timeout period, it will
artificially drop the PMTU by 10 and try again. Thus this will slowly
reduce the PMTU until a working PMTU is found. This is slow, but it
does work and works around ICMP packets being dropped.
docs/testing.txt
+161
View File
@@ -0,0 +1,161 @@
http://baturin.org/docs/iproute2/ is a very useful cheat-sheet for the
"ip" command generally (especially the bit on namespaces).
The following assumes you have two VMs set up on your computer and
they are both on the same bridge.
From the host, the bridge (virbr0) has IP 192.168.122.1
VM1 has eth0 at 192.168.122.2
VM2 has eth0 at 192.168.122.3
All 3 can ping each other and generally contact each other. You do not
need to worry about forwarding on the host as all traffic will stay on
this bridge and not leave it.
The following work for me, but there seem to be a variety of bugs and
weirdnesses in the networking stack so the following is a bit odd...
Setting up VM1:
VM1> brctl addbr br0 # create bridge
VM1> ip netns add ns1 # create ns1
VM1> ip netns add ns2 # create ns2
VM1> ip link add name veth1 type veth peer name veth2 # create veth pair veth1/veth2
VM1> ip link add name veth3 type veth peer name veth4 # create veth pair veth3/veth4
VM1> ip link add name veth5 type veth peer name veth6 # create veth pair veth5/veth6
VM1> ip link set dev veth2 netns ns1 # move veth2 into ns1
VM1> ip link set dev veth4 netns ns2 # move veth4 into ns2
VM1> ip netns exec ns1 ip link set dev veth2 up # switch on veth2
VM1> ip netns exec ns2 ip link set dev veth4 up # switch on veth4
VM1> ip link set dev veth6 up # switch on veth6
VM1> ifconfig br0 10.0.0.1/24 up # configure and enable br0
VM1> ip netns exec ns1 ip address add 10.0.0.2/24 dev veth2 # configure veth2
VM1> ip netns exec ns2 ip address add 10.0.0.3/24 dev veth4 # configure veth4
VM1> ip address add 10.0.0.4/24 dev veth6
VM1> ip link set dev veth1 up # switch on veth1
VM1> ip link set dev veth3 up # switch on veth3
VM1> ip link set dev veth5 up # switch on veth5
VM1> brctl addif br0 veth1 # add veth1 to br0
VM1> brctl addif br0 veth3 # add veth3 to br0
VM1> brctl addif br0 veth5 # add veth5 to br0
Now we need to turn off checksum offloading on sending interfaces:
VM1> ip netns exec ns1 ethtool -K veth2 tx off
VM1> ip netns exec ns2 ethtool -K veth4 tx off
We've created 3 veth pairs, but only put an end of two of them inside
name spaces. We're not going to run weave inside a namespace. We're
instead going to run it on veth6. To avoid confusing the kernel
routing tables, we need to remove an entry it will have added:
VM1> route del -net 10.0.0.0/24 dev veth6
The kernel should (and will) use br0 as the entry point to our
10.0.0.0/24 network, not veth6.
At this point, you should be able to successfully ping 10.0.0.1,
10.0.0.2, 10.0.0.3 and 10.0.0.4 from VM1. But you can also try pinging
from inside the two namespaces (ns1 and ns2).
VM1> ping -c1 10.0.0.1
VM1> ping -c1 10.0.0.2
VM1> ping -c1 10.0.0.3
VM1> ping -c1 10.0.0.4
VM1> ip netns exec ns1 ping -c1 10.0.0.1
VM1> ip netns exec ns1 ping -c1 10.0.0.2
VM1> ip netns exec ns1 ping -c1 10.0.0.3
VM1> ip netns exec ns1 ping -c1 10.0.0.4
etc
In these VMs, we need the kernel to forward traffic between interfaces
(outside of bridges) so we need forwarding on:
VM1> echo 1 > /proc/sys/net/ipv4/ip_forward
Setting up VM2:
Pretty much the same as VM1, but we use 10.0.0.11, 10.0.0.12,
10.0.0.13 and 10.0.0.14 instead:
VM2> brctl addbr br0
VM2> ip netns add ns1
VM2> ip netns add ns2
VM2> ip link add name veth1 type veth peer name veth2
VM2> ip link add name veth3 type veth peer name veth4
VM2> ip link add name veth5 type veth peer name veth6
VM2> ip link set dev veth2 netns ns1
VM2> ip link set dev veth4 netns ns2
VM2> ip netns exec ns1 ip link set dev veth2 up
VM2> ip netns exec ns2 ip link set dev veth4 up
VM2> ip link set dev veth6 up
VM2> ifconfig br0 10.0.0.11/24 up
VM2> ip netns exec ns1 ip address add 10.0.0.12/24 dev veth2
VM2> ip netns exec ns2 ip address add 10.0.0.13/24 dev veth4
VM2> ip address add 10.0.0.14/24 dev veth6
VM2> ip link set dev veth1 up
VM2> ip link set dev veth3 up
VM2> ip link set dev veth5 up
VM2> brctl addif br0 veth1
VM2> brctl addif br0 veth3
VM2> brctl addif br0 veth5
VM2> echo 1 > /proc/sys/net/ipv4/ip_forward
VM2> ip netns exec ns1 ethtool --offload veth2 tx off
VM2> ip netns exec ns2 ethtool --offload veth4 tx off
Some final checks:
- check VM2 can't ping anything on VM1
VM2> ping 10.0.0.1 # should fail
VM2> ip netns exec ns1 ping 10.0.0.1 # should fail
...etc...
- and vice versa
We're going to have weave on VM2 connect to weave on VM1. Because
weave is not in any namespace, we don't need any iptables rules.
You should now find you can initiate a connection from VM2 to VM1 on
port 6783:
VM1> nc -l -p 6783
VM2> nc 192.168.122.2 6783
...type...
...type...
If all is well, you should now be able to launch weave on VM1
then on VM2:
VM1> weave -iface veth6 -name vm1
VM2> weave -iface veth6 -name vm2 192.168.122.1
Once the connection is up and running, you should be able to connect
from VM1/ns2 to VM2/ns2 and vice versa:
VM1> ip netns exec ns2 nc -l -p 10000
VM2> ip netns exec ns2 nc 10.0.0.3 10000
...type more...
...type more...
VM2> ip netns exec ns2 nc -l -p 10000
VM1> ip netns exec ns2 nc 10.0.0.6 10000
...type even more...
...type even more...
MTU considerations:
MTU is the payload size of an ethernet frame. The ethernet headers are
not included in this and add a further 14 bytes (an ethernet header is
really at least 22 bytes. However only 14 bytes actually carry data -
the two mac fields and a type/length field. The other parts of the
header are there for signalling at the electrical layer and thus don't
get captured by pcap, nor need forwarding). We are then passing this
over UDP which adds a further 8 bytes, and over IP, which adds at
least another 20 bytes of headers (IP can have variable header
length. Yay). We also include the namehash as a 32 byte prefix.
Therefore, we have 20(ip) + 8(udp) + 32(namehash) + 2(length prefix) +
14(ethernet) = 76 bytes of overhead. Thus we should make sure the
internal MTU is 76 bytes lower than the external MTU. So on a normal
1500 byte ethernet MTU, we should ensure the other containers use
nothing greater than 1424.