Merge remote-tracking branch 'upstream/master' into 11_http_interface

Conflicts:
	consts.go
	weaver/weave

README.md

@@ -8,13 +8,15 @@ plugged into the same network switch, with no need to configure port
 mappings, links, etc. Weave can optionally encrypt traffic, allowing
 hosts to be connected across an untrusted network.
 
+Services provided by application containers on the weave network can
+be made accessible to the outside world, regardless of where those
+containers are running.
+
 With weave you can easily construct applications consisting of
 multiple containers, running anywhere.
 
-Weave complements Docker's existing networking capabilities - use
-weave to establish network connectivity *between* containers, and
-Docker port mappings etc to establish network connectivity *to*
-individual containers from the outside world.
+Weave works alongside Docker's existing (single host) networking
+capabilities, so these can continue to be used by containers.
 
 ## Installation
 
@@ -28,8 +30,15 @@ To run weave on a host, you need to install...
           https://raw.githubusercontent.com/zettio/weave/master/weaver/weave
         sudo chmod a+x /usr/local/bin/weave
 
-3. ethtool. On many systems that is installed already; if not then
-grab it via your favourite package manager.
+3. (recommended) ethtool. On many systems this is installed already;
+   if not then grab it via your favourite package manager. On some
+   systems, weave application container networking may not operate
+   correctly unless ethtool is available.
+
+4. (optional) conntrack. Install this via your favourite package
+   manager. Without conntrack, the weave network may not re-establish
+   itself fully when individual weave instances are stopped (with
+   `weave stop`) and restarted quickly (typically within ~3 minutes).
 
 ## Example
 
@@ -39,8 +48,8 @@ two containers, one on each host.
 
 On $HOST1 run (as root)
 
-    host1# WEAVE=$(weave launch 10.0.0.1/16)
-    host1# C=$(weave run 10.0.1.1/24 -t -i debian /bin/bash)
+    host1# weave launch 10.0.0.1/16
+    host1# C=$(weave run 10.0.1.1/24 -t -i ubuntu /bin/bash)
 
 The first line starts the weave router, in a container. This needs to
 be done once on each host. We tell weave that its IP address should
@@ -50,28 +59,29 @@ The second line starts our application container. We give it an IP
 address and network (a subnet of the weave network). `weave run`
 invokes `docker run -d` with all the parameter following the IP
 address and netmask. So we could be launching any container this way;
-here we just take a stock debian container and launch a shell in it.
+here we just take a stock ubuntu container and launch a shell in it.
 
 If our application consists of more than one container on this host we
 simply launch them with a variation on that 2nd line.
 
 The IP addresses and netmasks can be anything you like which doesn't
-conflict with any IP ranges of 'external' services your containers
-need to connect to. The same IP range must be used everywhere, and the
-individual IP addresses must, of course, be unique.
+conflict with any IP ranges of 'external' services the hosts or your
+containers need to connect to. The same IP range must be used
+everywhere, and the individual IP addresses must, of course, be
+unique.
 
 We repeat similar steps on $HOST2...
 
-    host2# WEAVE=$(weave launch 10.0.0.2/16 $HOST1)
-    host2# C=$(weave run 10.0.1.2/24 -t -i debian /bin/bash)
+    host2# weave launch 10.0.0.2/16 $HOST1
+    host2# C=$(weave run 10.0.1.2/24 -t -i ubuntu /bin/bash)
 
 The only difference, apart from the IP addresses, is that we tell our
 weave that it should peer with the weave running on $HOST1. We could
 instead have told the weave on $HOST1 to connect to $HOST2, or told
 both about each other. Order doesn't matter here; weave automatically
-(re)connects to peers when they are become available. Also, we can
-tell weave to connect to multiple peers by supplying multiple
-addresses.
+(re)connects to peers when they become available. Also, we can tell
+weave to connect to multiple peers by supplying multiple
+addresses. And we can [add peers dynamically](#dynamic-topologies).
 
 Now that we've got everything set up, let's see whether our containers
 can talk to each other...
@@ -108,15 +118,21 @@ To application containers, the network established by weave looks
 like a giant Ethernet switch to which all the containers are
 connected.
 
-That means *any* protocol is supported. Doesn't even have to be over
-TCP/IP, e.g. in the above example,
+Containers can easily access services from each other, e.g. in the
+container on $HOST1 we can start a netcat "service" with
 
-    root@28841bd02eff:/# nc -lu -p 4422
+    root@28841bd02eff:/# nc -lk -p 4422
 
-    root@f76829496120:/# echo 'Hello, world.' | nc -u 10.0.1.1 4422
+and then connect to it from the container on $HOST2 with
 
-sends some data from the 2nd container to the first over a UDP
-connection on port 4422.
+    root@f76829496120:/# echo 'Hello, world.' | nc 10.0.1.1 4422
+
+Note that *any* protocol is supported. Doesn't even have to be over
+TCP/IP, e.g. a netcat UDP service would be run with
+
+    root@28841bd02eff:/# nc -lu -p 5533
+
+    root@f76829496120:/# echo 'Hello, world.' | nc -u 10.0.1.1 5533
 
 We can deploy the entire arsenal of standard network tools and
 applications, developed over decades, to configure, secure, monitor,
@@ -135,8 +151,8 @@ To accomplish that, we assign each application a different subnet. So,
 in the above example, if we wanted to add another application similar
 to, but isolated from, our first, we'd launch the containers with...
 
-    host1# D=$(weave run 10.0.2.1/24 -t -i debian /bin/bash)
-    host2# D=$(weave run 10.0.2.2/24 -t -i debian /bin/bash)
+    host1# D=$(weave run 10.0.2.1/24 -t -i ubuntu /bin/bash)
+    host2# D=$(weave run 10.0.2.2/24 -t -i ubuntu /bin/bash)
 
 A quick 'ping' test in the containers confirms that they can talk to
 each other but not the containers of our first application...
@@ -165,12 +181,109 @@ In order to connect containers across untrusted networks, weave peers
 can be told to encrypt traffic by supplying a `-password` option when
 launching weave, e.g.
 
-    host1# WEAVE=$(weave launch 10.0.0.1/16 -password wEaVe)
+    host1# weave launch 10.0.0.1/16 -password wEaVe
 
-The same password must be specified for all weave peers; it is the a
+The same password must be specified for all weave peers; it is a
 component in the creation of ephemeral session keys for connections
 between peers.
 
+### Host network integration
+
+Weave application networks can be integrated with a host's network,
+establishing connectivity between the host and application containers
+anywhere.
+
+Let's say that in our example we we want $HOST2 to have access to the
+application containers. On $HOST2 we run
+
+    host2# weave expose 10.0.1.102/24
+
+choosing an unused IP address in the application subnet. (There is a
+corresponding 'hide' command to revert this step.)
+
+Now
+
+    host2# ping 10.0.1.2
+
+will work. And, more interestingly,
+
+    host2# ping 10.0.1.1
+
+will work too, which is talking to a container that resides on $HOST1.
+
+### Service export
+
+Services running in containers on a weave network can be made
+accessible to the outside world (and, more generally, other networks)
+from any weave host, irrespective of where the service containers are
+located.
+
+Say we want to make our example netcat "service", which is running in
+a container on $HOST1, accessible to the outside world via $HOST2.
+
+First we need to expose the application network to $HOST2, as
+explained [above](#host-network-integration), i.e.
+
+    host2# weave expose 10.0.1.102/24
+
+Then we add a NAT rule to route from the outside world to the
+destination container service.
+
+    host2# iptables -t nat -A PREROUTING -p tcp -i eth0 --dport 2211 \
+           -j DNAT --to-destination 10.0.1.1:4422
+
+Here we are assuming that the "outside world" is connecting to $HOST2
+via 'eth0'. We want TCP traffic to port 2211 on the external IPs to be
+routed to our 'nc' service, which is running on port 4422 in the
+container with IP 10.0.1.1.
+
+With the above in place, we can connect to our 'nc' service from
+anywhere with
+
+    echo 'Hello, world.' | nc $HOST2 2211
+
+(NB: due to the way routing is handled in the Linux kernel, this won't
+work when run *on* $HOST2.)
+
+Similar NAT rules to the above can used to expose services not just to
+outside world but also other, internal, networks.
+
+### Service import
+
+Applications running in containers on a weave network can be given
+access to services which are only reachable from certain weave hosts,
+irrespective of where the application containers are located.
+
+Say that, as an extension of our example, we have a netcat service
+running on $HOST3, port 2211, and that $HOST3 is not part of the weave
+network and is only reachable from $HOST1, not $HOST2. Nevertheless we
+want to make the service accessible to an application running in a
+container on $HOST2.
+
+First we need to expose the application network to the host, as
+explained [above](#host-network-integration), this time on $HOST1,
+i.e.
+
+    host1# weave expose 10.0.1.101/24
+
+Then we add a NAT rule to route from the above IP to the destination
+service.
+
+    host1# iptables -t nat -A PREROUTING -p tcp -d 10.0.1.101 --dport 3322 \
+           -j DNAT --to-destination $HOST3:2211
+
+This allows any application container to reach the service by
+connecting to 10.0.0.1:3322. So if $HOST3 is indeed running a netcat
+service on port 2211, e.g.
+
+    host3# nc -kl 2211
+
+then we can connect to it from our application container on $HOST2 with
+
+    root@f76829496120:/# echo 'Hello, world.' | nc 10.0.1.101 3322
+
+The same command will work from any application container.
+
 ### Multi-cloud networking
 
 Weave can network containers hosted in different cloud providers /
@@ -218,17 +331,17 @@ the partition heals.
 
 The weave container is very light-weight - just over 8MB image size
 and a few 10s of MBs of runtime memory - and disposable. I.e. should
-weave ever run into difficulty, one can simply bounce the weave
-container. Application containers do *not* have to be restarted in
-that event, and indeed may not even experience a temporary
-connectivity failure if the weave container is restarted quickly
-enough.
+weave ever run into difficulty, one can simply stop it (with `weave
+stop`) and restart it. Application containers do *not* have to be
+restarted in that event, and indeed may not even experience a
+temporary connectivity failure if the weave container is restarted
+quickly enough.
 
 ## Troubleshooting
 
 Check the weave container logs with
 
-    docker logs $WEAVE
+    docker logs weave
 
 A reasonable amount of information, and all errors, get logged there.
 
@@ -239,7 +352,7 @@ basis, so can produce a lot of output.
 One can ask a weave router to log a status report by sending it a USR1
 signal, e.g.
 
-    kill -USR1 `docker inspect --format='{{ .State.Pid }}' $WEAVE`
+    kill -USR1 `docker inspect --format='{{ .State.Pid }}' weave`
 
 and then inspect the logs.
 
@@ -259,11 +372,11 @@ make it executable like this:
 Then, if we were trying to create the same containers as in the first
 example above, the 'launch' command would be run like this:
 
-    host1$ WEAVE=$(boot2docker ssh "sudo ./weave launch 10.0.0.1/16")
+    host1$ boot2docker ssh "sudo ./weave launch 10.0.0.1/16"
 
 and the 'run' command like this:
 
-    host1# C=$(boot2docker ssh "sudo ./weave run 10.0.1.1/24 -t -i debian /bin/bash")
+    host1# C=$(boot2docker ssh "sudo ./weave run 10.0.1.1/24 -t -i ubuntu /bin/bash")
 
 ## Building
 
@@ -279,3 +392,12 @@ Then simply run
 
 This will build the weave router, produce a docker image
 `zettio/weave` and export that image to /tmp/weave.tar
+
+If you are interested in the inner workings of weave, read the
+[architecture documentation](docs/architecture.txt).
+
+## Contact Us
+
+Found a bug, want to suggest a feature, or have a question?
+[File an issue](https://github.com/zettio/weave/issues), or email us
+at weave@zett.io.

connection.go

@@ -99,26 +99,12 @@ func (conn *LocalConnection) CheckFatal(err error) error {
 func (conn *LocalConnection) setEffectivePMTU(pmtu int) {
 	conn.Lock()
 	defer conn.Unlock()
-	conn.effectivePMTU = pmtu
-	conn.pmtuVerified = false
-	conn.log("Effective PMTU set to", pmtu)
-}
-
-func (conn *LocalConnection) effectivePMTUVerification(pmtu int) {
-	conn.Lock()
-	defer conn.Unlock()
-	if pmtu == conn.effectivePMTU && !conn.pmtuVerified {
-		conn.pmtuVerified = true
-		conn.log("Effective PMTU verified at", pmtu)
+	if (conn.effectivePMTU != pmtu) {
+		conn.effectivePMTU = pmtu
+		conn.log("Effective PMTU set to", pmtu)
 	}
 }
 
-func (conn *LocalConnection) isEffectivePMTUVerfied() bool {
-	conn.RLock()
-	defer conn.RUnlock()
-	return conn.pmtuVerified
-}
-
 // Called by the connection's actor process, by the connection's
 // heartbeat process and by the connection's TCP received
 // process. StackFrag is read in conn.Forward (called by router udp
@@ -506,7 +492,7 @@ func (conn *LocalConnection) receiveTCP(decoder *gob.Decoder, usingPassword bool
 			}
 			conn.local.BroadcastTCP(Concat(ProtocolUpdateByte, newUpdate))
 		} else if msg[0] == ProtocolPMTUVerified {
-			conn.effectivePMTUVerification(int(binary.BigEndian.Uint16(msg[1:])))
+			conn.verifyPMTU <- int(binary.BigEndian.Uint16(msg[1:]))
 		} else {
 			conn.log("received unknown msg:\n", msg)
 		}

consts.go

@@ -5,23 +5,24 @@ import (
 )
 
 const (
-	Protocol          = "weave"
-	ProtocolVersion   = 8
-	EthernetOverhead  = 14
-	UDPOverhead       = 28 // 20 bytes for IPv4, 8 bytes for UDP
-	Port              = 6783
-	StatusPort        = Port + 1
-	DefaultPMTU       = 65535
-	MaxUDPPacketSize  = 65536
-	ChannelSize       = 16
-	UDPNonceSendAt    = 8192
-	FragTestSize      = 60001
-	PMTUDiscoverySize = 60000
-	FastHeartbeat     = 500 * time.Millisecond
-	SlowHeartbeat     = 10 * time.Second
-	FetchAllInterval  = 30 * time.Second
-	FragTestInterval  = 5 * time.Minute
-	PMTUVerifyTimeout = 5 * time.Second
+	Protocol           = "weave"
+	ProtocolVersion    = 8
+	EthernetOverhead   = 14
+	UDPOverhead        = 28 // 20 bytes for IPv4, 8 bytes for UDP
+	Port               = 6783
+	StatusPort         = Port + 1
+	DefaultPMTU        = 65535
+	MaxUDPPacketSize   = 65536
+	ChannelSize        = 16
+	UDPNonceSendAt     = 8192
+	FragTestSize       = 60001
+	PMTUDiscoverySize  = 60000
+	FastHeartbeat      = 500 * time.Millisecond
+	SlowHeartbeat      = 10 * time.Second
+	FetchAllInterval   = 30 * time.Second
+	FragTestInterval   = 5 * time.Minute
+	PMTUVerifyAttempts = 8
+	PMTUVerifyTimeout  = 10 * time.Millisecond // gets doubled with every attempt
 )
 
 const (

docs/architecture.txt

@@ -203,7 +203,6 @@ dropped by firewalls. Whenever weave sends between peers with DF set
 and gets an error informing it of a new PMTU, it will attempt to
 verify that PMTU by sending packets of exactly that size to the remote
 peer. If it gets no indication from the remote weave that these
-packets have been received within a timeout period, it will
-artificially drop the PMTU by 8 and try again. Thus this will slowly
-reduce the PMTU until a working PMTU is found. This is slow, but it
-does work and works around ICMP packets being dropped.
+packets have been received within a timeout period, it will conduct a
+binary search - sending packets of different sizes - to determine the
+PMTU.

forwarder.go

@@ -34,18 +34,22 @@ func (conn *LocalConnection) ensureForwarders() error {
 		forwardChanDF = make(chan *ForwardedFrame, ChannelSize)
 		stopForward   = make(chan interface{}, 0)
 		stopForwardDF = make(chan interface{}, 0)
+		verifyPMTU    = make(chan int, ChannelSize)
 	)
-	forwarder := NewForwarder(conn, forwardChan, stopForward, encryptor, udpSender, DefaultPMTU)
-	forwarderDF := NewForwarder(conn, forwardChanDF, stopForwardDF, encryptorDF, udpSenderDF, DefaultPMTU)
+	//NB: only forwarderDF can ever encounter EMSGSIZE errors, and
+	//thus perform PMTU verification
+	forwarder := NewForwarder(conn, forwardChan, stopForward, nil, encryptor, udpSender, DefaultPMTU)
+	forwarderDF := NewForwarder(conn, forwardChanDF, stopForwardDF, verifyPMTU, encryptorDF, udpSenderDF, DefaultPMTU)
 
-	// The forward chans and effectivePMTU in the conn struct are read
-	// by other processes, so we have to use locks.
+	// Various fields in the conn struct are read by other processes,
+	// so we have to use locks.
 	conn.Lock()
 	conn.forwardChan = forwardChan
 	conn.forwardChanDF = forwardChanDF
 	conn.stopForward = stopForward
 	conn.stopForwardDF = stopForwardDF
-	conn.effectivePMTU = forwarder.effectivePMTU
+	conn.verifyPMTU = verifyPMTU
+	conn.effectivePMTU = forwarder.unverifiedPMTU
 	conn.Unlock()
 
 	go forwarder.Run()
@@ -179,15 +183,17 @@ func fragment(eth layers.Ethernet, ip layers.IPv4, pmtu int, frame *ForwardedFra
 
 // Forwarder
 
-func NewForwarder(conn *LocalConnection, ch <-chan *ForwardedFrame, stop <-chan interface{}, enc Encryptor, udpSender UDPSender, pmtu int) *Forwarder {
-	forwarder := &Forwarder{
-		conn:      conn,
-		ch:        ch,
-		stop:      stop,
-		enc:       enc,
-		udpSender: udpSender}
-	forwarder.setMaxPayloadAndEffectivePMTU(pmtu)
-	return forwarder
+func NewForwarder(conn *LocalConnection, ch <-chan *ForwardedFrame, stop <-chan interface{}, verifyPMTU <-chan int, enc Encryptor, udpSender UDPSender, pmtu int) *Forwarder {
+	fwd := &Forwarder{
+		conn:       conn,
+		ch:         ch,
+		stop:       stop,
+		verifyPMTU: verifyPMTU,
+		enc:        enc,
+		udpSender:  udpSender}
+	fwd.unverifiedPMTU = pmtu - fwd.effectiveOverhead()
+	fwd.maxPayload = pmtu - UDPOverhead
+	return fwd
 }
 
 func (fwd *Forwarder) Run() {
@@ -200,15 +206,35 @@ func (fwd *Forwarder) Run() {
 		case <-fwd.stop:
 			fwd.drain()
 			return
-		case <-fwd.pmtuVerifyTick:
+		case <-fwd.verifyPMTUTick:
 			// We only do this case here when we know the buffers are
 			// all empty so that we don't risk appending verify-frames
 			// to other data.
-			fwd.pmtuVerifyTick = nil
-			if !fwd.conn.isEffectivePMTUVerfied() {
-				fwd.effectivePMTU -= 8
-				fwd.maxPayload -= 8
-				fwd.updateEffectivePMTU()
+			fwd.verifyPMTUTick = nil
+			if fwd.pmtuVerified {
+				continue
+			}
+			if fwd.pmtuVerifyCount > 0 {
+				fwd.pmtuVerifyCount--
+				fwd.attemptVerifyEffectivePMTU()
+			} else {
+				// we've exceeded the verification attempts of the
+				// unverifiedPMTU
+				fwd.lowestBadPMTU = fwd.unverifiedPMTU
+				fwd.verifyEffectivePMTU((fwd.highestGoodPMTU + fwd.lowestBadPMTU) / 2)
+			}
+		case epmtu := <-fwd.verifyPMTU:
+			if fwd.pmtuVerified || epmtu != fwd.unverifiedPMTU {
+				continue
+			}
+			if epmtu+1 < fwd.lowestBadPMTU {
+				fwd.highestGoodPMTU = fwd.unverifiedPMTU // = epmtu
+				fwd.verifyEffectivePMTU((fwd.highestGoodPMTU + fwd.lowestBadPMTU) / 2)
+			} else {
+				fwd.pmtuVerified = true
+				fwd.maxPayload = epmtu + fwd.effectiveOverhead() - UDPOverhead
+				fwd.conn.setEffectivePMTU(epmtu)
+				fwd.conn.log("Effective PMTU verified at", epmtu)
 			}
 		case frame = <-fwd.ch:
 			if !fwd.appendFrame(frame) {
@@ -237,32 +263,26 @@ func (fwd *Forwarder) Run() {
 	}
 }
 
-func (fwd *Forwarder) setMaxPayloadAndEffectivePMTU(pmtu int) {
-	fwd.maxPayload = pmtu - UDPOverhead
-	fwd.effectivePMTU = fwd.maxPayload - fwd.enc.PacketOverhead() - fwd.enc.FrameOverhead() - EthernetOverhead
+func (fwd *Forwarder) effectiveOverhead() int {
+	return UDPOverhead + fwd.enc.PacketOverhead() + fwd.enc.FrameOverhead() + EthernetOverhead
 }
 
-func (fwd *Forwarder) updateEffectivePMTU() {
-	for gotNewPMTU := true; gotNewPMTU; {
-		fwd.conn.setEffectivePMTU(fwd.effectivePMTU)
-		pmtuVerifyFrame := &ForwardedFrame{
-			srcPeer: fwd.conn.local,
-			dstPeer: fwd.conn.remote,
-			frame:   make([]byte, fwd.effectivePMTU+EthernetOverhead)}
-		gotNewPMTU = false
-		for cnt := 0; cnt < 10; cnt += 1 {
-			fwd.enc.AppendFrame(pmtuVerifyFrame)
-			err := fwd.udpSender.Send(fwd.enc.Bytes())
-			if err != nil {
-				if mtbe, ok := err.(MsgTooBigError); ok {
-					fwd.setMaxPayloadAndEffectivePMTU(mtbe.PMTU)
-					gotNewPMTU = true
-					break
-				}
-			}
-		}
+func (fwd *Forwarder) verifyEffectivePMTU(newUnverifiedPMTU int) {
+	fwd.unverifiedPMTU = newUnverifiedPMTU
+	fwd.pmtuVerifyCount = PMTUVerifyAttempts
+	fwd.attemptVerifyEffectivePMTU()
+}
+
+func (fwd *Forwarder) attemptVerifyEffectivePMTU() {
+	pmtuVerifyFrame := &ForwardedFrame{
+		srcPeer: fwd.conn.local,
+		dstPeer: fwd.conn.remote,
+		frame:   make([]byte, fwd.unverifiedPMTU+EthernetOverhead)}
+	fwd.enc.AppendFrame(pmtuVerifyFrame)
+	fwd.flush()
+	if fwd.verifyPMTUTick == nil {
+		fwd.verifyPMTUTick = time.After(PMTUVerifyTimeout << (PMTUVerifyAttempts - fwd.pmtuVerifyCount))
 	}
-	fwd.pmtuVerifyTick = time.After(PMTUVerifyTimeout)
 }
 
 func (fwd *Forwarder) appendFrame(frame *ForwardedFrame) bool {
@@ -278,8 +298,16 @@ func (fwd *Forwarder) flush() {
 	err := fwd.udpSender.Send(fwd.enc.Bytes())
 	if err != nil {
 		if mtbe, ok := err.(MsgTooBigError); ok {
-			fwd.setMaxPayloadAndEffectivePMTU(mtbe.PMTU)
-			fwd.updateEffectivePMTU()
+			newUnverifiedPMTU := mtbe.PMTU - fwd.effectiveOverhead()
+			if newUnverifiedPMTU >= fwd.unverifiedPMTU {
+				return
+			}
+			fwd.pmtuVerified = false
+			fwd.maxPayload = mtbe.PMTU - UDPOverhead
+			fwd.highestGoodPMTU = 8
+			fwd.lowestBadPMTU = newUnverifiedPMTU + 1
+			fwd.conn.setEffectivePMTU(newUnverifiedPMTU)
+			fwd.verifyEffectivePMTU(newUnverifiedPMTU)
 		} else if PosixError(err) == syscall.ENOBUFS {
 			// TODO handle this better
 		} else {
@@ -302,7 +330,7 @@ func (fwd *Forwarder) drain() {
 }
 
 func (fwd *Forwarder) logDrop(frame *ForwardedFrame) {
-	fwd.conn.log("dropping too big frame during forwarding: frame len:", len(frame.frame), "; effective PMTU:", fwd.effectivePMTU)
+	fwd.conn.log("Dropping too big frame during forwarding: frame len:", len(frame.frame), "; effective PMTU:", fwd.maxPayload+UDPOverhead-fwd.effectiveOverhead())
 }
 
 // UDP Senders

mac_cache.go

@@ -18,12 +18,14 @@ type MacCache struct {
 	table       map[uint64]*MacCacheEntry
 	maxAge      time.Duration
 	expiryTimer *time.Timer
+	onExpiry    func(net.HardwareAddr, *Peer)
 }
 
-func NewMacCache(maxAge time.Duration) *MacCache {
+func NewMacCache(maxAge time.Duration, onExpiry func(net.HardwareAddr, *Peer)) *MacCache {
 	res := &MacCache{
-		table:  make(map[uint64]*MacCacheEntry),
-		maxAge: maxAge}
+		table:    make(map[uint64]*MacCacheEntry),
+		maxAge:   maxAge,
+		onExpiry: onExpiry}
 	res.setExpiryTimer()
 	return res
 }
@@ -102,6 +104,7 @@ func (cache *MacCache) expire() {
 	for key, entry := range cache.table {
 		if now.After(entry.lastSeen.Add(cache.maxAge)) {
 			delete(cache.table, key)
+			cache.onExpiry(intmac(key), entry.peer)
 		}
 	}
 	cache.setExpiryTimer()

peer_cache.go

@@ -5,17 +5,17 @@ import (
 	"encoding/gob"
 	"fmt"
 	"io"
-	"log"
 	"sync"
 )
 
 type PeerCache struct {
 	sync.RWMutex
 	table map[PeerName]*Peer
+	onGC func(*Peer)
 }
 
-func NewPeerCache() *PeerCache {
-	return &PeerCache{table: make(map[PeerName]*Peer)}
+func NewPeerCache(onGC func(*Peer)) *PeerCache {
+	return &PeerCache{table: make(map[PeerName]*Peer), onGC: onGC}
 }
 
 func (cache *PeerCache) FetchWithDefault(peer *Peer) *Peer {
@@ -129,7 +129,7 @@ func (cache *PeerCache) garbageCollect(ourself *Peer) []*Peer {
 	for name, peer := range cache.table {
 		found, _ := ourself.HasPathTo(peer, false)
 		if !found && !peer.IsLocallyReferenced() {
-			log.Println("Removing unreachable", peer)
+			cache.onGC(peer)
 			delete(cache.table, name)
 			ourself.Router.Macs.Delete(peer)
 			removed = append(removed, peer)

router.go

@@ -15,10 +15,16 @@ import (
 const macMaxAge = 10 * time.Minute
 
 func NewRouter(iface *net.Interface, name PeerName, password []byte, connLimit int, bufSz int, logFrame func(string, []byte, *layers.Ethernet)) *Router {
+	onMacExpiry := func(mac net.HardwareAddr, peer *Peer) {
+		log.Println("Expired MAC", mac, "at", peer.Name)
+	}
+	onPeerGC := func(peer *Peer) {
+		log.Println("Removing unreachable", peer)
+	}
 	router := &Router{
 		Iface:     iface,
-		Macs:      NewMacCache(macMaxAge),
-		Peers:     NewPeerCache(),
+		Macs:      NewMacCache(macMaxAge, onMacExpiry),
+		Peers:     NewPeerCache(onPeerGC),
 		ConnLimit: connLimit,
 		BufSz:     bufSz,
 		LogFrame:  logFrame}
@@ -274,11 +280,12 @@ func (router *Router) handleUDPPacketFunc(dec *EthernetDecoder, po PacketSink) F
 		}
 		router.LogFrame("Injecting", frame, &dec.eth)
 		checkWarn(po.WritePacket(frame))
+
 		dstPeer, found = router.Macs.Lookup(dec.eth.DstMAC)
 		if !found || dec.BroadcastFrame() || dstPeer != router.Ourself {
-			df := decodedLen == 2 && (dec.ip.Flags&layers.IPv4DontFragment != 0)
-			checkFrameTooBig(router.Ourself.RelayBroadcast(srcPeer, df, frame, dec), srcPeer)
+			return checkFrameTooBig(router.Ourself.RelayBroadcast(srcPeer, df, frame, dec), srcPeer)
 		}
+
 		return nil
 	}
 }

types.go

@@ -66,13 +66,13 @@ type LocalConnection struct {
 	established   bool
 	stackFrag     bool
 	effectivePMTU int
-	pmtuVerified  bool
 	SessionKey    *[32]byte
 	heartbeatStop chan<- interface{}
 	forwardChan   chan<- *ForwardedFrame
 	forwardChanDF chan<- *ForwardedFrame
 	stopForward   chan<- interface{}
 	stopForwardDF chan<- interface{}
+	verifyPMTU    chan<- int
 	Decryptor     Decryptor
 	Router        *Router
 	UID           uint64
@@ -86,14 +86,19 @@ type ConnectionInteraction struct {
 }
 
 type Forwarder struct {
-	conn           *LocalConnection
-	ch             <-chan *ForwardedFrame
-	stop           <-chan interface{}
-	pmtuVerifyTick <-chan time.Time
-	enc            Encryptor
-	udpSender      UDPSender
-	maxPayload     int
-	effectivePMTU  int
+	conn            *LocalConnection
+	ch              <-chan *ForwardedFrame
+	stop            <-chan interface{}
+	verifyPMTUTick  <-chan time.Time
+	verifyPMTU      <-chan int
+	pmtuVerifyCount uint
+	enc             Encryptor
+	udpSender       UDPSender
+	maxPayload      int
+	pmtuVerified    bool
+	highestGoodPMTU int
+	unverifiedPMTU  int
+	lowestBadPMTU   int
 }
 
 type UDPPacket struct {

weaver/main.go

@@ -47,7 +47,13 @@ func main() {
 	log.SetPrefix(weave.Protocol + " ")
 	log.SetFlags(log.Ldate | log.Ltime | log.Lmicroseconds)
 
-	runtime.GOMAXPROCS(runtime.NumCPU())
+	procs := runtime.NumCPU()
+	// packet sniffing can block an OS thread, so we need one thread
+	// for that plus at least one more.
+	if procs < 2 {
+		procs = 2
+	}
+	runtime.GOMAXPROCS(procs)
 
 	var (
 		ifaceName  string

weaver/weave

@@ -1,18 +1,13 @@
 #!/bin/sh
 set -e
 
-BRIDGE=weave
-CONTAINER_IFNAME=ethwe
-MTU=65535
-
-SCRIPT_DIR=$(dirname $(readlink -f $0))
-COMMAND=$1
-IPADDR=$2
-
 usage() {
     echo "Usage:"
     echo "weave launch <ipaddr>/<subnet> [-passwd <passwd>] <peer_host> ..."
     echo "weave run    <ipaddr>/<subnet> <docker run args> ..."
+    echo "weave expose <ipaddr>/<subnet>"
+    echo "weave hide   <ipaddr>/<subnet>"
+    echo "weave stop"
     echo "weave status"
     exit 1
 }
@@ -23,41 +18,64 @@ command_exists () {
     command -v $1 >/dev/null 2>&1
 }
 
-if [ "$COMMAND" != "status" -a "$#" -lt 2 ]; then
-    usage
-    exit 1
-fi
-
 [ `id -u` = 0 ] || {
     echo "weave must be run as 'root'"
     exit 1
 }
 
-# pre-create bridge so we can set the MTU
-[ ! -d /sys/class/net/$BRIDGE ] && {
-    ip link add dev $BRIDGE type bridge
-    ip link set dev $BRIDGE address 7a:$(od -txC -An -N5 /dev/random | tr \  : | tail -c+2)
-    # attempting to set the bridge MTU to a high value directly
-    # fails. Bridges take the lowest MTU of their interfaces. So
-    # instead we create a temporary interface with the desired MTU,
-    # attach that to the bridge, and then remove it again.
-    ip link add name v${CONTAINER_IFNAME}du mtu $MTU type dummy
-    ip link set dev v${CONTAINER_IFNAME}du master $BRIDGE
-    ip link del dev v${CONTAINER_IFNAME}du
+[ $# -gt 0 ] || usage
+
+BRIDGE=weave
+CONTAINER_IFNAME=ethwe
+MTU=65535
+
+COMMAND=$1
+
+shift 1
+
+create_bridge() {
+    [ ! -d /sys/class/net/$BRIDGE ] && {
+        ip link add dev $BRIDGE type bridge
+        ip link set dev $BRIDGE address 7a:$(od -txC -An -N5 /dev/random | tr \  : | tail -c+2)
+        # Attempting to set the bridge MTU to a high value directly
+        # fails. Bridges take the lowest MTU of their interfaces. So
+        # instead we create a temporary interface with the desired
+        # MTU, attach that to the bridge, and then remove it again.
+        ip link add name v${CONTAINER_IFNAME}du mtu $MTU type dummy
+        ip link set dev v${CONTAINER_IFNAME}du master $BRIDGE
+        ip link del dev v${CONTAINER_IFNAME}du
+    }
     ip link set dev $BRIDGE up
 }
 
-# We set the router name to the bridge mac since that is stable across
-# re-creations of the containers.
-#
-# TODO use the mac of one of the physical host interfaces (eth0,
-# wlan0, etc) so the name becomes stable across host restarts.
-MACADDR=`cat /sys/class/net/$BRIDGE/address`
+add_nat_rule() {
+    if ! iptables -w -t nat -C $1 >/dev/null 2>&1
+    then
+        iptables -w -t nat -A $1 >/dev/null
+    fi
+}
+
+setup_masquerading() {
+    add_nat_rule "POSTROUTING -o $BRIDGE ! -s $IPADDR -j MASQUERADE"
+    add_nat_rule "POSTROUTING -s $IPADDR ! -o $BRIDGE -j MASQUERADE"
+}
 
 case "$COMMAND" in
     launch)
-        shift 2
+        [ $# -gt 0 ] || usage
+        IPADDR=$1
+        shift 1
+        create_bridge
+        setup_masquerading
+        # We set the router name to the bridge mac since that is
+        # stable across re-creations of the containers.
+        #
+        # TODO use the mac of one of the physical host interfaces
+        # (eth0, wlan0, etc) so the name becomes stable across host
+        # restarts.
+        MACADDR=`cat /sys/class/net/$BRIDGE/address`
         CONTAINER=$(docker run --privileged -d --name=weave -p 6783:6783/tcp -p 6783:6783/udp $WEAVE_DOCKER_ARGS zettio/weave -name $MACADDR "$@")
+        echo $CONTAINER
         IFACE=eth0
         ;;
     status)
@@ -74,10 +92,45 @@ case "$COMMAND" in
         exit $?
         ;;
     run)
-        shift 2
+        [ $# -gt 0 ] || usage
+        IPADDR=$1
+        shift 1
+        create_bridge
         CONTAINER=$(docker run -d "$@")
+        echo $CONTAINER
         IFACE=$CONTAINER_IFNAME
         ;;
+    expose)
+        [ $# -gt 0 ] || usage
+        IPADDR=$1
+        shift 1
+        if ! ip addr list dev $BRIDGE | grep -qF $IPADDR
+        then
+            ip addr add dev $BRIDGE $IPADDR
+        fi
+        exit 0
+        ;;
+    hide)
+        [ $# -gt 0 ] || usage
+        IPADDR=$1
+        shift 1
+        if ip addr list dev $BRIDGE | grep -qF $IPADDR
+        then
+            ip addr del dev $BRIDGE $IPADDR
+        fi
+        exit 0
+        ;;
+    stop)
+        docker kill weave > /dev/null
+        docker rm weave > /dev/null
+        if command -v conntrack >/dev/null 2>&1
+        then
+            conntrack -D -p udp --dport 6783 >/dev/null 2>&1
+        else
+            echo "Cannot find conntrack; please install it. Continuing without it." >&2
+        fi
+        exit 0
+        ;;
     *)
         echo "Unknown weave command '$COMMAND'"
         usage
@@ -116,9 +169,7 @@ if command_exists ethtool
 then
     ip netns exec $NSPID ethtool -K $IFACE tx off > /dev/null
 else
-    echo "Cannot find ethtool; continuing without it" >&2
+    echo "Cannot find ethtool; please install it. Continuing without it." >&2
 fi
 
 [ -f /var/run/netns/$NSPID ] && rm -f /var/run/netns/$NSPID
-
-echo $CONTAINER