diff --git a/README.md b/README.md index 6211ee08..abd2a532 100644 --- a/README.md +++ b/README.md @@ -8,13 +8,15 @@ plugged into the same network switch, with no need to configure port mappings, links, etc. Weave can optionally encrypt traffic, allowing hosts to be connected across an untrusted network. +Services provided by application containers on the weave network can +be made accessible to the outside world, regardless of where those +containers are running. + With weave you can easily construct applications consisting of multiple containers, running anywhere. -Weave complements Docker's existing networking capabilities - use -weave to establish network connectivity *between* containers, and -Docker port mappings etc to establish network connectivity *to* -individual containers from the outside world. +Weave works alongside Docker's existing (single host) networking +capabilities, so these can continue to be used by containers. ## Installation @@ -28,8 +30,15 @@ To run weave on a host, you need to install... https://raw.githubusercontent.com/zettio/weave/master/weaver/weave sudo chmod a+x /usr/local/bin/weave -3. ethtool. On many systems that is installed already; if not then -grab it via your favourite package manager. +3. (recommended) ethtool. On many systems this is installed already; + if not then grab it via your favourite package manager. On some + systems, weave application container networking may not operate + correctly unless ethtool is available. + +4. (optional) conntrack. Install this via your favourite package + manager. Without conntrack, the weave network may not re-establish + itself fully when individual weave instances are stopped (with + `weave stop`) and restarted quickly (typically within ~3 minutes). ## Example @@ -39,8 +48,8 @@ two containers, one on each host. On $HOST1 run (as root) - host1# WEAVE=$(weave launch 10.0.0.1/16) - host1# C=$(weave run 10.0.1.1/24 -t -i debian /bin/bash) + host1# weave launch 10.0.0.1/16 + host1# C=$(weave run 10.0.1.1/24 -t -i ubuntu /bin/bash) The first line starts the weave router, in a container. This needs to be done once on each host. We tell weave that its IP address should @@ -50,28 +59,29 @@ The second line starts our application container. We give it an IP address and network (a subnet of the weave network). `weave run` invokes `docker run -d` with all the parameter following the IP address and netmask. So we could be launching any container this way; -here we just take a stock debian container and launch a shell in it. +here we just take a stock ubuntu container and launch a shell in it. If our application consists of more than one container on this host we simply launch them with a variation on that 2nd line. The IP addresses and netmasks can be anything you like which doesn't -conflict with any IP ranges of 'external' services your containers -need to connect to. The same IP range must be used everywhere, and the -individual IP addresses must, of course, be unique. +conflict with any IP ranges of 'external' services the hosts or your +containers need to connect to. The same IP range must be used +everywhere, and the individual IP addresses must, of course, be +unique. We repeat similar steps on $HOST2... - host2# WEAVE=$(weave launch 10.0.0.2/16 $HOST1) - host2# C=$(weave run 10.0.1.2/24 -t -i debian /bin/bash) + host2# weave launch 10.0.0.2/16 $HOST1 + host2# C=$(weave run 10.0.1.2/24 -t -i ubuntu /bin/bash) The only difference, apart from the IP addresses, is that we tell our weave that it should peer with the weave running on $HOST1. We could instead have told the weave on $HOST1 to connect to $HOST2, or told both about each other. Order doesn't matter here; weave automatically -(re)connects to peers when they are become available. Also, we can -tell weave to connect to multiple peers by supplying multiple -addresses. +(re)connects to peers when they become available. Also, we can tell +weave to connect to multiple peers by supplying multiple +addresses. And we can [add peers dynamically](#dynamic-topologies). Now that we've got everything set up, let's see whether our containers can talk to each other... @@ -108,15 +118,21 @@ To application containers, the network established by weave looks like a giant Ethernet switch to which all the containers are connected. -That means *any* protocol is supported. Doesn't even have to be over -TCP/IP, e.g. in the above example, +Containers can easily access services from each other, e.g. in the +container on $HOST1 we can start a netcat "service" with - root@28841bd02eff:/# nc -lu -p 4422 + root@28841bd02eff:/# nc -lk -p 4422 - root@f76829496120:/# echo 'Hello, world.' | nc -u 10.0.1.1 4422 +and then connect to it from the container on $HOST2 with -sends some data from the 2nd container to the first over a UDP -connection on port 4422. + root@f76829496120:/# echo 'Hello, world.' | nc 10.0.1.1 4422 + +Note that *any* protocol is supported. Doesn't even have to be over +TCP/IP, e.g. a netcat UDP service would be run with + + root@28841bd02eff:/# nc -lu -p 5533 + + root@f76829496120:/# echo 'Hello, world.' | nc -u 10.0.1.1 5533 We can deploy the entire arsenal of standard network tools and applications, developed over decades, to configure, secure, monitor, @@ -135,8 +151,8 @@ To accomplish that, we assign each application a different subnet. So, in the above example, if we wanted to add another application similar to, but isolated from, our first, we'd launch the containers with... - host1# D=$(weave run 10.0.2.1/24 -t -i debian /bin/bash) - host2# D=$(weave run 10.0.2.2/24 -t -i debian /bin/bash) + host1# D=$(weave run 10.0.2.1/24 -t -i ubuntu /bin/bash) + host2# D=$(weave run 10.0.2.2/24 -t -i ubuntu /bin/bash) A quick 'ping' test in the containers confirms that they can talk to each other but not the containers of our first application... @@ -165,12 +181,109 @@ In order to connect containers across untrusted networks, weave peers can be told to encrypt traffic by supplying a `-password` option when launching weave, e.g. - host1# WEAVE=$(weave launch 10.0.0.1/16 -password wEaVe) + host1# weave launch 10.0.0.1/16 -password wEaVe -The same password must be specified for all weave peers; it is the a +The same password must be specified for all weave peers; it is a component in the creation of ephemeral session keys for connections between peers. +### Host network integration + +Weave application networks can be integrated with a host's network, +establishing connectivity between the host and application containers +anywhere. + +Let's say that in our example we we want $HOST2 to have access to the +application containers. On $HOST2 we run + + host2# weave expose 10.0.1.102/24 + +choosing an unused IP address in the application subnet. (There is a +corresponding 'hide' command to revert this step.) + +Now + + host2# ping 10.0.1.2 + +will work. And, more interestingly, + + host2# ping 10.0.1.1 + +will work too, which is talking to a container that resides on $HOST1. + +### Service export + +Services running in containers on a weave network can be made +accessible to the outside world (and, more generally, other networks) +from any weave host, irrespective of where the service containers are +located. + +Say we want to make our example netcat "service", which is running in +a container on $HOST1, accessible to the outside world via $HOST2. + +First we need to expose the application network to $HOST2, as +explained [above](#host-network-integration), i.e. + + host2# weave expose 10.0.1.102/24 + +Then we add a NAT rule to route from the outside world to the +destination container service. + + host2# iptables -t nat -A PREROUTING -p tcp -i eth0 --dport 2211 \ + -j DNAT --to-destination 10.0.1.1:4422 + +Here we are assuming that the "outside world" is connecting to $HOST2 +via 'eth0'. We want TCP traffic to port 2211 on the external IPs to be +routed to our 'nc' service, which is running on port 4422 in the +container with IP 10.0.1.1. + +With the above in place, we can connect to our 'nc' service from +anywhere with + + echo 'Hello, world.' | nc $HOST2 2211 + +(NB: due to the way routing is handled in the Linux kernel, this won't +work when run *on* $HOST2.) + +Similar NAT rules to the above can used to expose services not just to +outside world but also other, internal, networks. + +### Service import + +Applications running in containers on a weave network can be given +access to services which are only reachable from certain weave hosts, +irrespective of where the application containers are located. + +Say that, as an extension of our example, we have a netcat service +running on $HOST3, port 2211, and that $HOST3 is not part of the weave +network and is only reachable from $HOST1, not $HOST2. Nevertheless we +want to make the service accessible to an application running in a +container on $HOST2. + +First we need to expose the application network to the host, as +explained [above](#host-network-integration), this time on $HOST1, +i.e. + + host1# weave expose 10.0.1.101/24 + +Then we add a NAT rule to route from the above IP to the destination +service. + + host1# iptables -t nat -A PREROUTING -p tcp -d 10.0.1.101 --dport 3322 \ + -j DNAT --to-destination $HOST3:2211 + +This allows any application container to reach the service by +connecting to 10.0.0.1:3322. So if $HOST3 is indeed running a netcat +service on port 2211, e.g. + + host3# nc -kl 2211 + +then we can connect to it from our application container on $HOST2 with + + root@f76829496120:/# echo 'Hello, world.' | nc 10.0.1.101 3322 + +The same command will work from any application container. + ### Multi-cloud networking Weave can network containers hosted in different cloud providers / @@ -218,17 +331,17 @@ the partition heals. The weave container is very light-weight - just over 8MB image size and a few 10s of MBs of runtime memory - and disposable. I.e. should -weave ever run into difficulty, one can simply bounce the weave -container. Application containers do *not* have to be restarted in -that event, and indeed may not even experience a temporary -connectivity failure if the weave container is restarted quickly -enough. +weave ever run into difficulty, one can simply stop it (with `weave +stop`) and restart it. Application containers do *not* have to be +restarted in that event, and indeed may not even experience a +temporary connectivity failure if the weave container is restarted +quickly enough. ## Troubleshooting Check the weave container logs with - docker logs $WEAVE + docker logs weave A reasonable amount of information, and all errors, get logged there. @@ -239,7 +352,7 @@ basis, so can produce a lot of output. One can ask a weave router to log a status report by sending it a USR1 signal, e.g. - kill -USR1 `docker inspect --format='{{ .State.Pid }}' $WEAVE` + kill -USR1 `docker inspect --format='{{ .State.Pid }}' weave` and then inspect the logs. @@ -259,11 +372,11 @@ make it executable like this: Then, if we were trying to create the same containers as in the first example above, the 'launch' command would be run like this: - host1$ WEAVE=$(boot2docker ssh "sudo ./weave launch 10.0.0.1/16") + host1$ boot2docker ssh "sudo ./weave launch 10.0.0.1/16" and the 'run' command like this: - host1# C=$(boot2docker ssh "sudo ./weave run 10.0.1.1/24 -t -i debian /bin/bash") + host1# C=$(boot2docker ssh "sudo ./weave run 10.0.1.1/24 -t -i ubuntu /bin/bash") ## Building @@ -279,3 +392,12 @@ Then simply run This will build the weave router, produce a docker image `zettio/weave` and export that image to /tmp/weave.tar + +If you are interested in the inner workings of weave, read the +[architecture documentation](docs/architecture.txt). + +## Contact Us + +Found a bug, want to suggest a feature, or have a question? +[File an issue](https://github.com/zettio/weave/issues), or email us +at weave@zett.io. diff --git a/connection.go b/connection.go index caa85061..0eb1bd63 100644 --- a/connection.go +++ b/connection.go @@ -99,26 +99,12 @@ func (conn *LocalConnection) CheckFatal(err error) error { func (conn *LocalConnection) setEffectivePMTU(pmtu int) { conn.Lock() defer conn.Unlock() - conn.effectivePMTU = pmtu - conn.pmtuVerified = false - conn.log("Effective PMTU set to", pmtu) -} - -func (conn *LocalConnection) effectivePMTUVerification(pmtu int) { - conn.Lock() - defer conn.Unlock() - if pmtu == conn.effectivePMTU && !conn.pmtuVerified { - conn.pmtuVerified = true - conn.log("Effective PMTU verified at", pmtu) + if (conn.effectivePMTU != pmtu) { + conn.effectivePMTU = pmtu + conn.log("Effective PMTU set to", pmtu) } } -func (conn *LocalConnection) isEffectivePMTUVerfied() bool { - conn.RLock() - defer conn.RUnlock() - return conn.pmtuVerified -} - // Called by the connection's actor process, by the connection's // heartbeat process and by the connection's TCP received // process. StackFrag is read in conn.Forward (called by router udp @@ -506,7 +492,7 @@ func (conn *LocalConnection) receiveTCP(decoder *gob.Decoder, usingPassword bool } conn.local.BroadcastTCP(Concat(ProtocolUpdateByte, newUpdate)) } else if msg[0] == ProtocolPMTUVerified { - conn.effectivePMTUVerification(int(binary.BigEndian.Uint16(msg[1:]))) + conn.verifyPMTU <- int(binary.BigEndian.Uint16(msg[1:])) } else { conn.log("received unknown msg:\n", msg) } diff --git a/consts.go b/consts.go index 30704df6..e90a27ad 100644 --- a/consts.go +++ b/consts.go @@ -5,23 +5,24 @@ import ( ) const ( - Protocol = "weave" - ProtocolVersion = 8 - EthernetOverhead = 14 - UDPOverhead = 28 // 20 bytes for IPv4, 8 bytes for UDP - Port = 6783 - StatusPort = Port + 1 - DefaultPMTU = 65535 - MaxUDPPacketSize = 65536 - ChannelSize = 16 - UDPNonceSendAt = 8192 - FragTestSize = 60001 - PMTUDiscoverySize = 60000 - FastHeartbeat = 500 * time.Millisecond - SlowHeartbeat = 10 * time.Second - FetchAllInterval = 30 * time.Second - FragTestInterval = 5 * time.Minute - PMTUVerifyTimeout = 5 * time.Second + Protocol = "weave" + ProtocolVersion = 8 + EthernetOverhead = 14 + UDPOverhead = 28 // 20 bytes for IPv4, 8 bytes for UDP + Port = 6783 + StatusPort = Port + 1 + DefaultPMTU = 65535 + MaxUDPPacketSize = 65536 + ChannelSize = 16 + UDPNonceSendAt = 8192 + FragTestSize = 60001 + PMTUDiscoverySize = 60000 + FastHeartbeat = 500 * time.Millisecond + SlowHeartbeat = 10 * time.Second + FetchAllInterval = 30 * time.Second + FragTestInterval = 5 * time.Minute + PMTUVerifyAttempts = 8 + PMTUVerifyTimeout = 10 * time.Millisecond // gets doubled with every attempt ) const ( diff --git a/docs/architecture.txt b/docs/architecture.txt index 5cfb4a73..79917b5f 100644 --- a/docs/architecture.txt +++ b/docs/architecture.txt @@ -203,7 +203,6 @@ dropped by firewalls. Whenever weave sends between peers with DF set and gets an error informing it of a new PMTU, it will attempt to verify that PMTU by sending packets of exactly that size to the remote peer. If it gets no indication from the remote weave that these -packets have been received within a timeout period, it will -artificially drop the PMTU by 8 and try again. Thus this will slowly -reduce the PMTU until a working PMTU is found. This is slow, but it -does work and works around ICMP packets being dropped. +packets have been received within a timeout period, it will conduct a +binary search - sending packets of different sizes - to determine the +PMTU. diff --git a/forwarder.go b/forwarder.go index 7a15a25b..74c168b8 100644 --- a/forwarder.go +++ b/forwarder.go @@ -34,18 +34,22 @@ func (conn *LocalConnection) ensureForwarders() error { forwardChanDF = make(chan *ForwardedFrame, ChannelSize) stopForward = make(chan interface{}, 0) stopForwardDF = make(chan interface{}, 0) + verifyPMTU = make(chan int, ChannelSize) ) - forwarder := NewForwarder(conn, forwardChan, stopForward, encryptor, udpSender, DefaultPMTU) - forwarderDF := NewForwarder(conn, forwardChanDF, stopForwardDF, encryptorDF, udpSenderDF, DefaultPMTU) + //NB: only forwarderDF can ever encounter EMSGSIZE errors, and + //thus perform PMTU verification + forwarder := NewForwarder(conn, forwardChan, stopForward, nil, encryptor, udpSender, DefaultPMTU) + forwarderDF := NewForwarder(conn, forwardChanDF, stopForwardDF, verifyPMTU, encryptorDF, udpSenderDF, DefaultPMTU) - // The forward chans and effectivePMTU in the conn struct are read - // by other processes, so we have to use locks. + // Various fields in the conn struct are read by other processes, + // so we have to use locks. conn.Lock() conn.forwardChan = forwardChan conn.forwardChanDF = forwardChanDF conn.stopForward = stopForward conn.stopForwardDF = stopForwardDF - conn.effectivePMTU = forwarder.effectivePMTU + conn.verifyPMTU = verifyPMTU + conn.effectivePMTU = forwarder.unverifiedPMTU conn.Unlock() go forwarder.Run() @@ -179,15 +183,17 @@ func fragment(eth layers.Ethernet, ip layers.IPv4, pmtu int, frame *ForwardedFra // Forwarder -func NewForwarder(conn *LocalConnection, ch <-chan *ForwardedFrame, stop <-chan interface{}, enc Encryptor, udpSender UDPSender, pmtu int) *Forwarder { - forwarder := &Forwarder{ - conn: conn, - ch: ch, - stop: stop, - enc: enc, - udpSender: udpSender} - forwarder.setMaxPayloadAndEffectivePMTU(pmtu) - return forwarder +func NewForwarder(conn *LocalConnection, ch <-chan *ForwardedFrame, stop <-chan interface{}, verifyPMTU <-chan int, enc Encryptor, udpSender UDPSender, pmtu int) *Forwarder { + fwd := &Forwarder{ + conn: conn, + ch: ch, + stop: stop, + verifyPMTU: verifyPMTU, + enc: enc, + udpSender: udpSender} + fwd.unverifiedPMTU = pmtu - fwd.effectiveOverhead() + fwd.maxPayload = pmtu - UDPOverhead + return fwd } func (fwd *Forwarder) Run() { @@ -200,15 +206,35 @@ func (fwd *Forwarder) Run() { case <-fwd.stop: fwd.drain() return - case <-fwd.pmtuVerifyTick: + case <-fwd.verifyPMTUTick: // We only do this case here when we know the buffers are // all empty so that we don't risk appending verify-frames // to other data. - fwd.pmtuVerifyTick = nil - if !fwd.conn.isEffectivePMTUVerfied() { - fwd.effectivePMTU -= 8 - fwd.maxPayload -= 8 - fwd.updateEffectivePMTU() + fwd.verifyPMTUTick = nil + if fwd.pmtuVerified { + continue + } + if fwd.pmtuVerifyCount > 0 { + fwd.pmtuVerifyCount-- + fwd.attemptVerifyEffectivePMTU() + } else { + // we've exceeded the verification attempts of the + // unverifiedPMTU + fwd.lowestBadPMTU = fwd.unverifiedPMTU + fwd.verifyEffectivePMTU((fwd.highestGoodPMTU + fwd.lowestBadPMTU) / 2) + } + case epmtu := <-fwd.verifyPMTU: + if fwd.pmtuVerified || epmtu != fwd.unverifiedPMTU { + continue + } + if epmtu+1 < fwd.lowestBadPMTU { + fwd.highestGoodPMTU = fwd.unverifiedPMTU // = epmtu + fwd.verifyEffectivePMTU((fwd.highestGoodPMTU + fwd.lowestBadPMTU) / 2) + } else { + fwd.pmtuVerified = true + fwd.maxPayload = epmtu + fwd.effectiveOverhead() - UDPOverhead + fwd.conn.setEffectivePMTU(epmtu) + fwd.conn.log("Effective PMTU verified at", epmtu) } case frame = <-fwd.ch: if !fwd.appendFrame(frame) { @@ -237,32 +263,26 @@ func (fwd *Forwarder) Run() { } } -func (fwd *Forwarder) setMaxPayloadAndEffectivePMTU(pmtu int) { - fwd.maxPayload = pmtu - UDPOverhead - fwd.effectivePMTU = fwd.maxPayload - fwd.enc.PacketOverhead() - fwd.enc.FrameOverhead() - EthernetOverhead +func (fwd *Forwarder) effectiveOverhead() int { + return UDPOverhead + fwd.enc.PacketOverhead() + fwd.enc.FrameOverhead() + EthernetOverhead } -func (fwd *Forwarder) updateEffectivePMTU() { - for gotNewPMTU := true; gotNewPMTU; { - fwd.conn.setEffectivePMTU(fwd.effectivePMTU) - pmtuVerifyFrame := &ForwardedFrame{ - srcPeer: fwd.conn.local, - dstPeer: fwd.conn.remote, - frame: make([]byte, fwd.effectivePMTU+EthernetOverhead)} - gotNewPMTU = false - for cnt := 0; cnt < 10; cnt += 1 { - fwd.enc.AppendFrame(pmtuVerifyFrame) - err := fwd.udpSender.Send(fwd.enc.Bytes()) - if err != nil { - if mtbe, ok := err.(MsgTooBigError); ok { - fwd.setMaxPayloadAndEffectivePMTU(mtbe.PMTU) - gotNewPMTU = true - break - } - } - } +func (fwd *Forwarder) verifyEffectivePMTU(newUnverifiedPMTU int) { + fwd.unverifiedPMTU = newUnverifiedPMTU + fwd.pmtuVerifyCount = PMTUVerifyAttempts + fwd.attemptVerifyEffectivePMTU() +} + +func (fwd *Forwarder) attemptVerifyEffectivePMTU() { + pmtuVerifyFrame := &ForwardedFrame{ + srcPeer: fwd.conn.local, + dstPeer: fwd.conn.remote, + frame: make([]byte, fwd.unverifiedPMTU+EthernetOverhead)} + fwd.enc.AppendFrame(pmtuVerifyFrame) + fwd.flush() + if fwd.verifyPMTUTick == nil { + fwd.verifyPMTUTick = time.After(PMTUVerifyTimeout << (PMTUVerifyAttempts - fwd.pmtuVerifyCount)) } - fwd.pmtuVerifyTick = time.After(PMTUVerifyTimeout) } func (fwd *Forwarder) appendFrame(frame *ForwardedFrame) bool { @@ -278,8 +298,16 @@ func (fwd *Forwarder) flush() { err := fwd.udpSender.Send(fwd.enc.Bytes()) if err != nil { if mtbe, ok := err.(MsgTooBigError); ok { - fwd.setMaxPayloadAndEffectivePMTU(mtbe.PMTU) - fwd.updateEffectivePMTU() + newUnverifiedPMTU := mtbe.PMTU - fwd.effectiveOverhead() + if newUnverifiedPMTU >= fwd.unverifiedPMTU { + return + } + fwd.pmtuVerified = false + fwd.maxPayload = mtbe.PMTU - UDPOverhead + fwd.highestGoodPMTU = 8 + fwd.lowestBadPMTU = newUnverifiedPMTU + 1 + fwd.conn.setEffectivePMTU(newUnverifiedPMTU) + fwd.verifyEffectivePMTU(newUnverifiedPMTU) } else if PosixError(err) == syscall.ENOBUFS { // TODO handle this better } else { @@ -302,7 +330,7 @@ func (fwd *Forwarder) drain() { } func (fwd *Forwarder) logDrop(frame *ForwardedFrame) { - fwd.conn.log("dropping too big frame during forwarding: frame len:", len(frame.frame), "; effective PMTU:", fwd.effectivePMTU) + fwd.conn.log("Dropping too big frame during forwarding: frame len:", len(frame.frame), "; effective PMTU:", fwd.maxPayload+UDPOverhead-fwd.effectiveOverhead()) } // UDP Senders diff --git a/mac_cache.go b/mac_cache.go index a387c136..c08e04c4 100644 --- a/mac_cache.go +++ b/mac_cache.go @@ -18,12 +18,14 @@ type MacCache struct { table map[uint64]*MacCacheEntry maxAge time.Duration expiryTimer *time.Timer + onExpiry func(net.HardwareAddr, *Peer) } -func NewMacCache(maxAge time.Duration) *MacCache { +func NewMacCache(maxAge time.Duration, onExpiry func(net.HardwareAddr, *Peer)) *MacCache { res := &MacCache{ - table: make(map[uint64]*MacCacheEntry), - maxAge: maxAge} + table: make(map[uint64]*MacCacheEntry), + maxAge: maxAge, + onExpiry: onExpiry} res.setExpiryTimer() return res } @@ -102,6 +104,7 @@ func (cache *MacCache) expire() { for key, entry := range cache.table { if now.After(entry.lastSeen.Add(cache.maxAge)) { delete(cache.table, key) + cache.onExpiry(intmac(key), entry.peer) } } cache.setExpiryTimer() diff --git a/peer_cache.go b/peer_cache.go index a6c29f4e..fc0c2b2a 100644 --- a/peer_cache.go +++ b/peer_cache.go @@ -5,17 +5,17 @@ import ( "encoding/gob" "fmt" "io" - "log" "sync" ) type PeerCache struct { sync.RWMutex table map[PeerName]*Peer + onGC func(*Peer) } -func NewPeerCache() *PeerCache { - return &PeerCache{table: make(map[PeerName]*Peer)} +func NewPeerCache(onGC func(*Peer)) *PeerCache { + return &PeerCache{table: make(map[PeerName]*Peer), onGC: onGC} } func (cache *PeerCache) FetchWithDefault(peer *Peer) *Peer { @@ -129,7 +129,7 @@ func (cache *PeerCache) garbageCollect(ourself *Peer) []*Peer { for name, peer := range cache.table { found, _ := ourself.HasPathTo(peer, false) if !found && !peer.IsLocallyReferenced() { - log.Println("Removing unreachable", peer) + cache.onGC(peer) delete(cache.table, name) ourself.Router.Macs.Delete(peer) removed = append(removed, peer) diff --git a/router.go b/router.go index 17d395dc..39a7bac6 100644 --- a/router.go +++ b/router.go @@ -15,10 +15,16 @@ import ( const macMaxAge = 10 * time.Minute func NewRouter(iface *net.Interface, name PeerName, password []byte, connLimit int, bufSz int, logFrame func(string, []byte, *layers.Ethernet)) *Router { + onMacExpiry := func(mac net.HardwareAddr, peer *Peer) { + log.Println("Expired MAC", mac, "at", peer.Name) + } + onPeerGC := func(peer *Peer) { + log.Println("Removing unreachable", peer) + } router := &Router{ Iface: iface, - Macs: NewMacCache(macMaxAge), - Peers: NewPeerCache(), + Macs: NewMacCache(macMaxAge, onMacExpiry), + Peers: NewPeerCache(onPeerGC), ConnLimit: connLimit, BufSz: bufSz, LogFrame: logFrame} @@ -274,11 +280,12 @@ func (router *Router) handleUDPPacketFunc(dec *EthernetDecoder, po PacketSink) F } router.LogFrame("Injecting", frame, &dec.eth) checkWarn(po.WritePacket(frame)) + dstPeer, found = router.Macs.Lookup(dec.eth.DstMAC) if !found || dec.BroadcastFrame() || dstPeer != router.Ourself { - df := decodedLen == 2 && (dec.ip.Flags&layers.IPv4DontFragment != 0) - checkFrameTooBig(router.Ourself.RelayBroadcast(srcPeer, df, frame, dec), srcPeer) + return checkFrameTooBig(router.Ourself.RelayBroadcast(srcPeer, df, frame, dec), srcPeer) } + return nil } } diff --git a/types.go b/types.go index cf675a21..020fba18 100644 --- a/types.go +++ b/types.go @@ -66,13 +66,13 @@ type LocalConnection struct { established bool stackFrag bool effectivePMTU int - pmtuVerified bool SessionKey *[32]byte heartbeatStop chan<- interface{} forwardChan chan<- *ForwardedFrame forwardChanDF chan<- *ForwardedFrame stopForward chan<- interface{} stopForwardDF chan<- interface{} + verifyPMTU chan<- int Decryptor Decryptor Router *Router UID uint64 @@ -86,14 +86,19 @@ type ConnectionInteraction struct { } type Forwarder struct { - conn *LocalConnection - ch <-chan *ForwardedFrame - stop <-chan interface{} - pmtuVerifyTick <-chan time.Time - enc Encryptor - udpSender UDPSender - maxPayload int - effectivePMTU int + conn *LocalConnection + ch <-chan *ForwardedFrame + stop <-chan interface{} + verifyPMTUTick <-chan time.Time + verifyPMTU <-chan int + pmtuVerifyCount uint + enc Encryptor + udpSender UDPSender + maxPayload int + pmtuVerified bool + highestGoodPMTU int + unverifiedPMTU int + lowestBadPMTU int } type UDPPacket struct { diff --git a/weaver/main.go b/weaver/main.go index 92758700..b9cd3670 100644 --- a/weaver/main.go +++ b/weaver/main.go @@ -47,7 +47,13 @@ func main() { log.SetPrefix(weave.Protocol + " ") log.SetFlags(log.Ldate | log.Ltime | log.Lmicroseconds) - runtime.GOMAXPROCS(runtime.NumCPU()) + procs := runtime.NumCPU() + // packet sniffing can block an OS thread, so we need one thread + // for that plus at least one more. + if procs < 2 { + procs = 2 + } + runtime.GOMAXPROCS(procs) var ( ifaceName string diff --git a/weaver/weave b/weaver/weave index 6ed51262..47a3196d 100755 --- a/weaver/weave +++ b/weaver/weave @@ -1,18 +1,13 @@ #!/bin/sh set -e -BRIDGE=weave -CONTAINER_IFNAME=ethwe -MTU=65535 - -SCRIPT_DIR=$(dirname $(readlink -f $0)) -COMMAND=$1 -IPADDR=$2 - usage() { echo "Usage:" echo "weave launch / [-passwd ] ..." echo "weave run / ..." + echo "weave expose /" + echo "weave hide /" + echo "weave stop" echo "weave status" exit 1 } @@ -23,41 +18,64 @@ command_exists () { command -v $1 >/dev/null 2>&1 } -if [ "$COMMAND" != "status" -a "$#" -lt 2 ]; then - usage - exit 1 -fi - [ `id -u` = 0 ] || { echo "weave must be run as 'root'" exit 1 } -# pre-create bridge so we can set the MTU -[ ! -d /sys/class/net/$BRIDGE ] && { - ip link add dev $BRIDGE type bridge - ip link set dev $BRIDGE address 7a:$(od -txC -An -N5 /dev/random | tr \ : | tail -c+2) - # attempting to set the bridge MTU to a high value directly - # fails. Bridges take the lowest MTU of their interfaces. So - # instead we create a temporary interface with the desired MTU, - # attach that to the bridge, and then remove it again. - ip link add name v${CONTAINER_IFNAME}du mtu $MTU type dummy - ip link set dev v${CONTAINER_IFNAME}du master $BRIDGE - ip link del dev v${CONTAINER_IFNAME}du +[ $# -gt 0 ] || usage + +BRIDGE=weave +CONTAINER_IFNAME=ethwe +MTU=65535 + +COMMAND=$1 + +shift 1 + +create_bridge() { + [ ! -d /sys/class/net/$BRIDGE ] && { + ip link add dev $BRIDGE type bridge + ip link set dev $BRIDGE address 7a:$(od -txC -An -N5 /dev/random | tr \ : | tail -c+2) + # Attempting to set the bridge MTU to a high value directly + # fails. Bridges take the lowest MTU of their interfaces. So + # instead we create a temporary interface with the desired + # MTU, attach that to the bridge, and then remove it again. + ip link add name v${CONTAINER_IFNAME}du mtu $MTU type dummy + ip link set dev v${CONTAINER_IFNAME}du master $BRIDGE + ip link del dev v${CONTAINER_IFNAME}du + } ip link set dev $BRIDGE up } -# We set the router name to the bridge mac since that is stable across -# re-creations of the containers. -# -# TODO use the mac of one of the physical host interfaces (eth0, -# wlan0, etc) so the name becomes stable across host restarts. -MACADDR=`cat /sys/class/net/$BRIDGE/address` +add_nat_rule() { + if ! iptables -w -t nat -C $1 >/dev/null 2>&1 + then + iptables -w -t nat -A $1 >/dev/null + fi +} + +setup_masquerading() { + add_nat_rule "POSTROUTING -o $BRIDGE ! -s $IPADDR -j MASQUERADE" + add_nat_rule "POSTROUTING -s $IPADDR ! -o $BRIDGE -j MASQUERADE" +} case "$COMMAND" in launch) - shift 2 + [ $# -gt 0 ] || usage + IPADDR=$1 + shift 1 + create_bridge + setup_masquerading + # We set the router name to the bridge mac since that is + # stable across re-creations of the containers. + # + # TODO use the mac of one of the physical host interfaces + # (eth0, wlan0, etc) so the name becomes stable across host + # restarts. + MACADDR=`cat /sys/class/net/$BRIDGE/address` CONTAINER=$(docker run --privileged -d --name=weave -p 6783:6783/tcp -p 6783:6783/udp $WEAVE_DOCKER_ARGS zettio/weave -name $MACADDR "$@") + echo $CONTAINER IFACE=eth0 ;; status) @@ -74,10 +92,45 @@ case "$COMMAND" in exit $? ;; run) - shift 2 + [ $# -gt 0 ] || usage + IPADDR=$1 + shift 1 + create_bridge CONTAINER=$(docker run -d "$@") + echo $CONTAINER IFACE=$CONTAINER_IFNAME ;; + expose) + [ $# -gt 0 ] || usage + IPADDR=$1 + shift 1 + if ! ip addr list dev $BRIDGE | grep -qF $IPADDR + then + ip addr add dev $BRIDGE $IPADDR + fi + exit 0 + ;; + hide) + [ $# -gt 0 ] || usage + IPADDR=$1 + shift 1 + if ip addr list dev $BRIDGE | grep -qF $IPADDR + then + ip addr del dev $BRIDGE $IPADDR + fi + exit 0 + ;; + stop) + docker kill weave > /dev/null + docker rm weave > /dev/null + if command -v conntrack >/dev/null 2>&1 + then + conntrack -D -p udp --dport 6783 >/dev/null 2>&1 + else + echo "Cannot find conntrack; please install it. Continuing without it." >&2 + fi + exit 0 + ;; *) echo "Unknown weave command '$COMMAND'" usage @@ -116,9 +169,7 @@ if command_exists ethtool then ip netns exec $NSPID ethtool -K $IFACE tx off > /dev/null else - echo "Cannot find ethtool; continuing without it" >&2 + echo "Cannot find ethtool; please install it. Continuing without it." >&2 fi [ -f /var/run/netns/$NSPID ] && rm -f /var/run/netns/$NSPID - -echo $CONTAINER