Add release 0.11.2 to CHANGELOG

Unit tests

.gitignore

@@ -6,6 +6,7 @@
 # Folders
 _obj
 _test
+.vagrant
 
 # Architecture specific extensions/prefixes
 *.[568vq]
@@ -22,6 +23,8 @@ _testmain.go
 *.exe
 *.test
 *.prof
+profile.cov
+coverage.html
 
 # Emacs backup files
 *~
@@ -29,5 +32,18 @@ _testmain.go
 # Project specific
 weaver/weaver
 weavedns/weavedns
+weaveproxy/weaveproxy
+sigproxy/sigproxy
+weavewait/weavewait
 tools/bin
+tools/build
 releases
+weave.tar
+weavedns.tar
+weaveexec.tar
+weaveexec/weave
+weaveexec/weaveproxy
+weaveexec/weavewait
+weaveexec/sigproxy
+weaveexec/docker*.tgz
+Vagrantfile.local

.travis.yml

@@ -0,0 +1,28 @@
+language: go
+sudo: false
+go:
+  - 1.4
+
+env:
+  - PATH="${PATH}:${HOME}/.local/bin"
+
+addons:
+  apt:
+    packages:
+    - libpcap-dev
+
+install:
+  - go clean -i net
+  - go install -tags netgo std
+  - pip install --user --upgrade gcloud gsutil
+  - bin/setup-ci-secrets $encrypted_5ba036e89377_key $encrypted_5ba036e89377_iv
+  - make travis
+
+script:
+  - bin/lint .
+  - make tests
+
+after_success:
+  - go get github.com/mattn/goveralls
+  - $HOME/gopath/bin/goveralls -coverprofile=profile.cov -service=travis-ci
+  - test "${TRAVIS_PULL_REQUEST}" = "false" && site/build-and-publish.sh

CHANGELOG.md

@@ -1,3 +1,121 @@
+## Release 0.11.2
+
+This is a bug fix release, addressing the following issues:
+
+- `weave run` did not respect DOCKER_CLIENT_ARGS. #855
+- Negative result cache did not expire if requeried within TTL. #845
+
+More details in the [change
+log](https://github.com/weaveworks/weave/issues?q=milestone%3A0.11.2).
+
+*NB: This release does not change the weave protocol version.
+Therefore, when upgrading an existing 0.11 installation incrementally,
+connectivity between peers will be retained.*
+
+## Release 0.11.1
+
+This is a bug fix release, addressing the following issues:
+
+- The IP Allocator could crash in some relatively rare
+  circumstances. #782/#783.
+- When the proxy failed to attach a container to the weave network,
+  there was no failure indication and descriptive error anywhere, and
+  the application process would still start. Now an error is reported
+  to the client (i.e. typically the Docker CLI), recorded in the proxy
+  logs, and the container is terminated. #788/#799.
+- `weave launch-proxy --with-ipam` failed to configure the entrypoint
+  and DNS unless a (possibly blank) `WEAVE_CIDR` was
+  specified. Affected containers could start the application process
+  w/o the weave network interface being available, and without
+  functioning name resolution for the weaveDNS domain.
+  #744/#747/#751/#752
+- The `weave status` output for the IP Allocator was misleadingly
+  conveying a sense of brokenness when no IP allocation requests had
+  been made yet. #787/#801
+- When invoking `weave launch-proxy` twice, the second invocation
+  would output a blank line and terminate with a success exit
+  status. Now it reports that the proxy is already running and exits
+  with a non-zero status. #767/#780
+- `weave launch-proxy` was not respecting WEAVEPROXY_DOCKER_ARGS, so
+  any user-supplied custom configuration for the weaveproxy container
+  was ignored. #755/#780
+- The proxy was not intercepting API calls to the unversioned (1.0)
+  Docker Remote API. Hence none of the weave functionality was
+  available when Docker clients were using that version of the
+  API. #770/#774/#777,#809
+- The proxy would crash when certain elements of the
+  `/containers/create` JSON were missing. We are not aware of any
+  Docker clients doing this, but it's been fixed regardless. #778/#777
+
+More details in the [change
+log](https://github.com/weaveworks/weave/issues?q=milestone%3A0.11.1).
+
+*NB: This release does not change the weave protocol version.
+Therefore, when upgrading an existing 0.11 installation incrementally,
+connectivity between peers will be retained.*
+
+## Release 0.11.0
+
+**Highlights**:
+
+- **automatic IP Address Management (IPAM)**, which allows application
+  containers to be started and attached to the weave network without
+  needing to supply an IP address.
+- **proxy** for automatically attaching containers started with
+  ordinary `docker run`, or the Docker remote API, to the weave
+  network.
+- ability to **add/remove extra DNS records**.
+- performance and scalability improvements
+- fixes for a small number of bugs discovered during testing
+
+More detail in the [change log](https://github.com/weaveworks/weave/issues?q=milestone%3A0.11.0)
+
+*NB: This release changes the weave protocol version. Therefore, when
+upgrading an existing installation, all hosts need to be upgraded in
+order to for them to be able to communicate and form a network.*
+
+## Release 0.10.0
+
+**Highlights**:
+
+- **bug fixes**, in particular eradicating several deadlocks that could
+  cause peer connection failures and hangs.
+- **performance and scalability improvements** to the weave control plane,
+  as a result of which it is now possible to construct much larger
+  weave networks than previously.
+- **improved installation and administration**, particularly the
+  introduction of remote execution of the weave script in a container,
+  permitting fully containerised deployment of weave.
+- **improved diagnostics**, such as the reporting of connection failures
+  in `weave status`.
+- **new weaveDNS features**, most notably the caching of DNS records
+  for application containers, which makes finding container IP
+  addresses via weaveDNS much faster.
+
+More detail in the [change log](https://github.com/weaveworks/weave/issues?q=milestone%3A0.10.0)
+
+*NB: This release changes the Weave Docker image names. To upgrade
+from an older version, 1) stop all application containers, 2) run
+`weave reset` from the old version to remove all traces of weave, and
+only then 3) install the new version.*
+
+## Release 0.9.0
+
+- Improve WeaveDNS to the point where it can act as the name server
+  for containers in nearly all situations.
+
+- Diagnose and report peer connectivity more comprehensively.
+
+- Adapt to changes in topology - adding & removing of weave peers,
+  disruption of connectivity - more rapidly.
+
+- Cope with delays in downloading/running docker images/containers
+  required for weave operation.
+
+See the
+[complete change log](https://github.com/weaveworks/weave/issues?q=milestone%3A0.9.0)
+for more details.
+
 ## Release 0.8.0
 
 - Align script and image version. When the `weave` script has a
@@ -22,9 +140,9 @@ This is the first release assigned a version number.
 
 When downloading weave you now have the following choices...
 
-1. a specific version, e.g. https://github.com/zettio/weave/releases/download/v0.7.0/weave
-2. latest released version: https://github.com/zettio/weave/releases/download/latest_release/weave
-3. most recent 'master' commit: https://raw.githubusercontent.com/zettio/weave/master/weave
+1. a specific version, e.g. https://github.com/weaveworks/weave/releases/download/v0.7.0/weave
+2. latest released version: https://github.com/weaveworks/weave/releases/download/latest_release/weave
+3. most recent 'master' commit: https://raw.githubusercontent.com/weaveworks/weave/master/weave
 
 Previously the only documented download location was (3). We recommend
 that any automated scripts using that be changed to either (1) or (2).

LICENSE

@@ -176,7 +176,7 @@
 
    END OF TERMS AND CONDITIONS
 
-   Copyright 2014 Zettio Ltd.
+   Copyright 2014-2015 Weaveworks Ltd.
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.

Makefile

@@ -1,28 +1,39 @@
+PUBLISH=publish_weave publish_weavedns publish_weaveexec
+
 .DEFAULT: all
-.PHONY: all update tests clean
+.PHONY: all update tests publish $(PUBLISH) clean prerequisites build travis run-smoketests
 
 # If you can use docker without being root, you can do "make SUDO="
 SUDO=sudo
 
+DOCKERHUB_USER=weaveworks
 WEAVE_VERSION=git-$(shell git rev-parse --short=12 HEAD)
 WEAVER_EXE=weaver/weaver
 WEAVEDNS_EXE=weavedns/weavedns
-WEAVETOOLS_EXES=tools/bin
-WEAVER_IMAGE=zettio/weave
-WEAVEDNS_IMAGE=zettio/weavedns
-WEAVETOOLS_IMAGE=zettio/weavetools
-WEAVER_EXPORT=/var/tmp/weave.tar
-WEAVEDNS_EXPORT=/var/tmp/weavedns.tar
-WEAVETOOLS_EXPORT=/var/tmp/weavetools.tar
+WEAVEPROXY_EXE=weaveproxy/weaveproxy
+SIGPROXY_EXE=sigproxy/sigproxy
+WEAVEWAIT_EXE=weavewait/weavewait
+WEAVER_IMAGE=$(DOCKERHUB_USER)/weave
+WEAVEDNS_IMAGE=$(DOCKERHUB_USER)/weavedns
+WEAVEEXEC_IMAGE=$(DOCKERHUB_USER)/weaveexec
+WEAVER_EXPORT=weave.tar
+WEAVEDNS_EXPORT=weavedns.tar
+WEAVEEXEC_EXPORT=weaveexec.tar
 
-all: $(WEAVER_EXPORT) $(WEAVEDNS_EXPORT) $(WEAVETOOLS_EXPORT)
+WEAVEEXEC_DOCKER_VERSION=1.3.1
+DOCKER_DISTRIB=weaveexec/docker-$(WEAVEEXEC_DOCKER_VERSION).tgz
+DOCKER_DISTRIB_URL=https://get.docker.com/builds/Linux/x86_64/docker-$(WEAVEEXEC_DOCKER_VERSION).tgz
+
+all: $(WEAVER_EXPORT) $(WEAVEDNS_EXPORT) $(WEAVEEXEC_EXPORT)
+
+travis: $(WEAVER_EXE) $(WEAVEDNS_EXE)
 
 update:
-	go get -u -f -v -tags -netgo ./$(dir $(WEAVER_EXE)) ./$(dir $(WEAVEDNS_EXE))
+	go get -u -f -v -tags -netgo ./$(dir $(WEAVER_EXE)) ./$(dir $(WEAVEDNS_EXE)) ./$(dir $(SIGPROXY_EXE)) ./$(dir $(WEAVEPROXY_EXE))
 
-$(WEAVER_EXE) $(WEAVEDNS_EXE):
+$(WEAVER_EXE) $(WEAVEDNS_EXE) $(WEAVEPROXY_EXE) $(WEAVEWAIT_EXE): common/*.go
 	go get -tags netgo ./$(@D)
-	go build -ldflags "-extldflags \"-static\" -X main.version $(WEAVE_VERSION)" -tags netgo -o $@ ./$(shell dirname $@)
+	go build -ldflags "-extldflags \"-static\" -X main.version $(WEAVE_VERSION)" -tags netgo -o $@ ./$(@D)
 	@strings $@ | grep cgo_stub\\\.go >/dev/null || { \
 		rm $@; \
 		echo "\nYour go standard library was built without the 'netgo' build tag."; \
@@ -32,11 +43,15 @@ $(WEAVER_EXE) $(WEAVEDNS_EXE):
 		false; \
 	}
 
-$(WEAVER_EXE): router/*.go weaver/main.go
+$(WEAVER_EXE): router/*.go ipam/*.go ipam/*/*.go weaver/main.go
 $(WEAVEDNS_EXE): nameserver/*.go weavedns/main.go
+$(WEAVEPROXY_EXE): proxy/*.go weaveproxy/main.go
+$(WEAVEWAIT_EXE): weavewait/*.go weavewait/main.go
 
-$(WEAVETOOLS_EXES): tools/build.sh
-	$(SUDO) docker run --rm -v $(realpath $(<D)):/home/weave ubuntu sh /home/weave/build.sh
+# Sigproxy needs separate rule as it fails the netgo check in the main
+# build stanza due to not importing net package
+$(SIGPROXY_EXE): sigproxy/main.go
+	go build -o $@ ./$(@D)
 
 $(WEAVER_EXPORT): weaver/Dockerfile $(WEAVER_EXE)
 	$(SUDO) docker build -t $(WEAVER_IMAGE) weaver
@@ -46,15 +61,49 @@ $(WEAVEDNS_EXPORT): weavedns/Dockerfile $(WEAVEDNS_EXE)
 	$(SUDO) docker build -t $(WEAVEDNS_IMAGE) weavedns
 	$(SUDO) docker save $(WEAVEDNS_IMAGE):latest > $@
 
-$(WEAVETOOLS_EXPORT): tools/Dockerfile $(WEAVETOOLS_EXES)
-	$(SUDO) docker build -t $(WEAVETOOLS_IMAGE) tools
-	$(SUDO) docker save $(WEAVETOOLS_IMAGE):latest > $@
+$(WEAVEEXEC_EXPORT): weaveexec/Dockerfile $(DOCKER_DISTRIB) weave $(SIGPROXY_EXE) $(WEAVEPROXY_EXE) $(WEAVEWAIT_EXE)
+	cp weave weaveexec/weave
+	cp $(SIGPROXY_EXE) weaveexec/sigproxy
+	cp $(WEAVEWAIT_EXE) weaveexec/weavewait
+	cp $(WEAVEPROXY_EXE) weaveexec/weaveproxy
+	cp $(DOCKER_DISTRIB) weaveexec/docker.tgz
+	$(SUDO) docker build -t $(WEAVEEXEC_IMAGE) weaveexec
+	$(SUDO) docker save $(WEAVEEXEC_IMAGE):latest > $@
+
+$(DOCKER_DISTRIB):
+	curl -o $(DOCKER_DISTRIB) $(DOCKER_DISTRIB_URL)
 
-# Add more directories in here as more tests are created
 tests:
-	cd nameserver; go test -tags netgo
+	echo "mode: count" > profile.cov
+	fail=0 ;                                                                              \
+	for dir in $$(find . -type f -name '*_test.go' | xargs -n1 dirname | sort -u); do     \
+	    output=$$(mktemp cover.XXXXXXXXXX) ;                                              \
+	    if ! go test -tags netgo -covermode=count -coverprofile=$$output $$dir ; then     \
+            fail=1 ;                                                                          \
+        fi ;                                                                                  \
+	    if [ -f $$output ]; then                                                          \
+	        tail -n +2 <$$output >>profile.cov;                                           \
+	        rm $$output;                                                                  \
+	    fi                                                                                \
+	done ;                                                                                \
+	exit $$fail
+	go tool cover -html=profile.cov -o=coverage.html
+
+$(PUBLISH): publish_%:
+	$(SUDO) docker tag -f $(DOCKERHUB_USER)/$* $(DOCKERHUB_USER)/$*:$(WEAVE_VERSION)
+	$(SUDO) docker push   $(DOCKERHUB_USER)/$*:$(WEAVE_VERSION)
+	$(SUDO) docker push   $(DOCKERHUB_USER)/$*:latest
+
+publish: $(PUBLISH)
 
 clean:
-	-$(SUDO) docker rmi $(WEAVER_IMAGE) $(WEAVEDNS_IMAGE) $(WEAVETOOLS_IMAGE)
-	rm -f $(WEAVER_EXE) $(WEAVEDNS_EXE) $(WEAVER_EXPORT) $(WEAVEDNS_EXPORT) $(WEAVETOOLS_EXPORT)
-	$(SUDO) rm -rf $(WEAVETOOLS_EXES)
+	-$(SUDO) docker rmi $(WEAVER_IMAGE) $(WEAVEDNS_IMAGE) $(WEAVEEXEC_IMAGE)
+	rm -f $(WEAVER_EXE) $(WEAVEDNS_EXE) $(SIGPROXY_EXE) $(WEAVEPROXY_EXE) $(WEAVEWAIT_EXE) $(WEAVER_EXPORT) $(WEAVEDNS_EXPORT) $(WEAVEEXEC_EXPORT)
+
+build:
+	$(SUDO) go clean -i net
+	$(SUDO) go install -tags netgo std
+	$(MAKE)
+
+run-smoketests: all
+	cd test && ./setup.sh && ./run_all.sh

NOTICE

@@ -1,3 +1,3 @@
 Weave
-Copyright 2012-2014 Zettio Ltd.
-This product includes software developed at Zettio Ltd.
+Copyright 2014-2015 Weaveworks Ltd.
+This product includes software developed at Weaveworks Ltd.

README.md

@@ -1,7 +1,9 @@
 # Weave - the Docker network
 
+[![Build Status](https://travis-ci.org/weaveworks/weave.svg?branch=master)](https://travis-ci.org/weaveworks/weave) [![Integration Tests](https://circleci.com/gh/weaveworks/weave/tree/master.svg?style=shield&circle-token=4933c7dabb3d0383e62117565cb9d16df7b1a811)](https://circleci.com/gh/weaveworks/weave) [![Coverage Status](https://coveralls.io/repos/weaveworks/weave/badge.svg)](https://coveralls.io/r/weaveworks/weave)
+
 Weave creates a virtual network that connects Docker containers
-deployed across multiple hosts.
+deployed across multiple hosts and enables their automatic discovery.
 
 ![Weave Virtual Network](/docs/virtual-network.png?raw=true "Weave Virtual Network")
 
@@ -27,13 +29,18 @@ capabilities, so these can continue to be used by containers.
 
 ## Installation
 
-Ensure you are running Linux (kernel 3.5 or later) and have Docker
-(version 0.9.1 or later) installed. Then install weave with
+Ensure you are running Linux (kernel 3.8 or later) and have Docker
+(version 1.3.1 or later) installed. Then install weave with
 
     sudo wget -O /usr/local/bin/weave \
-      https://github.com/zettio/weave/releases/download/latest_release/weave
+      https://github.com/weaveworks/weave/releases/download/latest_release/weave
     sudo chmod a+x /usr/local/bin/weave
 
+CoreOS users see [here](https://github.com/fintanr/weave-gs/blob/master/coreos-simple/user-data) for an example of installing weave using cloud-config.
+
+Weave respects the environment variable `DOCKER_HOST`, so you can run
+it locally to control a weave network on a remote host.
+
 ## Quick Start Screencast
 
 <a href="http://youtu.be/k6r7yuSr0hE" alt="Click to watch the screencast" target="_blank">
@@ -46,10 +53,10 @@ Say you have docker running on two hosts, accessible to each other as
 `$HOST1` and `$HOST2`, and want to deploy an application consisting of
 two containers, one on each host.
 
-On `$HOST1` run (as root)
+On `$HOST1` run
 
-    host1# weave launch
-    host1# C=$(weave run 10.0.1.1/24 -t -i ubuntu)
+    host1$ weave launch
+    host1$ C=$(weave run 10.2.1.1/24 -t -i ubuntu)
 
 The first line starts the weave router, in a container. This needs to
 be done once on each host. The required docker image for the weave
@@ -80,41 +87,42 @@ addresses must, of course, be unique.
 
 We repeat similar steps on `$HOST2`...
 
-    host2# weave launch $HOST1
-    host2# C=$(weave run 10.0.1.2/24 -t -i ubuntu)
+    host2$ weave launch $HOST1
+    host2$ C=$(weave run 10.2.1.2/24 -t -i ubuntu)
 
 The only difference, apart from the choice of IP address for the
 application container, is that we tell our weave that it should peer
-with the weave on `$HOST1` (specified as the IP address or hostname, and
-optional `:port`, by which `$HOST2` can reach it). NB: if there is a
-firewall between `$HOST1` and `$HOST2`, you must open port 6783 for TCP
-and UDP.
+with the weave on `$HOST1` (specified as the IP address or hostname,
+and optional `:port`, by which `$HOST2` can reach it). NB: if there is
+a firewall between `$HOST1` and `$HOST2`, you must open the weave port
+(6783 by default; this can be overriden by setting `WEAVE_PORT`) for
+TCP and UDP.
 
 Note that we could instead have told the weave on `$HOST1` to connect to
 `$HOST2`, or told both about each other. Order does not matter here;
 weave automatically (re)connects to peers when they become
 available. Also, we can tell weave to connect to multiple peers by
 supplying multiple addresses, separated by spaces. And we can
-[add peers dynamically](http://zettio.github.io/weave/features.html#dynamic-topologies).
+[add peers dynamically](http://docs.weave.works/weave/latest_release/features.html#dynamic-topologies).
 
 Now that we've got everything set up, let's see whether our containers
 can talk to each other...
 
 On `$HOST1`...
 
-    host1# docker attach $C
-    root@28841bd02eff:/# ping -c 1 -q 10.0.1.2
-    PING 10.0.1.2 (10.0.1.2): 48 data bytes
-    --- 10.0.1.2 ping statistics ---
+    host1$ docker attach $C
+    root@28841bd02eff:/# ping -c 1 -q 10.2.1.2
+    PING 10.2.1.2 (10.2.1.2): 48 data bytes
+    --- 10.2.1.2 ping statistics ---
     1 packets transmitted, 1 packets received, 0% packet loss
     round-trip min/avg/max/stddev = 1.048/1.048/1.048/0.000 ms
 
 Similarly, on `$HOST2`...
 
-    host2# docker attach $C
-    root@f76829496120:/# ping -c 1 -q 10.0.1.1
-    PING 10.0.1.1 (10.0.1.1): 48 data bytes
-    --- 10.0.1.1 ping statistics ---
+    host2$ docker attach $C
+    root@f76829496120:/# ping -c 1 -q 10.2.1.1
+    PING 10.2.1.1 (10.2.1.1): 48 data bytes
+    --- 10.2.1.1 ping statistics ---
     1 packets transmitted, 1 packets received, 0% packet loss
     round-trip min/avg/max/stddev = 1.034/1.034/1.034/0.000 ms
 
@@ -123,22 +131,22 @@ to each other.
 
 ## Find out more
 
- * [Documentation homepage](http://zettio.github.io/weave/)
- * [Features](http://zettio.github.io/weave/features.html)
- * [Troubleshooting](http://zettio.github.io/weave/troubleshooting.html)
- * [Building](http://zettio.github.io/weave/building.html)
- * [How it works](http://zettio.github.io/weave/how-it-works.html)
- * [Preview of weaveDNS](http://weaveblog.com/2014/11/04/have-you-met-weavedns/)
+ * [Documentation homepage](http://docs.weave.works/weave/latest_release/)
+ * [Features](http://docs.weave.works/weave/latest_release/features.html)
+ * [Automatic Discovery with WeaveDNS](http://docs.weave.works/weave/latest_release/weavedns.html)
+ * [Troubleshooting](http://docs.weave.works/weave/latest_release/troubleshooting.html)
+ * [Building](http://docs.weave.works/weave/latest_release/building.html)
+ * [How it works](http://docs.weave.works/weave/latest_release/how-it-works.html)
 
 ## Contact Us
 
 Found a bug, want to suggest a feature, or have a question?
-[File an issue](https://github.com/zettio/weave/issues), or email
+[File an issue](https://github.com/weaveworks/weave/issues), or email
 help@weave.works. When reporting a bug, please include which version of
 weave you are running, as shown by `weave version`.
 
 Follow weave on Twitter:
-[@weavenetwork](https://twitter.com/weavenetwork).
+[@weaveworks](https://twitter.com/weaveworks).
 
 Read the Weave blog:
 [Weaveblog](http://weaveblog.com/).

Vagrantfile

@@ -1,32 +1,40 @@
 VAGRANTFILE_API_VERSION = "2"
 
+require './vagrant-common.rb'
+
 vm_ip = "172.16.0.3" # arbitrary private IP
-pkgs = "lxc-docker build-essential ethtool libpcap-dev git mercurial"
+
+pkgs = %w(
+  lxc-docker
+  aufs-tools
+  build-essential
+  ethtool
+  iputils-arping
+  libpcap-dev
+  git
+  mercurial
+  bc
+)
 
 Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
-  config.vm.box = "phusion/ubuntu-14.04-amd64"
-  config.vm.box_url = "https://oss-binaries.phusionpassenger.com/vagrant/boxes/latest/ubuntu-14.04-amd64-vbox.box"
+
+  config.vm.box = "ubuntu/ubuntu-14.10-amd64"
+  config.vm.box_url = "https://cloud-images.ubuntu.com/vagrant/utopic/current/utopic-server-cloudimg-amd64-vagrant-disk1.box"
 
   config.vm.network "private_network", ip: vm_ip
+  config.vm.provider :virtualbox do |vb|
+    vb.customize ["modifyvm", :id, "--natdnshostresolver1", "off"]
+    vb.customize ["modifyvm", :id, "--natdnsproxy1", "off"]
+  end
 
-  config.vm.synced_folder "./", "/home/vagrant/src/github.com/zettio/weave"
+  config.vm.synced_folder ".", "/vagrant", disabled: true
+  config.vm.synced_folder ".", "/home/vagrant/src/github.com/weaveworks/weave"
 
-  config.vm.provision :shell, :inline => "sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9"
-  config.vm.provision :shell, :inline => "echo deb https://get.docker.io/ubuntu docker main > /etc/apt/sources.list.d/docker.list"
-
-  pkg_cmd = "export DEBIAN_FRONTEND=noninteractive; " \
-    "apt-get update -qq; " \
-    "apt-get install -qq --no-install-recommends #{pkgs}"
-
-  install_go_toolchain = "curl -s https://storage.googleapis.com/golang/go1.4.linux-amd64.tar.gz | tar xz -C /usr/local"
-
-  config.vm.provision :shell, :inline => pkg_cmd
-  config.vm.provision :shell, :inline => install_go_toolchain
-  config.vm.provision :shell, :inline => "usermod -a -G docker vagrant; "
-  config.vm.provision :shell, :inline => "echo export GOPATH=/home/vagrant >> /home/vagrant/.bashrc"
-  config.vm.provision :shell, :inline => "echo export PATH=/usr/local/go/bin:${PATH} >> /home/vagrant/.bashrc"
-  config.vm.provision :shell, :inline => "chown -R vagrant:vagrant /home/vagrant/src"
-  config.vm.provision :shell, :inline => "/usr/local/go/bin/go clean -i net; /usr/local/go/bin/go install -tags netgo std"
+  install_build_deps config.vm, pkgs
+  install_go_toochain config.vm
+  tweak_user_env config.vm
+  tweak_docker_daemon config.vm
+  cleanup config.vm
 
 end
 

bin/lint

@@ -0,0 +1,81 @@
+#!/bin/bash
+# This scipt lints go files for common errors.
+#
+# Its runs gofmt and go vet, and optionally golint and
+# gocyclo, if they are installed.
+#
+# With no arguments, it lints the current files staged
+# for git commit.  Or you can pass it explicit filenames
+# (or directories) and it will lint them.
+#
+# To use this script automatically, run:
+#   ln -s ../../bin/lint .git/hooks/pre-commit
+
+set -eu
+
+function lint_go {
+	filename="$1"
+	result=0
+
+	if [ -n "$(gofmt -l "${filename}")" ]; then
+		result=1
+		echo "${filename}: run gofmt -w ${filename}!"
+	fi
+
+	go tool vet "${filename}" || result=$?
+
+	# golint is completely optional.  If you don't like it
+	# don't have it installed.
+	if type golint >/dev/null 2>&1; then
+		# golint doesn't set an exit code it seems
+		lintoutput=$(golint "${filename}")
+		if [ "$lintoutput" != "" ]; then
+			result=1
+			echo "$lintoutput"
+		fi
+	fi
+
+	# gocyclo is completely optional.  If you don't like it
+	# don't have it installed.  Also never blocks a commit,
+	# it just warns.
+	if type gocyclo >/dev/null 2>&1; then
+		gocyclo -over 25 "${filename}" | while read line; do
+			echo "${filename}": higher than 25 cyclomatic complexity - "${line}"
+		done
+	fi
+
+	return $result
+}
+
+function lint {
+	filename="$1"
+	ext="${filename##*\.}"
+
+	# Don't lint this script
+	if [ "${filename}" = "bin/lint" ]; then
+		return
+	fi
+
+	case "$ext" in
+		go) lint_go "${filename}"
+		;;
+	esac
+}
+
+function lint_files {
+	lint_result=0
+	while read filename; do
+		lint "${filename}" || lint_result=1
+	done
+	exit $lint_result
+}
+
+function list_files {
+	if [ $# -gt 0 ]; then
+		find "$@" -type f
+	else
+		git diff --cached --name-only
+	fi
+}
+
+list_files "$@" | lint_files

bin/merge-into-gh-pages

@@ -1,20 +0,0 @@
-#! /bin/bash
-set -e
-
-# For info on why/how to use this script, see
-# https://github.com/zettio/weave/wiki/WorkingOnWeave#working-on-docs
-
-MERGE_ARGS="-X subtree=site"
-
-merge_dir=$(mktemp -d -t weave-merge.XXXXXX)/weave
-git clone -b gh-pages . $merge_dir
-
-pushd $merge_dir
-git pull --ff-only git@github.com:zettio/weave gh-pages
-git fetch origin master
-git merge $MERGE_ARGS -e -m "Merge master into gh-pages" origin/master
-
-popd
-git fetch $merge_dir gh-pages:gh-pages
-
-# rm -rf $merge_dir

bin/multiweave

@@ -0,0 +1,54 @@
+#!/bin/bash
+set -e
+
+# You may need to adjust
+#    echo 4096 >/proc/sys/net/ipv4/neigh/default/gc_thresh3
+# where the number is greater than NUM_WEAVES*(NUM_WEAVES-1) plus some
+# fudge.
+#
+# Without that, some connections between the weaves may fail to get
+# established, and networking on the host in general becomes
+# broken. There will also be some "net_ratelimit: nnn callbacks
+# suppressed" messages in the syslog.
+
+START=${FIRST_WEAVE:-1}
+COUNT=${NUM_WEAVES:-28}
+FINISH=$((START+COUNT-1))
+PORT=6783
+
+weavedir=$(dirname $0)/..
+
+# Generate a random MAC value - copied from 'weave' script
+random_mac() {
+    # In the first byte of the MAC, the 'multicast' bit should be
+    # clear and 'locally administered' bit should be set.  All other
+    # bits should be random.
+    od -txC -An -N6 /dev/urandom | sed 's|^ ||;s| |:|;s|[0-3]:|2:|;s|[4-7]:|6:|;s|[89ab]:|a:|;s|[c-f]:|e:|;s| |:|g'
+}
+
+case "$1" in
+    launch)
+        shift 1
+        for i in $(seq $START $FINISH); do
+            WEAVE_CONTAINER_NAME=weave$i WEAVE_PORT=$((5000+i)) $weavedir/weave launch -iface '' -name $(random_mac) -nickname weave$i -port $PORT "$@"
+        done
+        if [ $START -ne 1 ] ; then
+            status=$(docker inspect --format='{{.State.Running}} {{.NetworkSettings.IPAddress}}' weave$((START-1)))
+            PREV="${status#true }"
+        fi
+        for i in $(seq $START $FINISH); do
+            if [ -n "$PREV" ] ; then
+                echo connecting weave$i to $PREV
+                WEAVE_CONTAINER_NAME=weave$i $weavedir/weave connect $PREV
+            fi
+            status=$(docker inspect --format='{{.State.Running}} {{.NetworkSettings.IPAddress}}' weave$i)
+            PREV="${status#true }"
+        done
+        ;;
+    stop)
+        for i in $(seq $START $FINISH); do
+            echo Stopping weave$i
+            WEAVE_CONTAINER_NAME=weave$i $weavedir/weave stop
+        done
+        ;;
+esac

bin/publish-gh-pages

@@ -1,3 +0,0 @@
-#! /bin/sh
-set -e
-git push git@github.com:zettio/weave gh-pages:gh-pages

bin/release

@@ -2,11 +2,13 @@
 
 set -e
 
+SUDO=${SUDO-sudo}
+GITHUB_USER=${GITHUB_USER:-weaveworks}
+DOCKERHUB_USER=${DOCKERHUB_USER:-weaveworks}
+RELEASE_NAME=${RELEASE_NAME:-"Weave"}
+RELEASE_DESCRIPTION=${RELEASE_DESCRIPTION:-"Weaving Containers into Applications"}
+
 PWD=`pwd`
-GITHUB_USER=zettio
-DOCKERHUB_USER=zettio
-RELEASE_NAME="Weave"
-RELEASE_DESCRIPTION="Weaving Containers into Applications"
 
 setup() {
     ## Get the new version number from the most recent tag
@@ -49,10 +51,7 @@ build() {
     ## Inject the version numbers and build the distributables
     ## (library versions?)
     sed -i "/SCRIPT_VERSION=/ c\SCRIPT_VERSION=\"$VERSION\"" ./weave
-    make \
-        WEAVE_VERSION=$VERSION \
-        WEAVER_IMAGE=$DOCKERHUB_USER/weave \
-        WEAVEDNS_IMAGE=$DOCKERHUB_USER/weavedns
+    make SUDO=$SUDO WEAVE_VERSION=$VERSION DOCKERHUB_USER=$DOCKERHUB_USER
 
     if make tests; then
         echo -e '\u2713 Tests pass'
@@ -121,13 +120,8 @@ publish() {
 
     echo '** Sanity checks OK for publishing tag' $LATEST_TAG as $DOCKERHUB_USER/weave:$VERSION
 
-    for IMAGE in weave weavedns weavetools ; do
-        echo == Tagging $DOCKERHUB_USER/$IMAGE as $VERSION
-        docker tag  $DOCKERHUB_USER/$IMAGE $DOCKERHUB_USER/$IMAGE:$VERSION
-        docker push $DOCKERHUB_USER/$IMAGE:$VERSION
-        docker push $DOCKERHUB_USER/$IMAGE:latest
-        echo "** Pushed $DOCKERHUB_USER/$IMAGE:$VERSION to docker hub"
-    done
+    echo "== Tagging and publishing images on docker hub as user $DOCKERHUB_USER"
+    make SUDO=$SUDO WEAVE_VERSION=$VERSION DOCKERHUB_USER=$DOCKERHUB_USER publish
 
     echo "== Creating GitHub release $RELEASE_NAME $VERSION"
     # This cannot be done as a draft because of a bug in

bin/setup-ci-secrets

@@ -0,0 +1,57 @@
+#!/bin/sh
+
+## This is the shape of unencrypted `do-setup-ci-secrets`
+## 
+## cat > /tmp/boto_gsutil_cert.pem << EOF
+## -----BEGIN PRIVATE KEY-----
+## <...>
+## -----END PRIVATE KEY-----
+## EOF
+## 
+## cat > ~/.boto << EOF
+## [Credentials]
+## gs_service_client_id = <...>
+## gs_service_key_file = /tmp/boto_gsutil_cert.pem
+## 
+## [Boto]
+## https_validate_certificates = True
+## 
+## [GSUtil]
+## content_language = en
+## default_api_version = 2
+## default_project_id = <...>
+## EOF
+
+openssl base64 -d << EOF \
+  | openssl aes-256-cbc -d -K $1 -iv $2 \
+    -out bin/do-setup-ci-secrets
+mOYNzNsAdwva0LbB2YFMY9VgL0SITFu1bPrbOujyLtOrBNjmOYbzR5d/sG+ARMFr
+7NZ5OKcvMbItllz0n7TtNkzkCOMztMC870Wcx3nGJzbBephdcYwos4tBlp5bihTU
+AXgxLUiDT1broYAFKnbE1aZC/XyNxuLTN+nXJWIWeijaXTgvhkEEm3tab+z7ioCb
+tlg71W1BrUuocctAJAuVwH/uTZAaXaJGJj1s2S7F2DrFnG6sFlrp8rnCucPLWKnC
+4IwJFC5saR5gxK6wECffwmwbpeXVOoQ58OgzxoZTEyAr9nhVbaBt5UZz3tfHlx8W
+Rn5jHehACw5kg5eMtGPqMEksJ63LX4h3mfSTLJXJHEmOrzCRpFrigVrsbA408U2u
+8jJTHJmKrGfwBh5ZFzBg6NuV4VplOp+ZquJLgegv2WH/iu/ciopXqB2YDCdx5ZBi
+nHV2Puth7kjS4EMCKUGfeqpesjL9Djb8cygzv/J3ZBmLjNH9dqlSXWy2DcFZA1Iy
+pFTTHkVU3w5sNz3nYOuT2UPQvULBW0FUndDKX6olyVa0mlxPwGBP9N3yGg918/rw
+7rNV/r54lA334F5QMhDrVxar6sl9wfLA+Z+8qemBbNFlLog1uQGhuwWK2nFu1tRM
+9uyvQIq0oq5NHKpogWc7NEuk7CPe1JyfRIcbMVj8+ilmKjUPB2HPC43IDmKt9/56
+rY3IXAcxZ/ClGfGpeNHXqop9T/WhfaowponOZFcdGSUddG0OswqkTE4KIv71mgTF
+jXooMkTQBOG+gBnCe+8bCRdRCG3anxHX74za4dGD58cOJtl6DYWnjWuR91NWnxdT
+1Zr3iUe41txg9hCNY9LHo2QHGLPxe6pmNnSClQ1mcFhuY6NF8RmCX3HAGKJQkxJb
+vpLGbXTXjAIFgIYgLNgKP2RLvCif/LmSWgkvE3ZU9GXtCUbm6xgJodwOqXiI9uuO
+4H4+/vpGHnSFENiclkgMFrLG4yiMu8ArgmnCBzg9E2AoOX5zCrh/2ZRTU8Dk4mt7
+jUd5SKQLsjB9QtB+y3tRO7zY9RDvR6iBOPRsjx6t8EJMCfPb06KchfMJw+VGxvMM
+48jtmepREFnhd/8XDRGnw5bznjRuwohGik8BToltjIviMkkpifKWaLwlaOiDXaHh
+VBFNXnUKvOBwRMU+PZqv4+DP2qff1gSXnGKqA5GIURYrK9P7TqHHNTnmjBdfJLzc
+0x+W1DTiYhLT2vrLrkEpZ99o6qrg+lwvjat6IsJF8xoQEo4pLD/ZRPJ0WzoDyctT
+oWZ5rMsN1TzTvJrBrTAWIJyhIzZcuNwChMrU6qogMr0fd6TSxWhWn0sOJ5y8AaTR
+ej4JHx0qroS/9JPmgFYU61pX3c0ypdef/sSHWG23mcFzLTgwe2d4ty9VakypZy3T
+sXCl6EHVVXlUFWhY4zQCjdZpYJv/rtoKL1BL4fPE2NV5WTjCL2GNvaO5SfHR7pA0
+yQJR/iU+cS2n8UXsWNjyV+/qEn6+QXuyVHGLbxyH8EeqlliAbyHsc2rzVI8Xctzo
+W5DuIqxwHAkNNurhmx27ktIIQ6HhaCp028SWwvLttkbBBgF4v1mmd4u/2bKbKjUd
+dOaetaIUx1hU9OAeaePvVkeTDcc3mlFtPARCS9Fqj16hgJQ43onZyC0H1K2WAxmU
+GsmzjCHlTNquzFTuuQihYGZKfsWIh9z6HEOkvNJ3VEQ=
+EOF
+
+exec sh bin/do-setup-ci-secrets

bin/setup-circleci-secrets

@@ -0,0 +1,115 @@
+#!/bin/bash
+
+set -eu
+
+# openssl enc -in do-setup-circleci-secrets.orig -out setup-circleci-secrets.orig -e -aes256 -pass stdin
+# openssl base64 < setup-circleci-secrets.orig
+
+openssl base64 -d << EOF \
+  | openssl enc \
+    -out bin/do-setup-circleci-secrets \
+		-d -aes256 -pass pass:$1
+U2FsdGVkX1/3QHnDgpZn0srp+GiEQBTlag0CBdsw6Bp6iaIKy6Jy8gFshrsmvl3+
+c+7hn1nqsBVpzVH0S5QA3Ex5nx9rl7/dcGoogsySFcwMmBFcauT0UTMK1bvX0Hqx
+QrshYSndUkww72KA7EsSnjR1BBJWm3AhCDOTIcnO6nQhig1Fc+SBEj5FGS5xTQ5Z
+/8hDsKf54Ms49vOO6QeE5w5ndZ+k7IvVJfq9Evhb7htD20RntDhR86foAwMnLV7R
+Wh9qAqI3gnRq0GmbeRKaHmB7pcLq9f0OdScqM/Tn1D+q0556fqJjBziXpZiGbxVs
+DxEp8zDJ8yNDAi0Wb2NXjoA79SXJ8Q7iTieaST4y2OmXQpY+Vw8RVDXxi6ERyVnH
+41OeMzyLm0qvCoMl32pe1sSG5DRCUiAm7wOtDTVvNDRqmGJYUgwwexyayecJI1wQ
+LlhYLJavUvw3UGDZOld8BaMQhfOKBVCdFrjmA4HuVxmOf3029qkgzOCSZiQVNqs+
+fvM6J05zBuk+GPy8YInvRkzfxum/x00vmUyGjMs/lHDFfrmYDFK3ux41EoVqvWLG
+S5uYxdZJFdR3Xsrs+sMr51XNBQGMmz6KoEEzFYyHs5fZgtdpRdPAuQS1m6FqgZyi
+wSz6+EKsALtmGorEp+CF8JN5YzKJCrKzNkg0fILBqsqmJZGP64lmX69950lRLNGM
+/l3KVR2zH31b4AnnzTcvldVSCKDXjaMw3JA2Ce/MK6XhOWOcpEv4oDt/62lbhUKP
+QUQTPbfpQRvVfBcoIrfAm3ghi/CZc2TDVEuD7DhasLTZD3r7LoK261BWC9O5Wubi
+vlXuFIF+uyrpTDf6EJKE/+XywtArppffMP6RkHu1Iqnd8JtsqPaIZsmMmZbjgb6j
+BX4ZLb5U9R7jYoSgIke4+u6geFcrMQ+Lny4r7MxXaS0Y1UJ6Jo1bwpYw1TBIobZc
+eQ30+xhbSeRC9vEw1NXI6qM2Ai2gIHtAZvEiBkBBXTG5Ld7N/mBAmDIJEzVRxh6F
+jRkB3dzESF6eeTBm61OQepwV3aW6lslMaf26drPGv2W2q+K/WOpQOejfggtHUgVs
+9SWqtn7hHYSpIASe1Ihi14HAWv3kPhQien0zY1DLKJWf4ilbRaICn1QSJjNuKwtl
+A2ZwEw7FFddFcsF4okHPkxEoCre3shq0DEw9UA1gnwEaEF1Q+i9cvLtQg56zK3+R
+MfTzFUv2K1z3j1s5tC6Z6LzL8j682EFOll7MkihqiIWh+po1zcNs6Ln0SkjB3v9T
+OFRYVJis9GqXYO9PUW1mI69udR4nXbKlBl5tWNyAxn9hufJHLSAmbGMwActKLRM5
+kROapKrN1Ra/sHUmhoSSDUK9AXeYKkqPXVMbhly9xC1Kya4hFaFokjRQIMLPmXdQ
+lP46RL3HOq0v1BkXg05poSwDm+BtitZkcIhOzcmeucUqn1whsz39y3VoOhJqQu2j
+0dUcj19Fue1CDZfwzACb9mLZr643h0yCIHQaVT7is/yiN2vEt0S0zynvOA1D+Xgy
+4bFdjqed9DgBShLwF/lh4/hur5iZPpVr9A8rt3Vq0k5A1tdjsZpXP3qv5jLc7lJv
+cAQxdVLEuv8WPwwTV/EQsoH7lpoEdjktw/gu0v5PkUgR/WBxEqJq4liTekViQ+8X
+BAmdm/FGWeIxSSEm0a8I+muiBpIuKjuQoJit1f0k+lLUohoVDqpDDfeYHh1TxXHE
+xSKAKNmFSc2XOr5Zg2mOInxeH7WoxH6iylXKADBgEgAQ4C55DoAPRjFNtGGz2iJl
+WkzhQOfgsDss1VxpPZ4SWDQO+xxuuEnYcCYdJ6Tyi5hAwZ0+RZmXx0g9BCK+5qzc
+Ffei066kCItBcayekwCs0i86r8JvDtsMUUkzEKDQTSWHpP8ypIZoz3P+Skdqcs6p
+IcZHy2fODIsClctRAsHOjZ9BnWSY2rqf1lysqvRfOkNTViTOUVx2hf+LuL+8rT6g
+K4GvS39Q+c9FYSMjCNxv1m37NfHJv2W7AlqVV95gMBEFB97M3388T7PfvmSLsyPv
+qAKDE7genTOycQvEX9oeEACxPo08ai2Mgs/YXCig0g0btZoScoKPelQKtD0HLpZ+
+j7ruaKFy4/vySDMostB3AKSkCE/SkGu/W9oI37n1guSYC/9yJjtTcwZULXAClfwt
+LyHvLAw4+7BdGJqDe9Tf8z4zqVLax+BcBHkW7pbP+pvxC6fhIA4PO8k5HwwAVlRw
+68lG6e3z9+Vst4zrFLewHJvkDRSOTSecs9h9SPOjReTK0ovlDzMHVakfUvqpk9mv
+nCj8SWaMPF70z138AuRuKN7X4exNXPBFnOcseKp/NpT1DyvgoHm/oIO65mq9vvo+
+42uSbyyegYiTdVK4SaXBc2hixkhaTteqdEy6GxfWqKCvHtYuRD2LyUSjZn/0Y6CG
+3wZapylKL1sxfH+b1gogv2wAA8KtZAfiTlMISuymwQr5RbOFma7UVMVecCa7XrNG
+fZwO8R+6++ru0efDzeC/gtLOItjn+yVfsbk2gfu0FikHIV1sogOjnsv3dl6IpH/s
+w6WbuK/yZhyrrmhIIueHF30ES512Peogf/Gxp8U/ZAmqBa7p4qsv93XWDR52IPYz
+yU1MtboLD02J39daBNrYZuO4Yfo/xAvU7Ijg2KIR9HS0bSWNuSFAZaZXlJ2bSag3
+8l0dZMx2fZa0FALxWvY2+nd1tWcH+Wo99RSxX+4TH3guCpe1x/58yKOKL4d1xRpu
+02nGaqerV9bHU6WjCUYTrrZurTp6yqivpSFCALDMhAU5Ihd/Qjz69zg4uZ7iqbNd
+WqIYJ3SAOei2tSfoOJbdh8mohotnMFFbupck3Fv+lXAXYDp0V/sSuK32MSeJxan1
+q3t0eenAh6zoBP0Tj53N8B9YmrgZ1i1AljQqYsAdb8N5Z3vecuzQa946U8H4axEc
+BL5fgmqSfsbj1EjJVUrw3cE9hgqeqAYBBvZsyW5zSAsLq/RJKYtBxDGADqr0TOOu
+TjVqfPObb1u9EAQj15ALBhhPORa2gBAK5gnhJWW5deNgA0UOYoWARFYfATuQvr1T
+9xTVbqzZ0VJR4n7X5JyxJo2tZ44QYVzA0fqX0lt+uRkarrkbZMgplKp4MJTFCp2N
+tLciWS0YSHPbKTm7svw5mROazUvEoj1N3SYrJT+Ai9UgSL1cA4FmQ5Jsi+GOeQbH
+7E4pzjeST4b1aATY9ZZSy4TzK+UNYBcCVqY1dN5Xnk0e5yPMnrkkcYI41s/6oOaW
+lQkp/yrHkvpzQWZa7vF2yWg54j3eX8YWx/TsLvmuJmSQqiIHfu9TXcAIPDispCpF
+gp4ZbYPZ8RDQLw3ROc+bZ9E0J8e+FHwgXEELsMLW9n4d3PTj9pLFej1vmBKa03o3
+g+JViNxb1gUPNmC6ADk4o7TQzBrjV2pmmuCza16kOmq6oAbJPOeCLtVWA5HJTX8d
+AuOzBLO37tWP457dNqoNDaCMik3LCy8cslZjAz02D6VG/QGOAXP9x8bJ3zg9N7vW
+EOn1BMtIeMI4yIxpC6HgotNWDNbUBWremAFWYQRgYQqo9L1SdW4hz6X8FqdlBFvK
+ml7sdUwjEy2tNW+aiZI4cgAlI/VvMWYr5kPfhvaUu1wGYeS1TD1B+DPcjcXpOGWo
+vB1f/uvGc61zZWD0ph9f3S9OnvEko5qV3VhxOnxYUZTrqlM7LVgy/sHrSaPdR7Eo
+nb4QGdi5wEGMxVgQlDGwOmEmS5nCaA0RcfHbkQV4SebRleocLB9qcSwj5JzPICAB
+sn1qeSv45LnGdTzUnCQErw1EzgrK1FXOknujjZ3GX5dM7a6Gag0E6cAxoQeC8Mh0
+aUWolyBo9aSXltn7Vnywoe4X3v96eVH6tUQtN3wt51LBlfSDTWOQDeXLFyW57oYL
+/ErbmJLaSHpANBcC9InI+rxNYePTTba7K8rdPDppmMEmdOXjaL+5aPay2Bay9K8T
+FUyZdoyrbzxFZYT9/YyVJV/ukVcT9ru3HpzLTg9Vgc/CyRfIvCZY/Gzh+pPUbRsR
+pHYDT09BFS+mjszTkcdbceweYXJvknXpajMiISrPUUkb9rwCTmLS4QR4y7I9zAgJ
+9U72mCZB9yhhzS17kyKAANihXvOFrMXCO7QTWZbtSkobMBS2SybDxYP3rM5b6824
+HQaY4MZW5qCndydfz4fa908XbPTDka9Nu3CZfI3fhGiiGDFWViotGEYE4i9BuGZJ
+vmtzII7zeIoHwM4t5fb5mYDd9H+Rkct1yATGEIFEx7cu1fj4Ht2177EzhdkF3Q6I
+Fccn43yGDW06nFvEjnjb+Lz4EvtGp6dDefUhNBohdzd4Rm2o+etKobm0lZ1xOX74
+2gOGBLfXS6PF+msNkWP76mqKE6hG0+R1iPrN06Arti+ZRBhNGJDp1pnOERl7IyB0
+hiLUqae7xVSMVRDeq4O04EUpMXh5ixs3RJXjthYILt7vX+6qskJC3rMuyThrsvMi
+Kk0wErDpzR5U7KFnM2fk4jY5by/0b/jHZ/PWFGnaFrXNOiwK/mV1RIJtg5FCrvMv
+xJN5Qo7o2ZYzUL+IBf5mP5O49FCnz3e++9r4ySY/K57ybhQft1l+cQPWS7xuvVAg
+wZGM9DPl9AhBus0I3V3XzfkcWWHkMynMRcfTnkBHrmKjMopIOggsq05JPalM9b0V
+NfwZngVvTFbQejbGjgQQQ+G7spav/icCIA8Iu8JPHaKZt2IX77L1xMcQOouqMrx8
+JCXnJ7zsuU+8W3GQGm70H2kcen4bf4beA8TSkFmbVsaqDNtGaG+ZopWO8ctZX3CP
+jFgIYgEGi137IWvs1KD5QNFHZI6l0JMzglmvcfnViRPDHLi+EQXAoL/98j/YzA9t
+a9L4xMO0oucovKkqYOS2cdFaQKVIUM9L5ibcbW7Tol4ugMo5ibIsTgjZoyFohG6t
+acg+h2wTTyxtrIapvOOBeAo9V/iQowTAPQf38LKQZ5tApksiCG1qvJerKALTMp1J
+yUHjTCPCPsaiXVVO0mfa9OPtzVvi/icLmlz7umk82mew13djedgMS3/ghq5aEHbg
+e7rtI5snFlm/GQnV6IaCDytb+jYxqQ0DZ9Mb0CHcMpnLyiBO6nzHfadpZYlrxrpx
+dPx9MVqfLavj1UVjKuVj4Mgec9J38IBBRGMJZ4MGhde2hrw/s2jVzG1BzaIzcvFa
+bWiJOkKKRC4BY9DBDYfae5c/obwzkxLmvkbWAVzoFXc+46K1IK6bbC2a4EerOAxc
+C7B3X8BucD8OtiPORwXgjctd9AJ7cXq0EEVGoZZ8jkILnklsK6RXsxrT4gSOGFpx
+cBU7TF4kHpH+o4KGQmjlFA2VYP1tXirjEep57LOHfL+zy+HiSXdsbexf6odkqLYA
+1RR0TzSkmqJYvP0GfHPMvp2xredx1kP934fhTIUcWvErrG3NPd/Dod4BzQye1nNK
+8Y8aFYF6fACcD4SFZ11ecK9DJmB+WXGlsxtO3w1LeEysny2lVkkH6HuMRIzPyXAM
+/sEpwh3hwPJosAh6vGM6JxuL7CXP8hC1oUvK91VR0hxeDcO/LgdcHtDnlNMqt1wC
+jvYt9Hg+3aY5GehBnQPRppnIMaGhiBrRC6HjBg8Umt6UTUr5PXNY2aia5wRcx4M6
+Tj0O0dAt8WWZ9ySgzOoq4SRoaAp6m4E7CpRNMgXH031OWEeHqtt8rsYqAcvcQLuW
+vIwts5hiXZOfgXyR6SBY3ANE1SjDY46na2G4oiG8FI7eAqYgehjHrXqCnGrzjefE
+V1eVcIr7DBYuM8wq7C9x+qdMir6RNV4MrVD+7rO/drivnd3MaqDHlG0LfMZo6GIK
+QZGIy0z1W5lOXm9x3RAz7/Ea11ynkZyPDDxBE5NRcV6JfukdjDrvsaQA/Qk2Wo7h
+PdUZjv17H4syMdap6tUDTc8Cohut6n1UxnDnhaAYEDGR3TgPqI5vEaJ+yoaM9itv
+b2I/K+0uK6GfWwca7PeTtYn92/5etO4czG99xEBFzw7tiEmBTA8KfseZIUe/i3fr
+9LO+rKEGalFB4mVTYyCKPYYx4hOIvjdCox/jEBTFyYUEhgGnZvArgw9XabvII7eM
+c1FmwnN2CM/MHUzM0Kw7cS/oaoRFcG/PqioO/krL2Bd+xThwSH4Zn+Y1UVO7aXNy
+90MOehHSc2nhN984Y2re3ffVsjPhC3KdWxr8jbho/Az65rUkYg64roiO2ZqdqRvf
+aYtX9fCe8N8SzgLTlWZt+l8sjY8HP2k9NoajphV1nG4iC/pX0ZJiVQ+VBRNaGhSl
+6IhM9gM1ZsI86cULJpOTAhNu1fZBv+Mq8IXYD8SqlNVwVtjvshtlzBN4InSGEKiO
+HM7coo7aDuu+Go5AEiIVMbR09jAa0VINsCVScfSbTFhlGRTrZr2DSO/s8WxbV+Mb
+EQAfZmyrLF71Z3MfIukg+ps5iuqFhZ+gJSXXSFwumpg=
+EOF
+
+exec sh bin/do-setup-circleci-secrets

build/Dockerfile

@@ -0,0 +1,21 @@
+FROM ubuntu
+
+RUN apt-get -y update && apt-get -y install --no-install-recommends build-essential git ca-certificates docker.io mercurial libpcap-dev curl make pkg-config gcc bison flex
+
+# When doing a build in a container, "apt-get update" happens twice,
+# which can be a very significant overhead for incremental builds.
+# And it's unnecessary, because if any of the prerequisites do change,
+# the right thing to do is to rebuild the container image.  Rather
+# than suppressing the "apt-get update" commands explicitly, we clear
+# out sources.list so that 'apt-get update' doesn't do anything.
+RUN echo >/etc/apt/sources.list
+
+ENV GO_VERSION 1.4.2
+RUN curl -sSL https://golang.org/dl/go${GO_VERSION}.linux-amd64.tar.gz | tar -C /usr/local -xz
+ENV PATH /usr/local/go/bin:$PATH
+
+# So that sudo does not trample on PATH
+RUN sed -ie '/secure_path/d' /etc/sudoers
+
+COPY build.sh /
+ENTRYPOINT ["sh", "/build.sh"]

build/build.sh

@@ -0,0 +1,44 @@
+#!/bin/sh
+set -e
+
+GOPATH=/home/go
+export GOPATH
+
+WEAVE_SRC=$GOPATH/src/github.com/weaveworks/weave
+
+if [ $# -eq 0 ] ; then
+    # No arguments.  Expect that the weave repo will be bind-mounted
+    # into $GOPATH
+    if ! [ -e $WEAVE_SRC ] ; then
+        cat 2>&1 <<EOF
+No container arguments supplied, and nothing at ${WEAVE_SRC}.  Please
+either bind-mount the golang workspace containing weave with the
+docker run -v option, e.g.:
+
+    $ docker run -v <host gopath>:${GOPATH} \\
+          -v /var/run/docker.sock:/var/run/docker.sock weaveworks/weave-build
+
+Or supply git clone arguments to retrieve it, e.g.:
+
+    $ docker run -v /var/run/docker.sock:/var/run/docker.sock \\
+          weaveworks/weave-build https://github.com/weaveworks/weave.git
+EOF
+        exit 1
+    fi
+
+    # If we run make directly, any files created on the bind mount
+    # will have awkward ownership.  So we switch to a user with the
+    # same user and group IDs as source directory.  We have to set a
+    # few things up so that sudo works without complaining later on.
+    uid=$(stat --format="%u" $WEAVE_SRC)
+    gid=$(stat --format="%g" $WEAVE_SRC)
+    echo "weave:x:$uid:$gid::$WEAVE_SRC:/bin/sh" >>/etc/passwd
+    echo "weave:*:::::::" >>/etc/shadow
+    echo "weave	ALL=(ALL)	NOPASSWD: ALL" >>/etc/sudoers
+    su weave -c "PATH=$PATH make -C $WEAVE_SRC build"
+else
+    # There are arguments to pass to git-clone
+    mkdir -p ${WEAVE_SRC%/*}
+    git clone "$@" $WEAVE_SRC
+    make -C $WEAVE_SRC build
+fi

circle.yml

@@ -0,0 +1,46 @@
+general:
+  branches:
+    ignore:
+      - gh\-pages
+      - /pull\/.*/
+
+machine:
+  services:
+    - docker
+  environment:
+    GOPATH: /home/ubuntu:$GOPATH
+    SRCDIR: /home/ubuntu/src/github.com/weaveworks/weave
+    PATH: $PATH:$HOME/.local/bin:$HOME/google-cloud-sdk/bin/
+    CLOUDSDK_CORE_DISABLE_PROMPTS: 1
+    WEAVE_BUILD: $HOME/docker/weave-build.tar
+
+dependencies:
+  cache_directories:
+    - "~/docker"
+  override:
+    - if [[ -e "$WEAVE_BUILD" ]]; then
+        docker load -i $WEAVE_BUILD;
+      else
+        docker pull weaveworks/weave-build;
+        mkdir -p $(dirname "$WEAVE_BUILD");
+        docker save weaveworks/weave-build >$WEAVE_BUILD;
+      fi
+  post:
+    - curl https://sdk.cloud.google.com | bash
+    - bin/setup-circleci-secrets "$SECRET_PASSWORD"
+    - mkdir -p $(dirname $SRCDIR)
+    - cp -r $(pwd)/ $SRCDIR
+
+test:
+  override:
+    - docker run -v /var/run/docker.sock:/run/docker.sock -v /home/ubuntu:/home/go weaveworks/weave-build:
+        parallel: true
+    - cd $SRCDIR/test; ./gce.sh setup:
+        parallel: true
+    - cd $SRCDIR/test; . ./gce.sh hosts; ./setup.sh:
+        parallel: true
+    - cd $SRCDIR/test; . ./gce.sh hosts; ./run_all.sh:
+        parallel: true
+  post:
+    - cd $SRCDIR/test; ./gce.sh destroy:
+        parallel: true

common/errors.go

@@ -0,0 +1,20 @@
+package common
+
+func CheckFatal(e error) {
+	if e != nil {
+		Error.Fatal(e)
+	}
+}
+
+func CheckWarn(e error) {
+	if e != nil {
+		Warning.Println(e)
+	}
+}
+
+// Assert test is true, panic otherwise
+func Assert(test bool) {
+	if !test {
+		panic("Assertion failure")
+	}
+}

common/signals.go

@@ -0,0 +1,37 @@
+package common
+
+import (
+	"os"
+	"os/signal"
+	"runtime"
+	"syscall"
+)
+
+// A subsystem/server/... that can be stopped or queried about the status with a signal
+type SignalReceiver interface {
+	Stop() error
+	Status() string
+}
+
+func SignalHandlerLoop(ss ...SignalReceiver) {
+	sigs := make(chan os.Signal, 1)
+	signal.Notify(sigs, syscall.SIGINT, syscall.SIGQUIT, syscall.SIGUSR1)
+	buf := make([]byte, 1<<20)
+	for {
+		switch <-sigs {
+		case syscall.SIGINT:
+			Info.Printf("=== received SIGINT ===\n*** exiting\n")
+			for _, subsystem := range ss {
+				subsystem.Stop()
+			}
+			return
+		case syscall.SIGQUIT:
+			stacklen := runtime.Stack(buf, true)
+			Info.Printf("=== received SIGQUIT ===\n*** goroutine dump...\n%s\n*** end\n", buf[:stacklen])
+		case syscall.SIGUSR1:
+			for _, subsystem := range ss {
+				Info.Printf("=== received SIGUSR1 ===\n*** status...\n%s\n*** end\n", subsystem.Status())
+			}
+		}
+	}
+}

common/testing_utils.go

@@ -1,51 +0,0 @@
-package common
-
-import (
-	"fmt"
-	"reflect"
-	"runtime"
-	"strings"
-	"testing"
-)
-
-func CallSite() string {
-	_, file, line, ok := runtime.Caller(2)
-	if ok {
-		// Truncate file name at last file name separator.
-		if index := strings.LastIndex(file, "/"); index >= 0 {
-			file = file[index+1:]
-		} else if index = strings.LastIndex(file, "\\"); index >= 0 {
-			file = file[index+1:]
-		}
-	} else {
-		file = "???"
-		line = 1
-	}
-	return fmt.Sprintf("%s:%d: ", file, line)
-}
-
-func AssertNoErr(t *testing.T, err error) {
-	if err != nil {
-		t.Fatal(CallSite(), err)
-	}
-}
-
-func AssertStatus(t *testing.T, got int, wanted int, desc string) {
-	if got != wanted {
-		t.Fatalf("%s: Expected %s %d but got %d", CallSite(), desc, wanted, got)
-	}
-}
-
-func AssertErrorInterface(t *testing.T, got interface{}, wanted interface{}, desc string) {
-	gotT, wantedT := reflect.TypeOf(got), reflect.TypeOf(wanted).Elem()
-	if !gotT.Implements(wantedT) {
-		t.Fatalf("%s: Expected %s but got %s (%s)", CallSite(), wantedT.String(), gotT.String(), desc)
-	}
-}
-
-func AssertErrorType(t *testing.T, got interface{}, wanted interface{}, desc string) {
-	gotT, wantedT := reflect.TypeOf(got), reflect.TypeOf(wanted).Elem()
-	if gotT != wantedT {
-		t.Fatalf("%s: Expected %s but got %s (%s)", CallSite(), wantedT.String(), gotT.String(), desc)
-	}
-}

common/updater/updater.go

@@ -0,0 +1,47 @@
+package updater
+
+import (
+	"github.com/fsouza/go-dockerclient"
+	. "github.com/weaveworks/weave/common"
+)
+
+type ContainerObserver interface {
+	ContainerDied(ident string) error
+}
+
+func checkError(err error, apiPath string) {
+	if err != nil {
+		Error.Fatalf("[updater] Unable to connect to Docker API on %s: %s",
+			apiPath, err)
+	}
+}
+
+func Start(apiPath string, ob ContainerObserver) error {
+	client, err := docker.NewClient(apiPath)
+	checkError(err, apiPath)
+
+	env, err := client.Version()
+	checkError(err, apiPath)
+
+	events := make(chan *docker.APIEvents)
+	err = client.AddEventListener(events)
+	checkError(err, apiPath)
+
+	Info.Printf("[updater] Using Docker API on %s: %v", apiPath, env)
+
+	go func() {
+		for event := range events {
+			handleEvent(ob, event, client)
+		}
+	}()
+	return nil
+}
+
+func handleEvent(ob ContainerObserver, event *docker.APIEvents, client *docker.Client) error {
+	switch event.Status {
+	case "die":
+		id := event.ID
+		ob.ContainerDied(id)
+	}
+	return nil
+}

docs/architecture.txt

@@ -83,13 +83,17 @@ Router:
     local peer. It will tell the local connection (communicating to
     the actor thread) about the UDP address of the remote peer. The
     local connection will then start its forwarder threads as
-    described in 6a, and start sending slow heartbeats. The local connection
-    is marked as established which means it is included in network updates
-    broadcast to our peers. We also send to the remote peer via TCP a
-    ConnectionEstablished message. The remote peer receives this (on
-    the TCP receiver process), tells the connection actor process,
-    which then replaces the fast heartbeater with a slow heartbeater
-    and similarly marks the connection as established.
+    described in 6a, and start sending fast heartbeats. We send to the
+    remote peer via TCP a ConnectionEstablished message. The remote
+    peer receives this (on the TCP receiver process), tells the
+    connection actor process, which then replaces the fast heartbeater
+    with a slow heartbeater and marks the connection as established
+    (which means it is included in network updates broadcast to our
+    peers).
+  6c. When the connection initiator receives the fast heartbeat from
+    the remote peer it sends to the remote peer via a TCP a
+    ConnectionEstablished message. This is handled by the remote peer
+    as described in 6b.
   7. Whenever a connection is established or terminated, the local
     peer's version is incremented. Whenever this happens, the peer
     generates a network update message which is broadcast to its
@@ -156,7 +160,7 @@ if a node receives a packet that is bigger than the next MTU, it is
 required to drop the packet and send back an ICMP 3,4 packet which
 informs the sending node of the next hop MTU (RFC 1191). In theory,
 these ICMP packets should have all the reverse NAT and so forth
-applied, to make it all the way back to the sender. In practise this
+applied, to make it all the way back to the sender. In practice, this
 does often work, but in some cases all ICMP are sometimes blocked,
 often by idiots who don't understand that ICMP is really the error
 channel of all network traffic.
@@ -183,16 +187,16 @@ If weave determines that fragmentation is reliable then when weave
 sniffs large packets (i.e. packets larger than the MTU and thus don't
 have DF set) then weave will encapsulate these packets as necessary
 and similarly send them out as-is, without DF set, trusting to the
-network to do all necessary fragementation and reassembly.
+network to do all necessary fragmentation and reassembly.
 
-However, if weave determines that fragementation is not reliable
+However, if weave determines that fragmentation is not reliable
 between any two peers then it will manually fragment larger packets
 correctly according to the IP spec, and will then send them between
 weave peers with the DF flag set (i.e. the fragmentation will ensure
-that the encapsulatated traffic will not be greater than the PMTU
+that the encapsulated traffic will not be greater than the PMTU
 between weave peers). Because the fragmentation is done according to
 the IP spec, we don't need to do reassembly ourselves - on the
-receiving weave, we jus inject all the fragments and rely on the
+receiving weave, we just inject all the fragments and rely on the
 stack to do reassembly.
 
 Sometimes PMTU discovery doesn't work as ICMP packets sometimes get

docs/dockerless.txt

@@ -126,7 +126,7 @@ If all is well, you should now be able to launch weave on VM1
 then on VM2:
 
 VM1> weaver -iface veth6 -name vm1
-VM2> weaver -iface veth6 -name vm2 192.168.122.1
+VM2> weaver -iface veth6 -name vm2 192.168.122.2
 
 Once the connection is up and running, you should be able to connect
 from VM1/ns2 to VM2/ns2 and vice versa:

docs/ipam.md

@@ -0,0 +1,214 @@
+See the [requirements](https://github.com/zettio/weave/wiki/IP-allocation-requirements).
+
+At its highest level, the idea is that we start with a certain IP
+address space, known to all peers, and divide it up between the
+peers. This allows peers to allocate and free individual IPs locally
+until they run out.
+
+We use a CRDT to represent shared knowledge about the space,
+transmitted over the Weave Gossip mechanism, together with
+point-to-point messages for one peer to request more space from
+another.
+
+The allocator running at each peer also has an http interface which
+the container infrastructure (e.g. the Weave script, or a Docker
+plugin) uses to request IP addresses.
+
+![Schematic of IP allocation in operation](https://docs.google.com/drawings/d/1-EUIRKYxwfKTpBJ7v_LMcdvSpodIMSz4lT3wgEfWKl4/pub?w=701&h=310)
+
+## Commands
+
+The commands supported by the allocator via the http interface are:
+
+- Allocate: request one IP address for a container
+- Free: return an IP address that is currently allocated
+- Claim: request a specific IP address for a container (e.g. because
+  it is already using that address)
+
+The allocator also watches via the Docker event mechanism: if a
+container dies then all IP addresses allocated to that container are
+freed.
+
+## Definitions
+
+1. Allocations. We use the word 'allocation' to refer to a specific
+   IP address being assigned, e.g. so it can be given to a container.
+
+2. Range. Most of the time, instead of dealing with individual IP
+   addresses, we operate on them in contiguous groups, for which we
+   use the word "range".
+
+3. Ring. We consider the address space as a ring, so ranges wrap
+   around from the highest address to the lowest address.
+
+4. Peer. A Peer is a node on the Weave network. It can own zero or
+   more ranges.
+
+### The Allocation Process
+
+When a peer owns some range(s), it can allocate freely from within
+those ranges to containers on the same machine. If it runs out of
+space (all owned ranges are full), it will ask another peer for space:
+  - it picks a peer to ask at random, weighted by the amount of space
+    owned by each peer
+    - if the target peer decides to give up space, it unicasts a message
+      back to the asker with the newly-updated ring.
+    - if the target peer has no space, it unicasts a message back to the
+      asker with its current copy of the ring, on the basis that the
+      requestor must have acted on out-of-date information.
+  - it will continue to ask for space until it receives some, or its
+    copy of the ring tells it all peers are full.
+
+### Claiming an address
+
+If a Weave process is restarted, in most cases it will hear from
+another peer which ranges it used to own, but it needs to know which
+individual IP addresses are assigned to containers in order to avoid
+giving the same address out in subsequent allocation requests. The
+weave script invokes the `claim` command in `populate_ipam` to do
+this.
+
+When the Allocator is told to claim an address, there are four
+possibilities:
+  - the address is outside of the space managed by this Allocator, in
+    which case we ignore the request.
+  - the address is owned by this peer, in which case we record the
+    address as being assigned to a particular container and return
+    success.
+  - the address is owned by a different peer, in which case we return
+    failure.
+  - we have not yet heard of any address ranges being owned by anyone,
+    in which case we wait until we do hear.
+
+This approach fails if the peer does not hear from another peer about
+the ranges it used to own, e.g. if all peers in a network partition
+are restarted at once.
+
+### The Ring CRDT
+
+We use a Convergent Replicated Data Type - a CRDT - so that peers can
+make changes concurrently and communicate them without central
+coordination. To achieve this, we arrange that peers only make changes
+to the data structure in ranges that they own (except under
+administrator command - see later).
+
+The data structure is a set of tokens, each containing the name of an
+owning peer. A peer can own many ranges. Each token is placed at the
+start address of a range, and the set is kept ordered so each range
+goes from one token to the next. Each range on the ring includes the
+start address but does not include the end address (which is the start
+of the next range).  Ranges wrap, so the 'next' token after the last
+one is the first token.
+
+![Tokens on the Ring](https://docs.google.com/drawings/d/1hp--q2vmxbBAnPjhza4Kqjr1ugrw2iS1M1GerhH-IKY/pub?w=960&h=288)
+
+In more detail:
+- Each token is a tuple {peer name, version}, placed
+  at an IP address.
+- Peer names are taken from Weave: they are unique and survive across restarts.
+- The contents of a token can only be updated by the owning peer, and
+  when this is done the version is incremented
+- The ring data structure is always gossiped in its entirety
+- The merge operation when a peer receives a ring via gossip is:
+  - Tokens with unique addresses are just copied into the combined ring
+  - For tokens at the same address, pick the one with the highest
+    version number
+- The data gossiped about the ring also includes the amount of free
+  space in each range: this is not essential but it improves the
+  selection of which peer to ask for space.
+- When a peer is asked for space, there are four scenarios:
+  1. It has an empty range; it can change the peer associated with
+     the token at the beginning of the range and increment the version.
+  2. It has a range which can be subdivided by a single token to form
+     a free range.  It inserts said token, owned by the peer requesting
+     space.
+  3. It has a 'hole' in the middle of a range; an empty range can be
+     created by inserting two tokens, one at the beginning of the hole
+     owned by the peer requesting the space, and one at the end of the
+     hole owned by the requestee.
+  4. It has no space.
+
+## Initialisation
+
+The previous sections describe how peers coordinate changes to the
+ring.  But how is the initial state of the ring established?  If a new
+peer joins a long-standing cluster, it can learn about the ring state
+from other peers.  But in a freshly started cluster, the initial ring
+state must be established from a clean slate.  And it must be
+consistent across all peers.  This is a distributed consensus problem,
+and we solve it using the Paxos algorithm.
+
+Although Paxos has a reputation for being hard to understand and to
+implement, the implementation used for ring initialization is
+relatively straightforward:
+
+- We only need to establish consensus for a single value, rather than
+  a succession of values (as in a replicated transaction log).  Thus
+  we implement basic Paxos, rather than multi-Paxos, and the
+  implementation closely follows the outline of basic Paxos described
+  [on
+  wikipedia](http://en.wikipedia.org/wiki/Paxos_%28computer_science%29#Basic_Paxos).
+
+- All peers play all the roles described in basic Paxos: Proposer,
+  acceptor and listener.
+
+- Paxos is usually described in terms of unicast messages from one
+  agent to another, although three of the four messages are fanned out
+  to all agents in a certain role ("prepare" and "accept request" from
+  a proposer to all acceptors; "accepted" from an acceptor to all
+  learners).  So most implementations involve a communications layer
+  to implement the required communication patterns.  But our
+  implementation is built on top of the weave gossip layer, which
+  gives us a broadcast medium.  Using broadcast for the one truly
+  unicast message ("promise") may seem wasteful, but as we only
+  establish consensus for a single value, it is not a major concern.
+
+The value the peers obtain consensus on is the set of peers that will
+be represented in the initial ring (with each peer getting an equal
+share of the address space).  When a proposing peer gets to choose the
+value, it includes all the peers it has heard from during the Paxos
+phase, which is at least a quorum.
+
+Once a consensus is reached, it is used to initialize the ring.  If a
+peer hears about an initialized ring via gossip, that implies that a
+consensus was reached, so it will stop participating in Paxos.
+
+Paxos requires that we know the quorum size for the cluster.  As weave
+clusters are a dynamic entity, there is no fixed quorum size.  But the
+Paxos phase used to initialize the ring should normally be
+short-lived, so we just need to know the initial cluster size.  A
+heuristic is used to derive this from the list of addresses passed to
+`weave launch`, but the user can set it explicitly.
+
+## Peer shutdown
+
+When a peer leaves (a `weave reset` command), it grants all its own
+tokens to another peer, then broadcasts the updated ring.
+
+After sending the message, the peer terminates - it does not wait for
+any response.
+
+Failures:
+- message lost
+  - the space will be unusable by other peers because it will still be
+    seen as owned.
+
+To cope with the situation where a peer has left or died without
+managing to tell its peers, an administrator may go to any other peer
+and command that it take over the dead peer's tokens (with `weave
+rmpeer`).  This information will then be gossipped out to the network.
+
+(This operation is not guaranteed to be safe - if the dead peer made
+some transfers which have are known to other peers but have not
+reached the peer that does the `rmpeer`, then we get an inconsistent
+ring.  We could make it safe, up to partitions, by inquiring of every
+live peer what its view of the dead peer is, and making sure we use
+the latest info.)
+
+## Limitations
+
+Nothing is persisted. If one peer restarts it should receive gossip
+from other peers showing what it used to own, and in that case it can
+recover quite well. If, however, there are no peers left alive, or
+this peer cannot establish network communication with those that are,
+then it cannot recover.

ipam/address/address.go

@@ -0,0 +1,61 @@
+package address
+
+import (
+	"github.com/weaveworks/weave/common"
+	"net"
+)
+
+// Using 32-bit integer to represent IPv4 address
+type Address uint32
+type Offset uint32
+
+type Range struct {
+	Start, End Address // [Start, End); Start <= End
+}
+
+func ParseIP(s string) (Address, error) {
+	if ip := net.ParseIP(s); ip == nil {
+		return 0, &net.ParseError{Type: "IP Address", Text: s}
+	} else {
+		return FromIP4(ip), nil
+	}
+}
+
+// FromIP4 converts an ipv4 address to our integer address type
+func FromIP4(ip4 net.IP) (r Address) {
+	for _, b := range ip4.To4() {
+		r <<= 8
+		r |= Address(b)
+	}
+	return
+}
+
+// IP4 converts our integer address type to an ipv4 address
+func (addr Address) IP4() (r net.IP) {
+	r = make([]byte, net.IPv4len)
+	for i := 3; i >= 0; i-- {
+		r[i] = byte(addr)
+		addr >>= 8
+	}
+	return
+}
+
+func (addr Address) String() string {
+	return addr.IP4().String()
+}
+
+func Add(addr Address, i Offset) Address {
+	return addr + Address(i)
+}
+
+func Subtract(a, b Address) Offset {
+	common.Assert(a >= b)
+	return Offset(a - b)
+}
+
+func Min(a, b Offset) Offset {
+	if a > b {
+		return b
+	}
+	return a
+}

ipam/allocate.go

@@ -0,0 +1,58 @@
+package ipam
+
+import (
+	"fmt"
+	"github.com/weaveworks/weave/ipam/address"
+)
+
+type allocateResult struct {
+	addr address.Address
+	err  error
+}
+
+type allocate struct {
+	resultChan       chan<- allocateResult
+	hasBeenCancelled func() bool
+	ident            string
+}
+
+// Try returns true if the request is completed, false if pending
+func (g *allocate) Try(alloc *Allocator) bool {
+	if g.hasBeenCancelled() {
+		g.Cancel()
+		return true
+	}
+
+	// If we have previously stored an address for this container, return it.
+	if addr, found := alloc.owned[g.ident]; found {
+		g.resultChan <- allocateResult{addr, nil}
+		return true
+	}
+
+	if ok, addr := alloc.space.Allocate(); ok {
+		alloc.debugln("Allocated", addr, "for", g.ident)
+		alloc.addOwned(g.ident, addr)
+		g.resultChan <- allocateResult{addr, nil}
+		return true
+	}
+
+	// out of space
+	if donor, err := alloc.ring.ChoosePeerToAskForSpace(); err == nil {
+		alloc.debugln("Decided to ask peer", donor, "for space")
+		alloc.sendRequest(donor, msgSpaceRequest)
+	}
+
+	return false
+}
+
+func (g *allocate) Cancel() {
+	g.resultChan <- allocateResult{0, fmt.Errorf("Allocate request for %s cancelled", g.ident)}
+}
+
+func (g *allocate) String() string {
+	return fmt.Sprintf("Allocate for %s", g.ident)
+}
+
+func (g *allocate) ForContainer(ident string) bool {
+	return g.ident == ident
+}

ipam/allocator.go

@@ -0,0 +1,696 @@
+package ipam
+
+import (
+	"bytes"
+	"encoding/gob"
+	"errors"
+	"fmt"
+	"net"
+	"sort"
+	"time"
+
+	"github.com/weaveworks/weave/common"
+	"github.com/weaveworks/weave/ipam/address"
+	"github.com/weaveworks/weave/ipam/paxos"
+	"github.com/weaveworks/weave/ipam/ring"
+	"github.com/weaveworks/weave/ipam/space"
+	"github.com/weaveworks/weave/router"
+)
+
+// Kinds of message we can unicast to other peers
+const (
+	msgSpaceRequest = iota
+	msgRingUpdate
+
+	paxosInterval = time.Second * 5
+)
+
+// operation represents something which Allocator wants to do, but
+// which may need to wait until some other message arrives.
+type operation interface {
+	// Try attempts this operations and returns false if needs to be tried again.
+	Try(alloc *Allocator) bool
+
+	Cancel()
+
+	String() string
+
+	// Does this operation pertain to the given container id?
+	// Used for tidying up pending operations when containers die.
+	ForContainer(ident string) bool
+}
+
+// Allocator brings together Ring and space.Set, and does the
+// necessary plumbing.  Runs as a single-threaded Actor, so no locks
+// are used around data structures.
+type Allocator struct {
+	actionChan       chan<- func()
+	ourName          router.PeerName
+	subnetStart      address.Address            // start address of space all peers are allocating from
+	subnetSize       address.Offset             // length of space all peers are allocating from
+	prefixLen        int                        // network prefix length, e.g. 24 for a /24 network
+	ring             *ring.Ring                 // information on ranges owned by all peers
+	space            space.Space                // more detail on ranges owned by us
+	owned            map[string]address.Address // who owns what address, indexed by container-ID
+	nicknames        map[router.PeerName]string // so we can map nicknames for rmpeer
+	pendingAllocates []operation                // held until we get some free space
+	pendingClaims    []operation                // held until we know who owns the space
+	gossip           router.Gossip              // our link to the outside world for sending messages
+	paxos            *paxos.Node
+	paxosTicker      *time.Ticker
+	shuttingDown     bool // to avoid doing any requests while trying to shut down
+	now              func() time.Time
+}
+
+// NewAllocator creates and initialises a new Allocator
+func NewAllocator(ourName router.PeerName, ourUID router.PeerUID, ourNickname string, subnetCIDR string, quorum uint) (*Allocator, error) {
+	_, subnet, err := net.ParseCIDR(subnetCIDR)
+	if err != nil {
+		return nil, err
+	}
+	if subnet.IP.To4() == nil {
+		return nil, errors.New("Non-IPv4 address not supported")
+	}
+	// Get the size of the network from the mask
+	ones, bits := subnet.Mask.Size()
+	var subnetSize address.Offset = 1 << uint(bits-ones)
+	if subnetSize < 4 {
+		return nil, errors.New("Allocation subnet too small")
+	}
+	subnetStart := address.FromIP4(subnet.IP)
+	alloc := &Allocator{
+		ourName:     ourName,
+		subnetStart: subnetStart,
+		subnetSize:  subnetSize,
+		prefixLen:   ones,
+		// per RFC 1122, don't allocate the first and last address in the subnet
+		ring:      ring.New(address.Add(subnetStart, 1), address.Add(subnetStart, subnetSize-1), ourName),
+		owned:     make(map[string]address.Address),
+		paxos:     paxos.NewNode(ourName, ourUID, quorum),
+		nicknames: map[router.PeerName]string{ourName: ourNickname},
+		now:       time.Now,
+	}
+	return alloc, nil
+}
+
+// Start runs the allocator goroutine
+func (alloc *Allocator) Start() {
+	actionChan := make(chan func(), router.ChannelSize)
+	alloc.actionChan = actionChan
+	go alloc.actorLoop(actionChan)
+}
+
+// Stop makes the actor routine exit, for test purposes ONLY because any
+// calls after this is processed will hang. Async.
+func (alloc *Allocator) Stop() {
+	alloc.actionChan <- nil
+}
+
+// Operation life cycle
+
+// Given an operation, try it, and add it to the pending queue if it didn't succeed
+func (alloc *Allocator) doOperation(op operation, ops *[]operation) {
+	alloc.actionChan <- func() {
+		if alloc.shuttingDown {
+			op.Cancel()
+			return
+		}
+		alloc.establishRing()
+		if !op.Try(alloc) {
+			*ops = append(*ops, op)
+		}
+	}
+}
+
+// Given an operation, remove it from the pending queue
+//  Note the op may not be on the queue; it may have
+//  already succeeded.  If it is on the queue, we call
+//  cancel on it, allowing callers waiting for the resultChans
+//  to unblock.
+func (alloc *Allocator) cancelOp(op operation, ops *[]operation) {
+	for i, op := range *ops {
+		if op == op {
+			*ops = append((*ops)[:i], (*ops)[i+1:]...)
+			op.Cancel()
+			break
+		}
+	}
+}
+
+// Cancel all operations in a queue
+func (alloc *Allocator) cancelOps(ops *[]operation) {
+	for _, op := range *ops {
+		op.Cancel()
+	}
+	*ops = []operation{}
+}
+
+// Cancel all operations for a given container id, returns true
+// if we found any.
+func (alloc *Allocator) cancelOpsFor(ops *[]operation, ident string) bool {
+	var found bool
+	for i, op := range *ops {
+		if op.ForContainer(ident) {
+			found = true
+			op.Cancel()
+			*ops = append((*ops)[:i], (*ops)[i+1:]...)
+		}
+	}
+	return found
+}
+
+// Try all pending operations
+func (alloc *Allocator) tryPendingOps() {
+	// The slightly different semantics requires us to operate on 'claims' and
+	// 'allocates' separately:
+	// Claims must be tried before Allocates
+	for i := 0; i < len(alloc.pendingClaims); {
+		op := alloc.pendingClaims[i]
+		if !op.Try(alloc) {
+			i++
+			continue
+		}
+		alloc.pendingClaims = append(alloc.pendingClaims[:i], alloc.pendingClaims[i+1:]...)
+	}
+
+	// When the first Allocate fails, bail - no need to
+	// send too many begs for space.
+	for i := 0; i < len(alloc.pendingAllocates); {
+		op := alloc.pendingAllocates[i]
+		if !op.Try(alloc) {
+			break
+		}
+		alloc.pendingAllocates = append(alloc.pendingAllocates[:i], alloc.pendingAllocates[i+1:]...)
+	}
+}
+
+func hasBeenCancelled(cancelChan <-chan bool) func() bool {
+	return func() bool {
+		select {
+		case <-cancelChan:
+			return true
+		default:
+			return false
+		}
+	}
+}
+
+// Actor client API
+
+// Allocate (Sync) - get IP address for container with given name
+// if there isn't any space we block indefinitely
+func (alloc *Allocator) Allocate(ident string, cancelChan <-chan bool) (address.Address, error) {
+	resultChan := make(chan allocateResult)
+	op := &allocate{resultChan: resultChan, ident: ident,
+		hasBeenCancelled: hasBeenCancelled(cancelChan)}
+	alloc.doOperation(op, &alloc.pendingAllocates)
+	result := <-resultChan
+	return result.addr, result.err
+}
+
+// Claim an address that we think we should own (Sync)
+func (alloc *Allocator) Claim(ident string, addr address.Address, cancelChan <-chan bool) error {
+	resultChan := make(chan error)
+	op := &claim{resultChan: resultChan, ident: ident, addr: addr,
+		hasBeenCancelled: hasBeenCancelled(cancelChan)}
+	alloc.doOperation(op, &alloc.pendingClaims)
+	return <-resultChan
+}
+
+// Free (Sync) - release IP address for container with given name
+func (alloc *Allocator) Free(ident string) error {
+	return alloc.free(ident)
+}
+
+// ContainerDied is provided to satisfy the updater interface; does a free underneath.  Async.
+func (alloc *Allocator) ContainerDied(ident string) error {
+	alloc.debugln("Container", ident, "died; releasing addresses")
+	return alloc.free(ident)
+}
+
+func (alloc *Allocator) free(ident string) error {
+	errChan := make(chan error)
+	alloc.actionChan <- func() {
+		addr, found := alloc.owned[ident]
+		if found {
+			alloc.space.Free(addr)
+		}
+		delete(alloc.owned, ident)
+
+		// Also remove any pending ops
+		found = alloc.cancelOpsFor(&alloc.pendingAllocates, ident) || found
+		found = alloc.cancelOpsFor(&alloc.pendingClaims, ident) || found
+
+		if !found {
+			errChan <- fmt.Errorf("Free: no addresses for %s", ident)
+			return
+		}
+		errChan <- nil
+	}
+	return <-errChan
+}
+
+// Sync.
+func (alloc *Allocator) String() string {
+	resultChan := make(chan string)
+	alloc.actionChan <- func() {
+		resultChan <- alloc.string()
+	}
+	return <-resultChan
+}
+
+// Shutdown (Sync)
+func (alloc *Allocator) Shutdown() {
+	alloc.infof("Shutdown")
+	doneChan := make(chan struct{})
+	alloc.actionChan <- func() {
+		alloc.shuttingDown = true
+		alloc.cancelOps(&alloc.pendingClaims)
+		alloc.cancelOps(&alloc.pendingAllocates)
+		if heir := alloc.ring.PickPeerForTransfer(); heir != router.UnknownPeerName {
+			alloc.ring.Transfer(alloc.ourName, heir)
+			alloc.space.Clear()
+			alloc.gossip.GossipBroadcast(alloc.Gossip())
+			time.Sleep(100 * time.Millisecond)
+		}
+		doneChan <- struct{}{}
+	}
+	<-doneChan
+}
+
+// AdminTakeoverRanges (Sync) - take over the ranges owned by a given peer.
+// Only done on adminstrator command.
+func (alloc *Allocator) AdminTakeoverRanges(peerNameOrNickname string) error {
+	resultChan := make(chan error)
+	alloc.actionChan <- func() {
+		peername, err := alloc.lookupPeername(peerNameOrNickname)
+		if err != nil {
+			resultChan <- fmt.Errorf("Cannot find peer '%s'", peerNameOrNickname)
+			return
+		}
+
+		alloc.debugln("AdminTakeoverRanges:", peername)
+		if peername == alloc.ourName {
+			resultChan <- fmt.Errorf("Cannot take over ranges from yourself!")
+			return
+		}
+
+		delete(alloc.nicknames, peername)
+		err, newRanges := alloc.ring.Transfer(peername, alloc.ourName)
+		alloc.space.AddRanges(newRanges)
+		resultChan <- err
+	}
+	return <-resultChan
+}
+
+// Lookup a PeerName by nickname or stringified PeerName.  We can't
+// call into the router for this because we are interested in peers
+// that have gone away but are still in the ring, which is why we
+// maintain our own nicknames map.
+func (alloc *Allocator) lookupPeername(name string) (router.PeerName, error) {
+	for peername, nickname := range alloc.nicknames {
+		if nickname == name {
+			return peername, nil
+		}
+	}
+
+	return router.PeerNameFromString(name)
+}
+
+// Restrict the peers in "nicknames" to those in the ring and our own
+func (alloc *Allocator) pruneNicknames() {
+	ringPeers := alloc.ring.PeerNames()
+	for name, _ := range alloc.nicknames {
+		if _, ok := ringPeers[name]; !ok && name != alloc.ourName {
+			delete(alloc.nicknames, name)
+		}
+	}
+}
+
+// OnGossipUnicast (Sync)
+func (alloc *Allocator) OnGossipUnicast(sender router.PeerName, msg []byte) error {
+	alloc.debugln("OnGossipUnicast from", sender, ": ", len(msg), "bytes")
+	resultChan := make(chan error)
+	alloc.actionChan <- func() {
+		switch msg[0] {
+		case msgSpaceRequest:
+			// some other peer asked us for space
+			alloc.donateSpace(sender)
+			resultChan <- nil
+		case msgRingUpdate:
+			resultChan <- alloc.update(msg[1:])
+		}
+	}
+	return <-resultChan
+}
+
+// OnGossipBroadcast (Sync)
+func (alloc *Allocator) OnGossipBroadcast(msg []byte) (router.GossipData, error) {
+	alloc.debugln("OnGossipBroadcast:", len(msg), "bytes")
+	resultChan := make(chan error)
+	alloc.actionChan <- func() {
+		resultChan <- alloc.update(msg)
+	}
+	return alloc.Gossip(), <-resultChan
+}
+
+type gossipState struct {
+	// We send a timstamp along with the information to be
+	// gossipped in order to do detect skewed clocks
+	Now       int64
+	Nicknames map[router.PeerName]string
+
+	Paxos paxos.GossipState
+	Ring  *ring.Ring
+}
+
+func (alloc *Allocator) encode() []byte {
+	data := gossipState{
+		Now:       alloc.now().Unix(),
+		Nicknames: alloc.nicknames,
+	}
+
+	// We're only interested in Paxos until we have a Ring.
+	if alloc.ring.Empty() {
+		data.Paxos = alloc.paxos.GossipState()
+	} else {
+		data.Ring = alloc.ring
+	}
+	buf := new(bytes.Buffer)
+	enc := gob.NewEncoder(buf)
+	if err := enc.Encode(data); err != nil {
+		panic(err)
+	}
+	return buf.Bytes()
+}
+
+// Encode (Sync)
+func (alloc *Allocator) Encode() []byte {
+	resultChan := make(chan []byte)
+	alloc.actionChan <- func() {
+		resultChan <- alloc.encode()
+	}
+	return <-resultChan
+}
+
+// OnGossip (Sync)
+func (alloc *Allocator) OnGossip(msg []byte) (router.GossipData, error) {
+	alloc.debugln("Allocator.OnGossip:", len(msg), "bytes")
+	resultChan := make(chan error)
+	alloc.actionChan <- func() {
+		resultChan <- alloc.update(msg)
+	}
+	return nil, <-resultChan // for now, we never propagate updates. TBD
+}
+
+// GossipData implementation is trivial - we always gossip the latest
+// data we have at time of sending
+type ipamGossipData struct {
+	alloc *Allocator
+}
+
+func (d *ipamGossipData) Merge(other router.GossipData) {
+	// no-op
+}
+
+func (d *ipamGossipData) Encode() []byte {
+	return d.alloc.Encode()
+}
+
+// Gossip returns a GossipData implementation, which in this case always
+// returns the latest ring state (and does nothing on merge)
+func (alloc *Allocator) Gossip() router.GossipData {
+	return &ipamGossipData{alloc}
+}
+
+// SetInterfaces gives the allocator two interfaces for talking to the outside world
+func (alloc *Allocator) SetInterfaces(gossip router.Gossip) {
+	alloc.gossip = gossip
+}
+
+// ACTOR server
+
+func (alloc *Allocator) actorLoop(actionChan <-chan func()) {
+	for {
+		var tickChan <-chan time.Time
+		if alloc.paxosTicker != nil {
+			tickChan = alloc.paxosTicker.C
+		}
+
+		select {
+		case action := <-actionChan:
+			if action == nil {
+				return
+			}
+			action()
+		case <-tickChan:
+			alloc.propose()
+		}
+
+		alloc.assertInvariants()
+		alloc.reportFreeSpace()
+	}
+}
+
+// Helper functions
+
+func (alloc *Allocator) string() string {
+	var buf bytes.Buffer
+	fmt.Fprintf(&buf, "Allocator subnet %s/%d", alloc.subnetStart.String(), alloc.prefixLen)
+
+	if alloc.ring.Empty() {
+		if alloc.paxosTicker != nil {
+			fmt.Fprintf(&buf, " awaiting consensus: %s", alloc.paxos.String())
+		}
+	} else {
+		localFreeSpace := alloc.space.NumFreeAddresses()
+		remoteFreeSpace := alloc.ring.TotalRemoteFree()
+		percentFree := 100 * float64(localFreeSpace+remoteFreeSpace) / float64(alloc.subnetSize)
+		fmt.Fprintf(&buf, "  Free IPs: ~%.1f%%, %d local, ~%d remote\n",
+			percentFree, localFreeSpace, remoteFreeSpace)
+
+		fmt.Fprint(&buf, "Owned Ranges:")
+		alloc.ring.FprintWithNicknames(&buf, alloc.nicknames)
+	}
+	if len(alloc.pendingAllocates)+len(alloc.pendingClaims) > 0 {
+		fmt.Fprintf(&buf, "\nPending requests:")
+		for _, op := range alloc.pendingAllocates {
+			fmt.Fprintf(&buf, "\n  %s", op.String())
+		}
+		for _, op := range alloc.pendingClaims {
+			fmt.Fprintf(&buf, "\n  %s", op.String())
+		}
+	}
+	return buf.String()
+}
+
+func (alloc *Allocator) consensus() []router.PeerName {
+	_, val := alloc.paxos.Consensus()
+	return val.Value
+}
+
+// Ensure we are making progress towards an established ring
+func (alloc *Allocator) establishRing() {
+	if !alloc.ring.Empty() || alloc.paxosTicker != nil {
+		return
+	}
+
+	alloc.propose()
+	if cons := alloc.consensus(); cons != nil {
+		// If the quorum was 1, then proposing immediately
+		// leads to consensus
+		alloc.createRing(cons)
+	} else {
+		// re-try until we get consensus
+		alloc.paxosTicker = time.NewTicker(paxosInterval)
+	}
+}
+
+func (alloc *Allocator) createRing(peers []router.PeerName) {
+	alloc.debugln("Paxos consensus:", peers)
+	alloc.ring.ClaimForPeers(normalizeConsensus(peers))
+	alloc.gossip.GossipBroadcast(alloc.Gossip())
+	alloc.ringUpdated()
+}
+
+func (alloc *Allocator) ringUpdated() {
+	// When we have a ring, we don't need paxos any more
+	if alloc.paxos != nil {
+		alloc.paxos = nil
+
+		if alloc.paxosTicker != nil {
+			alloc.paxosTicker.Stop()
+			alloc.paxosTicker = nil
+		}
+	}
+
+	alloc.space.UpdateRanges(alloc.ring.OwnedRanges())
+	alloc.tryPendingOps()
+}
+
+// For compatibility with sort.Interface
+type peerNames []router.PeerName
+
+func (a peerNames) Len() int           { return len(a) }
+func (a peerNames) Less(i, j int) bool { return a[i] < a[j] }
+func (a peerNames) Swap(i, j int)      { a[i], a[j] = a[j], a[i] }
+
+// When we get a consensus from Paxos, the peer names are not in a
+// defined order and may contain duplicates.  This function sorts them
+// and de-dupes.
+func normalizeConsensus(consensus []router.PeerName) []router.PeerName {
+	if len(consensus) == 0 {
+		return nil
+	}
+
+	peers := make(peerNames, len(consensus))
+	copy(peers, consensus)
+	sort.Sort(peers)
+
+	dst := 0
+	for src := 1; src < len(peers); src++ {
+		if peers[dst] != peers[src] {
+			dst++
+			peers[dst] = peers[src]
+		}
+	}
+
+	return peers[:dst+1]
+}
+
+func (alloc *Allocator) propose() {
+	alloc.debugf("Paxos proposing")
+	alloc.paxos.Propose()
+	alloc.gossip.GossipBroadcast(alloc.Gossip())
+}
+
+func (alloc *Allocator) sendRequest(dest router.PeerName, kind byte) {
+	msg := router.Concat([]byte{kind}, alloc.encode())
+	alloc.gossip.GossipUnicast(dest, msg)
+}
+
+func (alloc *Allocator) update(msg []byte) error {
+	reader := bytes.NewReader(msg)
+	decoder := gob.NewDecoder(reader)
+	var data gossipState
+	var err error
+
+	if err := decoder.Decode(&data); err != nil {
+		return err
+	}
+
+	deltat := time.Unix(data.Now, 0).Sub(alloc.now())
+	if deltat > time.Hour || -deltat > time.Hour {
+		return fmt.Errorf("clock skew of %v detected, ignoring update", deltat)
+	}
+
+	// Merge nicknames
+	for peer, nickname := range data.Nicknames {
+		alloc.nicknames[peer] = nickname
+	}
+
+	// only one of Ring and Paxos should be present.  And we
+	// shouldn't get updates for a empty Ring. But tolerate
+	// them just in case.
+	if data.Ring != nil {
+		err = alloc.ring.Merge(*data.Ring)
+		if !alloc.ring.Empty() {
+			alloc.pruneNicknames()
+			alloc.ringUpdated()
+		}
+		return err
+	}
+
+	if data.Paxos != nil && alloc.ring.Empty() {
+		if alloc.paxos.Update(data.Paxos) {
+			if alloc.paxos.Think() {
+				// If something important changed, broadcast
+				alloc.gossip.GossipBroadcast(alloc.Gossip())
+			}
+
+			if cons := alloc.consensus(); cons != nil {
+				alloc.createRing(cons)
+			}
+		}
+	}
+
+	return nil
+}
+
+func (alloc *Allocator) donateSpace(to router.PeerName) {
+	// No matter what we do, we'll send a unicast gossip
+	// of our ring back to tha chap who asked for space.
+	// This serves to both tell him of any space we might
+	// have given him, or tell him where he might find some
+	// more.
+	defer alloc.sendRequest(to, msgRingUpdate)
+
+	alloc.debugln("Peer", to, "asked me for space")
+	start, size, ok := alloc.space.Donate()
+	if !ok {
+		free := alloc.space.NumFreeAddresses()
+		common.Assert(free == 0)
+		alloc.debugln("No space to give to peer", to)
+		return
+	}
+	end := address.Add(start, size)
+	alloc.debugln("Giving range", start, end, size, "to", to)
+	alloc.ring.GrantRangeToHost(start, end, to)
+}
+
+func (alloc *Allocator) assertInvariants() {
+	// We need to ensure all ranges the ring thinks we own have
+	// a corresponding space in the space set, and vice versa
+	checkSpace := space.New()
+	checkSpace.AddRanges(alloc.ring.OwnedRanges())
+	ranges := checkSpace.OwnedRanges()
+	spaces := alloc.space.OwnedRanges()
+
+	common.Assert(len(ranges) == len(spaces))
+
+	for i := 0; i < len(ranges); i++ {
+		r := ranges[i]
+		s := spaces[i]
+		common.Assert(s.Start == r.Start && s.End == r.End)
+	}
+}
+
+func (alloc *Allocator) reportFreeSpace() {
+	ranges := alloc.ring.OwnedRanges()
+	if len(ranges) == 0 {
+		return
+	}
+
+	freespace := make(map[address.Address]address.Offset)
+	for _, r := range ranges {
+		freespace[r.Start] = alloc.space.NumFreeAddressesInRange(r.Start, r.End)
+	}
+	alloc.ring.ReportFree(freespace)
+}
+
+// Owned addresses
+
+func (alloc *Allocator) addOwned(ident string, addr address.Address) {
+	alloc.owned[ident] = addr
+}
+
+func (alloc *Allocator) findOwner(addr address.Address) string {
+	for ident, candidate := range alloc.owned {
+		if candidate == addr {
+			return ident
+		}
+	}
+	return ""
+}
+
+// Logging
+
+func (alloc *Allocator) infof(fmt string, args ...interface{}) {
+	common.Info.Printf("[allocator %s] "+fmt, append([]interface{}{alloc.ourName}, args...)...)
+}
+func (alloc *Allocator) debugln(args ...interface{}) {
+	common.Debug.Println(append([]interface{}{fmt.Sprintf("[allocator %s]:", alloc.ourName)}, args...)...)
+}
+func (alloc *Allocator) debugf(fmt string, args ...interface{}) {
+	common.Debug.Printf("[allocator %s] "+fmt, append([]interface{}{alloc.ourName}, args...)...)
+}

ipam/allocator_test.go

@@ -0,0 +1,453 @@
+package ipam
+
+import (
+	"fmt"
+	"math/rand"
+	"sync"
+	"testing"
+	"time"
+
+	"github.com/weaveworks/weave/common"
+	"github.com/weaveworks/weave/ipam/address"
+	"github.com/weaveworks/weave/router"
+	wt "github.com/weaveworks/weave/testing"
+)
+
+const (
+	testStart1 = "10.0.1.0"
+	testStart2 = "10.0.2.0"
+	testStart3 = "10.0.3.0"
+)
+
+func TestAllocFree(t *testing.T) {
+	const (
+		container1 = "abcdef"
+		container2 = "baddf00d"
+		container3 = "b01df00d"
+		universe   = "10.0.3.0/28"
+		testAddr1  = "10.0.3.1" // first address allocated should be .1 because .0 is network addr
+		spaceSize  = 14         // 16 IP addresses in /28, minus .0 and .15
+	)
+
+	alloc := makeAllocatorWithMockGossip(t, "01:00:00:01:00:00", universe, 1)
+	defer alloc.Stop()
+
+	alloc.claimRingForTesting()
+	addr1, _ := alloc.Allocate(container1, nil)
+	wt.AssertEqualString(t, addr1.String(), testAddr1, "address")
+
+	// Ask for another address for a different container and check it's different
+	addr2, _ := alloc.Allocate(container2, nil)
+	if addr2.String() == testAddr1 {
+		t.Fatalf("Expected different address but got %s", addr2.String())
+	}
+
+	// Ask for the first container again and we should get the same address again
+	addr1a, _ := alloc.Allocate(container1, nil)
+	wt.AssertEqualString(t, addr1a.String(), testAddr1, "address")
+
+	// Now free the first one, and we should get it back when we ask
+	wt.AssertSuccess(t, alloc.Free(container1))
+	addr3, _ := alloc.Allocate(container3, nil)
+	wt.AssertEqualString(t, addr3.String(), testAddr1, "address")
+
+	alloc.ContainerDied(container2)
+	alloc.ContainerDied(container3)
+	alloc.String() // force sync-up after async call
+	wt.AssertEquals(t, alloc.space.NumFreeAddresses(), address.Offset(spaceSize))
+}
+
+func TestBootstrap(t *testing.T) {
+	common.InitDefaultLogging(false)
+	const (
+		donateSize     = 5
+		donateStart    = "10.0.1.7"
+		ourNameString  = "01:00:00:01:00:00"
+		peerNameString = "02:00:00:02:00:00"
+	)
+
+	alloc1 := makeAllocatorWithMockGossip(t, ourNameString, testStart1+"/22", 2)
+	defer alloc1.Stop()
+
+	// Simulate another peer on the gossip network
+	alloc2 := makeAllocatorWithMockGossip(t, peerNameString, testStart1+"/22", 2)
+	defer alloc2.Stop()
+
+	alloc1.OnGossipBroadcast(alloc2.encode())
+
+	alloc1.tryPendingOps()
+
+	ExpectBroadcastMessage(alloc1, nil) // alloc1 will try to form consensus
+	done := make(chan bool)
+	go func() {
+		alloc1.Allocate("somecontainer", nil)
+		done <- true
+	}()
+	time.Sleep(100 * time.Millisecond)
+	AssertNothingSent(t, done)
+
+	CheckAllExpectedMessagesSent(alloc1, alloc2)
+
+	alloc1.tryPendingOps()
+	AssertNothingSent(t, done)
+
+	CheckAllExpectedMessagesSent(alloc1, alloc2)
+
+	// alloc2 receives paxos update and broadcasts its reply
+	ExpectBroadcastMessage(alloc2, nil)
+	alloc2.OnGossipBroadcast(alloc1.encode())
+
+	ExpectBroadcastMessage(alloc1, nil)
+	alloc1.OnGossipBroadcast(alloc2.encode())
+
+	// both nodes will get consensus now so initialize the ring
+	ExpectBroadcastMessage(alloc2, nil)
+	ExpectBroadcastMessage(alloc2, nil)
+	alloc2.OnGossipBroadcast(alloc1.encode())
+
+	CheckAllExpectedMessagesSent(alloc1, alloc2)
+
+	alloc1.OnGossipBroadcast(alloc2.encode())
+	// now alloc1 should have space
+
+	AssertSent(t, done)
+
+	CheckAllExpectedMessagesSent(alloc1, alloc2)
+}
+
+func TestAllocatorClaim(t *testing.T) {
+	const (
+		container1 = "abcdef"
+		container2 = "baddf00d"
+		container3 = "b01df00d"
+		universe   = "10.0.3.0/30"
+		testAddr1  = "10.0.3.1" // first address allocated should be .1 because .0 is network addr
+	)
+
+	alloc := makeAllocatorWithMockGossip(t, "01:00:00:01:00:00", universe, 1)
+	defer alloc.Stop()
+
+	alloc.claimRingForTesting()
+	addr1, _ := alloc.Allocate(container1, nil)
+	alloc.Allocate(container2, nil)
+
+	// Now free the first one, and try to claim it
+	wt.AssertSuccess(t, alloc.Free(container1))
+	t.Log(alloc)
+	err := alloc.Claim(container3, addr1, nil)
+	wt.AssertNoErr(t, err)
+	addr3, _ := alloc.Allocate(container3, nil)
+	wt.AssertEqualString(t, addr3.String(), testAddr1, "address")
+}
+
+func (alloc *Allocator) pause() func() {
+	paused := make(chan struct{})
+	alloc.actionChan <- func() {
+		paused <- struct{}{}
+		<-paused
+	}
+	<-paused
+	return func() {
+		close(paused)
+	}
+}
+
+func TestCancel(t *testing.T) {
+	common.InitDefaultLogging(false)
+	const (
+		CIDR = "10.0.1.7/26"
+	)
+
+	router := TestGossipRouter{make(map[router.PeerName]chan gossipMessage), 0.0}
+
+	alloc1 := makeAllocator("01:00:00:02:00:00", CIDR, 2)
+	alloc1.SetInterfaces(router.connect(alloc1.ourName, alloc1))
+
+	alloc2 := makeAllocator("02:00:00:02:00:00", CIDR, 2)
+	alloc2.SetInterfaces(router.connect(alloc2.ourName, alloc2))
+	alloc1.claimRingForTesting(alloc1, alloc2)
+	alloc2.claimRingForTesting(alloc1, alloc2)
+
+	alloc1.Start()
+	alloc2.Start()
+
+	// tell peers about each other
+	alloc1.OnGossipBroadcast(alloc2.encode())
+
+	// Get some IPs, so each allocator has some space
+	res1, _ := alloc1.Allocate("foo", nil)
+	common.Debug.Printf("res1 = %s", res1.String())
+	res2, _ := alloc2.Allocate("bar", nil)
+	common.Debug.Printf("res2 = %s", res2.String())
+	if res1 == res2 {
+		wt.Fatalf(t, "Error: got same ips!")
+	}
+
+	// Now we're going to pause alloc2 and ask alloc1
+	// for an allocation
+	unpause := alloc2.pause()
+
+	// Use up all the IPs that alloc1 owns, so the allocation after this will prompt a request to alloc2
+	for i := 0; alloc1.space.NumFreeAddresses() > 0; i++ {
+		alloc1.Allocate(fmt.Sprintf("tmp%d", i), nil)
+	}
+	cancelChan := make(chan bool, 1)
+	doneChan := make(chan bool)
+	go func() {
+		_, ok := alloc1.Allocate("baz", cancelChan)
+		doneChan <- ok == nil
+	}()
+
+	AssertNothingSent(t, doneChan)
+	time.Sleep(100 * time.Millisecond)
+	AssertNothingSent(t, doneChan)
+
+	cancelChan <- true
+	unpause()
+	if <-doneChan {
+		wt.Fatalf(t, "Error: got result from Allocate")
+	}
+}
+
+func TestGossipShutdown(t *testing.T) {
+	const (
+		container1 = "abcdef"
+		container2 = "baddf00d"
+		universe   = "10.0.3.0/30"
+		testAddr1  = "10.0.3.1" // first address allocated should be .1 because .0 is network addr
+	)
+
+	alloc := makeAllocatorWithMockGossip(t, "01:00:00:01:00:00", universe, 1)
+	defer alloc.Stop()
+
+	alloc.claimRingForTesting()
+	addr1, _ := alloc.Allocate(container1, nil)
+	wt.AssertEqualString(t, addr1.String(), testAddr1, "address")
+
+	alloc.Shutdown()
+
+	_, err := alloc.Allocate(container2, nil) // trying to allocate after shutdown should fail
+	wt.AssertFalse(t, err == nil, "no address")
+
+	CheckAllExpectedMessagesSent(alloc)
+}
+
+func TestTransfer(t *testing.T) {
+	const (
+		cidr = "10.0.1.7/22"
+	)
+	allocs, router := makeNetworkOfAllocators(3, cidr)
+	alloc1 := allocs[0]
+	alloc2 := allocs[1]
+	alloc3 := allocs[2] // This will be 'master' and get the first range
+
+	_, err := alloc2.Allocate("foo", nil)
+	wt.AssertTrue(t, err == nil, "Failed to get address")
+
+	_, err = alloc3.Allocate("bar", nil)
+	wt.AssertTrue(t, err == nil, "Failed to get address")
+
+	router.GossipBroadcast(alloc2.Gossip())
+	router.GossipBroadcast(alloc3.Gossip())
+	router.removePeer(alloc2.ourName)
+	router.removePeer(alloc3.ourName)
+	alloc2.Stop()
+	alloc3.Stop()
+	wt.AssertSuccess(t, alloc1.AdminTakeoverRanges(alloc2.ourName.String()))
+	wt.AssertSuccess(t, alloc1.AdminTakeoverRanges(alloc3.ourName.String()))
+
+	wt.AssertEquals(t, alloc1.space.NumFreeAddresses(), address.Offset(1022))
+
+	_, err = alloc1.Allocate("foo", nil)
+	wt.AssertTrue(t, err == nil, "Failed to get address")
+	alloc1.Stop()
+}
+
+func TestFakeRouterSimple(t *testing.T) {
+	common.InitDefaultLogging(false)
+	const (
+		cidr = "10.0.1.7/22"
+	)
+	allocs, _ := makeNetworkOfAllocators(2, cidr)
+	defer stopNetworkOfAllocators(allocs)
+
+	alloc1 := allocs[0]
+	//alloc2 := allocs[1]
+
+	addr, _ := alloc1.Allocate("foo", nil)
+	println("Got addr", addr)
+}
+
+func TestAllocatorFuzz(t *testing.T) {
+	common.InitDefaultLogging(false)
+	const (
+		firstpass    = 1000
+		secondpass   = 5000
+		nodes        = 10
+		maxAddresses = 1000
+		concurrency  = 10
+		cidr         = "10.0.1.7/22"
+	)
+	allocs, _ := makeNetworkOfAllocators(nodes, cidr)
+	defer stopNetworkOfAllocators(allocs)
+
+	// Test state
+	// For each IP issued we store the allocator
+	// that issued it and the name of the container
+	// it was issued to.
+	type result struct {
+		name  string
+		alloc int32
+	}
+	stateLock := sync.Mutex{}
+	state := make(map[string]result)
+	// Keep a list of addresses issued, so we
+	// Can pick random ones
+	var addrs []string
+	numPending := 0
+
+	rand.Seed(0)
+
+	// Remove item from list by swapping it with last
+	// and reducing slice length by 1
+	rm := func(xs []string, i int32) []string {
+		ls := len(xs) - 1
+		xs[i] = xs[ls]
+		return xs[:ls]
+	}
+
+	// Do a Allocate and check the address
+	// is unique.  Needs a unique container
+	// name.
+	allocate := func(name string) {
+		stateLock.Lock()
+		if len(addrs)+numPending >= maxAddresses {
+			stateLock.Unlock()
+			return
+		}
+		numPending++
+		stateLock.Unlock()
+
+		allocIndex := rand.Int31n(nodes)
+		alloc := allocs[allocIndex]
+		//common.Info.Printf("Allocate: asking allocator %d", allocIndex)
+		addr, err := alloc.Allocate(name, nil)
+
+		if err != nil {
+			panic(fmt.Sprintf("Could not allocate addr"))
+		}
+
+		//common.Info.Printf("Allocate: got address %s for name %s", addr, name)
+		addrStr := addr.String()
+
+		stateLock.Lock()
+		defer stateLock.Unlock()
+
+		if res, existing := state[addrStr]; existing {
+			panic(fmt.Sprintf("Dup found for address %s - %s and %s", addrStr,
+				name, res.name))
+		}
+
+		state[addrStr] = result{name, allocIndex}
+		addrs = append(addrs, addrStr)
+		numPending--
+	}
+
+	// Free a random address.
+	free := func() {
+		stateLock.Lock()
+		if len(addrs) == 0 {
+			stateLock.Unlock()
+			return
+		}
+		// Delete an existing allocation
+		// Pick random addr
+		addrIndex := rand.Int31n(int32(len(addrs)))
+		addr := addrs[addrIndex]
+		res := state[addr]
+		addrs = rm(addrs, addrIndex)
+		delete(state, addr)
+		stateLock.Unlock()
+
+		alloc := allocs[res.alloc]
+		//common.Info.Printf("Freeing %s on allocator %d", addr, res.alloc)
+
+		wt.AssertSuccess(t, alloc.Free(res.name))
+	}
+
+	// Do a Allocate on an existing container & allocator
+	// and check we get the right answer.
+	allocateAgain := func() {
+		stateLock.Lock()
+		addrIndex := rand.Int31n(int32(len(addrs)))
+		addr := addrs[addrIndex]
+		res := state[addr]
+		stateLock.Unlock()
+		alloc := allocs[res.alloc]
+
+		//common.Info.Printf("Asking for %s on allocator %d again", addr, res.alloc)
+
+		newAddr, _ := alloc.Allocate(res.name, nil)
+		oldAddr, _ := address.ParseIP(addr)
+		if newAddr != oldAddr {
+			panic(fmt.Sprintf("Got different address for repeat request"))
+		}
+	}
+
+	// Run function _f_ _iterations_ times, in _concurrency_
+	// number of goroutines
+	doConcurrentIterations := func(iterations int, f func(int)) {
+		iterationsPerThread := iterations / concurrency
+
+		wg := sync.WaitGroup{}
+		for i := 0; i < concurrency; i++ {
+			wg.Add(1)
+			go func(j int) {
+				defer wg.Done()
+				for k := 0; k < iterationsPerThread; k++ {
+					f((j * iterationsPerThread) + k)
+				}
+			}(i)
+		}
+		wg.Wait()
+	}
+
+	// First pass, just allocate a bunch of ips
+	doConcurrentIterations(firstpass, func(iteration int) {
+		name := fmt.Sprintf("first%d", iteration)
+		allocate(name)
+	})
+
+	// Second pass, random ask for more allocations,
+	// or remove existing ones, or ask for allocation
+	// again.
+	doConcurrentIterations(secondpass, func(iteration int) {
+		r := rand.Float32()
+		switch {
+		case 0.0 <= r && r < 0.4:
+			// Ask for a new allocation
+			name := fmt.Sprintf("second%d", iteration)
+			allocate(name)
+
+		case (0.4 <= r && r < 0.8):
+			// free a random addr
+			free()
+
+		case 0.8 <= r && r < 1.0:
+			// ask for an existing name again, check we get same ip
+			allocateAgain()
+		}
+	})
+}
+
+func TestGossipSkew(t *testing.T) {
+	alloc1 := makeAllocatorWithMockGossip(t, "01:00:00:01:00:00", "10.0.1.0/22", 2)
+	defer alloc1.Stop()
+	alloc2 := makeAllocatorWithMockGossip(t, "02:00:00:02:00:00", "10.0.1.0/22", 2)
+	alloc2.now = func() time.Time { return time.Now().Add(time.Hour * 2) }
+	defer alloc2.Stop()
+
+	if _, err := alloc1.OnGossipBroadcast(alloc2.encode()); err == nil {
+		t.Fail()
+	}
+}

ipam/claim.go

@@ -0,0 +1,79 @@
+package ipam
+
+import (
+	"fmt"
+
+	"github.com/weaveworks/weave/ipam/address"
+	"github.com/weaveworks/weave/router"
+)
+
+type claim struct {
+	resultChan       chan<- error
+	hasBeenCancelled func() bool
+	ident            string
+	addr             address.Address
+}
+
+// Try returns true for success (or failure), false if we need to try again later
+func (c *claim) Try(alloc *Allocator) bool {
+	if (c.hasBeenCancelled)() {
+		c.Cancel()
+		return true
+	}
+
+	if !alloc.ring.Contains(c.addr) {
+		// Address not within our universe; assume user knows what they are doing
+		alloc.infof("Ignored address %s claimed by %s - not in our universe\n", c.addr, c.ident)
+		c.resultChan <- nil
+		return true
+	}
+
+	// If our ring doesn't know, it must be empty.  We will have initiated the
+	// bootstrap of the ring, so wait until we find some owner for this
+	// range (might be us).
+	owner := alloc.ring.Owner(c.addr)
+	if owner == router.UnknownPeerName {
+		alloc.infof("Ring is empty; will try later.\n", c.addr, owner)
+		return false
+	}
+	if owner != alloc.ourName {
+		name, found := alloc.nicknames[owner]
+		if found {
+			name = " (" + name + ")"
+		}
+		c.resultChan <- fmt.Errorf("address %s is owned by other peer %s%s", c.addr.String(), owner, name)
+		return true
+	}
+	// We are the owner, check we haven't given it to another container
+	existingIdent := alloc.findOwner(c.addr)
+	if existingIdent == c.ident {
+		// same identifier is claiming same address; that's OK
+		c.resultChan <- nil
+		return true
+	}
+	if existingIdent == "" {
+		err := alloc.space.Claim(c.addr)
+		if err != nil {
+			c.resultChan <- err
+			return true
+		}
+		alloc.addOwned(c.ident, c.addr)
+		c.resultChan <- nil
+		return true
+	}
+	// Addr already owned by container on this machine
+	c.resultChan <- fmt.Errorf("address %s is already owned by %s", c.addr.String(), existingIdent)
+	return true
+}
+
+func (c *claim) Cancel() {
+	c.resultChan <- fmt.Errorf("Operation cancelled.")
+}
+
+func (c *claim) String() string {
+	return fmt.Sprintf("Claim %s -> %s", c.ident, c.addr.String())
+}
+
+func (c *claim) ForContainer(ident string) bool {
+	return c.ident == ident
+}

ipam/dummy.go

@@ -0,0 +1,36 @@
+package ipam
+
+import (
+	"errors"
+
+	"github.com/weaveworks/weave/router"
+)
+
+// Exists solely for the purpose of detecting a network in which some
+// peers are configured with IPAM on and some with IPAM off.
+type DummyAllocator struct {
+}
+
+var (
+	ErrMixedConfig = errors.New("Mixed IPAM/non-IPAM configuration not supported")
+)
+
+func (alloc *DummyAllocator) OnGossipUnicast(sender router.PeerName, msg []byte) error {
+	return ErrMixedConfig
+}
+
+func (alloc *DummyAllocator) OnGossipBroadcast(msg []byte) (router.GossipData, error) {
+	return nil, ErrMixedConfig
+}
+
+func (alloc *DummyAllocator) OnGossip(msg []byte) (router.GossipData, error) {
+	return nil, ErrMixedConfig
+}
+
+func (alloc *DummyAllocator) Gossip() router.GossipData {
+	return nil
+}
+
+func (alloc *DummyAllocator) Encode() []byte {
+	return nil
+}

ipam/http.go

@@ -0,0 +1,72 @@
+package ipam
+
+import (
+	"fmt"
+	"net/http"
+
+	"github.com/gorilla/mux"
+
+	"github.com/weaveworks/weave/common"
+	"github.com/weaveworks/weave/ipam/address"
+)
+
+func badRequest(w http.ResponseWriter, err error) {
+	http.Error(w, err.Error(), http.StatusBadRequest)
+	common.Warning.Println("[allocator]:", err.Error())
+}
+
+// HandleHTTP wires up ipams HTTP endpoints to the provided mux.
+func (alloc *Allocator) HandleHTTP(router *mux.Router) {
+	router.Methods("PUT").Path("/ip/{id}/{ip}").HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		closedChan := w.(http.CloseNotifier).CloseNotify()
+		vars := mux.Vars(r)
+		ident := vars["id"]
+		ipStr := vars["ip"]
+		if ip, err := address.ParseIP(ipStr); err != nil {
+			badRequest(w, err)
+			return
+		} else if err := alloc.Claim(ident, ip, closedChan); err != nil {
+			badRequest(w, fmt.Errorf("Unable to claim: %s", err))
+			return
+		}
+
+		w.WriteHeader(204)
+	})
+
+	router.Methods("POST").Path("/ip/{id}").HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		closedChan := w.(http.CloseNotifier).CloseNotify()
+		ident := mux.Vars(r)["id"]
+		newAddr, err := alloc.Allocate(ident, closedChan)
+		if err != nil {
+			badRequest(w, err)
+			return
+		}
+
+		fmt.Fprintf(w, "%s/%d", newAddr.String(), alloc.prefixLen)
+	})
+
+	router.Methods("DELETE").Path("/ip/{id}").HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		ident := mux.Vars(r)["id"]
+		if err := alloc.Free(ident); err != nil {
+			badRequest(w, err)
+			return
+		}
+
+		w.WriteHeader(204)
+	})
+
+	router.Methods("DELETE").Path("/peer").HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		alloc.Shutdown()
+		w.WriteHeader(204)
+	})
+
+	router.Methods("DELETE").Path("/peer/{id}").HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		ident := mux.Vars(r)["id"]
+		if err := alloc.AdminTakeoverRanges(ident); err != nil {
+			badRequest(w, err)
+			return
+		}
+
+		w.WriteHeader(204)
+	})
+}

ipam/http_test.go

@@ -0,0 +1,164 @@
+package ipam
+
+import (
+	"fmt"
+	"io"
+	"io/ioutil"
+	"math/rand"
+	"net/http"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/gorilla/mux"
+
+	"github.com/weaveworks/weave/common"
+	wt "github.com/weaveworks/weave/testing"
+)
+
+func HTTPPost(t *testing.T, url string) string {
+	resp, err := http.Post(url, "", nil)
+	wt.AssertNoErr(t, err)
+	wt.AssertStatus(t, resp.StatusCode, http.StatusOK, "http response")
+	defer resp.Body.Close()
+	body, _ := ioutil.ReadAll(resp.Body)
+	return string(body)
+}
+
+func doHTTP(method string, url string) (resp *http.Response, err error) {
+	req, _ := http.NewRequest(method, url, nil)
+	return http.DefaultClient.Do(req)
+}
+
+func listenHTTP(port int, alloc *Allocator) {
+	router := mux.NewRouter()
+	router.HandleFunc("/status", func(w http.ResponseWriter, r *http.Request) {
+		io.WriteString(w, fmt.Sprintln(alloc))
+	})
+	alloc.HandleHTTP(router)
+
+	srv := &http.Server{
+		Addr:    fmt.Sprintf(":%d", port),
+		Handler: router,
+	}
+	if err := srv.ListenAndServe(); err != nil {
+		common.Error.Fatal("Unable to create http listener: ", err)
+	}
+}
+
+func allocURL(port int, containerID string) string {
+	return fmt.Sprintf("http://localhost:%d/ip/%s", port, containerID)
+}
+
+func TestHttp(t *testing.T) {
+	var (
+		containerID = "deadbeef"
+		container2  = "baddf00d"
+		container3  = "b01df00d"
+		testAddr1   = "10.0.3.9"
+		netSuffix   = "/29"
+		testCIDR1   = "10.0.3.8" + netSuffix
+	)
+
+	alloc := makeAllocatorWithMockGossip(t, "08:00:27:01:c3:9a", testCIDR1, 1)
+	port := rand.Intn(10000) + 32768
+	fmt.Println("Http test on port", port)
+	go listenHTTP(port, alloc)
+
+	time.Sleep(100 * time.Millisecond) // Allow for http server to get going
+
+	alloc.claimRingForTesting()
+	// Ask the http server for a new address
+	cidr1 := HTTPPost(t, allocURL(port, containerID))
+	wt.AssertEqualString(t, cidr1, testAddr1+netSuffix, "address")
+
+	// Ask the http server for another address and check it's different
+	cidr2 := HTTPPost(t, allocURL(port, container2))
+	wt.AssertFalse(t, cidr2 == testAddr1+netSuffix, "address")
+
+	// Ask for the first container again and we should get the same address again
+	cidr1a := HTTPPost(t, allocURL(port, containerID))
+	wt.AssertEqualString(t, cidr1a, testAddr1+netSuffix, "address")
+
+	// Now free the first one, and we should get it back when we ask
+	doHTTP("DELETE", allocURL(port, containerID))
+	cidr3 := HTTPPost(t, allocURL(port, container3))
+	wt.AssertEqualString(t, cidr3, testAddr1+netSuffix, "address")
+
+	// Would like to shut down the http server at the end of this test
+	// but it's complicated.
+	// See https://groups.google.com/forum/#!topic/golang-nuts/vLHWa5sHnCE
+}
+
+func TestBadHttp(t *testing.T) {
+	var (
+		containerID = "deadbeef"
+		testCIDR1   = "10.0.0.0/8"
+	)
+
+	alloc := makeAllocatorWithMockGossip(t, "08:00:27:01:c3:9a", testCIDR1, 1)
+	defer alloc.Stop()
+	port := rand.Intn(10000) + 32768
+	fmt.Println("BadHttp test on port", port)
+	go listenHTTP(port, alloc)
+
+	alloc.claimRingForTesting()
+	cidr1 := HTTPPost(t, allocURL(port, containerID))
+	parts := strings.Split(cidr1, "/")
+	testAddr1 := parts[0]
+	// Verb that's not handled
+	resp, err := doHTTP("HEAD", fmt.Sprintf("http://localhost:%d/ip/%s/%s", port, containerID, testAddr1))
+	wt.AssertNoErr(t, err)
+	wt.AssertStatus(t, resp.StatusCode, http.StatusNotFound, "http response")
+	// Mis-spelled URL
+	resp, err = doHTTP("POST", fmt.Sprintf("http://localhost:%d/xip/%s/", port, containerID))
+	wt.AssertNoErr(t, err)
+	wt.AssertStatus(t, resp.StatusCode, http.StatusNotFound, "http response")
+	// Malformed URL
+	resp, err = doHTTP("POST", fmt.Sprintf("http://localhost:%d/ip/%s/foo/bar/baz", port, containerID))
+	wt.AssertNoErr(t, err)
+	wt.AssertStatus(t, resp.StatusCode, http.StatusNotFound, "http response")
+}
+
+func TestHTTPCancel(t *testing.T) {
+	wt.RunWithTimeout(t, 2*time.Second, func() {
+		impTestHTTPCancel(t)
+	})
+}
+
+func impTestHTTPCancel(t *testing.T) {
+	common.InitDefaultLogging(true)
+	var (
+		containerID = "deadbeef"
+		testCIDR1   = "10.0.3.5/29"
+	)
+
+	alloc := makeAllocatorWithMockGossip(t, "08:00:27:01:c3:9a", testCIDR1, 1)
+	defer alloc.Stop()
+	alloc.claimRingForTesting()
+	port := rand.Intn(10000) + 32768
+	fmt.Println("Http test on port", port)
+	go listenHTTP(port, alloc)
+
+	time.Sleep(100 * time.Millisecond) // Allow for http server to get going
+
+	// Stop the alloc so nothing actually works
+	unpause := alloc.pause()
+
+	// Ask the http server for a new address
+	done := make(chan *http.Response)
+	req, _ := http.NewRequest("POST", allocURL(port, containerID), nil)
+	go func() {
+		res, _ := http.DefaultClient.Do(req)
+		done <- res
+	}()
+
+	time.Sleep(100 * time.Millisecond)
+	fmt.Println("Cancelling allocate")
+	http.DefaultTransport.(*http.Transport).CancelRequest(req)
+	unpause()
+	res := <-done
+	if res != nil {
+		wt.Fatalf(t, "Error: Allocate returned non-nil")
+	}
+}

ipam/paxos/paxos.go

@@ -0,0 +1,261 @@
+package paxos
+
+import (
+	"fmt"
+	"github.com/weaveworks/weave/router"
+)
+
+// The node identifier.  The use of the UID here is important: Paxos
+// acceptors must not forget their promises, so it's important that a
+// node does not restart and lose its Paxos state but claim to have
+// the same ID.
+type NodeID struct {
+	Name router.PeerName
+	UID  router.PeerUID
+}
+
+// note all fields exported in structs so we can Gob them
+type ProposalID struct {
+	// round numbers begin at 1.  round 0 indicates an
+	// uninitialized ProposalID, and precedes all other ProposalIDs
+	Round    uint
+	Proposer NodeID
+}
+
+func (a ProposalID) precedes(b ProposalID) bool {
+	switch {
+	case a.Round != b.Round:
+		return a.Round < b.Round
+	case a.Proposer.Name != b.Proposer.Name:
+		return a.Proposer.Name < b.Proposer.Name
+	case a.Proposer.UID != b.Proposer.UID:
+		return a.Proposer.UID < b.Proposer.UID
+	default:
+		return false
+	}
+}
+
+func (a ProposalID) valid() bool {
+	return a.Round > 0
+}
+
+// For seeding IPAM, the value we want consensus on is a set of peer names
+type Value []router.PeerName
+
+// An AcceptedValue is a Value plus the proposal which originated that
+// Value.  The origin is not essential, but makes comparing
+// AcceptedValues easy even if comparing Values is not.
+type AcceptedValue struct {
+	Value  Value
+	Origin ProposalID
+}
+
+type NodeClaims struct {
+	// The node promises not to accept a proposal with id less
+	// than this.
+	Promise ProposalID
+
+	// The accepted proposal, if valid
+	Accepted    ProposalID
+	AcceptedVal AcceptedValue
+}
+
+func (a NodeClaims) equals(b NodeClaims) bool {
+	return a.Promise == b.Promise && a.Accepted == b.Accepted &&
+		a.AcceptedVal.Origin == b.AcceptedVal.Origin
+}
+
+type GossipState map[NodeID]NodeClaims
+
+type Node struct {
+	id     NodeID
+	quorum uint
+	knows  GossipState
+}
+
+func NewNode(name router.PeerName, uid router.PeerUID, quorum uint) *Node {
+	return &Node{
+		id:     NodeID{name, uid},
+		quorum: quorum,
+		knows:  map[NodeID]NodeClaims{},
+	}
+}
+
+func (node *Node) GossipState() GossipState {
+	return node.knows
+}
+
+// Update this node's information about what other nodes know.
+// Returns true if we learned something new.
+func (node *Node) Update(from GossipState) bool {
+	changed := false
+
+	for i, fromClaims := range from {
+		claims, ok := node.knows[i]
+		if ok {
+			if claims.Promise.precedes(fromClaims.Promise) {
+				claims.Promise = fromClaims.Promise
+				changed = true
+			}
+
+			if claims.Accepted.precedes(fromClaims.Accepted) {
+				claims.Accepted = fromClaims.Accepted
+				claims.AcceptedVal = fromClaims.AcceptedVal
+				changed = true
+			}
+		} else {
+			claims = fromClaims
+			changed = true
+		}
+
+		node.knows[i] = claims
+	}
+
+	return changed
+}
+
+func max(a uint, b uint) uint {
+	if a > b {
+		return a
+	} else {
+		return b
+	}
+}
+
+// Initiate a new proposal, i.e. the Paxos "Prepare" step.  This is
+// simply a matter of gossipping a new proposal that supersedes all
+// others.
+func (node *Node) Propose() {
+	// Find the highest round number around
+	round := uint(0)
+
+	for _, claims := range node.knows {
+		round = max(round, claims.Promise.Round)
+		round = max(round, claims.Accepted.Round)
+	}
+
+	ourClaims := node.knows[node.id]
+	ourClaims.Promise = ProposalID{
+		Round:    round + 1,
+		Proposer: node.id,
+	}
+	node.knows[node.id] = ourClaims
+
+	// With a quorum of 1, we can immediately accept our proposal
+	if node.quorum == 1 {
+		node.Think()
+	}
+}
+
+// The heart of the consensus algorithm. Return true if we have
+// changed our claims.
+func (node *Node) Think() bool {
+	ourClaims := node.knows[node.id]
+
+	// The "Promise" step of Paxos: Copy the highest known
+	// promise.
+	for _, claims := range node.knows {
+		if ourClaims.Promise.precedes(claims.Promise) {
+			ourClaims.Promise = claims.Promise
+		}
+	}
+
+	// The "Accept Request" step of Paxos: Acting as a proposer,
+	// do we have a proposal that has been promised by a quorum?
+	//
+	// In Paxos, the "proposer" and "acceptor" roles are distinct,
+	// so in principle a node acting as a proposer could continue
+	// trying to get its proposal acccepted even after the same
+	// node as an acceptor has superseded that proposal.  But
+	// that's pointless in a gossip context: If our promise
+	// supersedes our own proposal, then anyone who hears about
+	// that promise will not accept that proposal.  So our
+	// proposal is only in the running if it is also our promise.
+	if ourClaims.Promise.Proposer == node.id {
+		// Determine whether a quorum has promised, and the
+		// best previously-accepted value if there is one.
+		count := uint(0)
+		var accepted ProposalID
+		var acceptedVal AcceptedValue
+
+		for _, claims := range node.knows {
+			if claims.Promise == ourClaims.Promise {
+				count++
+
+				if accepted.precedes(claims.Accepted) {
+					accepted = claims.Accepted
+					acceptedVal = claims.AcceptedVal
+				}
+			}
+		}
+
+		if count >= node.quorum {
+			if !accepted.valid() {
+				acceptedVal.Value = node.pickValue()
+				acceptedVal.Origin = ourClaims.Promise
+			}
+
+			// We automatically accept our own proposal,
+			// and that's how we communicate the "accept
+			// request" to other nodes.
+			ourClaims.Accepted = ourClaims.Promise
+			ourClaims.AcceptedVal = acceptedVal
+		}
+	}
+
+	// The "Accepted" step of Paxos: If the proposal we promised
+	// on got accepted by some other node, we accept it too.
+	for _, claims := range node.knows {
+		if claims.Accepted == ourClaims.Promise {
+			ourClaims.Accepted = claims.Accepted
+			ourClaims.AcceptedVal = claims.AcceptedVal
+			break
+		}
+	}
+
+	if ourClaims.equals(node.knows[node.id]) {
+		return false
+	}
+
+	node.knows[node.id] = ourClaims
+	return true
+}
+
+// When we get to pick a value, we use the set of peer names we know
+// about.  This is not necessarily all peer names, but it is at least
+// a quorum, and so good enough for seeding the ring.
+func (node *Node) pickValue() Value {
+	val := make([]router.PeerName, len(node.knows))
+	i := 0
+	for id := range node.knows {
+		val[i] = id.Name
+		i++
+	}
+	return val
+}
+
+// Has a consensus been reached, based on the known claims of other nodes?
+func (node *Node) Consensus() (bool, AcceptedValue) {
+	counts := map[ProposalID]uint{}
+
+	for _, claims := range node.knows {
+		if claims.Accepted.valid() {
+			origin := claims.AcceptedVal.Origin
+			count := counts[origin] + 1
+			counts[origin] = count
+			if count >= node.quorum {
+				return true, claims.AcceptedVal
+			}
+		}
+	}
+
+	return false, AcceptedValue{}
+}
+
+func (node *Node) String() string {
+	if ok, val := node.Consensus(); ok {
+		return fmt.Sprintf("Consensus reached with size %d", len(val.Value))
+	} else {
+		return fmt.Sprintf("Nodes known: %d, Quorum size: %d", len(node.knows), node.quorum)
+	}
+}

ipam/paxos/paxos_test.go

@@ -0,0 +1,372 @@
+package paxos
+
+import (
+	"fmt"
+	"github.com/weaveworks/weave/router"
+	"math/rand"
+	"testing"
+	"time"
+)
+
+type TestNode struct {
+	*Node
+
+	// Topology
+	links    []*Link
+	isolated bool
+
+	// The first consensus the Node observed
+	firstConsensus AcceptedValue
+}
+
+type Link struct {
+	from     *TestNode
+	to       *TestNode
+	converse *Link
+
+	// A link is considered ready if it is worthwhile gossipping
+	// along it (i.e. unless we already gossipped along it and the
+	// "from" node didn't change since then).  If this is <0, then
+	// the link is not ready.  Otherwise, this gives the index in
+	// the Model's readyLinks slice.
+	ready int
+}
+
+type Model struct {
+	t          *testing.T
+	r          *rand.Rand
+	quorum     uint
+	nodes      []TestNode
+	readyLinks []*Link
+	nextID     uint
+}
+
+func (m *Model) addLink(a, b *TestNode) {
+	if a == b {
+		// Never link a node to itself
+		return
+	}
+
+	ab := Link{from: a, to: b, ready: len(m.readyLinks)}
+	ba := Link{from: b, to: a, converse: &ab, ready: len(m.readyLinks) + 1}
+	ab.converse = &ba
+	a.links = append(a.links, &ab)
+	b.links = append(b.links, &ba)
+	m.readyLinks = append(m.readyLinks, &ab, &ba)
+}
+
+func (m *Model) linkExists(a, b *TestNode) bool {
+	for _, l := range a.links {
+		if l.to == b {
+			return true
+		}
+	}
+
+	return false
+}
+
+type TestParams struct {
+	// Number of nodes
+	nodeCount uint
+
+	// Probability that two nodes are connected.
+	connectedProb float32
+
+	// Probability that some node will re-propose at each
+	// step. Setting this too high makes it likely that we'll fail
+	// to converge.
+	reproposeProb float32
+
+	// Probability that some node will be isolated at each step.
+	isolateProb float32
+
+	// Probability that some node willbe restarted at each step.
+	restartProb float32
+}
+
+// Make a network of nodes with random topology
+func makeRandomModel(params *TestParams, r *rand.Rand, t *testing.T) *Model {
+	m := Model{
+		t:          t,
+		r:          r,
+		quorum:     params.nodeCount/2 + 1,
+		nodes:      make([]TestNode, params.nodeCount),
+		readyLinks: []*Link{},
+		nextID:     params.nodeCount + 1,
+	}
+
+	for i := range m.nodes {
+		m.nodes[i].Node = NewNode(router.PeerName(i/2+1),
+			router.PeerUID(r.Int63()), m.quorum)
+		m.nodes[i].Propose()
+	}
+
+	for i := 1; i < len(m.nodes); i++ {
+		// was node i connected to the other nodes yet?
+		connected := false
+
+		for j := 0; j < i; j++ {
+			if r.Float32() < params.connectedProb {
+				connected = true
+				m.addLink(&m.nodes[i], &m.nodes[j])
+			}
+		}
+
+		if !connected {
+			// node i must be connected into the graph
+			// somewhere.  So if we didn't connect it
+			// already, this is a last resort.
+			m.addLink(&m.nodes[i], &m.nodes[r.Intn(i)])
+		}
+	}
+
+	return &m
+}
+
+// Mark a link as unready
+func (m *Model) unreadyLink(link *Link) {
+	i := link.ready
+	if i >= 0 {
+		m.readyLinks[i] = m.readyLinks[len(m.readyLinks)-1]
+		m.readyLinks[i].ready = i
+		m.readyLinks = m.readyLinks[:len(m.readyLinks)-1]
+		link.ready = -1
+	}
+}
+
+// Mark a link as ready
+func (m *Model) readyLink(link *Link) {
+	if link.ready < 0 {
+		link.ready = len(m.readyLinks)
+		m.readyLinks = append(m.readyLinks, link)
+	}
+}
+
+// Mark all the outgoing links from a node as ready
+func (m *Model) nodeChanged(node *TestNode) {
+	for _, l := range node.links {
+		m.readyLink(l)
+	}
+}
+
+// Isolate a node
+func (m *Model) isolateNode(node *TestNode) {
+	node.isolated = true
+	for _, l := range node.links {
+		if l.ready >= 0 {
+			m.unreadyLink(l)
+			m.unreadyLink(l.converse)
+		}
+	}
+
+	// Isolating a node could partition the network.  We don't
+	// want to test such a case (because it could prevent
+	// consensus).  Checking for partitions would be more code
+	// than its worth, so just add some links to prevent the
+	// possibility of partitions.
+	for i := 1; i < len(node.links); i++ {
+		m.addLink(node.links[i-1].to, node.links[i].to)
+	}
+}
+
+// Restart a node
+func (m *Model) restart(node *TestNode) {
+	node.Node = NewNode(router.PeerName(m.nextID),
+		router.PeerUID(m.r.Int63()), m.quorum)
+	m.nextID++
+	node.Propose()
+
+	// The node is now ignorant, so we need to mark the links into
+	// the node as ready.
+	for _, l := range node.links {
+		if !l.to.isolated {
+			m.readyLink(l.converse)
+		}
+	}
+
+	// If a consensus was just accepted due to this node accepting
+	// it, without other nodes hearing of it, and we then restart
+	// this node, then a different consensus can occur later on.
+	// If a tree falls with no one to hear it, does it make a
+	// sound?
+	node.firstConsensus = AcceptedValue{}
+}
+
+func (m *Model) pickNode() *TestNode {
+	for {
+		node := &m.nodes[m.r.Intn(len(m.nodes))]
+		if !node.isolated {
+			return node
+		}
+	}
+}
+
+func (m *Model) simulate(params *TestParams) bool {
+	nodesLeft := uint(len(m.nodes))
+	restarts := uint(0)
+
+	for step := 0; step < 1000000; step++ {
+		if len(m.readyLinks) == 0 {
+			// Everything has converged.  This can be
+			// because consensus was reached, or because
+			// consensus became impossible, e.g. everyone
+			// promised on a particular proposal, but then
+			// the proposer restarted or was isolated.  So
+			// we detect the latter cases and force a new
+			// proposal
+			for i := range m.nodes {
+				ok, _ := m.nodes[i].Consensus()
+				if ok {
+					return true
+				}
+			}
+
+			node := m.pickNode()
+			node.Propose()
+			m.nodeChanged(node)
+		}
+
+		// Pick a ready link at random
+		i := m.r.Intn(len(m.readyLinks))
+		link := m.readyLinks[i]
+		if link.ready != i {
+			m.t.Fatal("Link in readyLinks was not ready")
+		}
+
+		// gossip across link
+		node := link.to
+		if node.Update(link.from.GossipState()) {
+			node.Think()
+
+			if !node.firstConsensus.Origin.valid() {
+				ok, val := node.Consensus()
+				if ok {
+					node.firstConsensus = val
+				}
+			}
+
+			m.nodeChanged(node)
+		}
+
+		m.unreadyLink(link)
+
+		// Re-propose?
+		if m.r.Float32() < params.reproposeProb {
+			node := m.pickNode()
+			node.Propose()
+			m.nodeChanged(node)
+		}
+
+		// Isolate?
+		if nodesLeft > m.quorum && m.r.Float32() < params.isolateProb {
+			m.isolateNode(m.pickNode())
+			nodesLeft--
+
+			// We isolated a node, so get another node to
+			// re-propose.  In reality the lack of
+			// consensus would be detected via a timeout
+			node := m.pickNode()
+			node.Propose()
+			m.nodeChanged(node)
+		}
+
+		// Restart?
+		if restarts < m.quorum && m.r.Float32() < params.restartProb {
+			restarts++
+			node := m.pickNode()
+			m.restart(node)
+			m.nodeChanged(node)
+		}
+	}
+
+	return false
+}
+
+func (m *Model) dump() {
+	for i := range m.nodes {
+		node := &m.nodes[i]
+		fmt.Println(node.id)
+		for n, claims := range node.knows {
+			fmt.Printf("\t%d %v\n", n, claims)
+		}
+	}
+}
+
+// Validate the final model state
+func (m *Model) validate() {
+	var origin ProposalID
+
+	for i := range m.nodes {
+		ok, val := m.nodes[i].Consensus()
+		if !ok {
+			//m.dump()
+			m.t.Fatal("Node doesn't know about consensus")
+		}
+
+		firstConsensus := m.nodes[i].firstConsensus
+		if firstConsensus.Origin.valid() &&
+			firstConsensus.Origin != val.Origin {
+			//m.dump()
+			m.t.Fatal("Consensus mismatch")
+		}
+
+		if i == 0 {
+			origin = val.Origin
+		} else if val.Origin != origin {
+			m.t.Fatal("Node disagrees about consensus")
+		}
+	}
+}
+
+func TestSingleNode(t *testing.T) {
+	r := rand.New(rand.NewSource(time.Now().UnixNano()))
+
+	// Test the single node case
+	params := &TestParams{
+		nodeCount:     1,
+		connectedProb: 0,
+		reproposeProb: 0,
+		isolateProb:   0,
+		restartProb:   0,
+	}
+
+	m := makeRandomModel(params, r, t)
+
+	if !m.simulate(params) {
+		m.t.Fatal("Failed to converge")
+	}
+
+	m.validate()
+}
+
+func TestPaxos(t *testing.T) {
+	r := rand.New(rand.NewSource(time.Now().UnixNano()))
+
+	params := &TestParams{
+		nodeCount:     10,
+		connectedProb: 0.5,
+		reproposeProb: 0.01,
+		isolateProb:   0.01,
+
+		// Restarts cause failures due to the case where 1)
+		// the gossip ordering is such that there is an
+		// effective partition from the start, with a quorum
+		// on one side and one node less than a quorum on the
+		// other; 2) the quorum reaches a consensus; 3) a node
+		// is restarted, and at the same time jumps the
+		// partition, so the side that was now just less than
+		// a quorum becomes a quorum; 4) the new quorum
+		// reaches a contradictory consensus.
+		restartProb: 0,
+	}
+
+	for i := 0; i < 1000; i++ {
+		m := makeRandomModel(params, r, t)
+
+		if !m.simulate(params) {
+			m.t.Fatal("Failed to converge")
+		}
+
+		m.validate()
+	}
+}

ipam/ring/entry.go

@@ -0,0 +1,102 @@
+package ring
+
+import (
+	"sort"
+
+	"github.com/weaveworks/weave/common"
+	"github.com/weaveworks/weave/ipam/address"
+	"github.com/weaveworks/weave/router"
+)
+
+// Entry represents entries around the ring
+type entry struct {
+	Token   address.Address // The start of this range
+	Peer    router.PeerName // Who owns this range
+	Version uint32          // Version of this range
+	Free    address.Offset  // Number of free IPs in this range
+}
+
+func (e *entry) Equal(e2 *entry) bool {
+	return e.Token == e2.Token && e.Peer == e2.Peer &&
+		e.Version == e2.Version
+}
+
+func (e *entry) update(peername router.PeerName, free address.Offset) {
+	e.Peer = peername
+	e.Version++
+	e.Free = free
+}
+
+// For compatibility with sort.Interface
+type entries []*entry
+
+func (es entries) Len() int           { return len(es) }
+func (es entries) Less(i, j int) bool { return es[i].Token < es[j].Token }
+func (es entries) Swap(i, j int)      { panic("Should never be swapping entries!") }
+
+func (es entries) entry(i int) *entry {
+	i = i % len(es)
+	if i < 0 {
+		i += len(es)
+	}
+	return es[i]
+}
+
+func (es *entries) insert(e entry) {
+	i := sort.Search(len(*es), func(j int) bool {
+		return (*es)[j].Token >= e.Token
+	})
+
+	if i < len(*es) && (*es)[i].Token == e.Token {
+		panic("Trying to insert an existing token!")
+	}
+
+	*es = append(*es, &entry{})
+	copy((*es)[i+1:], (*es)[i:])
+	(*es)[i] = &e
+}
+
+func (es entries) get(token address.Address) (*entry, bool) {
+	i := sort.Search(len(es), func(j int) bool {
+		return es[j].Token >= token
+	})
+
+	if i < len(es) && es[i].Token == token {
+		return es[i], true
+	}
+
+	return nil, false
+}
+
+func (es *entries) remove(i int) {
+	*es = (*es)[:i+copy((*es)[i:], (*es)[i+1:])]
+}
+
+// Is token between entries at i and j?
+// NB i and j can overflow and will wrap
+// NBB this function does not work very well if there is only one
+//     token on the ring; luckily an accurate answer is not needed
+//     by the call sites in this case.
+func (es entries) between(token address.Address, i, j int) bool {
+	common.Assert(i < j)
+
+	first := es.entry(i)
+	second := es.entry(j)
+
+	switch {
+	case first.Token == second.Token:
+		// This implies there is only one token
+		// on the ring (i < j and i.token == j.token)
+		// In which case everything is between, expect
+		// this one token
+		return token != first.Token
+
+	case first.Token < second.Token:
+		return first.Token <= token && token < second.Token
+
+	case second.Token < first.Token:
+		return first.Token <= token || token < second.Token
+	}
+
+	panic("Should never get here - switch covers all possibilities.")
+}

ipam/ring/expvar.go

@@ -0,0 +1,31 @@
+package ring
+
+/* Exported variables for monitoring and management */
+
+import (
+	"expvar"
+	"strconv"
+)
+
+var (
+	expRingSize    = expvar.NewMap("ipam.ringSize")
+	expRingEntries = expvar.NewMap("ipam.ringEntries")
+)
+
+type _int int
+
+func (i _int) String() string {
+	return strconv.Itoa(int(i))
+}
+
+type _uint32 uint32
+
+func (i _uint32) String() string {
+	return strconv.FormatUint(uint64(i), 10)
+}
+
+func (r *Ring) updateExportedVariables() {
+	ringName := r.Start.String()
+	expRingSize.Set(ringName, _uint32(r.End-r.Start))
+	expRingEntries.Set(ringName, _int(len(r.Entries)))
+}

ipam/ring/ring.go

@@ -0,0 +1,520 @@
+/*
+Package ring implements a simple ring CRDT.
+*/
+package ring
+
+import (
+	"bytes"
+	"errors"
+	"fmt"
+	"io"
+	"math/rand"
+	"sort"
+
+	"github.com/weaveworks/weave/common"
+	"github.com/weaveworks/weave/ipam/address"
+	"github.com/weaveworks/weave/router"
+)
+
+// Ring represents the ring itself
+type Ring struct {
+	Start, End address.Address // [min, max) tokens in this ring.  Due to wrapping, min == max (effectively)
+	Peer       router.PeerName // name of peer owning this ring instance
+	Entries    entries         // list of entries sorted by token
+}
+
+func (r *Ring) assertInvariants() {
+	err := r.checkInvariants()
+	if err != nil {
+		panic(err.Error())
+	}
+}
+
+// Errors returned by Merge
+var (
+	ErrNotSorted        = errors.New("Ring not sorted")
+	ErrTokenRepeated    = errors.New("Token appears twice in ring")
+	ErrTokenOutOfRange  = errors.New("Token is out of range")
+	ErrDifferentSubnets = errors.New("IP Allocator with different subnet detected")
+	ErrNewerVersion     = errors.New("Received new version for entry I own!")
+	ErrInvalidEntry     = errors.New("Received invalid state update!")
+	ErrEntryInMyRange   = errors.New("Received new entry in my range!")
+	ErrNoFreeSpace      = errors.New("No free space found!")
+	ErrInvalidTimeout   = errors.New("dt must be greater than 0")
+	ErrNotFound         = errors.New("No entries for peer found")
+)
+
+func (r *Ring) checkInvariants() error {
+	if !sort.IsSorted(r.Entries) {
+		return ErrNotSorted
+	}
+
+	// Check no token appears twice
+	// We know it's sorted...
+	for i := 1; i < len(r.Entries); i++ {
+		if r.Entries[i-1].Token == r.Entries[i].Token {
+			return ErrTokenRepeated
+		}
+	}
+
+	if len(r.Entries) == 0 {
+		return nil
+	}
+
+	// Check tokens are in range
+	if r.Entries.entry(0).Token < r.Start {
+		return ErrTokenOutOfRange
+	}
+	if r.Entries.entry(-1).Token >= r.End {
+		return ErrTokenOutOfRange
+	}
+
+	// Check all the freespaces are in range
+	for i, entry := range r.Entries {
+		next := r.Entries.entry(i + 1)
+		distance := r.distance(entry.Token, next.Token)
+
+		if entry.Free > distance {
+			return fmt.Errorf("Entry %s-%s reporting too much free space: %d > %d", entry.Token, next.Token, entry.Free, distance)
+		}
+	}
+
+	return nil
+}
+
+// New creates an empty ring belonging to peer.
+func New(start, end address.Address, peer router.PeerName) *Ring {
+	common.Assert(start < end)
+
+	ring := &Ring{Start: start, End: end, Peer: peer, Entries: make([]*entry, 0)}
+	ring.updateExportedVariables()
+	return ring
+}
+
+// TotalRemoteFree returns the approximate number of free IPs
+// on other hosts.
+func (r *Ring) TotalRemoteFree() address.Offset {
+	result := address.Offset(0)
+	for _, entry := range r.Entries {
+		if entry.Peer != r.Peer {
+			result += entry.Free
+		}
+	}
+	return result
+}
+
+// Returns the distance between two tokens on this ring, dealing
+// with ranges which cross the origin
+func (r *Ring) distance(start, end address.Address) address.Offset {
+	if end > start {
+		return address.Offset(end - start)
+	}
+
+	return address.Offset((r.End - start) + (end - r.Start))
+}
+
+// GrantRangeToHost modifies the ring such that range [start, end)
+// is assigned to peer.  This may insert up to two new tokens.
+// Preconditions:
+// - start < end
+// - [start, end) must be owned by the calling peer
+func (r *Ring) GrantRangeToHost(start, end address.Address, peer router.PeerName) {
+	//fmt.Printf("%s GrantRangeToHost [%v,%v) -> %s\n", r.Peer, start, end, peer)
+
+	r.assertInvariants()
+	defer r.assertInvariants()
+	defer r.updateExportedVariables()
+
+	// ----------------- Start of Checks -----------------
+
+	common.Assert(start < end)
+	common.Assert(r.Start <= start && start < r.End)
+	common.Assert(r.Start < end && end <= r.End)
+	common.Assert(len(r.Entries) > 0)
+
+	// Look for the left-most entry greater than start, then go one previous
+	// to get the right-most entry less than or equal to start
+	preceedingPos := sort.Search(len(r.Entries), func(j int) bool {
+		return r.Entries[j].Token > start
+	})
+	preceedingPos--
+
+	// Check all tokens up to end are owned by us
+	for pos := preceedingPos; pos < len(r.Entries) && r.Entries.entry(pos).Token < end; pos++ {
+		common.Assert(r.Entries.entry(pos).Peer == r.Peer)
+	}
+
+	// ----------------- End of Checks -----------------
+
+	// Free space at start is max(length of range, distance to next token)
+	startFree := r.distance(start, r.Entries.entry(preceedingPos+1).Token)
+	if length := r.distance(start, end); startFree > length {
+		startFree = length
+	}
+	// Is there already a token at start, update it
+	if previousEntry := r.Entries.entry(preceedingPos); previousEntry.Token == start {
+		previousEntry.update(peer, startFree)
+	} else {
+		// Otherwise, these isn't a token here, insert a new one.
+		r.Entries.insert(entry{Token: start, Peer: peer, Free: startFree})
+		preceedingPos++
+		// Reset free space on previous entry, which we own.
+		previousEntry.update(r.Peer, r.distance(previousEntry.Token, start))
+	}
+
+	// Give all intervening tokens to the other peer
+	pos := preceedingPos + 1
+	for ; pos < len(r.Entries) && r.Entries.entry(pos).Token < end; pos++ {
+		entry := r.Entries.entry(pos)
+		entry.update(peer, address.Min(entry.Free, r.distance(entry.Token, end)))
+	}
+
+	// There is never an entry with a token of r.End, as the end of
+	// the ring is exclusive.
+	if end == r.End {
+		end = r.Start
+	}
+
+	//  If there is a token equal to the end of the range, we don't need to do anything further
+	if _, found := r.Entries.get(end); found {
+		return
+	}
+
+	// If not, we need to insert a token such that we claim this bit on the end.
+	endFree := r.distance(end, r.Entries.entry(pos).Token)
+	r.Entries.insert(entry{Token: end, Peer: r.Peer, Free: endFree})
+}
+
+// Merge the given ring into this ring and return any new ranges added
+func (r *Ring) Merge(gossip Ring) error {
+	r.assertInvariants()
+	defer r.assertInvariants()
+	defer r.updateExportedVariables()
+
+	// Don't panic when checking the gossiped in ring.
+	// In this case just return any error found.
+	if err := gossip.checkInvariants(); err != nil {
+		return err
+	}
+
+	if r.Start != gossip.Start || r.End != gossip.End {
+		return ErrDifferentSubnets
+	}
+
+	// Now merge their ring with yours, in a temporary ring.
+	var result entries
+	addToResult := func(e entry) { result = append(result, &e) }
+
+	var mine, theirs *entry
+	var previousOwner *router.PeerName
+	// i is index into r.Entries; j is index into gossip.Entries
+	var i, j int
+	for i < len(r.Entries) && j < len(gossip.Entries) {
+		mine, theirs = r.Entries[i], gossip.Entries[j]
+		switch {
+		case mine.Token < theirs.Token:
+			addToResult(*mine)
+			previousOwner = &mine.Peer
+			i++
+		case mine.Token > theirs.Token:
+			// insert, checking that a range owned by us hasn't been split
+			if previousOwner != nil && *previousOwner == r.Peer && theirs.Peer != r.Peer {
+				return ErrEntryInMyRange
+			}
+			addToResult(*theirs)
+			previousOwner = nil
+			j++
+		case mine.Token == theirs.Token:
+			// merge
+			switch {
+			case mine.Version >= theirs.Version:
+				if mine.Version == theirs.Version && !mine.Equal(theirs) {
+					common.Debug.Printf("Error merging entries at %s - %v != %v\n", mine.Token, mine, theirs)
+					return ErrInvalidEntry
+				}
+				addToResult(*mine)
+				previousOwner = &mine.Peer
+			case mine.Version < theirs.Version:
+				if mine.Peer == r.Peer { // We shouldn't receive updates to our own tokens
+					return ErrNewerVersion
+				}
+				addToResult(*theirs)
+				previousOwner = nil
+			}
+			i++
+			j++
+		}
+	}
+
+	// At this point, either i is at the end of r or j is at the end
+	// of gossip, so copy over the remaining entries.
+
+	for ; i < len(r.Entries); i++ {
+		mine = r.Entries[i]
+		addToResult(*mine)
+	}
+
+	for ; j < len(gossip.Entries); j++ {
+		theirs = gossip.Entries[j]
+		if previousOwner != nil && *previousOwner == r.Peer && theirs.Peer != r.Peer {
+			return ErrEntryInMyRange
+		}
+		addToResult(*theirs)
+		previousOwner = nil
+	}
+
+	r.Entries = result
+	return nil
+}
+
+// Empty returns true if the ring has no entries
+func (r *Ring) Empty() bool {
+	return len(r.Entries) == 0
+}
+
+// Given a slice of ranges which are all in the right order except
+// possibly the last one spans zero, fix that up and return the slice
+func (r *Ring) splitRangesOverZero(ranges []address.Range) []address.Range {
+	if len(ranges) == 0 {
+		return ranges
+	}
+	lastRange := ranges[len(ranges)-1]
+	// if end token == start (ie last) entry on ring, we want to actually use r.End
+	if lastRange.End == r.Start {
+		ranges[len(ranges)-1].End = r.End
+	} else if lastRange.End <= lastRange.Start {
+		// We wrapped; want to split around 0
+		// First shuffle everything up as we want results to be sorted
+		ranges = append(ranges, address.Range{})
+		copy(ranges[1:], ranges[:len(ranges)-1])
+		ranges[0] = address.Range{Start: r.Start, End: lastRange.End}
+		ranges[len(ranges)-1].End = r.End
+	}
+	return ranges
+}
+
+// OwnedRanges returns slice of Ranges, ordered by IP, indicating which
+// ranges are owned by this peer.  Will split ranges which
+// span 0 in the ring.
+func (r *Ring) OwnedRanges() (result []address.Range) {
+	r.assertInvariants()
+
+	for i, entry := range r.Entries {
+		if entry.Peer == r.Peer {
+			nextEntry := r.Entries.entry(i + 1)
+			result = append(result, address.Range{Start: entry.Token, End: nextEntry.Token})
+		}
+	}
+
+	return r.splitRangesOverZero(result)
+}
+
+// ClaimForPeers claims the entire ring for the array of peers passed
+// in.  Only works for empty rings.
+func (r *Ring) ClaimForPeers(peers []router.PeerName) {
+	common.Assert(r.Empty())
+	defer r.assertInvariants()
+	defer r.updateExportedVariables()
+
+	totalSize := r.distance(r.Start, r.End)
+	share := totalSize/address.Offset(len(peers)) + 1
+	remainder := totalSize % address.Offset(len(peers))
+	pos := r.Start
+
+	for i, peer := range peers {
+		if address.Offset(i) == remainder {
+			share--
+			if share == 0 {
+				break
+			}
+		}
+
+		if e, found := r.Entries.get(pos); found {
+			e.update(peer, share)
+		} else {
+			r.Entries.insert(entry{Token: pos, Peer: peer, Free: share})
+		}
+
+		pos += address.Address(share)
+	}
+
+	common.Assert(pos == r.End)
+}
+
+func (r *Ring) ClaimItAll() {
+	r.ClaimForPeers([]router.PeerName{r.Peer})
+}
+
+func (r *Ring) FprintWithNicknames(w io.Writer, m map[router.PeerName]string) {
+	for _, entry := range r.Entries {
+		nickname, found := m[entry.Peer]
+		if found {
+			nickname = fmt.Sprintf(" (%s)", nickname)
+		}
+
+		fmt.Fprintf(w, "\n  %s -> %s%s (v%d)", entry.Token,
+			entry.Peer, nickname, entry.Version)
+	}
+}
+
+func (r *Ring) String() string {
+	var buffer bytes.Buffer
+	fmt.Fprintf(&buffer, "Ring [%s, %s)", r.Start, r.End)
+	r.FprintWithNicknames(&buffer, make(map[router.PeerName]string))
+	return buffer.String()
+}
+
+// ReportFree is used by the allocator to tell the ring
+// how many free ips are in a given range, so that ChoosePeerToAskForSpace
+// can make more intelligent decisions.
+func (r *Ring) ReportFree(freespace map[address.Address]address.Offset) {
+	r.assertInvariants()
+	defer r.assertInvariants()
+	defer r.updateExportedVariables()
+
+	common.Assert(!r.Empty())
+	entries := r.Entries
+
+	// As OwnedRanges splits around the origin, we need to
+	// detect that here and fix up freespace
+	if free, found := freespace[r.Start]; found && entries.entry(0).Token != r.Start {
+		lastToken := entries.entry(-1).Token
+		prevFree, found := freespace[lastToken]
+		common.Assert(found)
+		freespace[lastToken] = prevFree + free
+		delete(freespace, r.Start)
+	}
+
+	for start, free := range freespace {
+		// Look for entry
+		i := sort.Search(len(entries), func(j int) bool {
+			return entries[j].Token >= start
+		})
+
+		// Are you trying to report free on space I don't own?
+		common.Assert(i < len(entries) && entries[i].Token == start && entries[i].Peer == r.Peer)
+
+		// Check we're not reporting more space than the range
+		entry, next := entries.entry(i), entries.entry(i+1)
+		maxSize := r.distance(entry.Token, next.Token)
+		common.Assert(free <= maxSize)
+
+		if entries[i].Free == free {
+			return
+		}
+
+		entries[i].Free = free
+		entries[i].Version++
+	}
+}
+
+// ChoosePeerToAskForSpace chooses a weighted-random peer to ask
+// for space.
+func (r *Ring) ChoosePeerToAskForSpace() (result router.PeerName, err error) {
+	var (
+		sum               address.Offset
+		totalSpacePerPeer = make(map[router.PeerName]address.Offset) // Compute total free space per peer
+	)
+
+	// iterate through tokens
+	for _, entry := range r.Entries {
+		// Ignore ranges with no free space
+		if entry.Free <= 0 {
+			continue
+		}
+
+		// Don't talk to yourself
+		if entry.Peer == r.Peer {
+			continue
+		}
+
+		totalSpacePerPeer[entry.Peer] += entry.Free
+		sum += entry.Free
+	}
+
+	if sum == 0 {
+		err = ErrNoFreeSpace
+		return
+	}
+
+	// Pick random peer, weighted by total free space
+	rn := rand.Int63n(int64(sum))
+	for peername, space := range totalSpacePerPeer {
+		rn -= int64(space)
+		if rn < 0 {
+			return peername, nil
+		}
+	}
+
+	panic("Should never reach this point")
+}
+
+func (r *Ring) PickPeerForTransfer() router.PeerName {
+	for _, entry := range r.Entries {
+		if entry.Peer != r.Peer {
+			return entry.Peer
+		}
+	}
+	return router.UnknownPeerName
+}
+
+// Transfer will mark all entries associated with 'from' peer as owned by 'to' peer
+// and return ranges indicating the new space we picked up
+func (r *Ring) Transfer(from, to router.PeerName) (error, []address.Range) {
+	r.assertInvariants()
+	defer r.assertInvariants()
+	defer r.updateExportedVariables()
+
+	var newRanges []address.Range
+	found := false
+
+	for i, entry := range r.Entries {
+		if entry.Peer == from {
+			found = true
+			entry.Peer = to
+			entry.Version++
+			newRanges = append(newRanges, address.Range{Start: entry.Token, End: r.Entries.entry(i + 1).Token})
+		}
+	}
+
+	if !found {
+		return ErrNotFound, nil
+	}
+
+	return nil, r.splitRangesOverZero(newRanges)
+}
+
+// Contains returns true if addr is in this ring
+func (r *Ring) Contains(addr address.Address) bool {
+	return addr >= r.Start && addr < r.End
+}
+
+// Owner returns the peername which owns the range containing addr
+func (r *Ring) Owner(token address.Address) router.PeerName {
+	common.Assert(r.Start <= token && token < r.End)
+
+	r.assertInvariants()
+	// There can be no owners on an empty ring
+	if r.Empty() {
+		return router.UnknownPeerName
+	}
+
+	// Look for the right-most entry, less than or equal to token
+	preceedingEntry := sort.Search(len(r.Entries), func(j int) bool {
+		return r.Entries[j].Token > token
+	})
+	preceedingEntry--
+	entry := r.Entries.entry(preceedingEntry)
+	return entry.Peer
+}
+
+// Get the set of PeerNames mentioned in the ring
+func (r *Ring) PeerNames() map[router.PeerName]struct{} {
+	res := make(map[router.PeerName]struct{})
+
+	for _, entry := range r.Entries {
+		res[entry.Peer] = struct{}{}
+	}
+
+	return res
+}

ipam/ring/ring_test.go

@@ -0,0 +1,724 @@
+package ring
+
+import (
+	"bytes"
+	"fmt"
+	"math/rand"
+	"sort"
+	"testing"
+
+	"github.com/weaveworks/weave/common"
+	"github.com/weaveworks/weave/ipam/address"
+	"github.com/weaveworks/weave/router"
+	wt "github.com/weaveworks/weave/testing"
+)
+
+var (
+	peer1name, _ = router.PeerNameFromString("01:00:00:00:02:00")
+	peer2name, _ = router.PeerNameFromString("02:00:00:00:02:00")
+	peer3name, _ = router.PeerNameFromString("03:00:00:00:02:00")
+
+	start, end    = ParseIP("10.0.0.0"), ParseIP("10.0.0.255")
+	dot10, dot245 = ParseIP("10.0.0.10"), ParseIP("10.0.0.245")
+	dot250        = ParseIP("10.0.0.250")
+	middle        = ParseIP("10.0.0.128")
+)
+
+func ParseIP(s string) address.Address {
+	addr, _ := address.ParseIP(s)
+	return addr
+}
+
+func TestInvariants(t *testing.T) {
+	ring := New(start, end, peer1name)
+
+	// Check ring is sorted
+	ring.Entries = []*entry{{Token: dot245, Peer: peer1name}, {Token: dot10, Peer: peer2name}}
+	wt.AssertTrue(t, ring.checkInvariants() == ErrNotSorted, "Expected error")
+
+	// Check tokens don't appear twice
+	ring.Entries = []*entry{{Token: dot245, Peer: peer1name}, {Token: dot245, Peer: peer2name}}
+	wt.AssertTrue(t, ring.checkInvariants() == ErrTokenRepeated, "Expected error")
+
+	// Check tokens are in bounds
+	ring = New(dot10, dot245, peer1name)
+	ring.Entries = []*entry{{Token: start, Peer: peer1name}}
+	wt.AssertTrue(t, ring.checkInvariants() == ErrTokenOutOfRange, "Expected error")
+
+	ring.Entries = []*entry{{Token: end, Peer: peer1name}}
+	wt.AssertTrue(t, ring.checkInvariants() == ErrTokenOutOfRange, "Expected error")
+}
+
+func TestInsert(t *testing.T) {
+	ring := New(start, end, peer1name)
+	ring.Entries = []*entry{{Token: start, Peer: peer1name, Free: 255}}
+
+	wt.AssertPanic(t, func() {
+		ring.Entries.insert(entry{Token: start, Peer: peer1name})
+	})
+
+	ring.Entries.entry(0).Free = 0
+	ring.Entries.insert(entry{Token: dot245, Peer: peer1name})
+	ring2 := New(start, end, peer1name)
+	ring2.Entries = []*entry{{Token: start, Peer: peer1name, Free: 0}, {Token: dot245, Peer: peer1name}}
+	wt.AssertEquals(t, ring, ring2)
+
+	ring.Entries.insert(entry{Token: dot10, Peer: peer1name})
+	ring2.Entries = []*entry{{Token: start, Peer: peer1name, Free: 0}, {Token: dot10, Peer: peer1name}, {Token: dot245, Peer: peer1name}}
+	wt.AssertEquals(t, ring, ring2)
+}
+
+func TestBetween(t *testing.T) {
+	ring1 := New(start, end, peer1name)
+	ring1.Entries = []*entry{{Token: start, Peer: peer1name, Free: 255}}
+
+	// First off, in a ring where everything is owned by the peer
+	// between should return true for everything
+	for i := 1; i <= 255; i++ {
+		ip := ParseIP(fmt.Sprintf("10.0.0.%d", i))
+		wt.AssertTrue(t, ring1.Entries.between(ip, 0, 1), "between should be true!")
+	}
+
+	// Now, construct a ring with entries at +10 and -10
+	// And check the correct behaviour
+
+	ring1.Entries = []*entry{{Token: dot10, Peer: peer1name}, {Token: dot245, Peer: peer2name}}
+	ring1.assertInvariants()
+	for i := 10; i <= 244; i++ {
+		ipStr := fmt.Sprintf("10.0.0.%d", i)
+		ip := ParseIP(ipStr)
+		wt.AssertTrue(t, ring1.Entries.between(ip, 0, 1),
+			fmt.Sprintf("Between should be true for %s!", ipStr))
+		wt.AssertFalse(t, ring1.Entries.between(ip, 1, 2),
+			fmt.Sprintf("Between should be false for %s!", ipStr))
+	}
+	for i := 0; i <= 9; i++ {
+		ipStr := fmt.Sprintf("10.0.0.%d", i)
+		ip := ParseIP(ipStr)
+		wt.AssertFalse(t, ring1.Entries.between(ip, 0, 1),
+			fmt.Sprintf("Between should be false for %s!", ipStr))
+		wt.AssertTrue(t, ring1.Entries.between(ip, 1, 2),
+			fmt.Sprintf("Between should be true for %s!", ipStr))
+	}
+	for i := 245; i <= 255; i++ {
+		ipStr := fmt.Sprintf("10.0.0.%d", i)
+		ip := ParseIP(ipStr)
+		wt.AssertFalse(t, ring1.Entries.between(ip, 0, 1),
+			fmt.Sprintf("Between should be false for %s!", ipStr))
+		wt.AssertTrue(t, ring1.Entries.between(ip, 1, 2),
+			fmt.Sprintf("Between should be true for %s!", ipStr))
+	}
+}
+
+func TestGrantSimple(t *testing.T) {
+	ring1 := New(start, end, peer1name)
+	ring2 := New(start, end, peer2name)
+
+	// Claim everything for peer1
+	ring1.ClaimItAll()
+	wt.AssertEquals(t, ring1.Entries, entries{{Token: start, Peer: peer1name, Free: 255}})
+
+	// Now grant everything to peer2
+	ring1.GrantRangeToHost(start, end, peer2name)
+	ring2.Entries = []*entry{{Token: start, Peer: peer2name, Free: 255, Version: 1}}
+	wt.AssertEquals(t, ring1.Entries, ring2.Entries)
+
+	// Now spint back to peer 1
+	ring2.GrantRangeToHost(dot10, end, peer1name)
+	ring1.Entries = []*entry{{Token: start, Peer: peer2name, Free: 10, Version: 2},
+		{Token: dot10, Peer: peer1name, Free: 245}}
+	wt.AssertEquals(t, ring1.Entries, ring2.Entries)
+
+	// And spint back to peer 2 again
+	ring1.GrantRangeToHost(dot245, end, peer2name)
+	wt.AssertEquals(t, ring1.Entries, entries{{Token: start, Peer: peer2name, Free: 10, Version: 2},
+		{Token: dot10, Peer: peer1name, Free: 235, Version: 1},
+		{Token: dot245, Peer: peer2name, Free: 10}})
+
+	// Grant range spanning a live token
+	ring1.Entries = []*entry{{Token: start, Peer: peer1name, Free: 10, Version: 2},
+		{Token: dot10, Peer: peer1name, Free: 235}, {Token: dot245, Peer: peer1name, Free: 10}}
+	ring1.GrantRangeToHost(dot10, end, peer2name)
+	wt.AssertEquals(t, ring1.Entries, entries{{Token: start, Peer: peer1name, Free: 10, Version: 2},
+		{Token: dot10, Peer: peer2name, Free: 235, Version: 1},
+		{Token: dot245, Peer: peer2name, Free: 10, Version: 1}})
+}
+
+func TestGrantSplit(t *testing.T) {
+	ring1 := New(start, end, peer1name)
+	ring2 := New(start, end, peer2name)
+
+	// Claim everything for peer1
+	ring1.Entries = []*entry{{Token: start, Peer: peer1name, Free: 255}}
+	ring2.Merge(*ring1)
+	wt.AssertEquals(t, ring1.Entries, ring2.Entries)
+
+	// Now grant a split range to peer2
+	ring1.GrantRangeToHost(dot10, dot245, peer2name)
+	wt.AssertEquals(t, ring1.Entries, entries{{Token: start, Peer: peer1name, Free: 10, Version: 1},
+		{Token: dot10, Peer: peer2name, Free: 235},
+		{Token: dot245, Peer: peer1name, Free: 10}})
+	ring1.assertInvariants()
+
+	// Grant range spanning a live token, and inserting a new token
+	ring1.Entries = []*entry{{Token: start, Peer: peer1name, Free: 10, Version: 2},
+		{Token: dot10, Peer: peer1name, Free: 118}, {Token: middle, Peer: peer1name, Free: 127}}
+	ring1.GrantRangeToHost(dot10, dot245, peer2name)
+	wt.AssertEquals(t, ring1.Entries, entries{{Token: start, Peer: peer1name, Free: 10, Version: 2},
+		{Token: dot10, Peer: peer2name, Free: 118, Version: 1},
+		{Token: middle, Peer: peer2name, Free: 117, Version: 1},
+		{Token: dot245, Peer: peer1name, Free: 10, Version: 0}})
+	ring1.assertInvariants()
+}
+
+func TestMergeSimple(t *testing.T) {
+	ring1 := New(start, end, peer1name)
+	ring2 := New(start, end, peer2name)
+
+	// Claim everything for peer1
+	ring1.ClaimItAll()
+	ring1.GrantRangeToHost(middle, end, peer2name)
+	wt.AssertSuccess(t, ring2.Merge(*ring1))
+
+	wt.AssertEquals(t, ring1.Entries, entries{{Token: start, Peer: peer1name, Free: 128, Version: 1},
+		{Token: middle, Peer: peer2name, Free: 127}})
+	wt.AssertEquals(t, ring1.Entries, ring2.Entries)
+
+	// Now to two different operations on either side,
+	// check we can Merge again
+	ring1.GrantRangeToHost(start, middle, peer2name)
+	ring2.GrantRangeToHost(middle, end, peer1name)
+
+	wt.AssertSuccess(t, ring2.Merge(*ring1))
+	wt.AssertSuccess(t, ring1.Merge(*ring2))
+
+	wt.AssertEquals(t, ring1.Entries, entries{{Token: start, Peer: peer2name, Free: 128, Version: 2},
+		{Token: middle, Peer: peer1name, Free: 127, Version: 1}})
+	wt.AssertEquals(t, ring1.Entries, ring2.Entries)
+}
+
+func TestMergeErrors(t *testing.T) {
+	// Cannot Merge in an invalid ring
+	ring1 := New(start, end, peer1name)
+	ring2 := New(start, end, peer2name)
+	ring2.Entries = []*entry{{Token: middle, Peer: peer2name}, {Token: start, Peer: peer2name}}
+	wt.AssertTrue(t, ring1.Merge(*ring2) == ErrNotSorted, "Expected ErrNotSorted")
+
+	// Should Merge two rings for different ranges
+	ring2 = New(start, middle, peer2name)
+	ring2.Entries = []*entry{}
+	wt.AssertTrue(t, ring1.Merge(*ring2) == ErrDifferentSubnets, "Expected ErrDifferentSubnets")
+
+	// Cannot Merge newer version of entry I own
+	ring2 = New(start, end, peer2name)
+	ring1.Entries = []*entry{{Token: start, Peer: peer1name}}
+	ring2.Entries = []*entry{{Token: start, Peer: peer1name, Version: 1}}
+	wt.AssertTrue(t, ring1.Merge(*ring2) == ErrNewerVersion, "Expected ErrNewerVersion")
+
+	// Cannot Merge two entries with same version but different hosts
+	ring1.Entries = []*entry{{Token: start, Peer: peer1name}}
+	ring2.Entries = []*entry{{Token: start, Peer: peer2name}}
+	wt.AssertTrue(t, ring1.Merge(*ring2) == ErrInvalidEntry, "Expected ErrInvalidEntry")
+
+	// Cannot Merge an entry into a range I own
+	ring1.Entries = []*entry{{Token: start, Peer: peer1name}}
+	ring2.Entries = []*entry{{Token: middle, Peer: peer2name}}
+	wt.AssertTrue(t, ring1.Merge(*ring2) == ErrEntryInMyRange, "Expected ErrEntryInMyRange")
+}
+
+func TestMergeMore(t *testing.T) {
+	ring1 := New(start, end, peer1name)
+	ring2 := New(start, end, peer2name)
+
+	assertRing := func(ring *Ring, entries entries) {
+		wt.AssertEquals(t, ring.Entries, entries)
+	}
+
+	assertRing(ring1, []*entry{})
+	assertRing(ring2, []*entry{})
+
+	// Claim everything for peer1
+	ring1.ClaimItAll()
+	assertRing(ring1, []*entry{{Token: start, Peer: peer1name, Free: 255}})
+	assertRing(ring2, []*entry{})
+
+	// Check the Merge sends it to the other ring
+	wt.AssertSuccess(t, ring2.Merge(*ring1))
+	assertRing(ring1, []*entry{{Token: start, Peer: peer1name, Free: 255}})
+	assertRing(ring2, []*entry{{Token: start, Peer: peer1name, Free: 255}})
+
+	// Give everything to peer2
+	ring1.GrantRangeToHost(start, end, peer2name)
+	assertRing(ring1, []*entry{{Token: start, Peer: peer2name, Free: 255, Version: 1}})
+	assertRing(ring2, []*entry{{Token: start, Peer: peer1name, Free: 255}})
+
+	wt.AssertSuccess(t, ring2.Merge(*ring1))
+	assertRing(ring1, []*entry{{Token: start, Peer: peer2name, Free: 255, Version: 1}})
+	assertRing(ring2, []*entry{{Token: start, Peer: peer2name, Free: 255, Version: 1}})
+
+	// And carve off some space
+	ring2.GrantRangeToHost(middle, end, peer1name)
+	assertRing(ring2, []*entry{{Token: start, Peer: peer2name, Free: 128, Version: 2},
+		{Token: middle, Peer: peer1name, Free: 127}})
+	assertRing(ring1, []*entry{{Token: start, Peer: peer2name, Free: 255, Version: 1}})
+
+	// And Merge back
+	wt.AssertSuccess(t, ring1.Merge(*ring2))
+	assertRing(ring1, []*entry{{Token: start, Peer: peer2name, Free: 128, Version: 2},
+		{Token: middle, Peer: peer1name, Free: 127}})
+	assertRing(ring2, []*entry{{Token: start, Peer: peer2name, Free: 128, Version: 2},
+		{Token: middle, Peer: peer1name, Free: 127}})
+
+	// This should be a no-op
+	wt.AssertSuccess(t, ring2.Merge(*ring1))
+	assertRing(ring1, []*entry{{Token: start, Peer: peer2name, Free: 128, Version: 2},
+		{Token: middle, Peer: peer1name, Free: 127}})
+	assertRing(ring2, []*entry{{Token: start, Peer: peer2name, Free: 128, Version: 2},
+		{Token: middle, Peer: peer1name, Free: 127}})
+}
+
+func TestMergeSplit(t *testing.T) {
+	ring1 := New(start, end, peer1name)
+	ring2 := New(start, end, peer2name)
+
+	// Claim everything for peer2
+	ring1.Entries = []*entry{{Token: start, Peer: peer2name, Free: 255}}
+	wt.AssertSuccess(t, ring2.Merge(*ring1))
+	wt.AssertEquals(t, ring1.Entries, ring2.Entries)
+
+	// Now grant a split range to peer1
+	ring2.GrantRangeToHost(dot10, dot245, peer1name)
+	wt.AssertEquals(t, ring2.Entries, entries{{Token: start, Peer: peer2name, Free: 10, Version: 1},
+		{Token: dot10, Peer: peer1name, Free: 235},
+		{Token: dot245, Peer: peer2name, Free: 10}})
+	wt.AssertSuccess(t, ring1.Merge(*ring2))
+	wt.AssertEquals(t, ring1.Entries, entries{{Token: start, Peer: peer2name, Free: 10, Version: 1},
+		{Token: dot10, Peer: peer1name, Free: 235},
+		{Token: dot245, Peer: peer2name, Free: 10}})
+	wt.AssertEquals(t, ring1.Entries, ring2.Entries)
+}
+
+func TestMergeSplit2(t *testing.T) {
+	ring1 := New(start, end, peer1name)
+	ring2 := New(start, end, peer2name)
+
+	// Claim everything for peer2
+	ring1.Entries = []*entry{{Token: start, Peer: peer2name, Free: 250}, {Token: dot250, Peer: peer2name, Free: 5}}
+	wt.AssertSuccess(t, ring2.Merge(*ring1))
+	wt.AssertEquals(t, ring1.Entries, ring2.Entries)
+
+	// Now grant a split range to peer1
+	ring2.GrantRangeToHost(dot10, dot245, peer1name)
+	wt.AssertEquals(t, ring2.Entries, entries{{Token: start, Peer: peer2name, Free: 10, Version: 1},
+		{Token: dot10, Peer: peer1name, Free: 235},
+		{Token: dot245, Peer: peer2name, Free: 5}, {Token: dot250, Peer: peer2name, Free: 5}})
+	wt.AssertSuccess(t, ring1.Merge(*ring2))
+	wt.AssertEquals(t, ring1.Entries, entries{{Token: start, Peer: peer2name, Free: 10, Version: 1},
+		{Token: dot10, Peer: peer1name, Free: 235},
+		{Token: dot245, Peer: peer2name, Free: 5}, {Token: dot250, Peer: peer2name, Free: 5}})
+	wt.AssertEquals(t, ring1.Entries, ring2.Entries)
+}
+
+// A simple test, very similar to above, but using the marshalling to byte[]s
+func TestGossip(t *testing.T) {
+	ring1 := New(start, end, peer1name)
+	ring2 := New(start, end, peer2name)
+
+	assertRing := func(ring *Ring, entries entries) {
+		wt.AssertEquals(t, ring.Entries, entries)
+	}
+
+	assertRing(ring1, []*entry{})
+	assertRing(ring2, []*entry{})
+
+	// Claim everything for peer1
+	ring1.ClaimItAll()
+	assertRing(ring1, []*entry{{Token: start, Peer: peer1name, Free: 255}})
+	assertRing(ring2, []*entry{})
+
+	// Check the Merge sends it to the other ring
+	wt.AssertSuccess(t, ring2.Merge(*ring1))
+	assertRing(ring1, []*entry{{Token: start, Peer: peer1name, Free: 255}})
+	assertRing(ring2, []*entry{{Token: start, Peer: peer1name, Free: 255}})
+}
+
+func TestFindFree(t *testing.T) {
+	ring1 := New(start, end, peer1name)
+
+	_, err := ring1.ChoosePeerToAskForSpace()
+	wt.AssertTrue(t, err == ErrNoFreeSpace, "Expected ErrNoFreeSpace")
+
+	ring1.Entries = []*entry{{Token: start, Peer: peer1name}}
+	_, err = ring1.ChoosePeerToAskForSpace()
+	wt.AssertTrue(t, err == ErrNoFreeSpace, "Expected ErrNoFreeSpace")
+
+	// We shouldn't return outselves
+	ring1.ReportFree(map[address.Address]address.Offset{start: 10})
+	_, err = ring1.ChoosePeerToAskForSpace()
+	wt.AssertTrue(t, err == ErrNoFreeSpace, "Expected ErrNoFreeSpace")
+
+	ring1.Entries = []*entry{{Token: start, Peer: peer1name, Free: 1},
+		{Token: start, Peer: peer1name, Free: 1}}
+	_, err = ring1.ChoosePeerToAskForSpace()
+	wt.AssertTrue(t, err == ErrNoFreeSpace, "Expected ErrNoFreeSpace")
+
+	// We should return others
+	ring1.Entries = []*entry{{Token: start, Peer: peer2name, Free: 1}}
+	peer, err := ring1.ChoosePeerToAskForSpace()
+	wt.AssertSuccess(t, err)
+	wt.AssertEquals(t, peer, peer2name)
+
+	ring1.Entries = []*entry{{Token: start, Peer: peer2name, Free: 1},
+		{Token: start, Peer: peer2name, Free: 1}}
+	peer, err = ring1.ChoosePeerToAskForSpace()
+	wt.AssertSuccess(t, err)
+	wt.AssertEquals(t, peer, peer2name)
+}
+
+func TestReportFree(t *testing.T) {
+	ring1 := New(start, end, peer1name)
+	ring2 := New(start, end, peer2name)
+
+	ring1.ClaimItAll()
+	ring1.GrantRangeToHost(middle, end, peer2name)
+	wt.AssertSuccess(t, ring2.Merge(*ring1))
+
+	freespace := make(map[address.Address]address.Offset)
+	for _, r := range ring2.OwnedRanges() {
+		freespace[r.Start] = 0
+	}
+	ring2.ReportFree(freespace)
+}
+
+func TestMisc(t *testing.T) {
+	ring := New(start, end, peer1name)
+
+	wt.AssertTrue(t, ring.Empty(), "empty")
+
+	ring.ClaimItAll()
+	println(ring.String())
+}
+
+func TestEmptyGossip(t *testing.T) {
+	ring1 := New(start, end, peer1name)
+	ring2 := New(start, end, peer2name)
+
+	ring1.ClaimItAll()
+	// This used to panic, and it shouldn't
+	wt.AssertSuccess(t, ring1.Merge(*ring2))
+}
+
+func TestMergeOldMessage(t *testing.T) {
+	ring1 := New(start, end, peer1name)
+	ring2 := New(start, end, peer2name)
+
+	ring1.ClaimItAll()
+	wt.AssertSuccess(t, ring2.Merge(*ring1))
+
+	ring1.GrantRangeToHost(middle, end, peer1name)
+	wt.AssertSuccess(t, ring1.Merge(*ring2))
+}
+
+func TestSplitRangeAtBeginning(t *testing.T) {
+	ring1 := New(start, end, peer1name)
+	ring2 := New(start, end, peer2name)
+
+	ring1.ClaimItAll()
+	wt.AssertSuccess(t, ring2.Merge(*ring1))
+
+	ring1.GrantRangeToHost(start, middle, peer2name)
+	wt.AssertSuccess(t, ring2.Merge(*ring1))
+}
+
+func TestOwnedRange(t *testing.T) {
+	ring1 := New(start, end, peer1name)
+	ring1.ClaimItAll()
+
+	wt.AssertEquals(t, ring1.OwnedRanges(), []address.Range{{Start: start, End: end}})
+
+	ring1.GrantRangeToHost(middle, end, peer2name)
+	wt.AssertEquals(t, ring1.OwnedRanges(), []address.Range{{Start: start, End: middle}})
+
+	ring2 := New(start, end, peer2name)
+	ring2.Merge(*ring1)
+	wt.AssertEquals(t, ring2.OwnedRanges(), []address.Range{{Start: middle, End: end}})
+
+	ring2.Entries = []*entry{{Token: middle, Peer: peer2name}}
+	wt.AssertEquals(t, ring2.OwnedRanges(),
+		[]address.Range{{Start: start, End: middle}, {Start: middle, End: end}})
+
+	ring2.Entries = []*entry{{Token: dot10, Peer: peer2name}, {Token: middle, Peer: peer2name}}
+	wt.AssertEquals(t, ring2.OwnedRanges(),
+		[]address.Range{{Start: start, End: dot10}, {Start: dot10, End: middle},
+			{Start: middle, End: end}})
+}
+
+func TestTransfer(t *testing.T) {
+	// First test just checks if we can grant some range to a host, when we transfer it, we get it back
+	ring1 := New(start, end, peer1name)
+	ring1.ClaimItAll()
+	ring1.GrantRangeToHost(middle, end, peer2name)
+	ring1.Transfer(peer2name, peer1name)
+	wt.AssertEquals(t, ring1.OwnedRanges(), []address.Range{{start, middle}, {middle, end}})
+
+	// Second test is what happens when a token exists at the end of a range but is transferred
+	// - does it get resurrected correctly?
+	ring1 = New(start, end, peer1name)
+	ring1.ClaimItAll()
+	ring1.GrantRangeToHost(middle, end, peer2name)
+	ring1.Transfer(peer2name, peer1name)
+	ring1.GrantRangeToHost(dot10, middle, peer2name)
+	wt.AssertEquals(t, ring1.OwnedRanges(), []address.Range{{start, dot10}, {middle, end}})
+}
+
+func TestOwner(t *testing.T) {
+	ring1 := New(start, end, peer1name)
+	wt.AssertTrue(t, ring1.Contains(start), "start should be in ring")
+	wt.AssertFalse(t, ring1.Contains(end), "end should not be in ring")
+
+	wt.AssertEquals(t, ring1.Owner(start), router.UnknownPeerName)
+
+	ring1.ClaimItAll()
+	ring1.GrantRangeToHost(middle, end, peer2name)
+	wt.AssertEquals(t, ring1.Owner(start), peer1name)
+	wt.AssertEquals(t, ring1.Owner(middle), peer2name)
+	wt.AssertPanic(t, func() {
+		ring1.Owner(end)
+	})
+}
+
+type addressSlice []address.Address
+
+func (s addressSlice) Len() int           { return len(s) }
+func (s addressSlice) Less(i, j int) bool { return s[i] < s[j] }
+func (s addressSlice) Swap(i, j int)      { s[i], s[j] = s[j], s[i] }
+
+func TestFuzzRing(t *testing.T) {
+	var (
+		numPeers   = 25
+		iterations = 1000
+	)
+
+	peers := make([]router.PeerName, numPeers)
+	for i := 0; i < numPeers; i++ {
+		peer, _ := router.PeerNameFromString(fmt.Sprintf("%02d:00:00:00:02:00", i))
+		peers[i] = peer
+	}
+
+	// Make a valid, random ring
+	makeGoodRandomRing := func() *Ring {
+		addressSpace := end - start
+		numTokens := rand.Intn(int(addressSpace))
+
+		tokenMap := make(map[address.Address]bool)
+		for i := 0; i < numTokens; i++ {
+			tokenMap[address.Address(rand.Intn(int(addressSpace)))] = true
+		}
+		var tokens []address.Address
+		for token := range tokenMap {
+			tokens = append(tokens, token)
+		}
+		sort.Sort(addressSlice(tokens))
+
+		peer := peers[rand.Intn(len(peers))]
+		ring := New(start, end, peer)
+		for _, token := range tokens {
+			peer = peers[rand.Intn(len(peers))]
+			ring.Entries = append(ring.Entries, &entry{Token: start + token, Peer: peer})
+		}
+
+		ring.assertInvariants()
+		return ring
+	}
+
+	for i := 0; i < iterations; i++ {
+		// make 2 random rings
+		ring1 := makeGoodRandomRing()
+		ring2 := makeGoodRandomRing()
+
+		// Merge them - this might fail, we don't care
+		// We just want to make sure it doesn't panic
+		ring1.Merge(*ring2)
+
+		// Check whats left still passes assertions
+		ring1.assertInvariants()
+		ring2.assertInvariants()
+	}
+
+	// Make an invalid, random ring
+	makeBadRandomRing := func() *Ring {
+		addressSpace := end - start
+		numTokens := rand.Intn(int(addressSpace))
+		tokens := make([]address.Address, numTokens)
+		for i := 0; i < numTokens; i++ {
+			tokens[i] = address.Address(rand.Intn(int(addressSpace)))
+		}
+
+		peer := peers[rand.Intn(len(peers))]
+		ring := New(start, end, peer)
+		for _, token := range tokens {
+			peer = peers[rand.Intn(len(peers))]
+			ring.Entries = append(ring.Entries, &entry{Token: start + token, Peer: peer})
+		}
+
+		return ring
+	}
+
+	for i := 0; i < iterations; i++ {
+		// make 2 random rings
+		ring1 := makeGoodRandomRing()
+		ring2 := makeBadRandomRing()
+
+		// Merge them - this might fail, we don't care
+		// We just want to make sure it doesn't panic
+		ring1.Merge(*ring2)
+
+		// Check whats left still passes assertions
+		ring1.assertInvariants()
+	}
+}
+
+func TestFuzzRingHard(t *testing.T) {
+	//common.InitDefaultLogging(true)
+	var (
+		numPeers   = 100
+		iterations = 3000
+		peers      []router.PeerName
+		rings      []*Ring
+		nextPeerID = 0
+	)
+
+	addPeer := func() {
+		peer, _ := router.PeerNameFromString(fmt.Sprintf("%02d:%02d:00:00:00:00", nextPeerID/10, nextPeerID%10))
+		common.Debug.Printf("%s: Adding peer", peer)
+		nextPeerID++
+		peers = append(peers, peer)
+		rings = append(rings, New(start, end, peer))
+	}
+
+	for i := 0; i < numPeers; i++ {
+		addPeer()
+	}
+
+	rings[0].ClaimItAll()
+
+	randomPeer := func(exclude int) (int, router.PeerName, *Ring) {
+		var peerIndex int
+		if exclude >= 0 {
+			peerIndex = rand.Intn(len(peers) - 1)
+			if peerIndex == exclude {
+				peerIndex++
+			}
+		} else {
+			peerIndex = rand.Intn(len(peers))
+		}
+		return peerIndex, peers[peerIndex], rings[peerIndex]
+	}
+
+	// Keep a map of index -> ranges, as these are a little expensive to
+	// calculate for every ring on every iteration.
+	var theRanges = make(map[int][]address.Range)
+	theRanges[0] = rings[0].OwnedRanges()
+
+	addOrRmPeer := func() {
+		if len(peers) < numPeers {
+			addPeer()
+			return
+		}
+
+		peerIndex, peername, _ := randomPeer(-1)
+		// Remove peer from our state
+		peers = append(peers[:peerIndex], peers[peerIndex+1:]...)
+		rings = append(rings[:peerIndex], rings[peerIndex+1:]...)
+		theRanges = make(map[int][]address.Range)
+
+		// Transfer the space for this peer on another peer, but not this one
+		_, otherPeername, otherRing := randomPeer(peerIndex)
+
+		// We need to be in a ~converged ring to rmpeer
+		for _, ring := range rings {
+			wt.AssertSuccess(t, otherRing.Merge(*ring))
+		}
+
+		common.Debug.Printf("%s: transferring from peer %s", otherPeername, peername)
+		otherRing.Transfer(peername, peername)
+
+		// And now tell everyone about the transfer - rmpeer is
+		// not partition safe
+		for i, ring := range rings {
+			wt.AssertSuccess(t, ring.Merge(*otherRing))
+			theRanges[i] = ring.OwnedRanges()
+		}
+	}
+
+	doGrantOrGossip := func() {
+		var ringsWithRanges = make([]int, 0, len(rings))
+		for index, ranges := range theRanges {
+			if len(ranges) > 0 {
+				ringsWithRanges = append(ringsWithRanges, index)
+			}
+		}
+
+		if len(ringsWithRanges) > 0 {
+			// Produce a random split in a random owned range, given to a random peer
+			indexWithRanges := ringsWithRanges[rand.Intn(len(ringsWithRanges))]
+			ownedRanges := theRanges[indexWithRanges]
+			ring := rings[indexWithRanges]
+
+			rangeToSplit := ownedRanges[rand.Intn(len(ownedRanges))]
+			size := address.Subtract(rangeToSplit.End, rangeToSplit.Start)
+			ipInRange := address.Add(rangeToSplit.Start, address.Offset(rand.Intn(int(size))))
+			_, peerToGiveTo, _ := randomPeer(-1)
+			common.Debug.Printf("%s: Granting [%v, %v) to %s", ring.Peer, ipInRange, rangeToSplit.End, peerToGiveTo)
+			ring.GrantRangeToHost(ipInRange, rangeToSplit.End, peerToGiveTo)
+
+			// Now 'gossip' this to a random host (note, note could be same host as above)
+			otherIndex, _, otherRing := randomPeer(-1)
+			common.Debug.Printf("%s: 'Gossiping' to %s", ring.Peer, otherRing.Peer)
+			wt.AssertSuccess(t, otherRing.Merge(*ring))
+
+			theRanges[indexWithRanges] = ring.OwnedRanges()
+			theRanges[otherIndex] = otherRing.OwnedRanges()
+			return
+		}
+
+		// No rings think they own anything (as gossip might be behind)
+		// We're going to pick a random host (which has entries) and gossip
+		// it to a random host (which may or may not have entries).
+		var ringsWithEntries = make([]*Ring, 0, len(rings))
+		for _, ring := range rings {
+			if len(ring.Entries) > 0 {
+				ringsWithEntries = append(ringsWithEntries, ring)
+			}
+		}
+		ring1 := ringsWithEntries[rand.Intn(len(ringsWithEntries))]
+		ring2index, _, ring2 := randomPeer(-1)
+		common.Debug.Printf("%s: 'Gossiping' to %s", ring1.Peer, ring2.Peer)
+		wt.AssertSuccess(t, ring2.Merge(*ring1))
+		theRanges[ring2index] = ring2.OwnedRanges()
+	}
+
+	for i := 0; i < iterations; i++ {
+		// about 1 in 10 times, rmpeer or add host
+		n := rand.Intn(10)
+		switch {
+		case n < 1:
+			addOrRmPeer()
+		default:
+			doGrantOrGossip()
+		}
+	}
+}
+
+func (es entries) String() string {
+	var buffer bytes.Buffer
+	fmt.Fprintf(&buffer, "[")
+	for i, entry := range es {
+		fmt.Fprintf(&buffer, "%+v", *entry)
+		if i+1 < len(es) {
+			fmt.Fprintf(&buffer, " ")
+		}
+	}
+	fmt.Fprintf(&buffer, "]")
+	return buffer.String()
+}

ipam/space/space.go

@@ -0,0 +1,248 @@
+package space
+
+import (
+	"bytes"
+	"fmt"
+	"sort"
+
+	"github.com/weaveworks/weave/common"
+	"github.com/weaveworks/weave/ipam/address"
+)
+
+type Addr address.Address
+
+type Space struct {
+	// ours and free represent a set of addresses as a sorted
+	// sequences of ranges.  Even elements give the inclusive
+	// starting points of ranges, and odd elements give the
+	// exclusive ending points.  Ranges in an array do not
+	// overlap, and neighbouring ranges are always coalesced if
+	// possible, so the arrays consist of sorted Addrs without
+	// repetition.
+	ours []address.Address
+	free []address.Address
+}
+
+func New() *Space {
+	return &Space{}
+}
+
+func (s *Space) Add(start address.Address, size address.Offset) {
+	s.free = add(s.free, start, address.Add(start, size))
+}
+
+// Clear removes all spaces from this space set.  Used during node shutdown.
+func (s *Space) Clear() {
+	s.free = s.free[:0]
+	s.ours = s.ours[:0]
+}
+
+func (s *Space) Allocate() (bool, address.Address) {
+	if len(s.free) == 0 {
+		return false, 0
+	} else {
+		res := s.free[0]
+		s.ours = add(s.ours, res, res+1)
+		s.free = subtract(s.free, res, res+1)
+		return true, res
+	}
+}
+
+func (s *Space) Claim(addr address.Address) error {
+	if !contains(s.free, addr) {
+		return fmt.Errorf("Address %v is not free to claim", addr)
+	}
+
+	s.ours = add(s.ours, addr, addr+1)
+	s.free = subtract(s.free, addr, addr+1)
+	return nil
+}
+
+func (s *Space) NumFreeAddresses() address.Offset {
+	res := address.Offset(0)
+	for i := 0; i < len(s.free); i += 2 {
+		res += address.Subtract(s.free[i+1], s.free[i])
+	}
+	return res
+}
+
+func (s *Space) NumFreeAddressesInRange(start, end address.Address) address.Offset {
+	res := address.Offset(0)
+	for i := 0; i < len(s.free); i += 2 {
+		s, e := s.free[i], s.free[i+1]
+		if s < start {
+			s = start
+		}
+		if e > end {
+			e = end
+		}
+		if s >= e {
+			continue
+		}
+		res += address.Subtract(e, s)
+	}
+	return res
+}
+
+func (s *Space) Free(addr address.Address) error {
+	if !contains(s.ours, addr) {
+		return fmt.Errorf("Address %v is not ours", addr)
+	}
+	if contains(s.free, addr) {
+		return fmt.Errorf("Address %v is already free", addr)
+	}
+
+	s.ours = subtract(s.ours, addr, addr+1)
+	s.free = add(s.free, addr, addr+1)
+	return nil
+}
+
+func (s *Space) biggestFreeRange() (int, address.Offset) {
+	pos := -1
+	biggest := address.Offset(0)
+
+	for i := 0; i < len(s.free); i += 2 {
+		size := address.Subtract(s.free[i+1], s.free[i])
+		if size >= biggest {
+			pos = i
+			biggest = size
+		}
+	}
+	return pos, biggest
+}
+
+func (s *Space) Donate() (address.Address, address.Offset, bool) {
+	if len(s.free) == 0 {
+		return 0, 0, false
+	}
+
+	pos, biggest := s.biggestFreeRange()
+
+	// Donate half of that biggest free range, rounding up so
+	// that the donation can't be empty
+	end := s.free[pos+1]
+	start := end - address.Address((biggest+1)/2)
+
+	s.ours = subtract(s.ours, start, end)
+	s.free = subtract(s.free, start, end)
+	return start, address.Subtract(end, start), true
+}
+
+func firstGreater(a []address.Address, x address.Address) int {
+	return sort.Search(len(a), func(i int) bool { return a[i] > x })
+}
+
+func firstGreaterOrEq(a []address.Address, x address.Address) int {
+	return sort.Search(len(a), func(i int) bool { return a[i] >= x })
+}
+
+// Do the ranges contain the given address?
+func contains(addrs []address.Address, addr address.Address) bool {
+	return firstGreater(addrs, addr)&1 != 0
+}
+
+// Take the union of the range [start, end) with the ranges in the array
+func add(addrs []address.Address, start address.Address, end address.Address) []address.Address {
+	return addSub(addrs, start, end, 0)
+}
+
+// Subtract the range [start, end) from the ranges in the array
+func subtract(addrs []address.Address, start address.Address, end address.Address) []address.Address {
+	return addSub(addrs, start, end, 1)
+}
+
+func addSub(addrs []address.Address, start address.Address, end address.Address, sense int) []address.Address {
+	start_pos := firstGreaterOrEq(addrs, start)
+	end_pos := firstGreater(addrs[start_pos:], end) + start_pos
+
+	// Boundaries up to start_pos are unaffected
+	res := make([]address.Address, start_pos, len(addrs)+2)
+	copy(res, addrs)
+
+	// Include start and end as new boundaries if they lie
+	// outside/inside existing ranges (according to sense).
+	if start_pos&1 == sense {
+		res = append(res, start)
+	}
+
+	if end_pos&1 == sense {
+		res = append(res, end)
+	}
+
+	// Boundaries after end_pos are unaffected
+	return append(res, addrs[end_pos:]...)
+}
+
+func (s *Space) String() string {
+	var buf bytes.Buffer
+	if len(s.ours) > 0 {
+		fmt.Fprint(&buf, "owned:")
+		for i := 0; i < len(s.ours); i += 2 {
+			fmt.Fprintf(&buf, " %s+%d ", s.ours[i], s.ours[i+1]-s.ours[i])
+		}
+	}
+	if len(s.free) > 0 {
+		fmt.Fprintf(&buf, "free:")
+		for i := 0; i < len(s.free); i += 2 {
+			fmt.Fprintf(&buf, " %s+%d ", s.free[i], s.free[i+1]-s.free[i])
+		}
+	}
+	if len(s.ours) == 0 && len(s.free) == 0 {
+		fmt.Fprintf(&buf, "No address ranges owned")
+	}
+	return buf.String()
+}
+
+type addressSlice []address.Address
+
+func (p addressSlice) Len() int           { return len(p) }
+func (p addressSlice) Less(i, j int) bool { return p[i] < p[j] }
+func (p addressSlice) Swap(i, j int)      { p[i], p[j] = p[j], p[i] }
+
+func (s *Space) assertInvariants() {
+	common.Assert(sort.IsSorted(addressSlice(s.ours)))
+	common.Assert(sort.IsSorted(addressSlice(s.free)))
+}
+
+// Return a slice representing everything we own, whether it is free or not
+func (s *Space) everything() []address.Address {
+	a := make([]address.Address, len(s.ours))
+	copy(a, s.ours)
+	for i := 0; i < len(s.free); i += 2 {
+		a = add(a, s.free[i], s.free[i+1])
+	}
+	return a
+}
+
+// OwnedRanges returns slice of Ranges, ordered by IP, gluing together
+// contiguous sequences of owned and free addresses
+func (s *Space) OwnedRanges() []address.Range {
+	everything := s.everything()
+	result := make([]address.Range, len(everything)/2)
+	for i := 0; i < len(everything); i += 2 {
+		result[i/2] = address.Range{Start: everything[i], End: everything[i+1]}
+	}
+	return result
+}
+
+// Create a Space that has free space in all the supplied Ranges.
+func (s *Space) AddRanges(ranges []address.Range) {
+	for _, r := range ranges {
+		s.free = add(s.free, r.Start, r.End)
+	}
+}
+
+// Taking ranges to be a set of all space we should own, add in any excess as free space
+func (s *Space) UpdateRanges(ranges []address.Range) {
+	new := []address.Address{}
+	for _, r := range ranges {
+		new = add(new, r.Start, r.End)
+	}
+	current := s.everything()
+	for i := 0; i < len(current); i += 2 {
+		new = subtract(new, current[i], current[i+1])
+	}
+	for i := 0; i < len(new); i += 2 {
+		s.free = add(s.free, new[i], new[i+1])
+	}
+}

ipam/space/space_test.go

@@ -0,0 +1,249 @@
+package space
+
+import (
+	"reflect"
+	"testing"
+
+	"github.com/weaveworks/weave/ipam/address"
+	wt "github.com/weaveworks/weave/testing"
+)
+
+func makeSpace(start address.Address, size address.Offset) *Space {
+	s := New()
+	s.Add(start, size)
+	return s
+}
+
+func ip(s string) address.Address {
+	addr, _ := address.ParseIP(s)
+	return addr
+}
+
+func TestLowlevel(t *testing.T) {
+	a := []address.Address{}
+	a = add(a, 100, 200)
+	wt.AssertEquals(t, a, []address.Address{100, 200})
+	wt.AssertTrue(t, !contains(a, 99), "")
+	wt.AssertTrue(t, contains(a, 100), "")
+	wt.AssertTrue(t, contains(a, 199), "")
+	wt.AssertTrue(t, !contains(a, 200), "")
+	a = add(a, 700, 800)
+	wt.AssertEquals(t, a, []address.Address{100, 200, 700, 800})
+	a = add(a, 300, 400)
+	wt.AssertEquals(t, a, []address.Address{100, 200, 300, 400, 700, 800})
+	a = add(a, 400, 500)
+	wt.AssertEquals(t, a, []address.Address{100, 200, 300, 500, 700, 800})
+	a = add(a, 600, 700)
+	wt.AssertEquals(t, a, []address.Address{100, 200, 300, 500, 600, 800})
+	a = add(a, 500, 600)
+	wt.AssertEquals(t, a, []address.Address{100, 200, 300, 800})
+	a = subtract(a, 500, 600)
+	wt.AssertEquals(t, a, []address.Address{100, 200, 300, 500, 600, 800})
+	a = subtract(a, 600, 700)
+	wt.AssertEquals(t, a, []address.Address{100, 200, 300, 500, 700, 800})
+	a = subtract(a, 400, 500)
+	wt.AssertEquals(t, a, []address.Address{100, 200, 300, 400, 700, 800})
+	a = subtract(a, 300, 400)
+	wt.AssertEquals(t, a, []address.Address{100, 200, 700, 800})
+	a = subtract(a, 700, 800)
+	wt.AssertEquals(t, a, []address.Address{100, 200})
+	a = subtract(a, 100, 200)
+	wt.AssertEquals(t, a, []address.Address{})
+
+	s := New()
+	wt.AssertEquals(t, s.NumFreeAddresses(), address.Offset(0))
+	ok, got := s.Allocate()
+	wt.AssertFalse(t, ok, "allocate in empty space should fail")
+
+	s.Add(100, 100)
+	wt.AssertEquals(t, s.NumFreeAddresses(), address.Offset(100))
+	ok, got = s.Allocate()
+	wt.AssertTrue(t, ok && got == 100, "allocate")
+	wt.AssertEquals(t, s.NumFreeAddresses(), address.Offset(99))
+	wt.AssertNoErr(t, s.Claim(150))
+	wt.AssertEquals(t, s.NumFreeAddresses(), address.Offset(98))
+	wt.AssertNoErr(t, s.Free(100))
+	wt.AssertEquals(t, s.NumFreeAddresses(), address.Offset(99))
+	wt.AssertErrorInterface(t, s.Free(0), (*error)(nil), "free not allocated")
+	wt.AssertErrorInterface(t, s.Free(100), (*error)(nil), "double free")
+
+	start, size, ok := s.Donate()
+	wt.AssertTrue(t, ok && start == 125 && size == 25, "donate")
+
+	// test Donate when addresses are scarce
+	s = New()
+	start, size, ok = s.Donate()
+	wt.AssertTrue(t, !ok, "donate on empty space should fail")
+	s.Add(0, 3)
+	wt.AssertNoErr(t, s.Claim(0))
+	wt.AssertNoErr(t, s.Claim(2))
+	start, size, ok = s.Donate()
+	wt.AssertTrue(t, ok && start == 1 && size == 1, "donate")
+	start, size, ok = s.Donate()
+	wt.AssertTrue(t, !ok, "donate should fail")
+}
+
+func TestSpaceAllocate(t *testing.T) {
+	const (
+		testAddr1   = "10.0.3.4"
+		testAddrx   = "10.0.3.19"
+		testAddry   = "10.0.9.19"
+		containerID = "deadbeef"
+	)
+
+	space1 := makeSpace(ip(testAddr1), 20)
+	wt.AssertEquals(t, space1.NumFreeAddresses(), address.Offset(20))
+	space1.assertInvariants()
+
+	_, addr1 := space1.Allocate()
+	wt.AssertEqualString(t, addr1.String(), testAddr1, "address")
+	wt.AssertEquals(t, space1.NumFreeAddresses(), address.Offset(19))
+	space1.assertInvariants()
+
+	_, addr2 := space1.Allocate()
+	wt.AssertFalse(t, addr2.String() == testAddr1, "address")
+	wt.AssertEquals(t, space1.NumFreeAddresses(), address.Offset(18))
+	wt.AssertEquals(t, space1.NumFreeAddressesInRange(ip(testAddr1), ip(testAddrx)), address.Offset(13))
+	wt.AssertEquals(t, space1.NumFreeAddressesInRange(ip(testAddr1), ip(testAddry)), address.Offset(18))
+	space1.assertInvariants()
+
+	space1.Free(addr2)
+	space1.assertInvariants()
+
+	wt.AssertErrorInterface(t, space1.Free(addr2), (*error)(nil), "double free")
+	wt.AssertErrorInterface(t, space1.Free(ip(testAddrx)), (*error)(nil), "address not allocated")
+	wt.AssertErrorInterface(t, space1.Free(ip(testAddry)), (*error)(nil), "wrong out of range")
+
+	space1.assertInvariants()
+}
+
+func TestSpaceFree(t *testing.T) {
+	const (
+		testAddr1   = "10.0.3.4"
+		testAddrx   = "10.0.3.19"
+		testAddry   = "10.0.9.19"
+		containerID = "deadbeef"
+	)
+
+	space := makeSpace(ip(testAddr1), 20)
+
+	// Check we are prepared to give up the entire space
+	pos, size := space.biggestFreeRange()
+	wt.AssertTrue(t, space.free[pos] == ip(testAddr1) && size == 20, "Wrong space")
+
+	for i := 0; i < 20; i++ {
+		ok, _ := space.Allocate()
+		wt.AssertTrue(t, ok, "Failed to get address")
+	}
+
+	// Check we are full
+	ok, _ := space.Allocate()
+	wt.AssertTrue(t, !ok, "Should have failed to get address")
+	_, size, ok = space.Donate()
+	wt.AssertTrue(t, size == 0, "Wrong space")
+
+	// Free in the middle
+	wt.AssertSuccess(t, space.Free(ip("10.0.3.13")))
+	pos, size = space.biggestFreeRange()
+	wt.AssertTrue(t, space.free[pos] == ip("10.0.3.13") && size == 1, "Wrong space")
+
+	// Free one at the end
+	wt.AssertSuccess(t, space.Free(ip("10.0.3.23")))
+	pos, size = space.biggestFreeRange()
+	wt.AssertTrue(t, space.free[pos] == ip("10.0.3.23") && size == 1, "Wrong space")
+
+	// Now free a few at the end
+	wt.AssertSuccess(t, space.Free(ip("10.0.3.22")))
+	wt.AssertSuccess(t, space.Free(ip("10.0.3.21")))
+
+	wt.AssertEquals(t, space.NumFreeAddresses(), address.Offset(4))
+
+	// Now get the biggest free space; should be 3.21
+	pos, size = space.biggestFreeRange()
+	wt.AssertTrue(t, space.free[pos] == ip("10.0.3.21") && size == 3, "Wrong space")
+
+	// Now free a few in the middle
+	wt.AssertSuccess(t, space.Free(ip("10.0.3.12")))
+	wt.AssertSuccess(t, space.Free(ip("10.0.3.11")))
+	wt.AssertSuccess(t, space.Free(ip("10.0.3.10")))
+
+	wt.AssertEquals(t, space.NumFreeAddresses(), address.Offset(7))
+
+	// Now get the biggest free space; should be 3.21
+	pos, size = space.biggestFreeRange()
+	wt.AssertTrue(t, space.free[pos] == ip("10.0.3.10") && size == 4, "Wrong space")
+
+	wt.AssertEquals(t, space.OwnedRanges(), []address.Range{address.Range{Start: ip("10.0.3.4"), End: ip("10.0.3.24")}})
+}
+
+func (s1 *Space) Equal(s2 *Space) bool {
+	return reflect.DeepEqual(s1.ours, s2.ours) && reflect.DeepEqual(s1.free, s2.free)
+}
+
+func TestDonateSimple(t *testing.T) {
+	const (
+		testAddr1 = "10.0.1.0"
+		testAddr2 = "10.0.1.32"
+	)
+
+	var (
+		ipAddr1 = ip(testAddr1)
+	)
+
+	ps1 := makeSpace(ipAddr1, 48)
+
+	// Empty space set should split in two and give me the second half
+	start, numGivenUp, ok := ps1.Donate()
+	wt.AssertTrue(t, ok, "Donate result")
+	wt.AssertEqualString(t, start.String(), "10.0.1.24", "Invalid start")
+	wt.AssertEquals(t, numGivenUp, address.Offset(24))
+	wt.AssertEquals(t, ps1.NumFreeAddresses(), address.Offset(24))
+
+	// Now check we can give the rest up.
+	count := 0 // count to avoid infinite loop
+	for ; count < 1000; count++ {
+		_, size, ok := ps1.Donate()
+		if !ok {
+			break
+		}
+		numGivenUp += size
+	}
+	wt.AssertEquals(t, ps1.NumFreeAddresses(), address.Offset(0))
+	wt.AssertEquals(t, numGivenUp, address.Offset(48))
+}
+
+func TestDonateHard(t *testing.T) {
+	//common.InitDefaultLogging(true)
+	var (
+		start                = ip("10.0.1.0")
+		size  address.Offset = 48
+	)
+
+	// Fill a fresh space
+	spaceset := makeSpace(start, size)
+	for i := address.Offset(0); i < size; i++ {
+		ok, _ := spaceset.Allocate()
+		wt.AssertTrue(t, ok, "Failed to get IP!")
+	}
+
+	wt.AssertEquals(t, spaceset.NumFreeAddresses(), address.Offset(0))
+
+	// Now free all but the last address
+	// this will force us to split the free list
+	for i := address.Offset(0); i < size-1; i++ {
+		wt.AssertSuccess(t, spaceset.Free(address.Add(start, i)))
+	}
+
+	// Now split
+	newRange, numGivenUp, ok := spaceset.Donate()
+	wt.AssertTrue(t, ok, "GiveUpSpace result")
+	wt.AssertEquals(t, newRange, ip("10.0.1.23"))
+	wt.AssertEquals(t, numGivenUp, address.Offset(24))
+	wt.AssertEquals(t, spaceset.NumFreeAddresses(), address.Offset(23))
+
+	//Space set should now have 2 spaces
+	expected := New()
+	expected.Add(start, 23)
+	expected.ours = add(nil, ip("10.0.1.47"), ip("10.0.1.48"))
+	wt.AssertEquals(t, spaceset, expected)
+}

ipam/testutils_test.go

@@ -0,0 +1,286 @@
+package ipam
+
+import (
+	"fmt"
+	"math/rand"
+	"testing"
+	"time"
+
+	"github.com/weaveworks/weave/common"
+	"github.com/weaveworks/weave/router"
+	wt "github.com/weaveworks/weave/testing"
+)
+
+type mockMessage struct {
+	dst     router.PeerName
+	msgType byte
+	buf     []byte
+}
+
+func (m *mockMessage) String() string {
+	return fmt.Sprintf("-> %s [%x]", m.dst, m.buf)
+}
+
+func toStringArray(messages []mockMessage) []string {
+	out := make([]string, len(messages))
+	for i := range out {
+		out[i] = messages[i].String()
+	}
+	return out
+}
+
+type mockGossipComms struct {
+	t        *testing.T
+	name     string
+	messages []mockMessage
+}
+
+func (m *mockGossipComms) String() string {
+	return fmt.Sprintf("[mockGossipComms %s]", m.name)
+}
+
+// Note: this style of verification, using equalByteBuffer, requires
+// that the contents of messages are never re-ordered.  Which, for instance,
+// requires they are not based off iterating through a map.
+
+func (m *mockGossipComms) GossipBroadcast(update router.GossipData) error {
+	buf := []byte{}
+	if len(m.messages) == 0 {
+		m.Fatalf("%s: Gossip broadcast message unexpected: \n%x", m.name, buf)
+	} else if msg := m.messages[0]; msg.dst != router.UnknownPeerName {
+		m.Fatalf("%s: Expected Gossip message to %s but got broadcast", m.name, msg.dst)
+	} else if msg.buf != nil && !equalByteBuffer(msg.buf, buf) {
+		m.Fatalf("%s: Gossip message not sent as expected: \nwant: %x\ngot : %x", m.name, msg.buf, buf)
+	} else {
+		// Swallow this message
+		m.messages = m.messages[1:]
+	}
+	return nil
+}
+
+func equalByteBuffer(a, b []byte) bool {
+	if len(a) != len(b) {
+		return false
+	}
+	for i := 0; i < len(a); i++ {
+		if a[i] != b[i] {
+			return false
+		}
+	}
+	return true
+}
+
+func (m *mockGossipComms) Fatalf(format string, args ...interface{}) {
+	// this sometimes hangs: wt.Fatalf(m.t, args...)
+	panic(fmt.Sprintf(format, args...))
+}
+
+func (m *mockGossipComms) GossipUnicast(dstPeerName router.PeerName, buf []byte) error {
+	if len(m.messages) == 0 {
+		m.Fatalf("%s: Gossip message to %s unexpected: \n%s", m.name, dstPeerName, buf)
+	} else if msg := m.messages[0]; msg.dst == router.UnknownPeerName {
+		m.Fatalf("%s: Expected Gossip broadcast message but got dest %s", m.name, dstPeerName)
+	} else if msg.dst != dstPeerName {
+		m.Fatalf("%s: Expected Gossip message to %s but got dest %s", m.name, msg.dst, dstPeerName)
+	} else if buf[0] != msg.msgType {
+		m.Fatalf("%s: Expected Gossip message of type %d but got type %d", m.name, msg.msgType, buf[0])
+	} else if msg.buf != nil && !equalByteBuffer(msg.buf, buf[1:]) {
+		m.Fatalf("%s: Gossip message not sent as expected: \nwant: %x\ngot : %x", m.name, msg.buf, buf[1:])
+	} else {
+		// Swallow this message
+		m.messages = m.messages[1:]
+	}
+	return nil
+}
+
+func ExpectMessage(alloc *Allocator, dst string, msgType byte, buf []byte) {
+	m := alloc.gossip.(*mockGossipComms)
+	dstPeerName, _ := router.PeerNameFromString(dst)
+	m.messages = append(m.messages, mockMessage{dstPeerName, msgType, buf})
+}
+
+func ExpectBroadcastMessage(alloc *Allocator, buf []byte) {
+	m := alloc.gossip.(*mockGossipComms)
+	m.messages = append(m.messages, mockMessage{router.UnknownPeerName, 0, buf})
+}
+
+func CheckAllExpectedMessagesSent(allocs ...*Allocator) {
+	for _, alloc := range allocs {
+		m := alloc.gossip.(*mockGossipComms)
+		if len(m.messages) > 0 {
+			wt.Fatalf(m.t, "%s: Gossip message(s) not sent as expected: \n%x", m.name, m.messages)
+		}
+	}
+}
+
+func makeAllocator(name string, cidr string, quorum uint) *Allocator {
+	peername, err := router.PeerNameFromString(name)
+	if err != nil {
+		panic(err)
+	}
+
+	alloc, err := NewAllocator(peername, router.PeerUID(rand.Int63()),
+		"nick-"+name, cidr, quorum)
+	if err != nil {
+		panic(err)
+	}
+
+	return alloc
+}
+
+func makeAllocatorWithMockGossip(t *testing.T, name string, universeCIDR string, quorum uint) *Allocator {
+	alloc := makeAllocator(name, universeCIDR, quorum)
+	gossip := &mockGossipComms{t: t, name: name}
+	alloc.SetInterfaces(gossip)
+	alloc.Start()
+	return alloc
+}
+
+func (alloc *Allocator) claimRingForTesting(allocs ...*Allocator) {
+	peers := []router.PeerName{alloc.ourName}
+	for _, alloc2 := range allocs {
+		peers = append(peers, alloc2.ourName)
+	}
+	alloc.ring.ClaimForPeers(normalizeConsensus(peers))
+	alloc.space.AddRanges(alloc.ring.OwnedRanges())
+}
+
+// Check whether or not something was sent on a channel
+func AssertSent(t *testing.T, ch <-chan bool) {
+	timeout := time.After(10 * time.Second)
+	select {
+	case <-ch:
+		// This case is ok
+	case <-timeout:
+		wt.Fatalf(t, "Nothing sent on channel")
+	}
+}
+
+func AssertNothingSent(t *testing.T, ch <-chan bool) {
+	select {
+	case val := <-ch:
+		wt.Fatalf(t, "Unexpected value on channel: %t", val)
+	default:
+		// no message received
+	}
+}
+
+func AssertNothingSentErr(t *testing.T, ch <-chan error) {
+	select {
+	case val := <-ch:
+		wt.Fatalf(t, "Unexpected value on channel: %t", val)
+	default:
+		// no message received
+	}
+}
+
+// Router to convey gossip from one gossiper to another, for testing
+type gossipMessage struct {
+	isUnicast bool
+	sender    *router.PeerName
+	buf       []byte
+	exitChan  chan bool
+}
+
+type TestGossipRouter struct {
+	gossipChans map[router.PeerName]chan gossipMessage
+	loss        float32 // 0.0 means no loss
+}
+
+func (grouter *TestGossipRouter) GossipBroadcast(update router.GossipData) error {
+	for _, gossipChan := range grouter.gossipChans {
+		select {
+		case gossipChan <- gossipMessage{buf: update.(*ipamGossipData).alloc.encode()}:
+		default: // drop the message if we cannot send it
+		}
+	}
+	return nil
+}
+
+func (grouter *TestGossipRouter) removePeer(peer router.PeerName) {
+	gossipChan := grouter.gossipChans[peer]
+	resultChan := make(chan bool)
+	gossipChan <- gossipMessage{exitChan: resultChan}
+	<-resultChan
+	delete(grouter.gossipChans, peer)
+}
+
+type TestGossipRouterClient struct {
+	router *TestGossipRouter
+	sender router.PeerName
+}
+
+func (grouter *TestGossipRouter) connect(sender router.PeerName, gossiper router.Gossiper) router.Gossip {
+	gossipChan := make(chan gossipMessage, 100)
+
+	go func() {
+		gossipTimer := time.Tick(10 * time.Second)
+		for {
+			select {
+			case message := <-gossipChan:
+				if message.exitChan != nil {
+					message.exitChan <- true
+					return
+				}
+
+				if rand.Float32() > (1.0 - grouter.loss) {
+					continue
+				}
+
+				if message.isUnicast {
+					err := gossiper.OnGossipUnicast(*message.sender, message.buf)
+					if err != nil {
+						panic(fmt.Sprintf("Error doing gossip unicast to %s: %s", sender, err))
+					}
+				} else {
+					_, err := gossiper.OnGossipBroadcast(message.buf)
+					if err != nil {
+						panic(fmt.Sprintf("Error doing gossip broadcast to %s: %s", sender, err))
+					}
+				}
+			case <-gossipTimer:
+				grouter.GossipBroadcast(gossiper.Gossip())
+			}
+		}
+	}()
+
+	grouter.gossipChans[sender] = gossipChan
+	return TestGossipRouterClient{grouter, sender}
+}
+
+func (client TestGossipRouterClient) GossipUnicast(dstPeerName router.PeerName, buf []byte) error {
+	select {
+	case client.router.gossipChans[dstPeerName] <- gossipMessage{isUnicast: true, sender: &client.sender, buf: buf}:
+	default: // drop the message if we cannot send it
+		common.Error.Printf("Dropping message")
+	}
+	return nil
+}
+
+func (client TestGossipRouterClient) GossipBroadcast(update router.GossipData) error {
+	return client.router.GossipBroadcast(update)
+}
+
+func makeNetworkOfAllocators(size int, cidr string) ([]*Allocator, TestGossipRouter) {
+
+	gossipRouter := TestGossipRouter{make(map[router.PeerName]chan gossipMessage), 0.0}
+	allocs := make([]*Allocator, size)
+
+	for i := 0; i < size; i++ {
+		alloc := makeAllocator(fmt.Sprintf("%02d:00:00:02:00:00", i),
+			cidr, uint(size/2+1))
+		alloc.SetInterfaces(gossipRouter.connect(alloc.ourName, alloc))
+		alloc.Start()
+		allocs[i] = alloc
+	}
+
+	gossipRouter.GossipBroadcast(allocs[size-1].Gossip())
+	time.Sleep(1000 * time.Millisecond)
+	return allocs, gossipRouter
+}
+
+func stopNetworkOfAllocators(allocs []*Allocator) {
+	for _, alloc := range allocs {
+		alloc.Stop()
+	}
+}

nameserver/addrs.go

@@ -0,0 +1,125 @@
+package nameserver
+
+import (
+	"bytes"
+	"fmt"
+	"net"
+)
+
+// A zone record
+// Note: we will try to keep all the zone records information is concentrated here. Maybe not all queries will
+//       use things like priorities or weights, but we do not want to create a hierarchy for dealing with all
+//       possible queries...
+type ZoneRecord interface {
+	Name() string  // The name for this IP
+	IP() net.IP    // The IP (v4) address
+	Priority() int // The priority
+	Weight() int   // The weight
+	TTL() int      // TTL
+}
+
+type ZoneLookup interface {
+	// Lookup for a name
+	LookupName(name string) ([]ZoneRecord, error)
+	// Lookup for an address
+	LookupInaddr(inaddr string) ([]ZoneRecord, error)
+}
+
+type ZoneObserver interface {
+	// Observe anything that affects a particular name in the zone
+	ObserveName(name string, observer ZoneRecordObserver) error
+	// Observe anything that affects a particular IP in the zone
+	ObserveInaddr(inaddr string, observer ZoneRecordObserver) error
+}
+
+type ZoneRecordObserver func()
+
+/////////////////////////////////////////////////////////////
+
+// A basic record struct (satisfying the ZoneRecord interface)
+type Record struct {
+	name     string
+	ip       net.IP
+	priority int
+	weight   int
+	ttl      int
+}
+
+func (r Record) Name() string  { return r.name }
+func (r Record) IP() net.IP    { return r.ip }
+func (r Record) Priority() int { return r.priority }
+func (r Record) Weight() int   { return r.weight }
+func (r Record) TTL() int      { return r.ttl }
+
+func (r Record) String() string {
+	var buf bytes.Buffer
+	if len(r.Name()) > 0 {
+		fmt.Fprintf(&buf, "%s", r.Name())
+	}
+	if !r.IP().IsUnspecified() {
+		fmt.Fprintf(&buf, "[%s]", r.IP())
+	}
+	if r.Priority() > 0 {
+		fmt.Fprintf(&buf, "/P:%d", r.Priority())
+	}
+	if r.Weight() > 0 {
+		fmt.Fprintf(&buf, "/W:%d", r.Weight())
+	}
+	if r.TTL() > 0 {
+		fmt.Fprintf(&buf, "/TTL:%d", r.TTL())
+	}
+
+	return buf.String()
+}
+
+/////////////////////////////////////////////////////////////
+
+func newParseError(reason string, addr string) *net.ParseError {
+	return &net.ParseError{Type: reason, Text: addr}
+}
+
+// simple IPv4 type that can be used as a key in a map (in contrast with net.IP), used for sets of IPs, etc...
+type IPv4 [4]byte
+
+func (ip IPv4) toNetIP() net.IP { return net.IPv4(ip[0], ip[1], ip[2], ip[3]) }
+func (ip IPv4) String() string  { return ip.toNetIP().String() }
+func ipToIPv4(nip net.IP) IPv4  { ip := nip.To4(); return IPv4([4]byte{ip[0], ip[1], ip[2], ip[3]}) }
+
+// Parse an address (eg, "10.13.12.11") and return the corresponding IPv4 (eg, "10.13.12.11")
+func addrToIPv4(addr string) (IPv4, error) {
+	ip := net.ParseIP(addr)
+	if ip == nil {
+		return IPv4{}, newParseError("IP address", addr)
+	}
+	return ipToIPv4(ip), nil
+}
+
+// Parse a reverse address (eg, "11.12.13.10.in-addr.arpa.") and return the corresponding IPv4 (eg, "10.13.12.11")
+func raddrToIPv4(addr string) (IPv4, error) {
+	l := len(addr)
+	if l < rdnsDomainLen+1 {
+		return IPv4{}, newParseError("too short reverse IP address", addr)
+	}
+
+	suffixLen := l - rdnsDomainLen
+	suffix := addr[suffixLen+1:]
+	if suffix != RDNSDomain {
+		return IPv4{}, newParseError("suffix of reverse IP address", addr)
+	}
+	ipStr := addr[:suffixLen]
+	revIP := net.ParseIP(ipStr)
+	if revIP == nil {
+		return IPv4{}, newParseError("reverse IP address", addr)
+	}
+
+	revIP4 := revIP.To4()
+	return IPv4([4]byte{revIP4[3], revIP4[2], revIP4[1], revIP4[0]}), nil
+}
+
+func raddrToIP(addr string) (net.IP, error) {
+	r, err := raddrToIPv4(addr)
+	if err != nil {
+		return net.IP{}, err
+	}
+	return r.toNetIP(), nil
+}

nameserver/addrs_test.go

@@ -0,0 +1,31 @@
+package nameserver
+
+import (
+	. "github.com/weaveworks/weave/common"
+	wt "github.com/weaveworks/weave/testing"
+	"net"
+	"testing"
+)
+
+func TestAddrs(t *testing.T) {
+	InitDefaultLogging(testing.Verbose())
+	Info.Println("TestAddrs starting")
+
+	ip, err := addrToIPv4("10.13.12.11")
+	wt.AssertNoErr(t, err)
+	wt.AssertTrue(t, net.ParseIP("10.13.12.11").Equal(ip.toNetIP()), "IP")
+
+	ip, err = raddrToIPv4("11.12.13.10.in-addr.arpa.")
+	wt.AssertNoErr(t, err)
+	wt.AssertTrue(t, net.ParseIP("10.13.12.11").Equal(ip.toNetIP()), "IP")
+
+	// some malformed addresses
+	ip, err = addrToIPv4("10.13.12")
+	wt.AssertTrue(t, err != nil, "when parsing malformed address")
+	ip, err = addrToIPv4("10.13.AA.12")
+	wt.AssertTrue(t, err != nil, "when parsing malformed address")
+	ip, err = raddrToIPv4("11.12.13.10.in-axxx.arpa.")
+	wt.AssertTrue(t, err != nil, "when parsing malformed address")
+	ip, err = raddrToIPv4("11.12.13.10.in-addr")
+	wt.AssertTrue(t, err != nil, "when parsing malformed address")
+}

nameserver/cache.go

@@ -0,0 +1,401 @@
+package nameserver
+
+import (
+	"bytes"
+	"container/heap"
+	"errors"
+	"fmt"
+	"math"
+	"math/rand"
+	"sync"
+	"time"
+
+	"github.com/benbjohnson/clock"
+	"github.com/miekg/dns"
+	. "github.com/weaveworks/weave/common"
+)
+
+var (
+	errInvalidCapacity = errors.New("Invalid cache capacity")
+	errCouldNotResolve = errors.New("Could not resolve")
+	errTimeout         = errors.New("Timeout while waiting for resolution")
+	errNoLocalReplies  = errors.New("No local replies")
+)
+
+const (
+	defPendingTimeout int = 5 // timeout for a resolution
+)
+
+const nullTTL = 0 // a null TTL
+
+type entryStatus uint8
+
+const (
+	stPending  entryStatus = iota // someone is waiting for the resolution
+	stResolved entryStatus = iota // resolved
+)
+
+var statusToString = map[entryStatus]string{
+	stPending:  "pending",
+	stResolved: "resolved",
+}
+
+const (
+	CacheNoLocalReplies uint8 = 1 << iota // not found in local network (stored in the cache so we skip another local lookup or some time)
+)
+
+// shuffleAnswers reorders answers for very basic load balancing
+func shuffleAnswers(answers []dns.RR) []dns.RR {
+	n := len(answers)
+	if n > 1 {
+		rand.Seed(time.Now().UTC().UnixNano())
+
+		for i := 0; i < n; i++ {
+			r := i + rand.Intn(n-i)
+			answers[r], answers[i] = answers[i], answers[r]
+		}
+	}
+
+	return answers
+}
+
+// a cache entry
+type cacheEntry struct {
+	Status   entryStatus // status of the entry
+	Flags    uint8       // some extra flags
+	ReplyLen int
+
+	question dns.Question
+	protocol dnsProtocol
+	reply    dns.Msg
+
+	validUntil time.Time // obtained from the reply and stored here for convenience/speed
+	putTime    time.Time
+
+	index int // for fast lookups in the heap
+}
+
+func newCacheEntry(question *dns.Question, now time.Time) *cacheEntry {
+	e := &cacheEntry{
+		Status:     stPending,
+		validUntil: now.Add(time.Second * time.Duration(defPendingTimeout)),
+		question:   *question,
+		index:      -1,
+	}
+
+	return e
+}
+
+// Get a copy of the reply stored in the entry, but with some values adjusted like the TTL
+func (e *cacheEntry) getReply(request *dns.Msg, maxLen int, now time.Time) (*dns.Msg, error) {
+	if e.Status != stResolved {
+		return nil, nil
+	}
+
+	// if the reply has expired or is invalid, force the caller to start a new resolution
+	if e.hasExpired(now) {
+		return nil, nil
+	}
+
+	if e.Flags&CacheNoLocalReplies != 0 {
+		return nil, errNoLocalReplies
+	}
+
+	if e.ReplyLen >= maxLen {
+		Debug.Printf("[cache msgid %d] returning truncated reponse: %d > %d", request.MsgHdr.Id, e.ReplyLen, maxLen)
+		return makeTruncatedReply(request), nil
+	}
+
+	// create a copy of the reply, with values for this particular query
+	reply := e.reply.Copy()
+	reply.SetReply(request)
+
+	// adjust the TTLs
+	passedSecs := uint32(now.Sub(e.putTime).Seconds())
+	for _, rr := range reply.Answer {
+		hdr := rr.Header()
+		ttl := hdr.Ttl
+		if passedSecs < ttl {
+			hdr.Ttl = ttl - passedSecs
+		} else {
+			return nil, nil // it is expired: do not spend more time and return nil...
+		}
+	}
+
+	reply.Rcode = e.reply.Rcode
+	reply.Authoritative = true
+
+	// shuffle the values, etc...
+	reply.Answer = shuffleAnswers(reply.Answer)
+
+	return reply, nil
+}
+
+func (e cacheEntry) hasExpired(now time.Time) bool {
+	return e.validUntil.Before(now) || e.validUntil == now
+}
+
+// set the reply for the entry
+// returns True if the entry has changed the validUntil time
+func (e *cacheEntry) setReply(reply *dns.Msg, ttl int, flags uint8, now time.Time) bool {
+	var prevValidUntil time.Time
+	if e.Status == stResolved {
+		if reply != nil {
+			Debug.Printf("[cache msgid %d] replacing response in cache", reply.MsgHdr.Id)
+		}
+		prevValidUntil = e.validUntil
+	}
+
+	// make sure we do not overwrite noLocalReplies entries
+	if flags&CacheNoLocalReplies != 0 {
+		if e.Flags&CacheNoLocalReplies != 0 {
+			return false
+		}
+	}
+
+	if ttl != nullTTL {
+		e.validUntil = now.Add(time.Second * time.Duration(ttl))
+	} else if reply != nil {
+		// calculate the validUntil from the reply TTL
+		var minTTL uint32 = math.MaxUint32
+		for _, rr := range reply.Answer {
+			ttl := rr.Header().Ttl
+			if ttl < minTTL {
+				minTTL = ttl // TODO: improve the minTTL calculation (maybe we should skip some RRs)
+			}
+		}
+		e.validUntil = now.Add(time.Second * time.Duration(minTTL))
+	} else {
+		Warning.Printf("[cache] no valid TTL could be calculated")
+	}
+
+	e.Status = stResolved
+	e.Flags = flags
+	e.putTime = now
+
+	if reply != nil {
+		e.reply = *reply
+		e.ReplyLen = reply.Len()
+	}
+
+	return (prevValidUntil != e.validUntil)
+}
+
+func (e *cacheEntry) String() string {
+	var buf bytes.Buffer
+	q := e.question
+	fmt.Fprintf(&buf, "'%s'[%s]: ", q.Name, dns.TypeToString[q.Qtype])
+	if e.Flags&CacheNoLocalReplies != 0 {
+		fmt.Fprintf(&buf, "neg-local")
+	} else {
+		fmt.Fprintf(&buf, "%s", statusToString[e.Status])
+	}
+	fmt.Fprintf(&buf, "(%d bytes)", e.ReplyLen)
+	return buf.String()
+}
+
+//////////////////////////////////////////////////////////////////////////////////////
+
+// An entriesPtrHeap is a min-heap of cache entries.
+type entriesPtrsHeap []*cacheEntry
+
+func (h entriesPtrsHeap) Len() int           { return len(h) }
+func (h entriesPtrsHeap) Less(i, j int) bool { return h[i].validUntil.Before(h[j].validUntil) }
+func (h entriesPtrsHeap) Swap(i, j int) {
+	h[i], h[j] = h[j], h[i]
+	h[i].index = i
+	h[j].index = j
+}
+
+func (h *entriesPtrsHeap) Push(x interface{}) {
+	// Push and Pop use pointer receivers because they modify the slice's length,
+	// not just its contents.
+	n := len(*h)
+	entry := x.(*cacheEntry)
+	entry.index = n
+	*h = append(*h, entry)
+}
+
+func (h *entriesPtrsHeap) Pop() interface{} {
+	old := *h
+	n := len(old)
+	item := old[n-1]
+	item.index = -1 // for safety
+	*h = old[0 : n-1]
+	return item
+}
+
+//////////////////////////////////////////////////////////////////////////////////////
+
+// The cache interface
+type ZoneCache interface {
+	// Get from the cache
+	Get(request *dns.Msg, maxLen int) (reply *dns.Msg, err error)
+	// Put in the cache
+	Put(request *dns.Msg, reply *dns.Msg, ttl int, flags uint8) int
+	// Remove from the cache
+	Remove(question *dns.Question)
+	// Purge all expired entries from the cache
+	Purge()
+	// Remove all entries
+	Clear()
+	// Return the cache length
+	Len() int
+	// Return the max capacity
+	Capacity() int
+}
+
+type cacheKey dns.Question
+type entries map[cacheKey]*cacheEntry
+
+// Cache is a thread-safe fixed capacity LRU cache.
+type Cache struct {
+	capacity int
+	entries  entries
+	entriesH entriesPtrsHeap // len(entriesH) <= len(entries), as pending entries can be in entries but not in entriesH
+	clock    clock.Clock
+	lock     sync.RWMutex
+}
+
+// NewCache creates a cache of the given capacity
+func NewCache(capacity int, clk clock.Clock) (*Cache, error) {
+	if capacity <= 0 {
+		return nil, errInvalidCapacity
+	}
+	c := &Cache{
+		capacity: capacity,
+		entries:  make(entries, capacity),
+		clock:    clk,
+	}
+
+	if c.clock == nil {
+		c.clock = clock.New()
+	}
+
+	heap.Init(&c.entriesH)
+	return c, nil
+}
+
+// Clear removes all the entries in the cache
+func (c *Cache) Clear() {
+	c.lock.Lock()
+	defer c.lock.Unlock()
+
+	c.entries = make(entries, c.capacity)
+	heap.Init(&c.entriesH)
+}
+
+// Purge removes the old elements in the cache
+func (c *Cache) Purge() {
+	c.lock.Lock()
+	defer c.lock.Unlock()
+
+	now := c.clock.Now()
+	for i, entry := range c.entriesH {
+		if entry.hasExpired(now) {
+			heap.Remove(&c.entriesH, i)
+			delete(c.entries, cacheKey(entry.question))
+		} else {
+			return // all remaining entries must be still valid...
+		}
+	}
+}
+
+// Add adds a reply to the cache.
+// When `ttl` is equal to `nullTTL`, the cache entry will be valid until the closest TTL in the `reply`
+func (c *Cache) Put(request *dns.Msg, reply *dns.Msg, ttl int, flags uint8) int {
+	c.lock.Lock()
+	defer c.lock.Unlock()
+
+	now := c.clock.Now()
+	question := request.Question[0]
+	key := cacheKey(question)
+	ent, found := c.entries[key]
+	if found {
+		updated := ent.setReply(reply, ttl, flags, now)
+		if updated {
+			heap.Fix(&c.entriesH, ent.index)
+		}
+	} else {
+		// If we will add a new item and the capacity has been exceeded, make some room...
+		if len(c.entriesH) >= c.capacity {
+			lowestEntry := heap.Pop(&c.entriesH).(*cacheEntry)
+			delete(c.entries, cacheKey(lowestEntry.question))
+		}
+		ent = newCacheEntry(&question, now)
+		ent.setReply(reply, ttl, flags, now)
+		heap.Push(&c.entriesH, ent)
+		c.entries[key] = ent
+	}
+	return ent.ReplyLen
+}
+
+// Look up for a question's reply from the cache.
+// If no reply is stored in the cache, it returns a `nil` reply and no error.
+func (c *Cache) Get(request *dns.Msg, maxLen int) (reply *dns.Msg, err error) {
+	c.lock.Lock()
+	defer c.lock.Unlock()
+
+	now := c.clock.Now()
+	question := request.Question[0]
+	key := cacheKey(question)
+	if ent, found := c.entries[key]; found {
+		reply, err = ent.getReply(request, maxLen, now)
+		if ent.hasExpired(now) {
+			Debug.Printf("[cache msgid %d] expired: removing", request.MsgHdr.Id)
+			if ent.index > 0 {
+				heap.Remove(&c.entriesH, ent.index)
+			}
+			delete(c.entries, key)
+			reply = nil
+		}
+	} else {
+		// we are the first asking for this name: create an entry with no reply... the caller must wait
+		Debug.Printf("[cache msgid %d] addind in pending state", request.MsgHdr.Id)
+		c.entries[key] = newCacheEntry(&question, now)
+	}
+	return
+}
+
+// Remove removes the provided question from the cache.
+func (c *Cache) Remove(question *dns.Question) {
+	c.lock.Lock()
+	defer c.lock.Unlock()
+
+	key := cacheKey(*question)
+	if entry, found := c.entries[key]; found {
+		Debug.Printf("[cache] removing %s-response for '%s'", dns.TypeToString[question.Qtype], question.Name)
+		if entry.index > 0 {
+			heap.Remove(&c.entriesH, entry.index)
+		}
+		delete(c.entries, key)
+	}
+}
+
+// Len returns the number of entries in the cache.
+func (c *Cache) Len() int {
+	c.lock.RLock()
+	defer c.lock.RUnlock()
+
+	return len(c.entries)
+}
+
+// Return the max capacity
+func (c *Cache) Capacity() int {
+	c.lock.RLock()
+	defer c.lock.RUnlock()
+
+	return c.capacity
+}
+
+// Return the max capacity
+func (c *Cache) String() string {
+	c.lock.RLock()
+	defer c.lock.RUnlock()
+
+	var buf bytes.Buffer
+	for _, entry := range c.entries {
+		fmt.Fprintf(&buf, "%s\n", entry)
+	}
+	return buf.String()
+}

nameserver/cache_test.go

@@ -0,0 +1,228 @@
+package nameserver
+
+import (
+	"fmt"
+	"net"
+	"testing"
+	"time"
+
+	"github.com/benbjohnson/clock"
+	"github.com/miekg/dns"
+	. "github.com/weaveworks/weave/common"
+	wt "github.com/weaveworks/weave/testing"
+)
+
+// Check that the cache keeps its intended capacity constant
+func TestCacheLength(t *testing.T) {
+	InitDefaultLogging(testing.Verbose())
+	Info.Println("TestCacheLength starting")
+
+	const cacheLen = 128
+
+	l, err := NewCache(cacheLen, nil)
+	wt.AssertNoErr(t, err)
+
+	insTime := time.Now()
+
+	t.Logf("Inserting 256 questions in the cache at '%s', with TTL from 0 to 255", insTime)
+	for i := 0; i < cacheLen*2; i++ {
+		name := fmt.Sprintf("name%d", i)
+		questionMsg := new(dns.Msg)
+		questionMsg.SetQuestion(name, dns.TypeA)
+		questionMsg.RecursionDesired = true
+
+		question := &questionMsg.Question[0]
+
+		ip := net.ParseIP(fmt.Sprintf("10.0.1.%d", i))
+		records := []ZoneRecord{Record{name, ip, 0, 0, 0}}
+
+		reply := makeAddressReply(questionMsg, question, records)
+		reply.Answer[0].Header().Ttl = uint32(i)
+
+		l.Put(questionMsg, reply, 0, 0)
+	}
+
+	wt.AssertEqualInt(t, l.Len(), cacheLen, "cache length")
+
+	minExpectedTime := insTime.Add(time.Duration(cacheLen) * time.Second)
+	t.Logf("Checking all remaining entries expire after insert_time + %d secs='%s'", cacheLen, minExpectedTime)
+	for _, entry := range l.entries {
+		if entry.validUntil.Before(minExpectedTime) {
+			t.Fatalf("Entry valid until %s", entry.validUntil)
+		}
+	}
+}
+
+// Check that the cache entries are ok
+func TestCacheEntries(t *testing.T) {
+	InitDefaultLogging(testing.Verbose())
+	Info.Println("TestCacheEntries starting")
+
+	Info.Println("Checking cache consistency")
+
+	const cacheLen = 128
+	clk := clock.NewMock()
+
+	l, err := NewCache(cacheLen, clk)
+	wt.AssertNoErr(t, err)
+
+	questionMsg := new(dns.Msg)
+	questionMsg.SetQuestion("some.name", dns.TypeA)
+	questionMsg.RecursionDesired = true
+
+	question := &questionMsg.Question[0]
+
+	t.Logf("Trying to get a name")
+	resp, err := l.Get(questionMsg, minUDPSize)
+	wt.AssertNoErr(t, err)
+	if resp != nil {
+		t.Logf("Got\n%s", resp)
+		t.Fatalf("ERROR: Did not expect a reponse from Get() yet")
+	}
+	t.Logf("Trying to get it again")
+	resp, err = l.Get(questionMsg, minUDPSize)
+	wt.AssertNoErr(t, err)
+	if resp != nil {
+		t.Logf("Got\n%s", resp)
+		t.Fatalf("ERROR: Did not expect a reponse from Get() yet")
+	}
+
+	t.Logf("Inserting the reply")
+	records := []ZoneRecord{Record{"some.name", net.ParseIP("10.0.1.1"), 0, 0, 0}}
+	reply1 := makeAddressReply(questionMsg, question, records)
+	l.Put(questionMsg, reply1, nullTTL, 0)
+
+	t.Logf("Checking we can Get() the reply now")
+	resp, err = l.Get(questionMsg, minUDPSize)
+	wt.AssertNoErr(t, err)
+	wt.AssertTrue(t, resp != nil, "reponse from Get()")
+	t.Logf("Received '%s'", resp.Answer[0])
+	wt.AssertType(t, resp.Answer[0], (*dns.A)(nil), "DNS record")
+	ttlGet1 := resp.Answer[0].Header().Ttl
+
+	clk.Add(time.Duration(1) * time.Second)
+	t.Logf("Checking that a second Get(), after 1 second, gets the same result, but with reduced TTL")
+	resp, err = l.Get(questionMsg, minUDPSize)
+	wt.AssertNoErr(t, err)
+	wt.AssertTrue(t, resp != nil, "reponse from a second Get()")
+	t.Logf("Received '%s'", resp.Answer[0])
+	wt.AssertType(t, resp.Answer[0], (*dns.A)(nil), "DNS record")
+	ttlGet2 := resp.Answer[0].Header().Ttl
+	wt.AssertEqualInt(t, int(ttlGet1-ttlGet2), 1, "TTL difference")
+
+	clk.Add(time.Duration(localTTL) * time.Second)
+	t.Logf("Checking that a third Get(), after %d second, gets no result", localTTL)
+	resp, err = l.Get(questionMsg, minUDPSize)
+	wt.AssertNoErr(t, err)
+	if resp != nil {
+		t.Logf("Got\n%s", resp)
+		t.Fatalf("ERROR: Did NOT expect a reponse from the second Get()")
+	}
+
+	t.Logf("Checking that an Remove() results in Get() returning nothing")
+	records = []ZoneRecord{Record{"some.name", net.ParseIP("10.0.9.9"), 0, 0, 0}}
+	replyTemp := makeAddressReply(questionMsg, question, records)
+	l.Put(questionMsg, replyTemp, nullTTL, 0)
+	lenBefore := l.Len()
+	l.Remove(question)
+	wt.AssertEqualInt(t, l.Len(), lenBefore-1, "cache length")
+	l.Remove(question) // do it again: should have no effect...
+	wt.AssertEqualInt(t, l.Len(), lenBefore-1, "cache length")
+
+	resp, err = l.Get(questionMsg, minUDPSize)
+	wt.AssertNoErr(t, err)
+	wt.AssertTrue(t, resp == nil, "reponse from the Get() after a Remove()")
+
+	t.Logf("Inserting a two replies for the same query")
+	records = []ZoneRecord{Record{"some.name", net.ParseIP("10.0.1.2"), 0, 0, 0}}
+	reply2 := makeAddressReply(questionMsg, question, records)
+	l.Put(questionMsg, reply2, nullTTL, 0)
+	clk.Add(time.Duration(1) * time.Second)
+	records = []ZoneRecord{Record{"some.name", net.ParseIP("10.0.1.3"), 0, 0, 0}}
+	reply3 := makeAddressReply(questionMsg, question, records)
+	l.Put(questionMsg, reply3, nullTTL, 0)
+
+	t.Logf("Checking we get the last one...")
+	resp, err = l.Get(questionMsg, minUDPSize)
+	wt.AssertNoErr(t, err)
+	wt.AssertTrue(t, resp != nil, "reponse from the Get()")
+	t.Logf("Received '%s'", resp.Answer[0])
+	wt.AssertType(t, resp.Answer[0], (*dns.A)(nil), "DNS record")
+	wt.AssertEqualString(t, resp.Answer[0].(*dns.A).A.String(), "10.0.1.3", "IP address")
+	wt.AssertEqualInt(t, int(resp.Answer[0].Header().Ttl), int(localTTL), "TTL")
+
+	clk.Add(time.Duration(localTTL-1) * time.Second)
+	resp, err = l.Get(questionMsg, minUDPSize)
+	wt.AssertNoErr(t, err)
+	wt.AssertTrue(t, resp != nil, "reponse from the Get()")
+	t.Logf("Received '%s'", resp.Answer[0])
+	wt.AssertType(t, resp.Answer[0], (*dns.A)(nil), "DNS record")
+	wt.AssertEqualString(t, resp.Answer[0].(*dns.A).A.String(), "10.0.1.3", "IP address")
+	wt.AssertEqualInt(t, int(resp.Answer[0].Header().Ttl), 1, "TTL")
+
+	t.Logf("Checking we get empty replies when they are expired...")
+	lenBefore = l.Len()
+	clk.Add(time.Duration(localTTL) * time.Second)
+	resp, err = l.Get(questionMsg, minUDPSize)
+	wt.AssertNoErr(t, err)
+	if resp != nil {
+		t.Logf("Got\n%s", resp.Answer[0])
+		t.Fatalf("ERROR: Did NOT expect a reponse from the Get()")
+	}
+	wt.AssertEqualInt(t, l.Len(), lenBefore-1, "cache length (after getting an expired entry)")
+
+	questionMsg2 := new(dns.Msg)
+	questionMsg2.SetQuestion("some.other.name", dns.TypeA)
+	questionMsg2.RecursionDesired = true
+	question2 := &questionMsg2.Question[0]
+
+	t.Logf("Trying to Get() a name")
+	resp, err = l.Get(questionMsg2, minUDPSize)
+	wt.AssertNoErr(t, err)
+	wt.AssertNil(t, resp, "reponse from Get() yet")
+
+	t.Logf("Checking that an Remove() between Get() and Put() does not break things")
+	records = []ZoneRecord{Record{"some.name", net.ParseIP("10.0.9.9"), 0, 0, 0}}
+	replyTemp2 := makeAddressReply(questionMsg2, question2, records)
+	l.Remove(question2)
+	l.Put(questionMsg2, replyTemp2, nullTTL, 0)
+	resp, err = l.Get(questionMsg2, minUDPSize)
+	wt.AssertNoErr(t, err)
+	wt.AssertNotNil(t, resp, "reponse from Get()")
+
+	questionMsg3 := new(dns.Msg)
+	questionMsg3.SetQuestion("some.other.name", dns.TypeA)
+	questionMsg3.RecursionDesired = true
+	question3 := &questionMsg3.Question[0]
+
+	t.Logf("Checking that a entry with CacheNoLocalReplies return an error")
+	l.Put(questionMsg3, nil, negLocalTTL, CacheNoLocalReplies)
+	resp, err = l.Get(questionMsg3, minUDPSize)
+	wt.AssertNil(t, resp, "Get() response with CacheNoLocalReplies")
+	wt.AssertTrue(t, err == errNoLocalReplies, "Get() error with CacheNoLocalReplies")
+
+	t.Logf("Checking that more Put's do not make the negative TTL longer")
+	clk.Add(time.Duration(negLocalTTL/2) * time.Second)
+	l.Put(questionMsg3, nil, negLocalTTL, CacheNoLocalReplies)
+	clk.Add(time.Duration(1) * time.Second)
+	l.Put(questionMsg3, nil, negLocalTTL, CacheNoLocalReplies)
+	clk.Add(time.Duration(1) * time.Second)
+	l.Put(questionMsg3, nil, negLocalTTL, CacheNoLocalReplies)
+	resp, err = l.Get(questionMsg3, minUDPSize)
+	wt.AssertNil(t, resp, "Get() response with CacheNoLocalReplies")
+	wt.AssertTrue(t, err == errNoLocalReplies, "Get() error with CacheNoLocalReplies")
+
+	clk.Add(time.Duration(negLocalTTL/2) * time.Second)
+	t.Logf("Checking that we get an expired response after %f seconds", negLocalTTL)
+	resp, err = l.Get(questionMsg3, minUDPSize)
+	wt.AssertNil(t, resp, "expired Get() response with CacheNoLocalReplies")
+	wt.AssertNil(t, err, "expired Get() error with CacheNoLocalReplies")
+
+	l.Remove(question3)
+	t.Logf("Checking that Put&Get with CacheNoLocalReplies with a Remove in the middle returns nothing")
+	l.Put(questionMsg3, nil, nullTTL, CacheNoLocalReplies)
+	l.Remove(question3)
+	resp, err = l.Get(questionMsg3, minUDPSize)
+	wt.AssertNil(t, resp, "Get() reponse with CacheNoLocalReplies")
+	wt.AssertNil(t, err, "Get() error with CacheNoLocalReplies")
+}

nameserver/dns.go

@@ -2,12 +2,13 @@ package nameserver
 
 import (
 	"github.com/miekg/dns"
-	"net"
 )
 
 const (
-	localTTL uint32 = 300 // somewhat arbitrary; we don't expect anyone
-	// downstream to cache results
+	localTTL    uint32 = 30 // somewhat arbitrary; we don't expect anyone downstream to cache results
+	negLocalTTL        = 30 // TTL for negative local resolutions
+	minUDPSize         = 512
+	maxUDPSize         = 65535
 )
 
 func makeHeader(r *dns.Msg, q *dns.Question) *dns.RR_Header {
@@ -24,20 +25,31 @@ func makeReply(r *dns.Msg, as []dns.RR) *dns.Msg {
 	return m
 }
 
-func makeAddressReply(r *dns.Msg, q *dns.Question, addrs []net.IP) *dns.Msg {
+func makeTruncatedReply(r *dns.Msg) *dns.Msg {
+	// for truncated response, we create a minimal reply with the Truncated bit set
+	reply := new(dns.Msg)
+	reply.SetReply(r)
+	reply.Truncated = true
+	return reply
+}
+
+func makeAddressReply(r *dns.Msg, q *dns.Question, addrs []ZoneRecord) *dns.Msg {
 	answers := make([]dns.RR, len(addrs))
 	header := makeHeader(r, q)
 	count := 0
 	for _, addr := range addrs {
+		ip := addr.IP()
+		ip4 := ip.To4()
+
 		switch q.Qtype {
 		case dns.TypeA:
-			if ip4 := addr.To4(); ip4 != nil {
-				answers[count] = &dns.A{*header, addr}
+			if ip4 != nil {
+				answers[count] = &dns.A{Hdr: *header, A: ip}
 				count++
 			}
 		case dns.TypeAAAA:
-			if ip4 := addr.To4(); ip4 == nil {
-				answers[count] = &dns.AAAA{*header, addr}
+			if ip4 == nil {
+				answers[count] = &dns.AAAA{Hdr: *header, AAAA: ip}
 				count++
 			}
 		}
@@ -45,11 +57,30 @@ func makeAddressReply(r *dns.Msg, q *dns.Question, addrs []net.IP) *dns.Msg {
 	return makeReply(r, answers[:count])
 }
 
-func makePTRReply(r *dns.Msg, q *dns.Question, names []string) *dns.Msg {
+func makePTRReply(r *dns.Msg, q *dns.Question, names []ZoneRecord) *dns.Msg {
 	answers := make([]dns.RR, len(names))
 	header := makeHeader(r, q)
 	for i, name := range names {
-		answers[i] = &dns.PTR{*header, name}
+		answers[i] = &dns.PTR{Hdr: *header, Ptr: name.Name()}
 	}
 	return makeReply(r, answers)
 }
+
+func makeDNSFailResponse(r *dns.Msg) *dns.Msg {
+	m := new(dns.Msg)
+	m.SetReply(r)
+	m.RecursionAvailable = true
+	m.Rcode = dns.RcodeNameError
+	return m
+}
+
+// get the maximum UDP-reply length
+func getMaxReplyLen(r *dns.Msg, proto dnsProtocol) int {
+	maxLen := minUDPSize
+	if proto == protTCP {
+		maxLen = maxUDPSize
+	} else if opt := r.IsEdns0(); opt != nil {
+		maxLen = int(opt.UDPSize())
+	}
+	return maxLen
+}

nameserver/http.go

@@ -1,108 +1,95 @@
 package nameserver
 
 import (
-	"errors"
 	"fmt"
+	"github.com/gorilla/mux"
 	"github.com/miekg/dns"
-	. "github.com/zettio/weave/common"
-	"io"
+	. "github.com/weaveworks/weave/common"
 	"log"
 	"net"
 	"net/http"
-	"strings"
 )
 
-// Parse a URL of the form PUT /name/<identifier>/<ip-address>
-func parseUrl(url string) (identifier string, ipaddr string, err error) {
-	parts := strings.Split(url, "/")
-	if len(parts) != 4 {
-		return "", "", errors.New("Unable to parse url: " + url)
-	}
-	return parts[2], parts[3], nil
-}
-
 func httpErrorAndLog(level *log.Logger, w http.ResponseWriter, msg string,
 	status int, logmsg string, logargs ...interface{}) {
 	http.Error(w, msg, status)
-	level.Printf(logmsg, logargs...)
+	level.Printf("[http] "+logmsg, logargs...)
 }
 
-func ListenHttp(domain string, db Zone, port int) {
+func ListenHTTP(version string, server *DNSServer, domain string, db Zone, port int) {
 
-	http.HandleFunc("/status", func(w http.ResponseWriter, r *http.Request) {
-		if r.Method != "GET" {
-			http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
-		} else {
-			io.WriteString(w, "ok")
-		}
+	muxRouter := mux.NewRouter()
+
+	muxRouter.Methods("GET").Path("/status").HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		fmt.Fprintln(w, "weave DNS", version)
+		fmt.Fprintln(w, server.Status())
 	})
 
-	http.HandleFunc("/name/", func(w http.ResponseWriter, r *http.Request) {
-
+	muxRouter.Methods("PUT").Path("/name/{id:.+}/{ip:.+}").HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 		reqError := func(msg string, logmsg string, logargs ...interface{}) {
-			httpErrorAndLog(Warning, w, msg, http.StatusBadRequest,
-				logmsg, logargs...)
+			httpErrorAndLog(Warning, w, msg, http.StatusBadRequest, logmsg, logargs...)
 		}
 
-		switch r.Method {
-		case "PUT":
-			ident, ipStr, err := parseUrl(r.URL.Path)
-			name := r.FormValue("fqdn")
-			if ident == "" || ipStr == "" || name == "" {
-				reqError("Invalid request", "Invalid request: %s, %s", r.URL, r.Form)
-				return
-			}
+		vars := mux.Vars(r)
+		idStr := vars["id"]
+		ipStr := vars["ip"]
+		name := r.FormValue("fqdn")
 
-			ip := net.ParseIP(ipStr)
-			if ip == nil {
-				reqError("Invalid IP", "Invalid IP in request: %s", ipStr)
-				return
-			}
+		if name == "" {
+			reqError("Invalid FQDN", "Invalid FQDN in request: %s, %s", r.URL, r.Form)
+			return
+		}
 
-			if dns.IsSubDomain(domain, name) {
-				Info.Printf("Adding %s -> %s", name, ipStr)
-				if err = db.AddRecord(ident, name, ip); err != nil {
-					if _, ok := err.(DuplicateError); !ok {
-						httpErrorAndLog(
-							Error, w, "Internal error", http.StatusInternalServerError,
-							"Unexpected error from DB: %s", err)
-						return
-					} // oh, I already know this. whatever.
-				}
-			} else {
-				Info.Printf("Ignoring name %s, not in %s", name, domain)
-			}
+		ip := net.ParseIP(ipStr)
+		if ip == nil {
+			reqError("Invalid IP", "Invalid IP in request: %s", ipStr)
+			return
+		}
 
-		case "DELETE":
-			ident, ipStr, err := parseUrl(r.URL.Path)
-			if ident == "" || ipStr == "" {
-				reqError("Invalid Request", "Invalid request: %s, %s", r.URL, r.Form)
-				return
-			}
-
-			ip := net.ParseIP(ipStr)
-			if ip == nil {
-				reqError("Invalid IP in request", "Invalid IP in request: %s", ipStr)
-				return
-			}
-			Info.Printf("Deleting %s (%s)", ident, ipStr)
-			if err = db.DeleteRecord(ident, ip); err != nil {
-				if _, ok := err.(LookupError); !ok {
+		if dns.IsSubDomain(domain, name) {
+			Info.Printf("[http] Adding %s -> %s", name, ipStr)
+			if err := db.AddRecord(idStr, name, ip); err != nil {
+				if _, ok := err.(DuplicateError); !ok {
 					httpErrorAndLog(
 						Error, w, "Internal error", http.StatusInternalServerError,
 						"Unexpected error from DB: %s", err)
 					return
-				}
+				} // oh, I already know this. whatever.
 			}
-		default:
-			msg := "Unexpected http method: " + r.Method
-			reqError(msg, msg)
-			return
+		} else {
+			Info.Printf("[http] Ignoring name %s, not in %s", name, domain)
 		}
 	})
 
+	muxRouter.Methods("DELETE").Path("/name/{id:.+}/{ip:.+}").HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		reqError := func(msg string, logmsg string, logargs ...interface{}) {
+			httpErrorAndLog(Warning, w, msg, http.StatusBadRequest, logmsg, logargs...)
+		}
+
+		vars := mux.Vars(r)
+		idStr := vars["id"]
+		ipStr := vars["ip"]
+
+		ip := net.ParseIP(ipStr)
+		if ip == nil {
+			reqError("Invalid IP in request", "Invalid IP in request: %s", ipStr)
+			return
+		}
+		Info.Printf("[http] Deleting %s (%s)", idStr, ipStr)
+		if err := db.DeleteRecord(idStr, ip); err != nil {
+			if _, ok := err.(LookupError); !ok {
+				httpErrorAndLog(
+					Error, w, "Internal error", http.StatusInternalServerError,
+					"Unexpected error from DB: %s", err)
+				return
+			}
+		}
+	})
+
+	http.Handle("/", muxRouter)
+
 	address := fmt.Sprintf(":%d", port)
 	if err := http.ListenAndServe(address, nil); err != nil {
-		Error.Fatal("Unable to create http listener: ", err)
+		Error.Fatal("[http] Unable to create http listener: ", err)
 	}
 }

nameserver/http_test.go

@@ -2,7 +2,7 @@ package nameserver
 
 import (
 	"fmt"
-	wt "github.com/zettio/weave/common"
+	wt "github.com/weaveworks/weave/testing"
 	"math/rand"
 	"net"
 	"net/http"
@@ -23,63 +23,66 @@ func TestHttp(t *testing.T) {
 		containerID     = "deadbeef"
 		testDomain      = "weave.local."
 		successTestName = "test1." + testDomain
-		testAddr1       = "10.0.2.1/24"
+		testAddr1       = "10.2.2.1/24"
 		dockerIP        = "9.8.7.6"
 	)
 
-	var zone = new(ZoneDb)
+	zone := NewZoneDb(DefaultLocalDomain)
+
 	port := rand.Intn(10000) + 32768
 	fmt.Println("Http test on port", port)
-	go ListenHttp(testDomain, zone, port)
+	go ListenHTTP("", nil, testDomain, zone, port)
 
 	time.Sleep(100 * time.Millisecond) // Allow for http server to get going
 
 	// Ask the http server to add our test address into the database
 	addrParts := strings.Split(testAddr1, "/")
-	addrUrl := fmt.Sprintf("http://localhost:%d/name/%s/%s", port, containerID, addrParts[0])
-	resp, err := genForm("PUT", addrUrl,
+	addrURL := fmt.Sprintf("http://localhost:%d/name/%s/%s", port, containerID, addrParts[0])
+	resp, err := genForm("PUT", addrURL,
 		url.Values{"fqdn": {successTestName}, "local_ip": {dockerIP}, "routing_prefix": {addrParts[1]}})
 	wt.AssertNoErr(t, err)
 	wt.AssertStatus(t, resp.StatusCode, http.StatusOK, "http response")
 
 	// Check that the address is now there.
-	foundIP, err := zone.LookupLocal(successTestName)
-	wt.AssertNoErr(t, err)
 	ip, _, _ := net.ParseCIDR(testAddr1)
-	if !foundIP.Equal(ip) {
-		t.Fatal("Unexpected result for", successTestName, foundIP)
+	foundIP, err := zone.LookupName(successTestName)
+	wt.AssertNoErr(t, err)
+	if !foundIP[0].IP().Equal(ip) {
+		t.Fatalf("Unexpected result for %s: received %s, expected %s", successTestName, foundIP, ip)
 	}
 
 	// Adding exactly the same address should be OK
-	resp, err = genForm("PUT", addrUrl,
+	resp, err = genForm("PUT", addrURL,
 		url.Values{"fqdn": {successTestName}, "local_ip": {dockerIP}, "routing_prefix": {addrParts[1]}})
 	wt.AssertNoErr(t, err)
 	wt.AssertStatus(t, resp.StatusCode, http.StatusOK, "http success response for duplicate add")
 
 	// Now try adding the same address again with a different ident --
 	// again should be fine
-	otherUrl := fmt.Sprintf("http://localhost:%d/name/%s/%s", port, "other", addrParts[0])
-	resp, err = genForm("PUT", otherUrl,
+	otherURL := fmt.Sprintf("http://localhost:%d/name/%s/%s", port, "other", addrParts[0])
+	resp, err = genForm("PUT", otherURL,
 		url.Values{"fqdn": {successTestName}, "local_ip": {dockerIP}, "routing_prefix": {addrParts[1]}})
 	wt.AssertNoErr(t, err)
 	wt.AssertStatus(t, resp.StatusCode, http.StatusOK, "http response")
 
 	// Delete the address
-	resp, err = genForm("DELETE", addrUrl, nil)
+	resp, err = genForm("DELETE", addrURL, nil)
 	wt.AssertNoErr(t, err)
 	wt.AssertStatus(t, resp.StatusCode, http.StatusOK, "http response")
 
 	// Check that the address is still resolvable.
-	_, err = zone.LookupLocal(successTestName)
+	x, err := zone.LookupName(successTestName)
+	t.Logf("Got %s", x)
 	wt.AssertNoErr(t, err)
 
 	// Delete the address record mentioning the other container
-	resp, err = genForm("DELETE", otherUrl, nil)
+	resp, err = genForm("DELETE", otherURL, nil)
 	wt.AssertNoErr(t, err)
 	wt.AssertStatus(t, resp.StatusCode, http.StatusOK, "http response")
 
 	// Check that the address is gone
-	_, err = zone.LookupLocal(successTestName)
+	x, err = zone.LookupName(successTestName)
+	t.Logf("Got %s", x)
 	wt.AssertErrorType(t, err, (*LookupError)(nil), "fully-removed address")
 
 	// Would like to shut down the http server at the end of this test

nameserver/mdns_client.go

@@ -1,6 +1,8 @@
 package nameserver
 
 import (
+	"bytes"
+	"fmt"
 	"github.com/miekg/dns"
 	"math"
 	"net"
@@ -26,15 +28,54 @@ var (
 	}
 )
 
-type ResponseA struct {
-	Name string
-	Addr net.IP
-	Err  error
+type Response struct {
+	name string
+	addr net.IP
+	ttl  int
+	err  error
+}
+
+func (r Response) Name() string  { return r.name }
+func (r Response) IP() net.IP    { return r.addr }
+func (r Response) Priority() int { return 0 }
+func (r Response) Weight() int   { return 0 }
+func (r Response) TTL() int      { return r.ttl }
+
+func (r Response) Equal(r2 *Response) bool {
+	if r.name != r2.name {
+		return false
+	}
+	if !r.addr.Equal(r2.addr) {
+		return false
+	}
+	if r.err != r2.err {
+		return false
+	}
+	return true
+}
+
+func (r Response) String() string {
+	var buf bytes.Buffer
+	if r.err != nil {
+		fmt.Fprintf(&buf, "%s", r.err)
+	} else {
+		if len(r.Name()) > 0 {
+			fmt.Fprintf(&buf, "%s", r.Name())
+		}
+		if !r.IP().IsUnspecified() {
+			fmt.Fprintf(&buf, "[%s]", r.IP())
+		}
+		if r.ttl > 0 {
+			fmt.Fprintf(&buf, "(TTL:%d)", r.TTL())
+		}
+	}
+	return buf.String()
 }
 
 type responseInfo struct {
-	timeout time.Time // if no answer by this time, give up
-	ch      chan<- *ResponseA
+	timeout   time.Time // if no answer by this time, give up
+	insistent bool      // insistent queries are not removed on the first reply
+	ch        chan<- *Response
 }
 
 // Represents one query that we have sent for one name.
@@ -50,31 +91,36 @@ type inflightQuery struct {
 type MDNSClient struct {
 	// note unorthodox use of 'Server' class in client logic - using it to
 	// listen on a multicast socket and call callback whenever a message comes in.
-	listener  *dns.Server
-	conn      *net.UDPConn
-	addr      *net.UDPAddr
-	inflight  map[string]*inflightQuery
-	queryChan chan<- *MDNSInteraction
+	listener   *dns.Server
+	conn       *net.UDPConn
+	addr       *net.UDPAddr
+	inflight   map[string]*inflightQuery
+	actionChan chan<- MDNSAction
+	running    bool
 }
 
 type mDNSQueryInfo struct {
 	name       string
 	querytype  uint16
-	responseCh chan<- *ResponseA
+	responseCh chan<- *Response
 }
 
 func NewMDNSClient() (*MDNSClient, error) {
-	if conn, err := net.ListenUDP("udp4", &net.UDPAddr{IP: net.IPv4zero, Port: 0}); err != nil {
-		return nil, err
-	} else {
-		return &MDNSClient{
-			conn:     conn,
-			addr:     ipv4Addr,
-			inflight: make(map[string]*inflightQuery)}, nil
-	}
+	return &MDNSClient{
+		addr:     ipv4Addr,
+		running:  false,
+		inflight: make(map[string]*inflightQuery)}, nil
 }
 
-func (c *MDNSClient) Start(ifi *net.Interface) error {
+func (c *MDNSClient) Start(ifi *net.Interface) (err error) {
+	if c.running {
+		return nil
+	}
+
+	if c.conn, err = net.ListenUDP("udp4", &net.UDPAddr{IP: net.IPv4zero, Port: 0}); err != nil {
+		return err
+	}
+
 	multicast, err := LinkLocalMulticastListener(ifi)
 	if err != nil {
 		return err
@@ -90,10 +136,16 @@ func (c *MDNSClient) Start(ifi *net.Interface) error {
 	c.listener = &dns.Server{Unsafe: true, PacketConn: multicast, Handler: dns.HandlerFunc(handleMDNS)}
 	go c.listener.ActivateAndServe()
 
-	queryChan := make(chan *MDNSInteraction, MailboxSize)
-	c.queryChan = queryChan
-	go c.queryLoop(queryChan)
+	actionChan := make(chan MDNSAction, MailboxSize)
+	c.actionChan = actionChan
+	go c.actorLoop(actionChan)
+	return nil
+}
 
+func (c *MDNSClient) Stop() error {
+	if c.running {
+		c.actionChan <- nil
+	}
 	return nil
 }
 
@@ -103,34 +155,86 @@ func LinkLocalMulticastListener(ifi *net.Interface) (net.PacketConn, error) {
 
 // ACTOR client API
 
-const (
-	CSendQuery       = iota
-	CShutdown        = iota
-	CMessageReceived = iota
-)
-
-type MDNSInteraction struct {
-	code       int
-	resultChan chan<- interface{}
-	payload    interface{}
-}
+type MDNSAction func()
 
 // Async
-func (c *MDNSClient) Shutdown() {
-	c.queryChan <- &MDNSInteraction{code: CShutdown}
-}
-
-// Async
-func (c *MDNSClient) SendQuery(name string, querytype uint16, responseCh chan<- *ResponseA) {
-	c.queryChan <- &MDNSInteraction{
-		code:    CSendQuery,
-		payload: mDNSQueryInfo{name, querytype, responseCh},
+func (c *MDNSClient) SendQuery(name string, querytype uint16, insistent bool, responseCh chan<- *Response) {
+	c.actionChan <- func() {
+		query, found := c.inflight[name]
+		if !found {
+			m := new(dns.Msg)
+			m.SetQuestion(name, querytype)
+			m.RecursionDesired = false
+			buf, err := m.Pack()
+			if err != nil {
+				responseCh <- &Response{err: err}
+				close(responseCh)
+				return
+			}
+			query = &inflightQuery{
+				name: name,
+				id:   m.Id,
+			}
+			if _, err = c.conn.WriteTo(buf, c.addr); err != nil {
+				responseCh <- &Response{err: err}
+				close(responseCh)
+				return
+			}
+			c.inflight[name] = query
+		}
+		info := &responseInfo{
+			ch:        responseCh,
+			timeout:   time.Now().Add(mDNSTimeout),
+			insistent: insistent,
+		}
+		// Invariant on responseInfos: they are in ascending order of
+		// timeout.  Since we use a fixed interval from Now(), this
+		// must be after all existing timeouts.
+		query.responseInfos = append(query.responseInfos, info)
 	}
 }
 
 // Async - called from dns library multiplexer
 func (c *MDNSClient) ResponseCallback(r *dns.Msg) {
-	c.queryChan <- &MDNSInteraction{code: CMessageReceived, payload: r}
+	c.actionChan <- func() {
+		for _, answer := range r.Answer {
+			var name string
+			var res *Response
+
+			switch rr := answer.(type) {
+			case *dns.A:
+				name = rr.Hdr.Name
+				res = &Response{name: name, addr: rr.A, ttl: int(rr.Hdr.Ttl)}
+			case *dns.PTR:
+				name = rr.Hdr.Name
+				raddr, _ := raddrToIP(name)
+				res = &Response{name: rr.Ptr, addr: raddr, ttl: int(rr.Hdr.Ttl)}
+			default:
+				return
+			}
+
+			if query, found := c.inflight[name]; found {
+				var newResponseInfos []*responseInfo
+				for _, resp := range query.responseInfos {
+					resp.ch <- res
+					// insistent queries are not removed on the first reply, but on the timeout
+					if resp.insistent {
+						newResponseInfos = append(newResponseInfos, resp)
+					} else {
+						close(resp.ch)
+					}
+				}
+				if len(newResponseInfos) == 0 {
+					delete(c.inflight, name)
+				} else {
+					query.responseInfos = newResponseInfos
+				}
+			} else {
+				// We've received a response that didn't match a query
+				// Do we want to cache it?
+			}
+		}
+	}
 }
 
 // ACTOR server
@@ -164,28 +268,18 @@ func (c *MDNSClient) checkInFlightQueries() time.Duration {
 	return after
 }
 
-func (c *MDNSClient) queryLoop(queryChan <-chan *MDNSInteraction) {
+func (c *MDNSClient) actorLoop(actionChan <-chan MDNSAction) {
 	timer := time.NewTimer(MaxDuration)
-	run := func() {
-		timer.Reset(c.checkInFlightQueries())
-	}
-
-	terminate := false
-	for !terminate {
+	run := func() { timer.Reset(c.checkInFlightQueries()) }
+	c.running = true
+	for c.running {
 		select {
-		case query, ok := <-queryChan:
-			if !ok {
-				break
-			}
-			switch query.code {
-			case CShutdown:
+		case action := <-actionChan:
+			if action == nil {
 				c.listener.Shutdown()
-				terminate = true
-			case CSendQuery:
-				c.handleSendQuery(query.payload.(mDNSQueryInfo))
-				run()
-			case CMessageReceived:
-				c.handleResponse(query.payload.(*dns.Msg))
+				c.running = false
+			} else {
+				action()
 				run()
 			}
 		case <-timer.C:
@@ -200,54 +294,3 @@ func (c *MDNSClient) queryLoop(queryChan <-chan *MDNSInteraction) {
 		}
 	}
 }
-
-func (c *MDNSClient) handleSendQuery(q mDNSQueryInfo) {
-	query, found := c.inflight[q.name]
-	if !found {
-		m := new(dns.Msg)
-		m.SetQuestion(q.name, q.querytype)
-		m.RecursionDesired = false
-		buf, err := m.Pack()
-		if err != nil {
-			q.responseCh <- &ResponseA{Err: err}
-			close(q.responseCh)
-			return
-		}
-		query = &inflightQuery{
-			name: q.name,
-			id:   m.Id,
-		}
-		if _, err = c.conn.WriteTo(buf, c.addr); err != nil {
-			q.responseCh <- &ResponseA{Err: err}
-			close(q.responseCh)
-			return
-		}
-		c.inflight[q.name] = query
-	}
-	info := &responseInfo{
-		ch:      q.responseCh,
-		timeout: time.Now().Add(mDNSTimeout),
-	}
-	// Invariant on responseInfos: they are in ascending order of timeout.
-	// Since we use a fixed interval from Now(), this must be after all existing timeouts.
-	query.responseInfos = append(query.responseInfos, info)
-}
-
-func (c *MDNSClient) handleResponse(r *dns.Msg) {
-	for _, answer := range r.Answer {
-		switch rr := answer.(type) {
-		case *dns.A:
-			name := rr.Hdr.Name
-			if query, found := c.inflight[name]; found {
-				for _, resp := range query.responseInfos {
-					resp.ch <- &ResponseA{Name: rr.Hdr.Name, Addr: rr.A}
-					close(resp.ch)
-				}
-				delete(c.inflight, name)
-			} else {
-				// We've received a response that didn't match a query
-				// Do we want to cache it?
-			}
-		}
-	}
-}

nameserver/mdns_client_test.go

@@ -2,7 +2,8 @@ package nameserver
 
 import (
 	"github.com/miekg/dns"
-	wt "github.com/zettio/weave/common"
+	. "github.com/weaveworks/weave/common"
+	wt "github.com/weaveworks/weave/testing"
 	"log"
 	"net"
 	"testing"
@@ -13,9 +14,27 @@ var (
 	successTestName = "test1.weave."
 	failTestName    = "test2.weave."
 	testAddr        = net.ParseIP("9.8.7.6")
+	testInAddr      = "6.7.8.9.in-addr.arpa."
 )
 
 func minimalServer(w dns.ResponseWriter, req *dns.Msg) {
+
+	sendAnswer := func(m *dns.Msg) {
+		buf, err := m.Pack()
+		CheckFatal(err)
+		if buf == nil {
+			log.Fatal("Nil buffer")
+		}
+		//log.Println("minimalServer sending:", buf)
+		// This is a bit of a kludge - per the RFC we should send responses from 5353, but that doesn't seem to work
+		sendconn, err := net.DialUDP("udp", nil, ipv4Addr)
+		CheckFatal(err)
+
+		_, err = sendconn.Write(buf)
+		sendconn.Close()
+		CheckFatal(err)
+	}
+
 	//log.Println("minimalServer received:", req)
 	if len(req.Answer) > 0 {
 		return // Only interested in questions.
@@ -23,74 +42,77 @@ func minimalServer(w dns.ResponseWriter, req *dns.Msg) {
 	if len(req.Question) != 1 {
 		return // We only handle single-question messages
 	}
-	if req.Question[0].Name != successTestName {
-		return // This is not the DNS record you are looking for
+
+	if req.Question[0].Qtype == dns.TypeA && req.Question[0].Name == successTestName {
+		m := new(dns.Msg)
+		m.SetReply(req)
+		hdr := dns.RR_Header{Name: m.Question[0].Name, Rrtype: dns.TypeA,
+			Class: dns.ClassINET, Ttl: 3600}
+		a := &dns.A{Hdr: hdr, A: testAddr}
+		m.Answer = append(m.Answer, a)
+		sendAnswer(m)
+	} else if req.Question[0].Qtype == dns.TypePTR && req.Question[0].Name == testInAddr {
+		m := new(dns.Msg)
+		m.SetReply(req)
+		hdr := dns.RR_Header{Name: m.Question[0].Name, Rrtype: dns.TypePTR,
+			Class: dns.ClassINET, Ttl: 3600}
+		a := &dns.PTR{Hdr: hdr, Ptr: successTestName}
+		m.Answer = append(m.Answer, a)
+		sendAnswer(m)
 	}
-
-	m := new(dns.Msg)
-	m.SetReply(req)
-
-	hdr := dns.RR_Header{Name: m.Question[0].Name, Rrtype: dns.TypeA,
-		Class: dns.ClassINET, Ttl: 3600}
-	a := &dns.A{hdr, testAddr}
-	m.Answer = append(m.Answer, a)
-
-	buf, err := m.Pack()
-	checkFatal(err)
-	if buf == nil {
-		log.Fatal("Nil buffer")
-	}
-	//log.Println("minimalServer sending:", buf)
-	// This is a bit of a kludge - per the RFC we should send responses from 5353, but that doesn't seem to work
-	sendconn, err := net.DialUDP("udp", nil, ipv4Addr)
-	checkFatal(err)
-
-	_, err = sendconn.Write(buf)
-	sendconn.Close()
-	checkFatal(err)
 }
 
 func RunLocalMulticastServer() (*dns.Server, error) {
-	multicast, err := net.ListenMulticastUDP("udp", nil, ipv4Addr)
+	multicast, err := LinkLocalMulticastListener(nil)
 	if err != nil {
 		return nil, err
 	}
-	server := &dns.Server{Listener: nil, PacketConn: multicast, Handler: dns.HandlerFunc(minimalServer)}
+	server := &dns.Server{
+		PacketConn:  multicast,
+		Handler:     dns.HandlerFunc(minimalServer),
+		ReadTimeout: 100 * time.Millisecond}
 	go server.ActivateAndServe()
 	return server, nil
 }
 
 func setup(t *testing.T) (*MDNSClient, *dns.Server, error) {
+	InitDefaultLogging(testing.Verbose())
+
+	server, err := RunLocalMulticastServer()
+	if err != nil {
+		t.Fatalf("Unable to run test server: %s. No default multicast interface?", err)
+	}
+
 	mdnsClient, err := NewMDNSClient()
 	wt.AssertNoErr(t, err)
 	err = mdnsClient.Start(nil)
 	wt.AssertNoErr(t, err)
 
-	server, err := RunLocalMulticastServer()
-	if err != nil {
-		t.Fatalf("Unable to run test server: %s", err)
-	}
 	return mdnsClient, server, err
 }
 
+func (c *MDNSClient) Shutdown() {
+	c.actionChan <- nil
+}
+
 type testContext struct {
 	receivedAddr  net.IP
 	receivedCount int
-	channel       chan *ResponseA
+	channel       chan *Response
 }
 
 func newTestContext() *testContext {
-	return &testContext{channel: make(chan *ResponseA, 4)}
+	return &testContext{channel: make(chan *Response, 4)}
 }
 
-func (c *testContext) checkResponse(t *testing.T, channelOk bool, resp *ResponseA) {
+func (c *testContext) checkResponse(t *testing.T, channelOk bool, resp *Response) {
 	if !channelOk {
 		c.channel = nil
 		return
 	}
-	wt.AssertNoErr(t, resp.Err)
-	log.Printf("Got address response %s addr %s", resp.Name, resp.Addr)
-	c.receivedAddr = resp.Addr
+	wt.AssertNoErr(t, resp.err)
+	log.Printf("Got address response %s addr %s", resp.Name(), resp.IP())
+	c.receivedAddr = resp.IP()
 	c.receivedCount++
 }
 
@@ -103,7 +125,7 @@ func TestSimpleQuery(t *testing.T) {
 	context := newTestContext()
 
 	// First, a test we expect to succeed
-	mdnsClient.SendQuery(successTestName, dns.TypeA, context.channel)
+	mdnsClient.SendQuery(successTestName, dns.TypeA, false, context.channel)
 	for resp := range context.channel {
 		context.checkResponse(t, true, resp)
 	}
@@ -114,7 +136,7 @@ func TestSimpleQuery(t *testing.T) {
 
 	// Now, a test we expect to time out with no responses
 	context = newTestContext()
-	mdnsClient.SendQuery("test2.weave.", dns.TypeA, context.channel)
+	mdnsClient.SendQuery("test2.weave.", dns.TypeA, false, context.channel)
 	for resp := range context.channel {
 		context.checkResponse(t, true, resp)
 	}
@@ -133,8 +155,8 @@ func TestParallelQuery(t *testing.T) {
 	context1 := newTestContext()
 	context2 := newTestContext()
 
-	go mdnsClient.SendQuery(successTestName, dns.TypeA, context1.channel)
-	go mdnsClient.SendQuery(successTestName, dns.TypeA, context2.channel)
+	go mdnsClient.SendQuery(successTestName, dns.TypeA, false, context1.channel)
+	go mdnsClient.SendQuery(successTestName, dns.TypeA, false, context2.channel)
 	timeout := time.After(2 * time.Second)
 outerloop:
 	for context1.channel != nil || context2.channel != nil {
@@ -152,3 +174,24 @@ outerloop:
 		t.Fatal("Unexpected result for", successTestName, context1.receivedAddr, context2.receivedAddr, context1.receivedCount, context2.receivedCount)
 	}
 }
+
+func TestAsLookup(t *testing.T) {
+	mdnsClient, server, _ := setup(t)
+	defer mdnsClient.Shutdown()
+	defer server.Shutdown()
+
+	ips, err := mdnsClient.LookupName(successTestName)
+	wt.AssertNoErr(t, err)
+	if !testAddr.Equal(ips[0].IP()) {
+		t.Fatalf("Returned address incorrect %s", ips)
+	}
+
+	ips, err = mdnsClient.LookupName("foo.example.com.")
+	wt.AssertErrorType(t, err, (*LookupError)(nil), "unknown hostname")
+
+	names, err := mdnsClient.LookupInaddr(testInAddr)
+	wt.AssertNoErr(t, err)
+	if !(successTestName == names[0].Name()) {
+		t.Fatalf("Expected name %s, got %s", successTestName, names)
+	}
+}

nameserver/mdns_lookup.go

@@ -0,0 +1,71 @@
+package nameserver
+
+import (
+	"github.com/miekg/dns"
+	. "github.com/weaveworks/weave/common"
+)
+
+func mdnsLookup(client *MDNSClient, name string, qtype uint16, insistent bool) ([]ZoneRecord, error) {
+	var responses []*Response
+	channel := make(chan *Response)
+	client.SendQuery(name, qtype, insistent, channel)
+
+	for resp := range channel {
+		if err := resp.err; err != nil {
+			Debug.Printf("[mdns] Error for query type %s name %s: %s", dns.TypeToString[qtype], name, err)
+			return nil, err
+		}
+		Debug.Printf("[mdns] Got response name for %s-query about name '%s': name '%s', addr '%s'",
+			dns.TypeToString[qtype], name, resp.name, resp.addr)
+
+		if !insistent {
+			return []ZoneRecord{resp}, nil // non-background queries return the first response...
+		}
+
+		// add the response, avoiding duplicates
+		// TODO: replace this linear search by something better... (not a big deal as we don't expect long responses)
+		duplicate := false
+		for _, r := range responses {
+			if r.Equal(resp) {
+				duplicate = true
+				break
+			}
+		}
+		if !duplicate {
+			responses = append(responses, resp)
+		}
+	}
+
+	if len(responses) == 0 {
+		return nil, LookupError(name)
+	}
+
+	// convert []Response to []ZoneRecord
+	res := make([]ZoneRecord, len(responses))
+	for i, rr := range responses {
+		res[i] = ZoneRecord(rr)
+	}
+	return res, nil
+}
+
+// Perform a lookup for a name, returning either an IP address or an error
+func (client *MDNSClient) LookupName(name string) ([]ZoneRecord, error) {
+	return mdnsLookup(client, name, dns.TypeA, false)
+}
+
+// Perform a lookup for an IP address, returning either a name or an error
+func (client *MDNSClient) LookupInaddr(inaddr string) ([]ZoneRecord, error) {
+	return mdnsLookup(client, inaddr, dns.TypePTR, false)
+}
+
+// Perform lookup for a name, returning either a list of IP addresses or an error
+// Insistent queries will wait up to `mDNSTimeout` milliseconds for responses
+func (client *MDNSClient) InsistentLookupName(name string) ([]ZoneRecord, error) {
+	return mdnsLookup(client, name, dns.TypeA, true)
+}
+
+// Perform an insistent lookup for an IP address, returning either a list of names or an error
+// Insistent queries will wait up to `mDNSTimeout` milliseconds for responses
+func (client *MDNSClient) InsistentLookupInaddr(inaddr string) ([]ZoneRecord, error) {
+	return mdnsLookup(client, inaddr, dns.TypePTR, true)
+}

nameserver/mdns_server.go

@@ -2,23 +2,22 @@ package nameserver
 
 import (
 	"github.com/miekg/dns"
-	. "github.com/zettio/weave/common"
+	. "github.com/weaveworks/weave/common"
 	"net"
 )
 
 type MDNSServer struct {
 	localAddrs []net.Addr
 	sendconn   *net.UDPConn
+	srv        *dns.Server
 	zone       Zone
+	running    bool
 }
 
+// Create a new mDNS server
+// Nothing will be done (including port bindings) until you `Start()` the server
 func NewMDNSServer(zone Zone) (*MDNSServer, error) {
-	// This is a bit of a kludge - per the RFC we should send responses from 5353, but that doesn't seem to work
-	sendconn, err := net.ListenUDP("udp4", &net.UDPAddr{IP: net.IPv4zero, Port: 0})
-	if err != nil {
-		return nil, err
-	}
-	return &MDNSServer{sendconn: sendconn, zone: zone}, nil
+	return &MDNSServer{zone: zone}, nil
 }
 
 // Return true if testaddr is a UDP address with IP matching my local i/f
@@ -35,7 +34,19 @@ func (s *MDNSServer) addrIsLocal(testaddr net.Addr) bool {
 	return false
 }
 
-func (s *MDNSServer) Start(ifi *net.Interface) error {
+// Start the mDNS server
+func (s *MDNSServer) Start(ifi *net.Interface) (err error) {
+	// skip double initialization
+	if s.running {
+		return nil
+	}
+
+	// This is a bit of a kludge - per the RFC we should send responses from 5353, but that doesn't seem to work
+	s.sendconn, err = net.ListenUDP("udp4", &net.UDPAddr{IP: net.IPv4zero, Port: 0})
+	if err != nil {
+		return err
+	}
+
 	conn, err := LinkLocalMulticastListener(ifi)
 	if err != nil {
 		return err
@@ -50,24 +61,76 @@ func (s *MDNSServer) Start(ifi *net.Interface) error {
 		return err
 	}
 
-	handleMDNS := func(w dns.ResponseWriter, r *dns.Msg) {
-		// Ignore answers to other questions
+	handleLocal := s.makeHandler(dns.TypeA,
+		func(zone ZoneLookup, r *dns.Msg, q *dns.Question) *dns.Msg {
+			if ips, err := zone.LookupName(q.Name); err == nil {
+				return makeAddressReply(r, q, ips)
+			}
+			return nil
+		})
+
+	handleReverse := s.makeHandler(dns.TypePTR,
+		func(zone ZoneLookup, r *dns.Msg, q *dns.Question) *dns.Msg {
+			if names, err := zone.LookupInaddr(q.Name); err == nil {
+				return makePTRReply(r, q, names)
+			}
+			return nil
+		})
+
+	mux := dns.NewServeMux()
+	mux.HandleFunc(s.zone.Domain(), handleLocal)
+	mux.HandleFunc(RDNSDomain, handleReverse)
+
+	s.srv = &dns.Server{
+		Listener:   nil,
+		PacketConn: conn,
+		Handler:    mux,
+	}
+	go s.srv.ActivateAndServe()
+	s.running = true
+	return err
+}
+
+// Stop the mDNS server
+func (s *MDNSServer) Stop() error {
+	s.running = false
+	return s.srv.Shutdown()
+}
+
+func (s *MDNSServer) Zone() Zone {
+	return s.zone
+}
+
+type LookupFunc func(ZoneLookup, *dns.Msg, *dns.Question) *dns.Msg
+
+func (s *MDNSServer) makeHandler(qtype uint16, lookup LookupFunc) dns.HandlerFunc {
+	return func(rw dns.ResponseWriter, r *dns.Msg) {
+		// Handle only questions, ignore answers. We might also ignore
+		// questions that arise locally (i.e., that come from an IP we
+		// think is local), but in the interest of avoiding
+		// complication, and easier testing, this is elided on the
+		// assumption that the client wouldn't ask if it already knew
+		// the answer, and if it does ask, it'll be happy to get an
+		// answer.
+		// TODO: use any Answer in the query for adding records in the Zone database...
 		if len(r.Answer) == 0 && len(r.Question) > 0 {
-			q := r.Question[0]
-			if ip, err := s.zone.LookupLocal(q.Name); err == nil {
-				m := makeAddressReply(r, &q, []net.IP{ip})
-				if err = s.sendResponse(m); err != nil {
-					Warning.Printf("Error writing to %s", w)
+			q := &r.Question[0]
+			if q.Qtype == qtype {
+				Debug.Printf("[mdns msgid %d] srv: trying to answer to mDNS query '%s'", r.MsgHdr.Id, q.Name)
+				if m := lookup(s.zone, r, q); m != nil {
+					Debug.Printf("[mdns msgid %d] srv: found local answer to mDNS query '%s'", r.MsgHdr.Id, q.Name)
+					if err := s.sendResponse(m); err != nil {
+						Warning.Printf("[mdns msgid %d] srv: error writing mDNS response to %v", r.MsgHdr.Id, s.sendconn)
+					} else {
+						Debug.Printf("[mdns msgid %d] srv: response sent: %d answers", r.MsgHdr.Id, len(m.Answer))
+					}
+				} else {
+					Debug.Printf("[mdns msgid %d] srv: no local answer for answering mDNS query '%s'",
+						r.MsgHdr.Id, q.Name)
 				}
-			} else if s.addrIsLocal(w.RemoteAddr()) {
-				// ignore this - it's our own query received via multicast
-			} else {
-				Debug.Printf("Failed MDNS lookup for %s", q.Name)
 			}
 		}
 	}
-	go dns.ActivateAndServe(nil, conn, dns.HandlerFunc(handleMDNS))
-	return err
 }
 
 func (s *MDNSServer) sendResponse(m *dns.Msg) error {

nameserver/mdns_server_test.go

@@ -2,46 +2,33 @@ package nameserver
 
 import (
 	"github.com/miekg/dns"
-	wt "github.com/zettio/weave/common"
-	"log"
+	. "github.com/weaveworks/weave/common"
+	wt "github.com/weaveworks/weave/testing"
 	"net"
 	"testing"
 	"time"
 )
 
-var (
-	containerID = "deadbeef"
-	testAddr1   = "10.0.2.1/24"
-)
-
-func sendQuery(name string, querytype uint16) error {
-	m := new(dns.Msg)
-	m.SetQuestion(name, querytype)
-	m.RecursionDesired = false
-	buf, err := m.Pack()
-	if err != nil {
-		return err
-	}
-	conn, err := net.ListenUDP("udp4", &net.UDPAddr{IP: net.IPv4zero, Port: 0})
-	if err != nil {
-		return err
-	}
-	_, err = conn.WriteTo(buf, ipv4Addr)
-	return err
-}
-
 func TestServerSimpleQuery(t *testing.T) {
-	log.Println("TestServerSimpleQuery starting")
-	var zone = new(ZoneDb)
-	ip, _, _ := net.ParseCIDR(testAddr1)
-	zone.AddRecord(containerID, "test.weave.", ip)
+	var (
+		testRecord1 = Record{"test.weave.local.", net.ParseIP("10.20.20.10"), 0, 0, 0}
+		testRecord2 = Record{"test.weave.local.", net.ParseIP("10.20.20.20"), 0, 0, 0}
+		testInAddr1 = "10.20.20.10.in-addr.arpa."
+	)
 
-	mdnsServer, err := NewMDNSServer(zone)
+	InitDefaultLogging(testing.Verbose())
+	Info.Println("TestServerSimpleQuery starting")
+
+	mzone := newMockedZoneWithRecords([]ZoneRecord{testRecord1, testRecord2})
+	mdnsServer, err := NewMDNSServer(mzone)
 	wt.AssertNoErr(t, err)
 	err = mdnsServer.Start(nil)
 	wt.AssertNoErr(t, err)
+	defer mdnsServer.Stop()
 
-	var receivedAddr net.IP
+	var receivedAddrs []net.IP
+	receivedName := ""
+	recvChan := make(chan interface{})
 	receivedCount := 0
 
 	// Implement a minimal listener for responses
@@ -51,38 +38,83 @@ func TestServerSimpleQuery(t *testing.T) {
 	handleMDNS := func(w dns.ResponseWriter, r *dns.Msg) {
 		// Only handle responses here
 		if len(r.Answer) > 0 {
+			t.Logf("Received %d answer(s)", len(r.Answer))
 			for _, answer := range r.Answer {
 				switch rr := answer.(type) {
 				case *dns.A:
-					receivedAddr = rr.A
+					t.Logf("... A:\n%+v", rr)
+					receivedAddrs = append(receivedAddrs, rr.A)
+					receivedCount++
+				case *dns.PTR:
+					t.Logf("... PTR:\n%+v", rr)
+					receivedName = rr.Ptr
 					receivedCount++
 				}
 			}
+			recvChan <- "ok"
 		}
 	}
 
-	listener := &dns.Server{Unsafe: true, PacketConn: multicast, Handler: dns.HandlerFunc(handleMDNS)}
+	sendQuery := func(name string, querytype uint16) {
+		receivedAddrs = make([]net.IP, 0)
+		receivedName = ""
+		receivedCount = 0
+		recvChan = make(chan interface{})
+
+		m := new(dns.Msg)
+		m.SetQuestion(name, querytype)
+		m.RecursionDesired = false
+		buf, err := m.Pack()
+		wt.AssertNoErr(t, err)
+		conn, err := net.ListenUDP("udp4", &net.UDPAddr{IP: net.IPv4zero, Port: 0})
+		wt.AssertNoErr(t, err)
+		Debug.Printf("Sending UDP packet to %s", ipv4Addr)
+		_, err = conn.WriteTo(buf, ipv4Addr)
+		wt.AssertNoErr(t, err)
+
+		Debug.Printf("Waiting for response")
+		select {
+		case <-recvChan:
+			return
+		case <-time.After(100 * time.Millisecond):
+			Debug.Printf("Timeout while waiting for response")
+			return
+		}
+	}
+
+	listener := &dns.Server{
+		Unsafe:      true,
+		PacketConn:  multicast,
+		Handler:     dns.HandlerFunc(handleMDNS),
+		ReadTimeout: 100 * time.Millisecond}
 	go listener.ActivateAndServe()
 	defer listener.Shutdown()
 
 	time.Sleep(100 * time.Millisecond) // Allow for server to get going
 
-	sendQuery("test.weave.", dns.TypeA)
-
-	time.Sleep(time.Second)
-
-	if receivedCount != 1 {
-		t.Fatal("Unexpected result count for test.weave", receivedCount)
+	Debug.Printf("Checking that we get 2 IPs fo name '%s' [A]", testRecord1.Name())
+	sendQuery(testRecord1.Name(), dns.TypeA)
+	if receivedCount != 2 {
+		t.Fatalf("Unexpected result count %d for %s", receivedCount, testRecord1.Name())
 	}
-	if !receivedAddr.Equal(ip) {
-		t.Fatal("Unexpected result for test.weave", receivedAddr)
+	if !(receivedAddrs[0].Equal(testRecord1.IP()) || receivedAddrs[0].Equal(testRecord2.IP())) {
+		t.Fatalf("Unexpected result %s for %s", receivedAddrs, testRecord1.Name())
+	}
+	if !(receivedAddrs[1].Equal(testRecord1.IP()) || receivedAddrs[1].Equal(testRecord2.IP())) {
+		t.Fatalf("Unexpected result %s for %s", receivedAddrs, testRecord1.Name())
 	}
 
-	receivedCount = 0
-
+	Debug.Printf("Checking that 'testfail.weave.' [A] gets no answers")
 	sendQuery("testfail.weave.", dns.TypeA)
-
 	if receivedCount != 0 {
-		t.Fatal("Unexpected result count for testfail.weave", receivedCount)
+		t.Fatalf("Unexpected result count %d for testfail.weave", receivedCount)
+	}
+
+	Debug.Printf("Checking that '%s' [PTR] gets one name", testInAddr1)
+	sendQuery(testInAddr1, dns.TypePTR)
+	if receivedCount != 1 {
+		t.Fatalf("Expected an answer to %s, got %d answers", testInAddr1, receivedCount)
+	} else if !(testRecord1.Name() == receivedName) {
+		t.Fatalf("Expected answer %s to query for %s, got %s", testRecord1.Name(), testInAddr1, receivedName)
 	}
 }

nameserver/mocks_test.go

@@ -0,0 +1,91 @@
+package nameserver
+
+import (
+	. "github.com/weaveworks/weave/common"
+	"net"
+	"sync"
+)
+
+const (
+	testSocketTimeout = 100 // in millisecs
+)
+
+// Warn about some methods that some day should be implemented...
+func notImplWarn() { Warning.Printf("Mocked method. Not implemented.") }
+
+// A mocked Zone that always returns the same records
+// * it does not send/receive any mDNS query
+type mockedZoneWithRecords struct {
+	sync.RWMutex
+
+	records []ZoneRecord
+
+	// Statistics
+	NumLookupsName   int
+	NumLookupsInaddr int
+}
+
+func newMockedZoneWithRecords(zr []ZoneRecord) *mockedZoneWithRecords {
+	return &mockedZoneWithRecords{records: zr}
+}
+func (mz *mockedZoneWithRecords) Domain() string { return DefaultLocalDomain }
+func (mz *mockedZoneWithRecords) LookupName(name string) ([]ZoneRecord, error) {
+	Debug.Printf("[mocked zone]: LookupName: returning records %s", mz.records)
+	mz.Lock()
+	defer mz.Unlock()
+
+	mz.NumLookupsName++
+	var res []ZoneRecord
+	for _, r := range mz.records {
+		if r.Name() == name {
+			res = append(res, r)
+		}
+	}
+	return res, nil
+}
+
+func (mz *mockedZoneWithRecords) LookupInaddr(inaddr string) ([]ZoneRecord, error) {
+	Debug.Printf("[mocked zone]: LookupInaddr: returning records %s", mz.records)
+	mz.Lock()
+	defer mz.Unlock()
+
+	mz.NumLookupsInaddr++
+	var res []ZoneRecord
+	for _, r := range mz.records {
+		revIP, err := raddrToIP(inaddr)
+		if err != nil {
+			return nil, newParseError("lookup address", inaddr)
+		}
+		if r.IP().Equal(revIP) {
+			res = append(res, r)
+		}
+	}
+	return res, nil
+}
+
+func (mz *mockedZoneWithRecords) DomainLookupName(name string) ([]ZoneRecord, error) {
+	return mz.LookupName(name)
+}
+func (mz *mockedZoneWithRecords) DomainLookupInaddr(inaddr string) ([]ZoneRecord, error) {
+	return mz.LookupInaddr(inaddr)
+}
+
+// the following methods are not currently needed...
+func (mz *mockedZoneWithRecords) AddRecord(ident string, name string, ip net.IP) error {
+	notImplWarn()
+	return nil
+}
+func (mz *mockedZoneWithRecords) DeleteRecord(ident string, ip net.IP) error {
+	notImplWarn()
+	return nil
+}
+func (mz *mockedZoneWithRecords) DeleteRecordsFor(ident string) error { notImplWarn(); return nil }
+func (mz *mockedZoneWithRecords) Status() string                      { notImplWarn(); return "nothing" }
+func (mz *mockedZoneWithRecords) ObserveName(name string, observer ZoneRecordObserver) error {
+	notImplWarn()
+	return nil
+}
+func (mz *mockedZoneWithRecords) ObserveInaddr(inaddr string, observer ZoneRecordObserver) error {
+	notImplWarn()
+	return nil
+}

nameserver/server.go

@@ -1,153 +1,339 @@
 package nameserver
 
 import (
+	"bytes"
 	"fmt"
 	"github.com/miekg/dns"
-	. "github.com/zettio/weave/common"
+	. "github.com/weaveworks/weave/common"
 	"net"
+	"sync"
 )
 
 const (
-	LOCAL_DOMAIN = "weave.local."
-	RDNS_DOMAIN  = "in-addr.arpa."
+	DefaultLocalDomain   = "weave.local."     // The default name used for the local domain
+	DefaultServerPort    = 53                 // The default server port
+	DefaultCLICfgFile    = "/etc/resolv.conf" // default "resolv.conf" file to try to load
+	DefaultUDPBuflen     = 4096               // bigger than the default 512
+	DefaultCacheLen      = 8192               // default cache capacity
+	DefaultResolvWorkers = 8                  // default number of resolution workers
+	DefaultTimeout       = 5                  // default timeout for DNS resolutions
 )
 
-// +1 to also exclude a dot
-var rdnsDomainLen = len(RDNS_DOMAIN) + 1
+type DNSServerConfig struct {
+	// (Optional) client config file for resolving upstream servers
+	UpstreamCfgFile string
+	// (Optional) DNS client config for the fallback server(s)
+	UpstreamCfg *dns.ClientConfig
+	// (Optional) port number (for TCP and UDP)
+	Port int
+	// (Optional) local domain (ie, "weave.local.")
+	LocalDomain string
+	// (Optional) cache size
+	CacheLen int
+	// (Optional) timeout for DNS queries
+	Timeout int
+	// (Optional) UDP buffer length
+	UDPBufLen int
+}
 
-func checkFatal(e error) {
-	if e != nil {
-		Error.Fatal(e)
+type dnsProtocol uint8
+
+const (
+	protUDP dnsProtocol = iota // UDP protocol
+	protTCP dnsProtocol = iota // TCP protocol
+)
+
+func (proto dnsProtocol) String() string {
+	switch proto {
+	case protUDP:
+		return "UDP"
+	case protTCP:
+		return "TCP"
 	}
+	return "unknown"
 }
 
-func checkWarn(e error) {
-	if e != nil {
-		Warning.Println(e)
+// get a new dns.Client for a protocol
+func (proto dnsProtocol) GetNewClient(bufsize int) *dns.Client {
+	switch proto {
+	case protTCP:
+		return &dns.Client{Net: "tcp"}
+	case protUDP:
+		return &dns.Client{Net: "udp", UDPSize: uint16(bufsize)}
 	}
+	return nil
 }
 
-func makeDNSFailResponse(r *dns.Msg) *dns.Msg {
-	m := new(dns.Msg)
-	m.SetReply(r)
-	m.RecursionAvailable = true
-	m.Rcode = dns.RcodeNameError
-	return m
+type DNSServer struct {
+	Zone        Zone
+	Iface       *net.Interface
+	Upstream    *dns.ClientConfig
+	udpSrv      *dns.Server
+	tcpSrv      *dns.Server
+	mdnsCli     *MDNSClient
+	mdnsSrv     *MDNSServer
+	cache       *Cache
+	timeout     int
+	udpBuf      int
+	listenersWg *sync.WaitGroup
+
+	Domain     string // the local domain
+	ListenAddr string // the address the server is listening at
 }
 
-func queryHandler(zone Zone, mdnsClient *MDNSClient) dns.HandlerFunc {
-	return func(w dns.ResponseWriter, r *dns.Msg) {
-		q := r.Question[0]
-		Debug.Printf("Local query: %+v", q)
-		if q.Qtype == dns.TypeA {
-			if ip, err := zone.LookupLocal(q.Name); err == nil {
-				m := makeAddressReply(r, &q, []net.IP{ip})
-				w.WriteMsg(m)
-			} else {
-				Debug.Printf("Failed lookup for %s; sending mDNS query", q.Name)
-				// We don't know the answer; see if someone else does
-				channel := make(chan *ResponseA)
-				replies := make([]net.IP, 0)
-				go func() {
-					for resp := range channel {
-						Debug.Printf("Got address response %s to query %s addr %s", resp.Name, q.Name, resp.Addr)
-						replies = append(replies, resp.Addr)
-					}
-					var responseMsg *dns.Msg
-					if len(replies) > 0 {
-						responseMsg = makeAddressReply(r, &q, replies)
-					} else {
-						responseMsg = makeDNSFailResponse(r)
-					}
-					w.WriteMsg(responseMsg)
-				}()
-				mdnsClient.SendQuery(q.Name, dns.TypeA, channel)
-			}
-		} else {
-			Warning.Printf("Local query not handled: %+v", q)
-			m := makeDNSFailResponse(r)
-			w.WriteMsg(m)
+// Creates a new DNS server
+func NewDNSServer(config DNSServerConfig, zone Zone, iface *net.Interface) (s *DNSServer, err error) {
+	s = &DNSServer{
+		Zone:        zone,
+		Iface:       iface,
+		listenersWg: new(sync.WaitGroup),
+
+		Domain:     DefaultLocalDomain,
+		ListenAddr: fmt.Sprintf(":%d", DefaultServerPort),
+	}
+
+	// fill empty parameters with defaults...
+	if config.Port != 0 {
+		s.ListenAddr = fmt.Sprintf(":%d", config.Port)
+	}
+	if len(config.LocalDomain) > 0 {
+		s.Domain = config.LocalDomain
+	}
+	if config.UpstreamCfg != nil {
+		s.Upstream = config.UpstreamCfg
+	} else {
+		cfgFile := DefaultCLICfgFile
+		if len(config.UpstreamCfgFile) > 0 {
+			cfgFile = config.UpstreamCfgFile
 		}
+		if s.Upstream, err = dns.ClientConfigFromFile(cfgFile); err != nil {
+			return nil, err
+		}
+	}
+	if config.Timeout > 0 {
+		s.timeout = config.Timeout
+	}
+	if config.UDPBufLen > 0 {
+		s.udpBuf = config.UDPBufLen
+	}
+	s.mdnsCli, err = NewMDNSClient()
+	if err != nil {
 		return
 	}
+	s.mdnsSrv, err = NewMDNSServer(s.Zone)
+	if err != nil {
+		return
+	}
+	cacheLen := DefaultCacheLen
+	if config.CacheLen > 0 {
+		cacheLen = config.CacheLen
+	}
+	Debug.Printf("[dns] Initializing cache: %d entries", config.CacheLen)
+	s.cache, err = NewCache(cacheLen, nil)
+	if err != nil {
+		return
+	}
+
+	// create two DNS request multiplexerers, depending on the protocol used by clients
+	// (we use the same protocol for asking upstream servers)
+	mux := func(proto dnsProtocol) *dns.ServeMux {
+		m := dns.NewServeMux()
+		m.HandleFunc(s.Domain, s.queryHandler(proto))
+		m.HandleFunc(RDNSDomain, s.rdnsHandler(proto))
+		m.HandleFunc(".", s.notUsHandler(proto))
+		return m
+	}
+
+	s.udpSrv = &dns.Server{Addr: s.ListenAddr, Net: "udp", Handler: mux(protUDP)}
+	s.tcpSrv = &dns.Server{Addr: s.ListenAddr, Net: "tcp", Handler: mux(protTCP)}
+
+	return
 }
 
-func rdnsHandler(zone Zone, mdnsClient *MDNSClient) dns.HandlerFunc {
-	return func(w dns.ResponseWriter, r *dns.Msg) {
-		q := r.Question[0]
-		Debug.Printf("Local rdns query: %+v", q)
-		if q.Qtype == dns.TypePTR {
-			if ip := net.ParseIP(q.Name[:len(q.Name)-rdnsDomainLen]); ip != nil {
-				ip4 := ip.To4()
-				revIP := []byte{ip4[3], ip4[2], ip4[1], ip4[0]}
-				Debug.Printf("Looking for address: %+v", revIP)
-				if name, err := zone.ReverseLookupLocal(revIP); err == nil {
-					Debug.Printf("Found name: %s", name)
-					m := makePTRReply(r, &q, []string{name})
-					w.WriteMsg(m)
-				} else {
-					Debug.Printf("Failed lookup for %s; sending mDNS query", q.Name)
-					// We don't know the answer; see if someone else does
-					// TODO
-				}
-			}
-		}
-	}
+// Start the DNS server
+func (s *DNSServer) Start() error {
+	Info.Printf("Using mDNS on %v", s.Iface)
+	err := s.mdnsCli.Start(s.Iface)
+	CheckFatal(err)
+	err = s.mdnsSrv.Start(s.Iface)
+	CheckFatal(err)
+
+	s.listenersWg.Add(2)
+
+	go func() {
+		defer s.listenersWg.Done()
+
+		Debug.Printf("Listening for DNS on %s (UDP)", s.ListenAddr)
+		err = s.udpSrv.ListenAndServe()
+		CheckFatal(err)
+		Debug.Printf("DNS UDP server exiting...")
+	}()
+
+	go func() {
+		defer s.listenersWg.Done()
+
+		Debug.Printf("Listening for DNS on %s (TCP)", s.ListenAddr)
+		err = s.tcpSrv.ListenAndServe()
+		CheckFatal(err)
+		Debug.Printf("DNS TCP server exiting...")
+	}()
+
+	// Waiting for all goroutines to finish (otherwise they die as main routine dies)
+	s.listenersWg.Wait()
+
+	Info.Printf("WeaveDNS server exiting...")
+	return nil
 }
 
-/* When we receive a request for a name outside of our '.weave' domain, call
-   the underlying lookup mechanism and return the answer(s) it gives.
-   Unfortunately, this means that TTLs from a real DNS server are lost - FIXME.
-*/
-func notUsHandler() dns.HandlerFunc {
-	return func(w dns.ResponseWriter, r *dns.Msg) {
-		q := r.Question[0]
-		Debug.Printf("Non-local query: %+v", q)
-		var responseMsg *dns.Msg
-		if q.Qtype == dns.TypeA || q.Qtype == dns.TypeAAAA {
-			if addrs, err := net.LookupIP(q.Name); err == nil {
-				responseMsg = makeAddressReply(r, &q, addrs)
-			} else {
-				responseMsg = makeDNSFailResponse(r)
-				Debug.Print("Failed fallback lookup ", err)
-			}
-		} else {
-			Warning.Printf("Non-local query not handled: %+v", q)
-			responseMsg = makeDNSFailResponse(r)
-		}
-		w.WriteMsg(responseMsg)
-	}
+// Return status string
+func (s *DNSServer) Status() string {
+	var buf bytes.Buffer
+	fmt.Fprintln(&buf, "Local domain", s.Domain)
+	fmt.Fprintln(&buf, "Listen address", s.ListenAddr)
+	fmt.Fprintln(&buf, "mDNS interface", s.Iface)
+	fmt.Fprintln(&buf, "Fallback DNS config", s.Upstream)
+	fmt.Fprintf(&buf, "Zone database:\n%s", s.Zone)
+	return buf.String()
 }
 
-func StartServer(zone Zone, iface *net.Interface, dnsPort int, httpPort int, wait int) error {
-	go ListenHttp(LOCAL_DOMAIN, zone, httpPort)
-
-	mdnsClient, err := NewMDNSClient()
-	checkFatal(err)
-
-	ifaceName := "default interface"
-	if iface != nil {
-		ifaceName = iface.Name
+// Perform a graceful shutdown
+func (s *DNSServer) Stop() error {
+	// Stop the listeners/handlers
+	if err := s.tcpSrv.Shutdown(); err != nil {
+		return err
 	}
-	Info.Printf("Using mDNS on %s", ifaceName)
-	err = mdnsClient.Start(iface)
-	checkFatal(err)
+	if err := s.udpSrv.Shutdown(); err != nil {
+		return err
+	}
+	s.listenersWg.Wait()
 
-	LocalServeMux := dns.NewServeMux()
-	LocalServeMux.HandleFunc(LOCAL_DOMAIN, queryHandler(zone, mdnsClient))
-	LocalServeMux.HandleFunc(RDNS_DOMAIN, rdnsHandler(zone, mdnsClient))
-	LocalServeMux.HandleFunc(".", notUsHandler())
-
-	mdnsServer, err := NewMDNSServer(zone)
-	checkFatal(err)
-
-	err = mdnsServer.Start(iface)
-	checkFatal(err)
-
-	address := fmt.Sprintf(":%d", dnsPort)
-	Info.Printf("Listening for DNS on %s", address)
-	err = dns.ListenAndServe(address, "udp", LocalServeMux)
-	checkFatal(err)
+	// shutdown the mDNS server
+	s.mdnsSrv.Stop()
 
 	return nil
 }
+
+func (s *DNSServer) queryHandler(proto dnsProtocol) dns.HandlerFunc {
+	zoneLookup := func(lookup ZoneLookup, q *dns.Question, r *dns.Msg) (*dns.Msg, []ZoneRecord, error) {
+		ips, err := lookup.LookupName(q.Name)
+		if err != nil {
+			return nil, nil, err
+		}
+		return makeAddressReply(r, q, ips), ips, nil
+	}
+
+	fallback := func(w dns.ResponseWriter, r *dns.Msg) {
+		w.WriteMsg(makeDNSFailResponse(r))
+	}
+
+	return s.commonQueryHandler(proto, "Query", dns.TypeA, zoneLookup, fallback)
+}
+
+func (s *DNSServer) rdnsHandler(proto dnsProtocol) dns.HandlerFunc {
+	zoneLookup := func(lookup ZoneLookup, q *dns.Question, r *dns.Msg) (*dns.Msg, []ZoneRecord, error) {
+		names, err := lookup.LookupInaddr(q.Name)
+		if err != nil {
+			return nil, nil, err
+		}
+		return makePTRReply(r, q, names), names, nil
+	}
+
+	notUsHandler := s.notUsHandler(proto)
+	fallback := func(w dns.ResponseWriter, r *dns.Msg) {
+		Info.Printf("[dns msgid %d] -> sending to fallback server", r.MsgHdr.Id)
+		notUsHandler(w, r)
+	}
+
+	return s.commonQueryHandler(proto, "Reverse query", dns.TypePTR, zoneLookup, fallback)
+}
+
+func (s *DNSServer) commonQueryHandler(proto dnsProtocol, kind string, qtype uint16,
+	zoneLookup func(ZoneLookup, *dns.Question, *dns.Msg) (*dns.Msg, []ZoneRecord, error), fallback dns.HandlerFunc) dns.HandlerFunc {
+
+	return func(w dns.ResponseWriter, r *dns.Msg) {
+		q := r.Question[0]
+		Debug.Printf("[dns msgid %d] %s: %+v", r.MsgHdr.Id, kind, q)
+		maxLen := getMaxReplyLen(r, proto)
+		lookups := []ZoneLookup{s.Zone, s.mdnsCli}
+
+		reply, err := s.cache.Get(r, maxLen)
+		if err != nil {
+			if err == errNoLocalReplies {
+				Debug.Printf("[dns msgid %d] Cached 'no local replies' - skipping local lookup", r.MsgHdr.Id)
+				lookups = []ZoneLookup{s.Zone}
+			} else {
+				Debug.Printf("[dns msgid %d] Error from cache: %s", r.MsgHdr.Id, err)
+				w.WriteMsg(makeDNSFailResponse(r))
+				return
+			}
+		}
+		if reply != nil {
+			Debug.Printf("[dns msgid %d] Returning reply from cache: %s/%d answers",
+				r.MsgHdr.Id, dns.RcodeToString[reply.MsgHdr.Rcode], len(reply.Answer))
+			w.WriteMsg(reply)
+			return
+		}
+
+		// catch unsupported queries
+		if q.Qtype != qtype {
+			Debug.Printf("[dns msgid %d] Unsupported query type %s", r.MsgHdr.Id, dns.TypeToString[q.Qtype])
+			m := makeDNSFailResponse(r)
+			s.cache.Put(r, m, negLocalTTL, 0)
+			w.WriteMsg(m)
+			return
+		}
+
+		for _, lookup := range lookups {
+			if m, answers, err := zoneLookup(lookup, &q, r); err == nil {
+				m.Authoritative = true
+				Debug.Printf("[dns msgid %d] Caching response for type %s query for '%s': %s [code:%s]",
+					m.MsgHdr.Id, dns.TypeToString[q.Qtype], q.Name, answers, dns.RcodeToString[m.Rcode])
+				s.cache.Put(r, m, nullTTL, 0)
+				w.WriteMsg(m)
+				return
+			}
+		}
+
+		Info.Printf("[dns msgid %d] No results for type %s query for '%s' [caching no-local]",
+			r.MsgHdr.Id, dns.TypeToString[q.Qtype], q.Name)
+		s.cache.Put(r, nil, negLocalTTL, CacheNoLocalReplies)
+
+		fallback(w, r)
+	}
+}
+
+// When we receive a request for a name outside of our '.weave.local.'
+// domain, ask the configured DNS server as a fallback.
+func (s *DNSServer) notUsHandler(proto dnsProtocol) dns.HandlerFunc {
+	dnsClient := proto.GetNewClient(DefaultUDPBuflen)
+
+	return func(w dns.ResponseWriter, r *dns.Msg) {
+		maxLen := getMaxReplyLen(r, proto)
+		q := r.Question[0]
+
+		// create a request where we announce our max payload size
+		rcopy := r
+		rcopy.SetEdns0(uint16(maxLen), false)
+
+		Debug.Printf("[dns msgid %d] Fallback query: %+v [%s, max:%d bytes]", rcopy.MsgHdr.Id, q, proto, maxLen)
+		for _, server := range s.Upstream.Servers {
+			reply, _, err := dnsClient.Exchange(rcopy, fmt.Sprintf("%s:%s", server, s.Upstream.Port))
+			if err != nil {
+				Debug.Printf("[dns msgid %d] Network error trying %s (%s)",
+					r.MsgHdr.Id, server, err)
+				continue
+			}
+			if reply != nil && reply.Rcode != dns.RcodeSuccess {
+				Debug.Printf("[dns msgid %d] Failure reported by %s for query %s",
+					r.MsgHdr.Id, server, q.Name)
+				continue
+			}
+			Debug.Printf("[dns msgid %d] Given answer by %s for query %s",
+				r.MsgHdr.Id, server, q.Name)
+			w.WriteMsg(reply)
+			return
+		}
+		Warning.Printf("[dns msgid %d] Failed lookup for external name %s", r.MsgHdr.Id, q.Name)
+		w.WriteMsg(makeDNSFailResponse(r))
+	}
+}

nameserver/server_test.go

@@ -0,0 +1,301 @@
+package nameserver
+
+import (
+	"encoding/binary"
+	"fmt"
+	"github.com/miekg/dns"
+	. "github.com/weaveworks/weave/common"
+	wt "github.com/weaveworks/weave/testing"
+	"net"
+	"testing"
+	"time"
+)
+
+const (
+	testRDNSsuccess  = "1.2.2.10.in-addr.arpa."
+	testRDNSfail     = "4.3.2.1.in-addr.arpa."
+	testRDNSnonlocal = "8.8.8.8.in-addr.arpa."
+	testUDPBufSize   = 16384
+)
+
+var (
+	testPort = 17625
+)
+
+func setupForTest(t *testing.T) {
+	// fail early if we cannot find a default multicast interface
+	multicast, err := LinkLocalMulticastListener(nil)
+	if err != nil {
+		t.Fatalf("Unable to create multicast listener: %s. No default multicast interface?", err)
+	}
+	multicast.Close()
+
+	testPort++
+}
+
+func TestUDPDNSServer(t *testing.T) {
+	setupForTest(t)
+	const (
+		containerID     = "foobar"
+		successTestName = "test1.weave.local."
+		failTestName    = "test2.weave.local."
+		nonLocalName    = "weave.works."
+		testAddr1       = "10.2.2.1"
+	)
+	testCIDR1 := testAddr1 + "/24"
+
+	InitDefaultLogging(true)
+	var zone = NewZoneDb(DefaultLocalDomain)
+	ip, _, _ := net.ParseCIDR(testCIDR1)
+	zone.AddRecord(containerID, successTestName, ip)
+
+	fallbackHandler := func(w dns.ResponseWriter, req *dns.Msg) {
+		m := new(dns.Msg)
+		m.SetReply(req)
+		if len(req.Question) == 1 {
+			q := req.Question[0]
+			if q.Name == nonLocalName && q.Qtype == dns.TypeMX {
+				m.Answer = make([]dns.RR, 1)
+				m.Answer[0] = &dns.MX{Hdr: dns.RR_Header{Name: m.Question[0].Name, Rrtype: dns.TypeMX, Class: dns.ClassINET, Ttl: 0}, Mx: "mail." + nonLocalName}
+			} else if q.Name == nonLocalName && q.Qtype == dns.TypeANY {
+				m.Answer = make([]dns.RR, 512/len("mailn."+nonLocalName)+1)
+				for i := range m.Answer {
+					m.Answer[i] = &dns.MX{Hdr: dns.RR_Header{Name: m.Question[0].Name, Rrtype: dns.TypeMX, Class: dns.ClassINET, Ttl: 0}, Mx: fmt.Sprintf("mail%d.%s", i, nonLocalName)}
+				}
+			} else if q.Name == testRDNSnonlocal && q.Qtype == dns.TypePTR {
+				m.Answer = make([]dns.RR, 1)
+				m.Answer[0] = &dns.PTR{Hdr: dns.RR_Header{Name: m.Question[0].Name, Rrtype: dns.TypePTR, Class: dns.ClassINET, Ttl: 0}, Ptr: "ns1.google.com."}
+			} else if q.Name == testRDNSfail && q.Qtype == dns.TypePTR {
+				m.Rcode = dns.RcodeNameError
+			}
+		}
+		w.WriteMsg(m)
+	}
+
+	// Run another DNS server for fallback
+	s, fallbackAddr, err := runLocalUDPServer(t, "127.0.0.1:0", fallbackHandler)
+	wt.AssertNoErr(t, err)
+	defer s.Shutdown()
+
+	_, fallbackPort, err := net.SplitHostPort(fallbackAddr)
+	wt.AssertNoErr(t, err)
+
+	config := &dns.ClientConfig{Servers: []string{"127.0.0.1"}, Port: fallbackPort}
+	srv, err := NewDNSServer(DNSServerConfig{UpstreamCfg: config, Port: testPort}, zone, nil)
+	wt.AssertNoErr(t, err)
+	defer srv.Stop()
+	go srv.Start()
+	time.Sleep(100 * time.Millisecond) // Allow sever goroutine to start
+
+	var r *dns.Msg
+
+	r = assertExchange(t, successTestName, dns.TypeA, 1, 1, 0)
+	wt.AssertType(t, r.Answer[0], (*dns.A)(nil), "DNS record")
+	wt.AssertEqualString(t, r.Answer[0].(*dns.A).A.String(), testAddr1, "IP address")
+
+	assertExchange(t, failTestName, dns.TypeA, 0, 0, dns.RcodeNameError)
+
+	r = assertExchange(t, testRDNSsuccess, dns.TypePTR, 1, 1, 0)
+	wt.AssertType(t, r.Answer[0], (*dns.PTR)(nil), "DNS record")
+	wt.AssertEqualString(t, r.Answer[0].(*dns.PTR).Ptr, successTestName, "IP address")
+
+	assertExchange(t, testRDNSfail, dns.TypePTR, 0, 0, dns.RcodeNameError)
+
+	// This should fail because we don't have information about MX records
+	assertExchange(t, successTestName, dns.TypeMX, 0, 0, dns.RcodeNameError)
+
+	// This non-local query for an MX record should succeed by being
+	// passed on to the fallback server
+	assertExchange(t, nonLocalName, dns.TypeMX, 1, -1, 0)
+
+	// Now ask a query that we expect to return a lot of data.
+	assertExchange(t, nonLocalName, dns.TypeANY, 5, -1, 0)
+
+	assertExchange(t, testRDNSnonlocal, dns.TypePTR, 1, -1, 0)
+
+	// Not testing MDNS functionality of server here (yet), since it
+	// needs two servers, each listening on its own address
+}
+
+func TestTCPDNSServer(t *testing.T) {
+	setupForTest(t)
+	const (
+		numAnswers   = 512
+		nonLocalName = "weave.works."
+	)
+	dnsAddr := fmt.Sprintf("localhost:%d", testPort)
+
+	InitDefaultLogging(true)
+	var zone = NewZoneDb(DefaultLocalDomain)
+
+	// generate a list of `numAnswers` IP addresses
+	var addrs []ZoneRecord
+	bs := make([]byte, 4)
+	for i := 0; i < numAnswers; i++ {
+		binary.LittleEndian.PutUint32(bs, uint32(i))
+		addrs = append(addrs, Record{"", net.IPv4(bs[0], bs[1], bs[2], bs[3]), 0, 0, 0})
+	}
+
+	// handler for the fallback server: it will just return a very long response
+	fallbackUDPHandler := func(w dns.ResponseWriter, req *dns.Msg) {
+		maxLen := getMaxReplyLen(req, protUDP)
+
+		t.Logf("Fallback UDP server got asked: returning %d answers", numAnswers)
+		q := req.Question[0]
+		m := makeAddressReply(req, &q, addrs)
+		mLen := m.Len()
+		m.SetEdns0(uint16(maxLen), false)
+
+		if mLen > maxLen {
+			t.Logf("... truncated response (%d > %d)", mLen, maxLen)
+			m.Truncated = true
+		}
+		w.WriteMsg(m)
+	}
+	fallbackTCPHandler := func(w dns.ResponseWriter, req *dns.Msg) {
+		t.Logf("Fallback TCP server got asked: returning %d answers", numAnswers)
+		q := req.Question[0]
+		m := makeAddressReply(req, &q, addrs)
+		w.WriteMsg(m)
+	}
+
+	t.Logf("Running a DNS fallback server with UDP")
+	us, fallbackUDPAddr, err := runLocalUDPServer(t, "127.0.0.1:0", fallbackUDPHandler)
+	wt.AssertNoErr(t, err)
+	defer us.Shutdown()
+
+	_, fallbackPort, err := net.SplitHostPort(fallbackUDPAddr)
+	wt.AssertNoErr(t, err)
+
+	t.Logf("Starting another fallback server, with TCP, on the same port as the UDP server")
+	fallbackTCPAddr := fmt.Sprintf("127.0.0.1:%s", fallbackPort)
+	ts, fallbackTCPAddr, err := runLocalTCPServer(t, fallbackTCPAddr, fallbackTCPHandler)
+	wt.AssertNoErr(t, err)
+	defer ts.Shutdown()
+
+	t.Logf("Creating a WeaveDNS server instance, falling back to 127.0.0.1:%s", fallbackPort)
+	config := &dns.ClientConfig{Servers: []string{"127.0.0.1"}, Port: fallbackPort}
+	srv, err := NewDNSServer(DNSServerConfig{UpstreamCfg: config, Port: testPort}, zone, nil)
+	wt.AssertNoErr(t, err)
+	defer srv.Stop()
+	go srv.Start()
+	time.Sleep(100 * time.Millisecond) // Allow sever goroutine to start
+
+	t.Logf("Creating a UDP and a TCP client")
+	uc := new(dns.Client)
+	uc.UDPSize = minUDPSize
+	tc := new(dns.Client)
+	tc.Net = "tcp"
+
+	t.Logf("Creating DNS query message")
+	m := new(dns.Msg)
+	m.RecursionDesired = true
+	m.SetQuestion(nonLocalName, dns.TypeA)
+
+	t.Logf("Checking the fallback server at %s returns a truncated response with UDP", fallbackUDPAddr)
+	r, _, err := uc.Exchange(m, fallbackUDPAddr)
+	t.Logf("Got response from fallback server (UDP) with %d answers", len(r.Answer))
+	t.Logf("Response:\n%+v\n", r)
+	wt.AssertNoErr(t, err)
+	wt.AssertTrue(t, r.MsgHdr.Truncated, "DNS truncated reponse flag")
+	wt.AssertNotEqualInt(t, len(r.Answer), numAnswers, "number of answers (UDP)")
+
+	t.Logf("Checking the WeaveDNS server at %s returns a truncated reponse with UDP", dnsAddr)
+	r, _, err = uc.Exchange(m, dnsAddr)
+	t.Logf("UDP Response:\n%+v\n", r)
+	wt.AssertNoErr(t, err)
+	wt.AssertNotNil(t, r, "response")
+	t.Logf("%d answers", len(r.Answer))
+	wt.AssertTrue(t, r.MsgHdr.Truncated, "DNS truncated reponse flag")
+	wt.AssertNotEqualInt(t, len(r.Answer), numAnswers, "number of answers (UDP)")
+
+	t.Logf("Checking the WeaveDNS server at %s does not return a truncated reponse with TCP", dnsAddr)
+	r, _, err = tc.Exchange(m, dnsAddr)
+	t.Logf("TCP Response:\n%+v\n", r)
+	wt.AssertNoErr(t, err)
+	wt.AssertNotNil(t, r, "response")
+	t.Logf("%d answers", len(r.Answer))
+	wt.AssertFalse(t, r.MsgHdr.Truncated, "DNS truncated response flag")
+	wt.AssertEqualInt(t, len(r.Answer), numAnswers, "number of answers (TCP)")
+
+	t.Logf("Checking the WeaveDNS server at %s does not return a truncated reponse with UDP with a bigger buffer", dnsAddr)
+	m.SetEdns0(testUDPBufSize, false)
+	r, _, err = uc.Exchange(m, dnsAddr)
+	t.Logf("UDP-large Response:\n%+v\n", r)
+	wt.AssertNoErr(t, err)
+	wt.AssertNotNil(t, r, "response")
+	t.Logf("%d answers", len(r.Answer))
+	wt.AssertNoErr(t, err)
+	wt.AssertFalse(t, r.MsgHdr.Truncated, "DNS truncated response flag")
+	wt.AssertEqualInt(t, len(r.Answer), numAnswers, "number of answers (UDP-long)")
+}
+
+//////////////////////////////////////////////////////////////////////////////
+
+// perform a DNS query and assert the reply code, number or answers, etc
+func assertExchange(t *testing.T, z string, ty uint16, minAnswers int, maxAnswers int, expErr int) *dns.Msg {
+	c := new(dns.Client)
+	c.UDPSize = testUDPBufSize
+	m := new(dns.Msg)
+	m.RecursionDesired = true
+	m.SetQuestion(z, ty)
+	m.SetEdns0(testUDPBufSize, false) // we don't want to play with truncation here...
+
+	r, _, err := c.Exchange(m, fmt.Sprintf("127.0.0.1:%d", testPort))
+	t.Logf("Response:\n%+v\n", r)
+	wt.AssertNoErr(t, err)
+	if minAnswers == 0 && maxAnswers == 0 {
+		wt.AssertStatus(t, r.Rcode, expErr, "DNS response code")
+	} else {
+		wt.AssertStatus(t, r.Rcode, dns.RcodeSuccess, "DNS response code")
+	}
+	answers := len(r.Answer)
+	if minAnswers >= 0 && answers < minAnswers {
+		wt.Fatalf(t, "Number of answers >= %d", minAnswers)
+	}
+	if maxAnswers >= 0 && answers > maxAnswers {
+		wt.Fatalf(t, "Number of answers <= %d", maxAnswers)
+	}
+	return r
+}
+
+// run a UDP fallback server
+func runLocalUDPServer(t *testing.T, laddr string, handler dns.HandlerFunc) (*dns.Server, string, error) {
+	t.Logf("Starting fallback UDP server at %s", laddr)
+	pc, err := net.ListenPacket("udp", laddr)
+	if err != nil {
+		return nil, "", err
+	}
+	server := &dns.Server{PacketConn: pc, Handler: handler, ReadTimeout: 100 * time.Millisecond}
+
+	go func() {
+		server.ActivateAndServe()
+		pc.Close()
+	}()
+
+	t.Logf("Fallback UDP server listening at %s", pc.LocalAddr())
+	return server, pc.LocalAddr().String(), nil
+}
+
+// run a TCP fallback server
+func runLocalTCPServer(t *testing.T, laddr string, handler dns.HandlerFunc) (*dns.Server, string, error) {
+	t.Logf("Starting fallback TCP server at %s", laddr)
+	laddrTCP, err := net.ResolveTCPAddr("tcp", laddr)
+	if err != nil {
+		return nil, "", err
+	}
+
+	l, err := net.ListenTCP("tcp", laddrTCP)
+	if err != nil {
+		return nil, "", err
+	}
+	server := &dns.Server{Listener: l, Handler: handler, ReadTimeout: 100 * time.Millisecond}
+
+	go func() {
+		server.ActivateAndServe()
+		l.Close()
+	}()
+
+	t.Logf("Fallback TCP server listening at %s", l.Addr().String())
+	return server, l.Addr().String(), nil
+}

nameserver/updater.go

@@ -1,43 +0,0 @@
-package nameserver
-
-import (
-	"github.com/fsouza/go-dockerclient"
-	. "github.com/zettio/weave/common"
-)
-
-func checkError(err error, apiPath string) {
-	if err != nil {
-		Error.Fatalf("Unable to connect to Docker API on %s: %s", apiPath, err)
-	}
-}
-
-func StartUpdater(apiPath string, zone Zone) error {
-	client, err := docker.NewClient(apiPath)
-	checkError(err, apiPath)
-
-	env, err := client.Version()
-	checkError(err, apiPath)
-
-	events := make(chan *docker.APIEvents)
-	err = client.AddEventListener(events)
-	checkError(err, apiPath)
-
-	Info.Printf("Using Docker API on %s: %v", apiPath, env)
-
-	go func() {
-		for event := range events {
-			handleEvent(zone, event, client)
-		}
-	}()
-	return nil
-}
-
-func handleEvent(zone Zone, event *docker.APIEvents, client *docker.Client) error {
-	switch event.Status {
-	case "die":
-		id := event.ID
-		Info.Printf("Container %s down. Removing records", id)
-		zone.DeleteRecordsFor(id)
-	}
-	return nil
-}

nameserver/zone.go

@@ -1,20 +1,30 @@
 package nameserver
 
 import (
+	"bytes"
+	"fmt"
 	"github.com/miekg/dns"
+	. "github.com/weaveworks/weave/common"
 	"net"
 	"sync"
 )
 
+const (
+	RDNSDomain = "in-addr.arpa."
+)
+
+// +1 to also exclude a dot
+var rdnsDomainLen = len(RDNSDomain) + 1
+
 type Zone interface {
 	AddRecord(ident string, name string, ip net.IP) error
 	DeleteRecord(ident string, ip net.IP) error
 	DeleteRecordsFor(ident string) error
-	LookupLocal(name string) (net.IP, error)
-	ReverseLookupLocal(ip net.IP) (string, error)
+	Domain() string
+	ZoneLookup
 }
 
-type Record struct {
+type dbRecord struct {
 	Ident string
 	Name  string
 	IP    net.IP
@@ -23,8 +33,9 @@ type Record struct {
 // Very simple data structure for now, with linear searching.
 // TODO: make more sophisticated to improve performance.
 type ZoneDb struct {
-	mx   sync.RWMutex
-	recs []Record
+	mx     sync.RWMutex
+	recs   []dbRecord
+	domain string
 }
 
 type LookupError string
@@ -40,7 +51,17 @@ func (dup DuplicateError) Error() string {
 	return "Tried to add a duplicate entry"
 }
 
-func (zone *ZoneDb) indexOf(match func(Record) bool) int {
+func NewZoneDb(domain string) *ZoneDb {
+	return &ZoneDb{
+		domain: domain,
+	}
+}
+
+func (zone *ZoneDb) Domain() string {
+	return zone.domain
+}
+
+func (zone *ZoneDb) indexOf(match func(dbRecord) bool) int {
 	for i, r := range zone.recs {
 		if match(r) {
 			return i
@@ -49,54 +70,66 @@ func (zone *ZoneDb) indexOf(match func(Record) bool) int {
 	return -1
 }
 
-func (zone *ZoneDb) LookupLocal(name string) (net.IP, error) {
+func (zone *ZoneDb) String() string {
+	zone.mx.RLock()
+	defer zone.mx.RUnlock()
+	var buf bytes.Buffer
+	for _, r := range zone.recs {
+		fmt.Fprintf(&buf, "%.12s %s %v\n", r.Ident, r.IP, r.Name)
+	}
+	return buf.String()
+}
+
+func (zone *ZoneDb) LookupName(name string) ([]ZoneRecord, error) {
 	zone.mx.RLock()
 	defer zone.mx.RUnlock()
 	for _, r := range zone.recs {
 		if r.Name == name {
-			return r.IP, nil
+			return []ZoneRecord{Record{r.Name, r.IP, 0, 0, 0}}, nil
 		}
 	}
 	return nil, LookupError(name)
 }
 
-func (zone *ZoneDb) ReverseLookupLocal(ip net.IP) (string, error) {
-	zone.mx.RLock()
-	defer zone.mx.RUnlock()
-	for _, r := range zone.recs {
-		if r.IP.Equal(ip) {
-			return r.Name, nil
+func (zone *ZoneDb) LookupInaddr(inaddr string) ([]ZoneRecord, error) {
+	if revIP := net.ParseIP(inaddr[:len(inaddr)-rdnsDomainLen]); revIP != nil {
+		revIP4 := revIP.To4()
+		ip := []byte{revIP4[3], revIP4[2], revIP4[1], revIP4[0]}
+		Debug.Printf("[zonedb] Looking for address: %+v", ip)
+		zone.mx.RLock()
+		defer zone.mx.RUnlock()
+		for _, r := range zone.recs {
+			if r.IP.Equal(ip) {
+				return []ZoneRecord{Record{r.Name, r.IP, 0, 0, 0}}, nil
+			}
 		}
+		return nil, LookupError(inaddr)
 	}
-	return "", LookupError(ip.String())
+	Warning.Printf("[zonedb] Asked to reverse lookup %s", inaddr)
+	return nil, LookupError(inaddr)
 }
 
 func (zone *ZoneDb) AddRecord(ident string, name string, ip net.IP) error {
 	zone.mx.Lock()
 	defer zone.mx.Unlock()
 	fqdn := dns.Fqdn(name)
-	if index := zone.indexOf(
-		func(r Record) bool {
-			return r.Name == fqdn && r.IP.Equal(ip) && r.Ident == ident
-		}); index != -1 {
+	pred := func(r dbRecord) bool { return r.Name == fqdn && r.IP.Equal(ip) && r.Ident == ident }
+	if index := zone.indexOf(pred); index != -1 {
 		return DuplicateError{}
 	}
-	zone.recs = append(zone.recs, Record{ident, fqdn, ip})
+	zone.recs = append(zone.recs, dbRecord{ident, fqdn, ip})
 	return nil
 }
 
 func (zone *ZoneDb) DeleteRecord(ident string, ip net.IP) error {
 	zone.mx.Lock()
 	defer zone.mx.Unlock()
-	if index := zone.indexOf(
-		func(r Record) bool {
-			return r.Ident == ident && r.IP.Equal(ip)
-		}); index == -1 {
-		return LookupError(ident)
-	} else {
+	pred := func(r dbRecord) bool { return r.Ident == ident && r.IP.Equal(ip) }
+	if index := zone.indexOf(pred); index != -1 {
 		zone.recs = append(zone.recs[:index], zone.recs[index+1:]...)
+		return nil
 	}
-	return nil
+	return LookupError(ident)
 }
 
 func (zone *ZoneDb) DeleteRecordsFor(ident string) error {
@@ -113,3 +146,8 @@ func (zone *ZoneDb) DeleteRecordsFor(ident string) error {
 	zone.recs = zone.recs[:w]
 	return nil
 }
+
+func (zone *ZoneDb) ContainerDied(ident string) error {
+	Info.Printf("[zonedb] Container %s down. Removing records", ident)
+	return zone.DeleteRecordsFor(ident)
+}

nameserver/zone_test.go

@@ -1,7 +1,7 @@
 package nameserver
 
 import (
-	wt "github.com/zettio/weave/common"
+	wt "github.com/weaveworks/weave/testing"
 	"net"
 	"testing"
 )
@@ -11,34 +11,34 @@ func TestZone(t *testing.T) {
 		containerID      = "deadbeef"
 		otherContainerID = "cowjuice"
 		successTestName  = "test1.weave."
-		testAddr1        = "10.0.2.1/24"
+		testAddr1        = "10.2.2.1/24"
 	)
 
-	var zone = new(ZoneDb)
+	var zone = NewZoneDb(DefaultLocalDomain)
 
 	ip, _, _ := net.ParseCIDR(testAddr1)
 	err := zone.AddRecord(containerID, successTestName, ip)
 	wt.AssertNoErr(t, err)
 
 	// Add a few more records to make the job harder
-	err = zone.AddRecord("abcdef0123", "adummy.weave.", net.ParseIP("10.0.0.1"))
+	err = zone.AddRecord("abcdef0123", "adummy.weave.", net.ParseIP("10.2.0.1"))
 	wt.AssertNoErr(t, err)
-	err = zone.AddRecord("0123abcdef", "zdummy.weave.", net.ParseIP("10.0.0.2"))
+	err = zone.AddRecord("0123abcdef", "zdummy.weave.", net.ParseIP("10.2.0.2"))
 	wt.AssertNoErr(t, err)
 
 	// Check that the address is now there.
-	foundIP, err := zone.LookupLocal(successTestName)
+	foundIP, err := zone.LookupName(successTestName)
 	wt.AssertNoErr(t, err)
 
-	if !foundIP.Equal(ip) {
+	if !foundIP[0].IP().Equal(ip) {
 		t.Fatal("Unexpected result for", successTestName, foundIP)
 	}
 
 	// See if we can find the address by IP.
-	foundName, err := zone.ReverseLookupLocal(ip)
+	foundName, err := zone.LookupInaddr("1.2.2.10.in-addr.arpa.")
 	wt.AssertNoErr(t, err)
 
-	if foundName != successTestName {
+	if foundName[0].Name() != successTestName {
 		t.Fatal("Unexpected result for", ip, foundName)
 	}
 
@@ -50,14 +50,14 @@ func TestZone(t *testing.T) {
 	err = zone.DeleteRecord(containerID, ip)
 	wt.AssertNoErr(t, err)
 
-	_, err = zone.LookupLocal(successTestName)
+	_, err = zone.LookupName(successTestName)
 	wt.AssertNoErr(t, err)
 
 	err = zone.DeleteRecord(otherContainerID, ip)
 	wt.AssertNoErr(t, err)
 
 	// Check that the address is not there now.
-	_, err = zone.LookupLocal(successTestName)
+	_, err = zone.LookupName(successTestName)
 	wt.AssertErrorType(t, err, (*LookupError)(nil), "after deleting record")
 
 	// Delete a record that isn't there
@@ -69,20 +69,20 @@ func TestDeleteFor(t *testing.T) {
 	var (
 		id    = "foobar"
 		name  = "foo.weave."
-		addr1 = "10.1.2.3/24"
+		addr1 = "10.2.2.3/24"
 		addr2 = "10.2.7.8/24"
 	)
-	zone := new(ZoneDb)
+	zone := NewZoneDb(DefaultLocalDomain)
 	for _, addr := range []string{addr1, addr2} {
 		ip, _, _ := net.ParseCIDR(addr)
 		err := zone.AddRecord(id, name, ip)
 		wt.AssertNoErr(t, err)
 	}
 
-	_, err := zone.LookupLocal(name)
+	_, err := zone.LookupName(name)
 	wt.AssertNoErr(t, err)
 
 	err = zone.DeleteRecordsFor(id)
-	_, err = zone.LookupLocal(name)
+	_, err = zone.LookupName(name)
 	wt.AssertErrorType(t, err, (*LookupError)(nil), "after deleting records for ident")
 }

net/if.go

@@ -10,7 +10,7 @@ func EnsureInterface(ifaceName string, wait int) (iface *net.Interface, err erro
 	if iface, err = findInterface(ifaceName); err == nil || wait == 0 {
 		return
 	}
-	for ; err != nil && wait > 0; wait -= 1 {
+	for ; err != nil && wait > 0; wait-- {
 		time.Sleep(1 * time.Second)
 		iface, err = findInterface(ifaceName)
 	}

proxy/client.go

@@ -0,0 +1,170 @@
+package proxy
+
+import (
+	"bufio"
+	"errors"
+	"fmt"
+	"io"
+	"io/ioutil"
+	"net"
+	"net/http"
+	"net/http/httputil"
+
+	"github.com/fsouza/go-dockerclient"
+	. "github.com/weaveworks/weave/common"
+)
+
+type client struct {
+	interceptor
+	Dial func() (net.Conn, error)
+}
+
+func newClient(dial func() (net.Conn, error), i interceptor) *client {
+	return &client{
+		interceptor: i,
+		Dial:        dial,
+	}
+}
+
+func (c *client) ServeHTTP(w http.ResponseWriter, r *http.Request) {
+	if err := c.InterceptRequest(r); err != nil {
+		_, isNoSuchContainer := err.(*docker.NoSuchContainer)
+		switch {
+		case isNoSuchContainer:
+			http.Error(w, err.Error(), http.StatusNotFound)
+		case err == docker.ErrNoSuchImage:
+			http.Error(w, err.Error(), http.StatusNotFound)
+		default:
+			http.Error(w, err.Error(), http.StatusInternalServerError)
+			Warning.Print("Error intercepting request: ", err)
+		}
+		return
+	}
+
+	conn, err := c.Dial()
+	if err != nil {
+		http.Error(w, "Could not connect to target", http.StatusInternalServerError)
+		Warning.Print(err)
+		return
+	}
+	client := httputil.NewClientConn(conn, nil)
+	defer client.Close()
+
+	resp, err := client.Do(r)
+	if err != nil && err != httputil.ErrPersistEOF {
+		http.Error(w, fmt.Sprintf("Could not make request to target: %v", err), http.StatusInternalServerError)
+		Warning.Print("Error forwarding request: ", err)
+		return
+	}
+	err = c.InterceptResponse(resp)
+	if err != nil {
+		http.Error(w, err.Error(), http.StatusInternalServerError)
+		Warning.Print("Error intercepting response: ", err)
+		return
+	}
+
+	hdr := w.Header()
+	for k, vs := range resp.Header {
+		for _, v := range vs {
+			hdr.Add(k, v)
+		}
+	}
+	Debug.Printf("Response from target: %s %v", resp.Status, w.Header())
+
+	if resp.Header.Get("Content-Type") == "application/vnd.docker.raw-stream" {
+		doRawStream(w, resp, client)
+	} else if resp.TransferEncoding != nil && resp.TransferEncoding[0] == "chunked" {
+		doChunkedResponse(w, resp, client)
+	} else {
+		w.WriteHeader(resp.StatusCode)
+		if _, err := io.Copy(w, resp.Body); err != nil {
+			Warning.Print(err)
+		}
+	}
+}
+
+func doRawStream(w http.ResponseWriter, resp *http.Response, client *httputil.ClientConn) {
+	down, downBuf, up, rem, err := hijack(w, client)
+	if err != nil {
+		http.Error(w, "Unable to hijack connection for raw stream mode", http.StatusInternalServerError)
+		return
+	}
+	defer down.Close()
+	defer up.Close()
+
+	if _, err := down.Write([]byte("HTTP/1.1 200 OK\n")); err != nil {
+		Warning.Print(err)
+		return
+	}
+
+	if err := resp.Header.Write(down); err != nil {
+		Warning.Print(err)
+		return
+	}
+
+	if _, err := down.Write([]byte("\n")); err != nil {
+		Warning.Print(err)
+		return
+	}
+
+	upDone := make(chan struct{})
+	downDone := make(chan struct{})
+	go copyStream(down, io.MultiReader(rem, up), upDone)
+	go copyStream(up, downBuf, downDone)
+	<-upDone
+	<-downDone
+}
+
+func copyStream(dst io.Writer, src io.Reader, done chan struct{}) {
+	defer close(done)
+	if _, err := io.Copy(dst, src); err != nil {
+		Warning.Print(err)
+	}
+	if c, ok := dst.(interface {
+		CloseWrite() error
+	}); ok {
+		if err := c.CloseWrite(); err != nil {
+			Warning.Printf("Error closing connection: %s", err)
+		}
+	}
+}
+
+func doChunkedResponse(w http.ResponseWriter, resp *http.Response, client *httputil.ClientConn) {
+	// Because we can't go back to request/response after we
+	// hijack the connection, we need to close it and make the
+	// client open another.
+	w.Header().Add("Connection", "close")
+	w.WriteHeader(resp.StatusCode)
+
+	down, _, up, rem, err := hijack(w, client)
+	if err != nil {
+		http.Error(w, "Unable to hijack response stream for chunked response", http.StatusInternalServerError)
+		return
+	}
+	defer up.Close()
+	defer down.Close()
+	// Copy the chunked response body to downstream,
+	// stopping at the end of the chunked section.
+	rawResponseBody := io.MultiReader(rem, up)
+	if _, err := io.Copy(ioutil.Discard, httputil.NewChunkedReader(io.TeeReader(rawResponseBody, down))); err != nil {
+		http.Error(w, "Error copying chunked response body", http.StatusInternalServerError)
+		return
+	}
+	resp.Trailer.Write(down)
+	// a chunked response ends with a CRLF
+	down.Write([]byte("\r\n"))
+}
+
+func hijack(w http.ResponseWriter, client *httputil.ClientConn) (down net.Conn, downBuf *bufio.ReadWriter, up net.Conn, rem io.Reader, err error) {
+	hj, ok := w.(http.Hijacker)
+	if !ok {
+		err = errors.New("Unable to cast to Hijack")
+		return
+	}
+	down, downBuf, err = hj.Hijack()
+	if err != nil {
+		return
+	}
+	up, rem = client.Hijack()
+	return
+}

proxy/common.go

@@ -0,0 +1,64 @@
+package proxy
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"io/ioutil"
+	"net/http"
+	"os/exec"
+	"regexp"
+	"strings"
+
+	"github.com/fsouza/go-dockerclient"
+	. "github.com/weaveworks/weave/common"
+)
+
+var (
+	containerIDRegexp   = regexp.MustCompile("^(/v[0-9\\.]*)?/containers/([^/]*)/.*")
+	weaveWaitEntrypoint = []string{"/home/weavewait/weavewait"}
+)
+
+func callWeave(args ...string) ([]byte, error) {
+	args = append([]string{"--local"}, args...)
+	Debug.Print("Calling weave", args)
+	cmd := exec.Command("./weave", args...)
+	cmd.Env = []string{"PROCFS=/hostproc", "PATH=/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"}
+	out, err := cmd.CombinedOutput()
+	return out, err
+}
+
+func weaveCIDRsFromConfig(config *docker.Config) ([]string, bool) {
+	for _, e := range config.Env {
+		if strings.HasPrefix(e, "WEAVE_CIDR=") {
+			return strings.Fields(e[11:]), true
+		}
+	}
+	return nil, false
+}
+
+func marshalRequestBody(r *http.Request, body interface{}) error {
+	newBody, err := json.Marshal(body)
+	if err != nil {
+		return err
+	}
+	r.Body = ioutil.NopCloser(bytes.NewReader(newBody))
+	r.ContentLength = int64(len(newBody))
+	return nil
+}
+
+func inspectContainerInPath(client *docker.Client, path string) (*docker.Container, error) {
+	subs := containerIDRegexp.FindStringSubmatch(path)
+	if subs == nil {
+		err := fmt.Errorf("No container id found in request with path %s", path)
+		Warning.Println(err)
+		return nil, err
+	}
+	containerID := subs[2]
+
+	container, err := client.InspectContainer(containerID)
+	if err != nil {
+		Warning.Printf("Error inspecting container %s: %v", containerID, err)
+	}
+	return container, err
+}

proxy/create_container_interceptor.go

@@ -0,0 +1,108 @@
+package proxy
+
+import (
+	"bytes"
+	"encoding/json"
+	"io/ioutil"
+	"net/http"
+	"strings"
+
+	"github.com/fsouza/go-dockerclient"
+	. "github.com/weaveworks/weave/common"
+)
+
+type createContainerInterceptor struct {
+	client         *docker.Client
+	withDNS        bool
+	dockerBridgeIP string
+	withIPAM       bool
+}
+
+type createContainerRequestBody struct {
+	*docker.Config
+	HostConfig *docker.HostConfig `json:"HostConfig,omitempty" yaml:"HostConfig,omitempty"`
+	MacAddress string             `json:"MacAddress,omitempty" yaml:"MacAddress,omitempty"`
+}
+
+func (i *createContainerInterceptor) InterceptRequest(r *http.Request) error {
+	body, err := ioutil.ReadAll(r.Body)
+	if err != nil {
+		return err
+	}
+	r.Body.Close()
+
+	container := createContainerRequestBody{}
+	if err := json.Unmarshal(body, &container); err != nil {
+		return err
+	}
+
+	if cidrs, ok := weaveCIDRsFromConfig(container.Config); ok || i.withIPAM {
+		Info.Printf("Creating container with WEAVE_CIDR \"%s\"", strings.Join(cidrs, " "))
+		if container.HostConfig == nil {
+			container.HostConfig = &docker.HostConfig{}
+		}
+		container.HostConfig.VolumesFrom = append(container.HostConfig.VolumesFrom, "weaveproxy")
+		if container.Config == nil {
+			container.Config = &docker.Config{}
+		}
+		if err := i.setWeaveWaitEntrypoint(container.Config); err != nil {
+			return err
+		}
+		if err := i.setWeaveDNS(&container); err != nil {
+			return err
+		}
+	}
+
+	newBody, err := json.Marshal(container)
+	if err != nil {
+		return err
+	}
+	r.Body = ioutil.NopCloser(bytes.NewReader(newBody))
+	r.ContentLength = int64(len(newBody))
+
+	return nil
+}
+
+func (i *createContainerInterceptor) setWeaveWaitEntrypoint(container *docker.Config) error {
+	if len(container.Entrypoint) == 0 {
+		image, err := i.client.InspectImage(container.Image)
+		if err != nil {
+			return err
+		}
+
+		if len(container.Cmd) == 0 {
+			container.Cmd = image.Config.Cmd
+		}
+
+		if container.Entrypoint == nil {
+			container.Entrypoint = image.Config.Entrypoint
+		}
+	}
+
+	container.Entrypoint = append(weaveWaitEntrypoint, container.Entrypoint...)
+	return nil
+}
+
+func configHasEntrypoint(c *docker.Config) bool {
+	return c != nil && len(c.Entrypoint) > 0
+}
+func configHasCmd(c *docker.Config) bool {
+	return c != nil && len(c.Cmd) > 0
+}
+
+func (i *createContainerInterceptor) setWeaveDNS(container *createContainerRequestBody) error {
+	if !i.withDNS {
+		return nil
+	}
+
+	container.HostConfig.DNS = append(container.HostConfig.DNS, i.dockerBridgeIP)
+
+	if len(container.HostConfig.DNSSearch) == 0 {
+		container.HostConfig.DNSSearch = []string{"."}
+	}
+	return nil
+}
+
+func (i *createContainerInterceptor) InterceptResponse(r *http.Response) error {
+	return nil
+}

proxy/create_exec_interceptor.go

@@ -0,0 +1,55 @@
+package proxy
+
+import (
+	"encoding/json"
+	"io/ioutil"
+	"net/http"
+	"strings"
+
+	"github.com/fsouza/go-dockerclient"
+	. "github.com/weaveworks/weave/common"
+)
+
+type createExecInterceptor struct {
+	client   *docker.Client
+	withIPAM bool
+}
+
+type createExecRequestBody struct {
+	*docker.Config
+	HostConfig *docker.HostConfig `json:"HostConfig,omitempty" yaml:"HostConfig,omitempty"`
+	MacAddress string             `json:"MacAddress,omitempty" yaml:"MacAddress,omitempty"`
+}
+
+func (i *createExecInterceptor) InterceptRequest(r *http.Request) error {
+	body, err := ioutil.ReadAll(r.Body)
+	if err != nil {
+		return err
+	}
+	r.Body.Close()
+
+	options := docker.CreateExecOptions{}
+	if err := json.Unmarshal(body, &options); err != nil {
+		return err
+	}
+
+	container, err := inspectContainerInPath(i.client, r.URL.Path)
+	if err != nil {
+		return err
+	}
+
+	if cidrs, ok := weaveCIDRsFromConfig(container.Config); ok || i.withIPAM {
+		Info.Printf("Exec in container %s with WEAVE_CIDR \"%s\"", container.ID, strings.Join(cidrs, " "))
+		options.Cmd = append(weaveWaitEntrypoint, options.Cmd...)
+	}
+
+	if err := marshalRequestBody(r, options); err != nil {
+		return err
+	}
+
+	return nil
+}
+
+func (i *createExecInterceptor) InterceptResponse(r *http.Response) error {
+	return nil
+}

proxy/interceptor.go

@@ -0,0 +1,19 @@
+package proxy
+
+import "net/http"
+
+type interceptor interface {
+	InterceptRequest(*http.Request) error
+	InterceptResponse(*http.Response) error
+}
+
+type nullInterceptor struct {
+}
+
+func (i nullInterceptor) InterceptRequest(r *http.Request) error {
+	return nil
+}
+
+func (i nullInterceptor) InterceptResponse(r *http.Response) error {
+	return nil
+}

proxy/proxy.go

@@ -0,0 +1,85 @@
+package proxy
+
+import (
+	"net"
+	"net/http"
+	"net/url"
+	"regexp"
+	"strings"
+
+	"github.com/fsouza/go-dockerclient"
+	. "github.com/weaveworks/weave/common"
+)
+
+var (
+	containerCreateRegexp = regexp.MustCompile("^(/v[0-9\\.]*)?/containers/create$")
+	containerStartRegexp  = regexp.MustCompile("^(/v[0-9\\.]*)?/containers/[^/]*/(re)?start$")
+	execCreateRegexp      = regexp.MustCompile("^(/v[0-9\\.]*)?/containers/[^/]*/exec$")
+)
+
+type Proxy struct {
+	Dial           func() (net.Conn, error)
+	client         *docker.Client
+	withDNS        bool
+	dockerBridgeIP string
+	withIPAM       bool
+}
+
+func NewProxy(targetURL string, withDNS, withIPAM bool) (*Proxy, error) {
+	u, err := url.Parse(targetURL)
+	if err != nil {
+		return nil, err
+	}
+
+	var dockerBridgeIP []byte
+	if withDNS {
+		dockerBridgeIP, err = callWeave("docker-bridge-ip")
+		if err != nil {
+			return nil, err
+		}
+	}
+
+	client, err := docker.NewClient(targetURL)
+	if err != nil {
+		return nil, err
+	}
+
+	targetAddr := ""
+	switch u.Scheme {
+	case "tcp":
+		targetAddr = u.Host
+	case "unix":
+		targetAddr = u.Path
+	}
+
+	return &Proxy{
+		Dial: func() (net.Conn, error) {
+			return net.Dial(u.Scheme, targetAddr)
+		},
+		client:         client,
+		withDNS:        withDNS,
+		dockerBridgeIP: string(dockerBridgeIP),
+		withIPAM:       withIPAM,
+	}, nil
+}
+
+func (proxy *Proxy) ServeHTTP(w http.ResponseWriter, r *http.Request) {
+	Info.Printf("%s %s", r.Method, r.URL)
+	path := r.URL.Path
+	switch {
+	case containerCreateRegexp.MatchString(path):
+		proxy.serveWithInterceptor(&createContainerInterceptor{proxy.client, proxy.withDNS, proxy.dockerBridgeIP, proxy.withIPAM}, w, r)
+	case containerStartRegexp.MatchString(path):
+		proxy.serveWithInterceptor(&startContainerInterceptor{proxy.client, proxy.withDNS, proxy.withIPAM}, w, r)
+	case execCreateRegexp.MatchString(path):
+		proxy.serveWithInterceptor(&createExecInterceptor{proxy.client, proxy.withIPAM}, w, r)
+	case strings.HasPrefix(path, "/weave"):
+		w.WriteHeader(http.StatusOK)
+	default:
+		proxy.serveWithInterceptor(&nullInterceptor{}, w, r)
+	}
+}
+
+func (proxy *Proxy) serveWithInterceptor(i interceptor, w http.ResponseWriter, r *http.Request) {
+	newClient(proxy.Dial, i).ServeHTTP(w, r)
+}

proxy/start_container_interceptor.go

@@ -0,0 +1,43 @@
+package proxy
+
+import (
+	"errors"
+	"net/http"
+	"strings"
+
+	"github.com/fsouza/go-dockerclient"
+	. "github.com/weaveworks/weave/common"
+)
+
+type startContainerInterceptor struct {
+	client   *docker.Client
+	withDNS  bool
+	withIPAM bool
+}
+
+func (i *startContainerInterceptor) InterceptRequest(r *http.Request) error {
+	return nil
+}
+
+func (i *startContainerInterceptor) InterceptResponse(r *http.Response) error {
+	container, err := inspectContainerInPath(i.client, r.Request.URL.Path)
+	if err != nil {
+		return err
+	}
+
+	cidrs, ok := weaveCIDRsFromConfig(container.Config)
+	if !ok && !i.withIPAM {
+		Debug.Print("No Weave CIDR, ignoring")
+		return nil
+	}
+	Info.Printf("Attaching container %s with WEAVE_CIDR \"%s\" to weave network", container.ID, strings.Join(cidrs, " "))
+	args := []string{"attach"}
+	args = append(args, cidrs...)
+	args = append(args, container.ID)
+	if output, err := callWeave(args...); err != nil {
+		i.client.KillContainer(docker.KillContainerOptions{ID: container.ID})
+		Warning.Printf("Attaching container %s to weave network failed: %s", container.ID, string(output))
+		return errors.New(string(output))
+	}
+	return nil
+}

router/connection.go

@@ -3,75 +3,134 @@ package router
 import (
 	"encoding/binary"
 	"encoding/gob"
-	"encoding/hex"
 	"fmt"
 	"log"
 	"net"
-	"strconv"
+	"sync"
 	"time"
 )
 
-func NewRemoteConnection(from, to *Peer, tcpAddr string) *RemoteConnection {
+type Connection interface {
+	Local() *Peer
+	Remote() *Peer
+	RemoteTCPAddr() string
+	Outbound() bool
+	Established() bool
+	BreakTie(Connection) ConnectionTieBreak
+	Shutdown(error)
+	Log(args ...interface{})
+}
+
+type ConnectionTieBreak int
+
+const (
+	TieBreakWon ConnectionTieBreak = iota
+	TieBreakLost
+	TieBreakTied
+)
+
+type RemoteConnection struct {
+	local         *Peer
+	remote        *Peer
+	remoteTCPAddr string
+	outbound      bool
+	established   bool
+}
+
+type LocalConnection struct {
+	sync.RWMutex
+	RemoteConnection
+	TCPConn           *net.TCPConn
+	tcpSender         TCPSender
+	remoteUDPAddr     *net.UDPAddr
+	receivedHeartbeat bool
+	stackFrag         bool
+	effectivePMTU     int
+	SessionKey        *[32]byte
+	heartbeatTCP      *time.Ticker
+	heartbeatTimeout  *time.Timer
+	heartbeatFrame    *ForwardedFrame
+	heartbeat         *time.Ticker
+	fragTest          *time.Ticker
+	forwarder         *Forwarder
+	forwarderDF       *ForwarderDF
+	Decryptor         Decryptor
+	Router            *Router
+	uid               uint64
+	actionChan        chan<- ConnectionAction
+	finished          <-chan struct{} // closed to signal that actorLoop has finished
+}
+
+type ConnectionAction func() error
+
+func NewRemoteConnection(from, to *Peer, tcpAddr string, outbound bool, established bool) *RemoteConnection {
 	return &RemoteConnection{
 		local:         from,
 		remote:        to,
-		remoteTCPAddr: tcpAddr}
+		remoteTCPAddr: tcpAddr,
+		outbound:      outbound,
+		established:   established,
+	}
 }
 
-func (conn *RemoteConnection) Local() *Peer {
-	return conn.local
-}
+func (conn *RemoteConnection) Local() *Peer                           { return conn.local }
+func (conn *RemoteConnection) Remote() *Peer                          { return conn.remote }
+func (conn *RemoteConnection) RemoteTCPAddr() string                  { return conn.remoteTCPAddr }
+func (conn *RemoteConnection) Outbound() bool                         { return conn.outbound }
+func (conn *RemoteConnection) Established() bool                      { return conn.established }
+func (conn *RemoteConnection) BreakTie(Connection) ConnectionTieBreak { return TieBreakTied }
+func (conn *RemoteConnection) Shutdown(error)                         {}
 
-func (conn *RemoteConnection) Remote() *Peer {
-	return conn.remote
-}
-
-func (conn *RemoteConnection) RemoteTCPAddr() string {
-	return conn.remoteTCPAddr
-}
-
-func (conn *RemoteConnection) Shutdown() {
-}
-
-func (conn *RemoteConnection) Established() bool {
-	return true
+func (conn *RemoteConnection) Log(args ...interface{}) {
+	log.Println(append(append([]interface{}{}, fmt.Sprintf("->[%s|%s]:", conn.remoteTCPAddr, conn.remote)), args...)...)
 }
 
 func (conn *RemoteConnection) String() string {
 	from := "<nil>"
 	if conn.local != nil {
-		from = conn.local.Name.String()
+		from = conn.local.String()
 	}
 	to := "<nil>"
 	if conn.remote != nil {
-		to = conn.remote.Name.String()
+		to = conn.remote.String()
 	}
 	return fmt.Sprint("Connection ", from, "->", to)
 }
 
-// Async. Does not return anything. If the connection is successful,
-// it will end up in the local peer's connections map.
-func NewLocalConnection(connRemote *RemoteConnection, acceptNewPeer bool, tcpConn *net.TCPConn, udpAddr *net.UDPAddr, router *Router) {
+func NewLocalConnection(connRemote *RemoteConnection, tcpConn *net.TCPConn, udpAddr *net.UDPAddr, router *Router) *LocalConnection {
 	if connRemote.local != router.Ourself.Peer {
 		log.Fatal("Attempt to create local connection from a peer which is not ourself")
 	}
-
-	queryChan := make(chan *ConnectionInteraction, ChannelSize)
 	// NB, we're taking a copy of connRemote here.
-	connLocal := &LocalConnection{
+	return &LocalConnection{
 		RemoteConnection: *connRemote,
 		Router:           router,
 		TCPConn:          tcpConn,
 		remoteUDPAddr:    udpAddr,
-		effectivePMTU:    DefaultPMTU,
-		queryChan:        queryChan}
-	go connLocal.queryLoop(queryChan, acceptNewPeer)
+		effectivePMTU:    DefaultPMTU}
 }
 
-func (conn *LocalConnection) Established() bool {
-	conn.RLock()
-	defer conn.RUnlock()
-	return conn.established
+// Async. Does not return anything. If the connection is successful,
+// it will end up in the local peer's connections map.
+func (conn *LocalConnection) Start(acceptNewPeer bool) {
+	actionChan := make(chan ConnectionAction, ChannelSize)
+	conn.actionChan = actionChan
+	finished := make(chan struct{})
+	conn.finished = finished
+	go conn.run(actionChan, finished, acceptNewPeer)
+}
+
+func (conn *LocalConnection) BreakTie(dupConn Connection) ConnectionTieBreak {
+	dupConnLocal := dupConn.(*LocalConnection)
+	// conn.uid is used as the tie breaker here, in the knowledge that
+	// both sides will make the same decision.
+	if conn.uid < dupConnLocal.uid {
+		return TieBreakWon
+	} else if dupConnLocal.uid < conn.uid {
+		return TieBreakLost
+	} else {
+		return TieBreakTied
+	}
 }
 
 // Read by the forwarder processes when in the UDP senders
@@ -81,17 +140,10 @@ func (conn *LocalConnection) RemoteUDPAddr() *net.UDPAddr {
 	return conn.remoteUDPAddr
 }
 
-// Called by the forwarder processes in a few places (including
-// crypto), but the connection TCP receiver process, and by the local
-// peer actor process. Do not call this from the connection's actor
-// process itself.
-func (conn *LocalConnection) CheckFatal(err error) error {
-	if err == nil {
-		return nil
-	}
-	conn.log("error:", err)
-	conn.Shutdown()
-	return err
+func (conn *LocalConnection) Established() bool {
+	conn.RLock()
+	defer conn.RUnlock()
+	return conn.established
 }
 
 // Called by forwarder processes, read in Forward (by sniffer and udp
@@ -101,7 +153,7 @@ func (conn *LocalConnection) setEffectivePMTU(pmtu int) {
 	defer conn.Unlock()
 	if conn.effectivePMTU != pmtu {
 		conn.effectivePMTU = pmtu
-		conn.log("Effective PMTU set to", pmtu)
+		conn.Log("Effective PMTU set to", pmtu)
 	}
 }
 
@@ -114,132 +166,91 @@ func (conn *LocalConnection) setStackFrag(frag bool) {
 	conn.stackFrag = frag
 }
 
-func (conn *LocalConnection) log(args ...interface{}) {
-	v := append([]interface{}{}, fmt.Sprintf("->[%s]:", conn.remote.Name))
-	v = append(v, args...)
-	log.Println(v...)
+// Called by the connection's TCP receiver process.
+func (conn *LocalConnection) pmtuVerified(pmtu int) {
+	conn.RLock()
+	fwd := conn.forwarderDF
+	conn.RUnlock()
+	if fwd != nil {
+		fwd.PMTUVerified(pmtu)
+	}
 }
 
-// ACTOR client API
+// Send directly, not via the Actor.  If it goes via the Actor we can
+// get a deadlock where LocalConnection is blocked talking to
+// LocalPeer and LocalPeer is blocked trying send a ProtocolMsg via
+// LocalConnection, and the channels are full in both directions so
+// nothing can proceed.
+func (conn *LocalConnection) SendProtocolMsg(m ProtocolMsg) {
+	if err := conn.sendProtocolMsg(m); err != nil {
+		conn.Shutdown(err)
+	}
+}
 
-const (
-	CSendTCP          = iota
-	CSetEstablished   = iota
-	CSetRemoteUDPAddr = iota
-	CShutdown         = iota
-)
+// ACTOR methods
+
+// NB: The conn.* fields are only written by the connection actor
+// process, which is the caller of the ConnectionAction funs. Hence we
+// do not need locks for reading, and only need write locks for fields
+// read by other processes.
 
 // Async
-func (conn *LocalConnection) Shutdown() {
-	conn.queryChan <- &ConnectionInteraction{Interaction: Interaction{code: CShutdown}}
+func (conn *LocalConnection) Shutdown(err error) {
+	// err should always be a real error, even if only io.EOF
+	if err == nil {
+		panic("nil error")
+	}
+
+	// Run on its own goroutine in case the channel is backed up
+	go func() { conn.sendAction(func() error { return err }) }()
 }
 
 // Async
-func (conn *LocalConnection) SetRemoteUDPAddr(remoteUDPAddr *net.UDPAddr) {
-	if remoteUDPAddr == nil {
+//
+// Heartbeating serves two purposes: a) keeping NAT paths alive, and
+// b) updating a remote peer's knowledge of our address, in the event
+// it changes (e.g. because NAT paths expired).
+func (conn *LocalConnection) ReceivedHeartbeat(remoteUDPAddr *net.UDPAddr, connUID uint64) {
+	if remoteUDPAddr == nil || connUID != conn.uid {
 		return
 	}
-	conn.queryChan <- &ConnectionInteraction{
-		Interaction: Interaction{code: CSetRemoteUDPAddr},
-		payload:     remoteUDPAddr}
+	conn.sendAction(func() error {
+		oldRemoteUDPAddr := conn.remoteUDPAddr
+		old := conn.receivedHeartbeat
+		conn.Lock()
+		conn.remoteUDPAddr = remoteUDPAddr
+		conn.receivedHeartbeat = true
+		conn.Unlock()
+		conn.heartbeatTimeout.Reset(HeartbeatTimeout)
+		if !old {
+			if err := conn.sendSimpleProtocolMsg(ProtocolConnectionEstablished); err != nil {
+				return err
+			}
+		}
+		if oldRemoteUDPAddr == nil {
+			return conn.sendFastHeartbeats()
+		} else if oldRemoteUDPAddr.String() != remoteUDPAddr.String() {
+			log.Println("Peer", conn.remote, "moved from", old, "to", remoteUDPAddr)
+		}
+		return nil
+	})
 }
 
 // Async
 func (conn *LocalConnection) SetEstablished() {
-	conn.queryChan <- &ConnectionInteraction{Interaction: Interaction{code: CSetEstablished}}
-}
-
-// Async
-func (conn *LocalConnection) SendTCP(msg []byte) {
-	conn.queryChan <- &ConnectionInteraction{
-		Interaction: Interaction{code: CSendTCP},
-		payload:     msg}
-}
-
-// ACTOR server
-
-func (conn *LocalConnection) queryLoop(queryChan <-chan *ConnectionInteraction, acceptNewPeer bool) {
-	err := conn.handshake(acceptNewPeer)
-	if err != nil {
-		log.Printf("->[%s] encountered error during handshake: %v\n", conn.remoteTCPAddr, err)
-		conn.handleShutdown()
-		return
-	}
-	conn.Router.Ourself.AddConnection(conn)
-	if conn.remoteUDPAddr != nil {
-		if err = conn.ensureForwarders(); err == nil {
-			conn.heartbeat = time.NewTicker(FastHeartbeat)
-			conn.forwardHeartbeatFrame() // avoid initial wait
+	conn.sendAction(func() error {
+		stopTicker(conn.heartbeat)
+		old := conn.established
+		conn.Lock()
+		conn.established = true
+		conn.Unlock()
+		if old {
+			return nil
 		}
-	}
-	terminate := false
-	for !terminate {
-		if err != nil {
-			conn.log("error:", err)
-			break
-		}
-		select {
-		case query, ok := <-queryChan:
-			if !ok {
-				break
-			}
-			switch query.code {
-			case CShutdown:
-				terminate = true
-			case CSetEstablished:
-				err = conn.handleSetEstablished()
-			case CSetRemoteUDPAddr:
-				err = conn.handleSetRemoteUDPAddr(query.payload.(*net.UDPAddr))
-			case CSendTCP:
-				err = conn.handleSendTCP(query.payload.([]byte))
-			}
-		case <-tickerChan(conn.heartbeat):
-			conn.forwardHeartbeatFrame()
-		case <-tickerChan(conn.fetchAll):
-			err = conn.handleSendTCP(ProtocolFetchAllByte)
-		case <-tickerChan(conn.fragTest):
-			conn.setStackFrag(false)
-			err = conn.handleSendTCP(ProtocolStartFragmentationTestByte)
-
-		}
-	}
-	conn.handleShutdown()
-}
-
-func (conn *LocalConnection) handleSetRemoteUDPAddr(remoteUDPAddr *net.UDPAddr) error {
-	if remoteUDPAddr == nil {
-		return nil
-	}
-	conn.Lock()
-	old := conn.remoteUDPAddr
-	conn.remoteUDPAddr = remoteUDPAddr
-	conn.Unlock()
-	if old == nil {
-		if err := conn.handleSendTCP(ProtocolConnectionEstablishedByte); err != nil {
-			return err
-		}
-		return conn.handleSetEstablished()
-	} else if old.String() != remoteUDPAddr.String() {
-		log.Println("Peer", conn.remote.Name, "moved from", old, "to", remoteUDPAddr)
-	}
-	return nil
-}
-
-func (conn *LocalConnection) handleSetEstablished() error {
-	conn.Lock()
-	old := conn.established
-	conn.established = true
-	conn.Unlock()
-	if !old {
 		conn.Router.Ourself.ConnectionEstablished(conn)
 		if err := conn.ensureForwarders(); err != nil {
 			return err
 		}
-		stopTicker(conn.heartbeat)
-		conn.heartbeat = time.NewTicker(SlowHeartbeat)
-		conn.fetchAll = time.NewTicker(FetchAllInterval)
-		conn.fragTest = time.NewTicker(FragTestInterval)
-		conn.forwardHeartbeatFrame() // avoid initial wait
 		// Send a large frame down the DF channel in order to prompt
 		// PMTU discovery to start.
 		conn.Forward(true, &ForwardedFrame{
@@ -247,196 +258,187 @@ func (conn *LocalConnection) handleSetEstablished() error {
 			dstPeer: conn.remote,
 			frame:   PMTUDiscovery},
 			nil)
+		conn.heartbeat = time.NewTicker(SlowHeartbeat)
+		conn.fragTest = time.NewTicker(FragTestInterval)
+		// avoid initial waits for timers to fire
+		conn.Forward(true, conn.heartbeatFrame, nil)
 		conn.setStackFrag(false)
-		return conn.handleSendTCP(ProtocolStartFragmentationTestByte)
-	}
-	return nil
+		if err := conn.sendSimpleProtocolMsg(ProtocolStartFragmentationTest); err != nil {
+			return err
+		}
+		return nil
+	})
 }
 
-func (conn *LocalConnection) handleSendTCP(msg []byte) error {
-	return conn.tcpSender.Send(msg)
+// Send an actor request to the actorLoop, but don't block if
+// actorLoop has exited - see http://blog.golang.org/pipelines for
+// pattern
+func (conn *LocalConnection) sendAction(action ConnectionAction) {
+	select {
+	case conn.actionChan <- action:
+	case <-conn.finished:
+	}
 }
 
-func (conn *LocalConnection) handleShutdown() {
-	if conn.remote != nil {
-		conn.log("connection shutting down")
-	}
+// ACTOR server
 
-	// Whilst some of these elements may have been written to whilst
-	// holding locks, they were only written to by the connection
-	// actor process. handleShutdown is only called by the connection
-	// actor process. So there is no need to take locks to read these
-	// (or write elements which are only read by the same
-	// process). Taking locks is only done for elements which are read
-	// by other processes.
-	if conn.shutdown {
+func (conn *LocalConnection) run(actionChan <-chan ConnectionAction, finished chan<- struct{}, acceptNewPeer bool) {
+	var err error // important to use this var and not create another one with 'err :='
+	defer func() { conn.shutdown(err) }()
+	defer close(finished)
+
+	tcpConn := conn.TCPConn
+	tcpConn.SetLinger(0)
+	enc := gob.NewEncoder(tcpConn)
+	dec := gob.NewDecoder(tcpConn)
+
+	if err = conn.handshake(enc, dec, acceptNewPeer); err != nil {
 		return
 	}
-	conn.shutdown = true
+	conn.Log("completed handshake")
+
+	// The ordering of the following is very important. [1]
+
+	if conn.remoteUDPAddr != nil {
+		if err = conn.ensureForwarders(); err != nil {
+			return
+		}
+	}
+	if err = conn.Router.Ourself.AddConnection(conn); err != nil {
+		return
+	}
+	if err = conn.initHeartbeats(); err != nil {
+		return
+	}
+	go conn.receiveTCP(dec)
+	err = conn.actorLoop(actionChan)
+}
+
+// [1] Ordering constraints:
+//
+// (a) AddConnections must precede initHeartbeats. It is only after
+// the former completes that we know the connection is valid, in
+// particular is not a duplicate connection to the same peer. Sending
+// heartbeats on a duplicate connection can trip up crypto at the
+// other end, since the associated UDP packets may get decoded by the
+// other connection. It is also generally wasteful to engage in any
+// interaction with the remote on a connection that turns out to be
+// invald.
+//
+// (b) AddConnection must precede receiveTCP. In the absence of any
+// indirect connectivity to the remote peer, the first we hear about
+// it (and any peers reachable from it) is through topology gossip it
+// sends us on the connection. We must ensure that the connection has
+// been added to Ourself prior to processing any such gossip,
+// otherwise we risk immediately gc'ing part of that newly received
+// portion of the topology (though not the remote peer itself, since
+// that will have a positive ref count), leaving behind dangling
+// references to peers. Hence we must invoke AddConnection, which is
+// *synchronous*, first.
+//
+// (c) AddConnection must precede actorLoop. More precisely, it must
+// precede shutdown, since that invokes DeleteConnection and is
+// invoked on termination of this entire function. Essentially this
+// boils down to a prohibition on running AddConnection in a separate
+// goroutine, at least not without some synchronisation. Which in turn
+// requires us the launching of the receiveTCP goroutine to precede
+// actorLoop.
+//
+// (d) AddConnection should precede receiveTCP. There is no point
+// starting the latter if the former fails.
+//
+// (e) initHeartbeats should precede actorLoop. The former is setting
+// LocalConnection fields accessed by the latter. Since the latter
+// runs in a separate goroutine, we'd have to add some synchronisation
+// if initHeartbeats isn't run first.
+//
+// (f) ensureForwarders should precede AddConnection. As soon as a
+// connection has been added to LocalPeer by the latter, it becomes
+// visible to the packet routing logic, which will end up dropping
+// packets if the forwarders haven't been created yet. We cannot
+// prevent that completely, since, for example, forwarder can only be
+// created when we know the remote UDP address, but it helps to try.
+
+func (conn *LocalConnection) initHeartbeats() error {
+	conn.heartbeatTCP = time.NewTicker(TCPHeartbeat)
+	conn.heartbeatTimeout = time.NewTimer(HeartbeatTimeout)
+	heartbeatFrameBytes := make([]byte, EthernetOverhead+8)
+	binary.BigEndian.PutUint64(heartbeatFrameBytes[EthernetOverhead:], conn.uid)
+	conn.heartbeatFrame = &ForwardedFrame{
+		srcPeer: conn.local,
+		dstPeer: conn.remote,
+		frame:   heartbeatFrameBytes}
+	if conn.remoteUDPAddr == nil {
+		return nil
+	}
+	return conn.sendFastHeartbeats()
+}
+
+func (conn *LocalConnection) actorLoop(actionChan <-chan ConnectionAction) (err error) {
+	for err == nil {
+		select {
+		case action := <-actionChan:
+			err = action()
+		case <-conn.heartbeatTCP.C:
+			err = conn.sendSimpleProtocolMsg(ProtocolHeartbeat)
+		case <-conn.heartbeatTimeout.C:
+			err = fmt.Errorf("timed out waiting for UDP heartbeat")
+		case <-tickerChan(conn.heartbeat):
+			conn.Forward(true, conn.heartbeatFrame, nil)
+		case <-tickerChan(conn.fragTest):
+			conn.setStackFrag(false)
+			err = conn.sendSimpleProtocolMsg(ProtocolStartFragmentationTest)
+		}
+	}
+	return
+}
+
+func (conn *LocalConnection) shutdown(err error) {
+	if conn.remote == nil {
+		log.Printf("->[%s] connection shutting down due to error during handshake: %v\n", conn.remoteTCPAddr, err)
+	} else {
+		conn.Log("connection shutting down due to error:", err)
+	}
 
 	if conn.TCPConn != nil {
 		checkWarn(conn.TCPConn.Close())
 	}
 
 	if conn.remote != nil {
-		conn.remote.DecrementLocalRefCount()
+		conn.Router.Peers.Dereference(conn.remote)
 		conn.Router.Ourself.DeleteConnection(conn)
 	}
 
+	if conn.heartbeatTimeout != nil {
+		conn.heartbeatTimeout.Stop()
+	}
+
+	stopTicker(conn.heartbeatTCP)
 	stopTicker(conn.heartbeat)
-	stopTicker(conn.fetchAll)
 	stopTicker(conn.fragTest)
 
 	// blank out the forwardChan so that the router processes don't
 	// try to send any more
 	conn.stopForwarders()
 
-	conn.Router.ConnectionMaker.ConnectionTerminated(conn.remoteTCPAddr)
+	conn.Router.ConnectionMaker.ConnectionTerminated(conn.remoteTCPAddr, err)
 }
 
-func (conn *LocalConnection) handshake(acceptNewPeer bool) error {
-	// We do not need to worry about locking in here as at this point,
-	// the connection is not reachable by any go-routine other than
-	// ourself. Only when we add this connection to the conn.local
-	// peer will it be visible from multiple go-routines.
-	tcpConn := conn.TCPConn
-	tcpConn.SetLinger(0)
-	conn.extendReadDeadline()
+// Helpers
 
-	enc := gob.NewEncoder(tcpConn)
-
-	localConnID := randUint64()
-
-	versionStr := fmt.Sprint(ProtocolVersion)
-	handshakeSend := map[string]string{
-		"Protocol":        Protocol,
-		"ProtocolVersion": versionStr,
-		"PeerNameFlavour": PeerNameFlavour,
-		"Name":            conn.local.Name.String(),
-		"UID":             fmt.Sprint(conn.local.UID),
-		"ConnID":          fmt.Sprint(localConnID)}
-	handshakeRecv := map[string]string{}
-
-	usingPassword := conn.Router.UsingPassword()
-	var public, private *[32]byte
-	var err error
-	if usingPassword {
-		public, private, err = GenerateKeyPair()
-		if err != nil {
-			return err
-		}
-		handshakeSend["PublicKey"] = hex.EncodeToString(public[:])
-	}
-	enc.Encode(handshakeSend)
-
-	dec := gob.NewDecoder(tcpConn)
-	err = dec.Decode(&handshakeRecv)
-	if err != nil {
-		return err
-	}
-	_, err = checkHandshakeStringField("Protocol", Protocol, handshakeRecv)
-	if err != nil {
-		return err
-	}
-	_, err = checkHandshakeStringField("ProtocolVersion", versionStr, handshakeRecv)
-	if err != nil {
-		return err
-	}
-	_, err = checkHandshakeStringField("PeerNameFlavour", PeerNameFlavour, handshakeRecv)
-	if err != nil {
-		return err
-	}
-	nameStr, err := checkHandshakeStringField("Name", "", handshakeRecv)
-	if err != nil {
-		return err
-	}
-	name, err := PeerNameFromString(nameStr)
-	if err != nil {
-		return err
-	}
-	if !acceptNewPeer {
-		if _, found := conn.Router.Peers.Fetch(name); !found {
-			return fmt.Errorf("Found unknown remote name: %s at %s", name, conn.remoteTCPAddr)
-		}
-	}
-	if existingConn, found := conn.local.ConnectionTo(name); found && existingConn.Established() {
-		return fmt.Errorf("Already have connection to %s at %s", name, existingConn.RemoteTCPAddr())
-	}
-
-	uidStr, err := checkHandshakeStringField("UID", "", handshakeRecv)
-	if err != nil {
-		return err
-	}
-	uid, err := strconv.ParseUint(uidStr, 10, 64)
-	if err != nil {
-		return err
-	}
-
-	remoteConnIdStr, err := checkHandshakeStringField("ConnID", "", handshakeRecv)
-	if err != nil {
-		return err
-	}
-	remoteConnID, err := strconv.ParseUint(remoteConnIdStr, 10, 64)
-	if err != nil {
-		return err
-	}
-	conn.UID = localConnID ^ remoteConnID
-
-	if usingPassword {
-		remotePublicStr, rpErr := checkHandshakeStringField("PublicKey", "", handshakeRecv)
-		if rpErr != nil {
-			return rpErr
-		}
-		remotePublicSlice, rpErr := hex.DecodeString(remotePublicStr)
-		if rpErr != nil {
-			return rpErr
-		}
-		remotePublic := [32]byte{}
-		for idx, elem := range remotePublicSlice {
-			remotePublic[idx] = elem
-		}
-		conn.SessionKey = FormSessionKey(&remotePublic, private, conn.Router.Password)
-		conn.tcpSender = NewEncryptedTCPSender(enc, conn)
-		conn.Decryptor = NewNaClDecryptor(conn)
-	} else {
-		if _, found := handshakeRecv["PublicKey"]; found {
-			return fmt.Errorf("Remote network is encrypted. Password required.")
-		}
-		conn.tcpSender = NewSimpleTCPSender(enc)
-		conn.Decryptor = NewNonDecryptor(conn)
-	}
-
-	toPeer := NewPeer(name, uid, 0)
-	toPeer = conn.Router.Peers.FetchWithDefault(toPeer)
-	if toPeer == nil {
-		return fmt.Errorf("Connection appears to be with different version of a peer we already know of")
-	} else if toPeer == conn.local {
-		// have to do assigment here to ensure Shutdown releases ref count
-		conn.remote = toPeer
-		return fmt.Errorf("Cannot connect to ourself")
-	}
-	conn.remote = toPeer
-
-	go conn.receiveTCP(dec, usingPassword)
-	return nil
+func (conn *LocalConnection) sendSimpleProtocolMsg(tag ProtocolTag) error {
+	return conn.sendProtocolMsg(ProtocolMsg{tag: tag})
 }
 
-func checkHandshakeStringField(fieldName string, expectedValue string, handshake map[string]string) (string, error) {
-	val, found := handshake[fieldName]
-	if !found {
-		return "", fmt.Errorf("Field %s is missing", fieldName)
-	}
-	if expectedValue != "" && val != expectedValue {
-		return "", fmt.Errorf("Field %s has wrong value; expected '%s', received '%s'", fieldName, expectedValue, val)
-	}
-	return val, nil
+func (conn *LocalConnection) sendProtocolMsg(m ProtocolMsg) error {
+	return conn.tcpSender.Send(Concat([]byte{byte(m.tag)}, m.msg))
 }
 
-func (conn *LocalConnection) receiveTCP(decoder *gob.Decoder, usingPassword bool) {
-	defer conn.Decryptor.Shutdown()
+func (conn *LocalConnection) receiveTCP(decoder *gob.Decoder) {
+	usingPassword := conn.SessionKey != nil
 	var receiver TCPReceiver
 	if usingPassword {
-		receiver = NewEncryptedTCPReceiver(conn)
+		receiver = NewEncryptedTCPReceiver(conn.SessionKey, conn.outbound)
 	} else {
 		receiver = NewSimpleTCPReceiver()
 	}
@@ -444,94 +446,62 @@ func (conn *LocalConnection) receiveTCP(decoder *gob.Decoder, usingPassword bool
 	for {
 		var msg []byte
 		conn.extendReadDeadline()
-		if conn.CheckFatal(decoder.Decode(&msg)) != nil {
-			return
+		if err = decoder.Decode(&msg); err != nil {
+			break
 		}
 		msg, err = receiver.Decode(msg)
 		if err != nil {
-			checkWarn(err)
-			// remote peer may be using wrong password. Or there could
-			// be some sort of injection attack going on. Just ignore
-			// the traffic rather than shutting down.
+			break
+		}
+		if len(msg) < 1 {
+			conn.Log("ignoring blank msg")
 			continue
 		}
-		if msg[0] == ProtocolConnectionEstablished {
-			// We initiated the connection. We sent fast heartbeats to
-			// the remote side, which has now received at least one of
-			// them and thus has informed us via TCP that it considers
-			// the connection is now up. We now do a fetchAll on it.
-			conn.SetEstablished()
-			conn.SendTCP(ProtocolFetchAllByte)
-		} else if msg[0] == ProtocolStartFragmentationTest {
-			conn.Forward(false, &ForwardedFrame{
-				srcPeer: conn.local,
-				dstPeer: conn.remote,
-				frame:   FragTest},
-				nil)
-		} else if msg[0] == ProtocolFragmentationReceived {
-			conn.setStackFrag(true)
-		} else if usingPassword && msg[0] == ProtocolNonce {
-			conn.Decryptor.ReceiveNonce(msg[1:])
-		} else if msg[0] == ProtocolFetchAll {
-			// There are exactly two messages that relate to topology
-			// updates.
-			//
-			// 1. FetchAll. This carries no payload. The receiver
-			// responds with the entire topology model as the receiver
-			// has it.
-			//
-			// 2. Update. This carries a topology payload. The
-			// receiver merges it with its own topology model. If the
-			// payload is a subset of the receiver's topology, no
-			// further action is taken. Otherwise, the receiver sends
-			// out to all its connections an "improved" update:
-			//  - elements which the original payload added to the
-			//    receiver are included
-			//  - elements which the original payload updated in the
-			//    receiver are included
-			//  - elements which are equal between the receiver and
-			//    the payload are not included
-			//  - elements where the payload was older than the
-			//    receiver's version are updated
-			conn.SendTCP(Concat(ProtocolUpdateByte, conn.Router.Peers.EncodeAllPeers()))
-		} else if msg[0] == ProtocolUpdate {
-			newUpdate, err := conn.Router.Peers.ApplyUpdate(msg[1:])
-			if _, ok := err.(UnknownPeersError); err != nil && ok {
-				// That update contained a peer we didn't know about;
-				// request full update
-				conn.SendTCP(ProtocolFetchAllByte)
-				continue
-			}
-			if conn.CheckFatal(err) != nil {
-				return
-			}
-			if len(newUpdate) != 0 {
-				conn.Router.ConnectionMaker.Refresh()
-				conn.Router.Routes.Recalculate()
-				conn.Router.Ourself.BroadcastTCP(Concat(ProtocolUpdateByte, newUpdate))
-			}
-		} else if msg[0] == ProtocolPMTUVerified {
-			conn.verifyPMTU <- int(binary.BigEndian.Uint16(msg[1:]))
-		} else {
-			conn.log("received unknown msg:\n", msg)
+		if err = conn.handleProtocolMsg(ProtocolTag(msg[0]), msg[1:]); err != nil {
+			break
 		}
 	}
+	conn.Shutdown(err)
+}
+
+func (conn *LocalConnection) handleProtocolMsg(tag ProtocolTag, payload []byte) error {
+	switch tag {
+	case ProtocolHeartbeat:
+	case ProtocolConnectionEstablished:
+		// We sent fast heartbeats to the remote peer, which has now
+		// received at least one of them and told us via this message.
+		// We can now consider the connection as established from our
+		// end.
+		conn.SetEstablished()
+	case ProtocolStartFragmentationTest:
+		conn.Forward(false, &ForwardedFrame{
+			srcPeer: conn.local,
+			dstPeer: conn.remote,
+			frame:   FragTest},
+			nil)
+	case ProtocolFragmentationReceived:
+		conn.setStackFrag(true)
+	case ProtocolPMTUVerified:
+		conn.pmtuVerified(int(binary.BigEndian.Uint16(payload)))
+	case ProtocolGossipUnicast, ProtocolGossipBroadcast, ProtocolGossip:
+		return conn.Router.handleGossip(tag, payload)
+	default:
+		conn.Log("ignoring unknown protocol tag:", tag)
+	}
+	return nil
 }
 
 func (conn *LocalConnection) extendReadDeadline() {
-	conn.TCPConn.SetReadDeadline(time.Now().Add(ReadTimeout))
+	conn.TCPConn.SetReadDeadline(time.Now().Add(TCPHeartbeat * 2))
 }
 
-// Heartbeating serves two purposes: a) keeping NAT paths alive, and
-// b) updating a remote peer's knowledge of our address, in the event
-// it changes (e.g. because NAT paths expired).
-// Called only by connection actor process.
-func (conn *LocalConnection) forwardHeartbeatFrame() {
-	heartbeatFrame := &ForwardedFrame{
-		srcPeer: conn.local,
-		dstPeer: conn.remote,
-		frame:   []byte{}}
-	conn.Forward(true, heartbeatFrame, nil)
+func (conn *LocalConnection) sendFastHeartbeats() error {
+	err := conn.ensureForwarders()
+	if err == nil {
+		conn.heartbeat = time.NewTicker(FastHeartbeat)
+		conn.Forward(true, conn.heartbeatFrame, nil) // avoid initial wait
+	}
+	return err
 }
 
 func tickerChan(ticker *time.Ticker) <-chan time.Time {

router/connection_maker.go

@@ -14,96 +14,118 @@ const (
 	MaxInterval     = 10 * time.Minute
 )
 
-const (
-	CMInitiate   = iota
-	CMTerminated = iota
-	CMRefresh    = iota
-	CMStatus     = iota
-)
-
 type ConnectionMaker struct {
-	ourself        *LocalPeer
-	peers          *Peers
-	targets        map[string]*Target
-	cmdLineAddress map[string]bool
-	queryChan      chan<- *ConnectionMakerInteraction
+	ourself      *LocalPeer
+	peers        *Peers
+	port         int
+	targets      map[string]*Target
+	cmdLinePeers map[string]*net.TCPAddr
+	actionChan   chan<- ConnectionMakerAction
 }
 
 // Information about an address where we may find a peer
 type Target struct {
 	attempting  bool          // are we currently attempting to connect there?
+	lastError   error         // reason for disconnection last time
 	tryAfter    time.Time     // next time to try this address
 	tryInterval time.Duration // backoff time on next failure
 }
 
-type ConnectionMakerInteraction struct {
-	Interaction
-	address string
+type ConnectionMakerAction func() bool
+
+func NewConnectionMaker(ourself *LocalPeer, peers *Peers, port int) *ConnectionMaker {
+	return &ConnectionMaker{
+		ourself:      ourself,
+		peers:        peers,
+		port:         port,
+		cmdLinePeers: make(map[string]*net.TCPAddr),
+		targets:      make(map[string]*Target)}
 }
 
-func StartConnectionMaker(ourself *LocalPeer, peers *Peers) *ConnectionMaker {
-	queryChan := make(chan *ConnectionMakerInteraction, ChannelSize)
-	state := &ConnectionMaker{
-		ourself:        ourself,
-		peers:          peers,
-		queryChan:      queryChan,
-		cmdLineAddress: make(map[string]bool),
-		targets:        make(map[string]*Target)}
-	go state.queryLoop(queryChan)
-	return state
+func (cm *ConnectionMaker) Start() {
+	actionChan := make(chan ConnectionMakerAction, ChannelSize)
+	cm.actionChan = actionChan
+	go cm.queryLoop(actionChan)
 }
 
-func (cm *ConnectionMaker) InitiateConnection(address string) {
-	cm.queryChan <- &ConnectionMakerInteraction{
-		Interaction: Interaction{code: CMInitiate},
-		address:     address}
+func (cm *ConnectionMaker) InitiateConnection(peer string) error {
+	host, port, err := net.SplitHostPort(peer)
+	if err != nil {
+		host = peer
+		port = "0" // we use that as an indication that "no port was supplied"
+	}
+	addr, err := net.ResolveTCPAddr("tcp4", fmt.Sprintf("%s:%s", host, port))
+	if err != nil {
+		return err
+	}
+	cm.actionChan <- func() bool {
+		cm.cmdLinePeers[peer] = addr
+		// curtail any existing reconnect interval
+		if target, found := cm.targets[addr.String()]; found {
+			target.tryAfter, target.tryInterval = tryImmediately()
+		}
+		return true
+	}
+	return nil
 }
 
-func (cm *ConnectionMaker) ConnectionTerminated(address string) {
-	cm.queryChan <- &ConnectionMakerInteraction{
-		Interaction: Interaction{code: CMTerminated},
-		address:     address}
+func (cm *ConnectionMaker) ForgetConnection(peer string) {
+	cm.actionChan <- func() bool {
+		delete(cm.cmdLinePeers, peer)
+		return false
+	}
+}
+
+func (cm *ConnectionMaker) ConnectionTerminated(address string, err error) {
+	cm.actionChan <- func() bool {
+		if target, found := cm.targets[address]; found {
+			target.attempting = false
+			target.lastError = err
+			target.tryAfter, target.tryInterval = tryAfter(target.tryInterval)
+		}
+		return true
+	}
 }
 
 func (cm *ConnectionMaker) Refresh() {
-	cm.queryChan <- &ConnectionMakerInteraction{
-		Interaction: Interaction{code: CMRefresh}}
+	cm.actionChan <- func() bool { return true }
 }
 
 func (cm *ConnectionMaker) String() string {
-	resultChan := make(chan interface{}, 0)
-	cm.queryChan <- &ConnectionMakerInteraction{
-		Interaction: Interaction{code: CMStatus, resultChan: resultChan}}
-	result := <-resultChan
-	return result.(string)
+	// We need to Refresh first in order to clear out any 'attempting'
+	// connections from cm.targets that have been established since
+	// the last run of cm.checkStateAndAttemptConnections. These
+	// entries are harmless but do represent stale state that we do
+	// not want to report.
+	cm.Refresh()
+	resultChan := make(chan string, 0)
+	cm.actionChan <- func() bool {
+		var buf bytes.Buffer
+		for address, target := range cm.targets {
+			fmt.Fprintf(&buf, "->[%s]", address)
+			if target.lastError != nil {
+				fmt.Fprintf(&buf, " (%s)", target.lastError)
+			}
+			if target.attempting {
+				fmt.Fprintf(&buf, " trying since %v\n", target.tryAfter)
+			} else {
+				fmt.Fprintf(&buf, " next try at %v\n", target.tryAfter)
+			}
+		}
+		resultChan <- buf.String()
+		return false
+	}
+	return <-resultChan
 }
 
-func (cm *ConnectionMaker) queryLoop(queryChan <-chan *ConnectionMakerInteraction) {
+func (cm *ConnectionMaker) queryLoop(actionChan <-chan ConnectionMakerAction) {
 	timer := time.NewTimer(MaxDuration)
 	run := func() { timer.Reset(cm.checkStateAndAttemptConnections()) }
 	for {
 		select {
-		case query, ok := <-queryChan:
-			if !ok {
-				return
-			}
-			switch query.code {
-			case CMInitiate:
-				cm.cmdLineAddress[NormalisePeerAddr(query.address)] = true
+		case action := <-actionChan:
+			if action() {
 				run()
-			case CMTerminated:
-				if target, found := cm.targets[query.address]; found {
-					target.attempting = false
-					target.tryAfter, target.tryInterval = tryAfter(target.tryInterval)
-				}
-				run()
-			case CMRefresh:
-				run()
-			case CMStatus:
-				run()
-				query.resultChan <- cm.status()
-			default:
-				log.Fatal("Unexpected connection maker query:", query)
 			}
 		case <-timer.C:
 			run()
@@ -112,62 +134,121 @@ func (cm *ConnectionMaker) queryLoop(queryChan <-chan *ConnectionMakerInteractio
 }
 
 func (cm *ConnectionMaker) checkStateAndAttemptConnections() time.Duration {
-	validTarget := make(map[string]bool)
-
-	// copy the set of things we are connected to, so we can access them without locking
-	our_connected_peers := make(map[PeerName]bool)
-	our_connected_targets := make(map[string]bool)
-	cm.ourself.ForEachConnection(func(peer PeerName, conn Connection) {
-		our_connected_peers[peer] = true
-		our_connected_targets[conn.RemoteTCPAddr()] = true
-	})
+	var (
+		validTarget   = make(map[string]struct{})
+		cmdLineTarget = make(map[string]struct{})
+	)
+	// Copy the set of things we are connected to, so we can access
+	// them without locking.  Also clear out any entries in cm.targets
+	// for existing connections.
+	ourConnectedPeers, ourConnectedTargets, ourInboundIPs := cm.ourConnections()
 
 	addTarget := func(address string) {
-		if !our_connected_targets[address] {
-			validTarget[address] = true
-			cm.addTarget(address)
+		if _, connected := ourConnectedTargets[address]; connected {
+			return
 		}
+		validTarget[address] = void
+		if _, found := cm.targets[address]; found {
+			return
+		}
+		target := &Target{}
+		target.tryAfter, target.tryInterval = tryImmediately()
+		cm.targets[address] = target
 	}
 
 	// Add command-line targets that are not connected
-	for address, _ := range cm.cmdLineAddress {
-		addTarget(address)
+	for _, addr := range cm.cmdLinePeers {
+		completeAddr := *addr
+		attempt := true
+		if completeAddr.Port == 0 {
+			completeAddr.Port = cm.port
+			// If a peer was specified w/o a port, then we do not
+			// attempt to connect to it if we have any inbound
+			// connections from that IP.
+			if _, connected := ourInboundIPs[completeAddr.IP.String()]; connected {
+				attempt = false
+			}
+		}
+		address := completeAddr.String()
+		cmdLineTarget[address] = void
+		if attempt {
+			addTarget(address)
+		}
 	}
 
 	// Add targets for peers that someone else is connected to, but we
 	// aren't
-	cm.peers.ForEach(func(name PeerName, peer *Peer) {
-		peer.ForEachConnection(func(otherPeer PeerName, conn Connection) {
-			if otherPeer == cm.ourself.Name || our_connected_peers[otherPeer] {
-				return
+	cm.addPeerTargets(ourConnectedPeers, addTarget)
+
+	return cm.connectToTargets(validTarget, cmdLineTarget)
+}
+
+func (cm *ConnectionMaker) ourConnections() (PeerNameSet, map[string]struct{}, map[string]struct{}) {
+	var (
+		ourConnectedPeers   = make(PeerNameSet)
+		ourConnectedTargets = make(map[string]struct{})
+		ourInboundIPs       = make(map[string]struct{})
+	)
+	for conn := range cm.ourself.Connections() {
+		address := conn.RemoteTCPAddr()
+		delete(cm.targets, address)
+		ourConnectedPeers[conn.Remote().Name] = void
+		ourConnectedTargets[address] = void
+		if conn.Outbound() {
+			continue
+		}
+		if ip, _, err := net.SplitHostPort(address); err == nil { // should always succeed
+			ourInboundIPs[ip] = void
+		}
+	}
+	return ourConnectedPeers, ourConnectedTargets, ourInboundIPs
+}
+
+func (cm *ConnectionMaker) addPeerTargets(ourConnectedPeers PeerNameSet, addTarget func(string)) {
+	cm.peers.ForEach(func(peer *Peer) {
+		if peer == cm.ourself.Peer {
+			return
+		}
+		// Modifying peer.connections requires a write lock on Peers,
+		// and since we are holding a read lock (due to the ForEach),
+		// access without locking the peer is safe.
+		for otherPeer, conn := range peer.connections {
+			if otherPeer == cm.ourself.Name {
+				continue
+			}
+			if _, connected := ourConnectedPeers[otherPeer]; connected {
+				continue
 			}
 			address := conn.RemoteTCPAddr()
-			// try both portnumber of connection and standard port
-			addTarget(address)
-			if host, _, err := net.SplitHostPort(address); err == nil {
-				addTarget(NormalisePeerAddr(host))
+			if conn.Outbound() {
+				addTarget(address)
+			} else if ip, _, err := net.SplitHostPort(address); err == nil {
+				// There is no point connecting to the (likely
+				// ephemeral) remote port of an inbound connection
+				// that some peer has. Let's try to connect to on the
+				// weave port instead.
+				addTarget(fmt.Sprintf("%s:%d", ip, cm.port))
 			}
-		})
+		}
 	})
+}
 
+func (cm *ConnectionMaker) connectToTargets(validTarget map[string]struct{}, cmdLineTarget map[string]struct{}) time.Duration {
 	now := time.Now() // make sure we catch items just added
 	after := MaxDuration
 	for address, target := range cm.targets {
-		if our_connected_targets[address] {
-			delete(cm.targets, address)
-			continue
-		}
 		if target.attempting {
 			continue
 		}
-		if !validTarget[address] {
+		if _, valid := validTarget[address]; !valid {
 			delete(cm.targets, address)
 			continue
 		}
 		switch duration := target.tryAfter.Sub(now); {
 		case duration <= 0:
 			target.attempting = true
-			go cm.attemptConnection(address, cm.cmdLineAddress[address])
+			_, isCmdLineTarget := cmdLineTarget[address]
+			go cm.attemptConnection(address, isCmdLineTarget)
 		case duration < after:
 			after = duration
 		}
@@ -175,33 +256,11 @@ func (cm *ConnectionMaker) checkStateAndAttemptConnections() time.Duration {
 	return after
 }
 
-func (cm *ConnectionMaker) addTarget(address string) {
-	if _, found := cm.targets[address]; !found {
-		target := &Target{}
-		target.tryAfter, target.tryInterval = tryImmediately()
-		cm.targets[address] = target
-	}
-}
-
-func (cm *ConnectionMaker) status() string {
-	var buf bytes.Buffer
-	for address, target := range cm.targets {
-		var fmtStr string
-		if target.attempting {
-			fmtStr = "%s (trying since %v)\n"
-		} else {
-			fmtStr = "%s (next try at %v)\n"
-		}
-		buf.WriteString(fmt.Sprintf(fmtStr, address, target.tryAfter))
-	}
-	return buf.String()
-}
-
 func (cm *ConnectionMaker) attemptConnection(address string, acceptNewPeer bool) {
-	log.Println("Attempting connection to", address)
+	log.Printf("->[%s] attempting connection\n", address)
 	if err := cm.ourself.CreateConnection(address, acceptNewPeer); err != nil {
-		log.Println(err)
-		cm.ConnectionTerminated(address)
+		log.Printf("->[%s] error during connection attempt: %v\n", address, err)
+		cm.ConnectionTerminated(address, err)
 	}
 }
 

router/consts.go

@@ -6,46 +6,27 @@ import (
 )
 
 const (
-	Protocol           = "weave"
-	ProtocolVersion    = 8
-	EthernetOverhead   = 14
-	UDPOverhead        = 28 // 20 bytes for IPv4, 8 bytes for UDP
-	Port               = 6783
-	HttpPort           = Port + 1
-	DefaultPMTU        = 65535
-	MaxUDPPacketSize   = 65536
-	ChannelSize        = 16
-	UDPNonceSendAt     = 8192
-	FragTestSize       = 60001
-	PMTUDiscoverySize  = 60000
-	FastHeartbeat      = 500 * time.Millisecond
-	SlowHeartbeat      = 10 * time.Second
-	FetchAllInterval   = 30 * time.Second
-	FragTestInterval   = 5 * time.Minute
-	ReadTimeout        = 2 * FetchAllInterval
-	PMTUVerifyAttempts = 8
-	PMTUVerifyTimeout  = 10 * time.Millisecond // gets doubled with every attempt
-	MaxDuration        = time.Duration(math.MaxInt64)
-)
-
-const (
-	ProtocolConnectionEstablished  = iota
-	ProtocolFragmentationReceived  = iota
-	ProtocolStartFragmentationTest = iota
-	ProtocolNonce                  = iota
-	ProtocolFetchAll               = iota
-	ProtocolUpdate                 = iota
-	ProtocolPMTUVerified           = iota
+	EthernetOverhead    = 14
+	UDPOverhead         = 28 // 20 bytes for IPv4, 8 bytes for UDP
+	Port                = 6783
+	HTTPPort            = Port + 1
+	DefaultPMTU         = 65535
+	MaxUDPPacketSize    = 65536
+	ChannelSize         = 16
+	FragTestSize        = 60001
+	PMTUDiscoverySize   = 60000
+	TCPHeartbeat        = 30 * time.Second
+	FastHeartbeat       = 500 * time.Millisecond
+	SlowHeartbeat       = 10 * time.Second
+	FragTestInterval    = 5 * time.Minute
+	PMTUVerifyAttempts  = 8
+	PMTUVerifyTimeout   = 10 * time.Millisecond // gets doubled with every attempt
+	MaxDuration         = time.Duration(math.MaxInt64)
+	MaxMissedHeartbeats = 6
+	HeartbeatTimeout    = MaxMissedHeartbeats * SlowHeartbeat
 )
 
 var (
-	FragTest                           = make([]byte, FragTestSize)
-	PMTUDiscovery                      = make([]byte, PMTUDiscoverySize)
-	ProtocolConnectionEstablishedByte  = []byte{ProtocolConnectionEstablished}
-	ProtocolFragmentationReceivedByte  = []byte{ProtocolFragmentationReceived}
-	ProtocolStartFragmentationTestByte = []byte{ProtocolStartFragmentationTest}
-	ProtocolNonceByte                  = []byte{ProtocolNonce}
-	ProtocolFetchAllByte               = []byte{ProtocolFetchAll}
-	ProtocolUpdateByte                 = []byte{ProtocolUpdate}
-	ProtocolPMTUVerifiedByte           = []byte{ProtocolPMTUVerified}
+	FragTest      = make([]byte, FragTestSize)
+	PMTUDiscovery = make([]byte, PMTUDiscoverySize)
 )

router/crypto.go

@@ -1,7 +1,6 @@
 package router
 
 import (
-	"bytes"
 	"code.google.com/p/go-bit/bit"
 	"code.google.com/p/go.crypto/nacl/box"
 	"code.google.com/p/go.crypto/nacl/secretbox"
@@ -10,95 +9,50 @@ import (
 	"encoding/binary"
 	"encoding/gob"
 	"fmt"
-	"log"
+	"sync"
 )
 
 func GenerateKeyPair() (publicKey, privateKey *[32]byte, err error) {
 	return box.GenerateKey(rand.Reader)
 }
 
-func FormSessionKey(remotePublicKey, localPrivateKey *[32]byte, secretKey *[]byte) *[32]byte {
+func FormSessionKey(remotePublicKey, localPrivateKey *[32]byte, secretKey []byte) *[32]byte {
 	var sharedKey [32]byte
 	box.Precompute(&sharedKey, remotePublicKey, localPrivateKey)
 	sharedKeySlice := sharedKey[:]
-	sharedKeySlice = append(sharedKeySlice, *secretKey...)
+	sharedKeySlice = append(sharedKeySlice, secretKey...)
 	sessionKey := sha256.Sum256(sharedKeySlice)
 	return &sessionKey
 }
 
-func GenerateRandomNonce() ([24]byte, error) {
-	var nonce [24]byte
-	n, err := rand.Read(nonce[:])
-	if err != nil {
-		return nonce, err
-	}
-	if n != 24 {
-		return nonce, fmt.Errorf("Did not read enough - wanted 24, got %v", n)
-	}
-	return nonce, nil
-}
-
-func EncryptPrefixNonce(plaintxt []byte, nonce *[24]byte, secret *[32]byte) []byte {
-	buf := make([]byte, 24, 24+len(plaintxt)+secretbox.Overhead)
-	copy(buf, nonce[:])
-	// Seal *appends* to buf
-	return secretbox.Seal(buf, plaintxt, nonce, secret)
-}
-
-func DecryptPrefixNonce(ciphertxt []byte, secret *[32]byte) ([]byte, bool) {
-	if len(ciphertxt) < secretbox.Overhead+24 {
-		return nil, false
-	}
-	// There is no way to nicely convert from a slice to an
-	// array. So have to used the following loop.
-	var nonce [24]byte
-	for idx, e := range ciphertxt[0:24] {
-		nonce[idx] = e
-	}
-	ciphertxt = ciphertxt[24:]
-	return secretbox.Open(nil, ciphertxt, &nonce, secret)
-}
-
-func SetNonceLow15Bits(nonce *[24]byte, offset uint16) {
-	// ensure top bit of offset is 0
-	offset = offset & ((1 << 15) - 1)
-	// grab top bit of nonce[22:24] (and clear out lower bits)
-	nonceBits := binary.BigEndian.Uint16(nonce[22:24]) & (1 << 15)
-	// Big endian => the MSB is stored at the *lowest* address. So
-	// that top bit in nonce[22] should stay as the top bit in
-	// nonce[22]
-	binary.BigEndian.PutUint16(nonce[22:24], nonceBits+offset)
-}
-
-// Nonce encoding/decoding
-
-func EncodeNonce(df bool) (*[24]byte, []byte, error) {
-	nonce, err := GenerateRandomNonce()
-	if err != nil {
-		return nil, []byte{}, err
-	}
-	// wipe out lowest 15 bits, but encode the df right at the bottom
-	flags := uint16(0)
-	if df {
-		flags = flags | 1
-	}
-	SetNonceLow15Bits(&nonce, flags)
-	return &nonce, Concat(ProtocolNonceByte, nonce[:]), nil
-}
-
-func DecodeNonce(msg []byte) (bool, *[24]byte) {
-	flags := uint16(msg[23])
-	df := 0 != (flags & 1)
-	nonce := [24]byte{}
-	// upper bound is exclusive so this avoids copying the flags
-	for idx, elem := range msg[:23] {
-		nonce[idx] = elem
-	}
-	return df, &nonce
-}
-
 // Frame Encryptors
 
+type Encryptor interface {
+	FrameOverhead() int
+	PacketOverhead() int
+	IsEmpty() bool
+	Bytes() ([]byte, error)
+	AppendFrame(src []byte, dst []byte, frame []byte)
+	TotalLen() int
+}
+
+type NonEncryptor struct {
+	buf       []byte
+	bufTail   []byte
+	buffered  int
+	prefixLen int
+}
+
+type NaClEncryptor struct {
+	NonEncryptor
+	buf        []byte
+	prefixLen  int
+	sessionKey *[32]byte
+	nonce      [24]byte
+	seqNo      uint64
+	df         bool
+}
+
 func NewNonEncryptor(prefix []byte) *NonEncryptor {
 	buf := make([]byte, MaxUDPPacketSize)
 	prefixLen := copy(buf, prefix)
@@ -121,89 +75,71 @@ func (ne *NonEncryptor) IsEmpty() bool {
 	return ne.buffered == ne.prefixLen
 }
 
-func (ne *NonEncryptor) Bytes() []byte {
+func (ne *NonEncryptor) Bytes() ([]byte, error) {
 	buf := ne.buf[:ne.buffered]
 	ne.buffered = ne.prefixLen
 	ne.bufTail = ne.buf[ne.prefixLen:]
-	return buf
+	return buf, nil
 }
 
-func (ne *NonEncryptor) AppendFrame(frame *ForwardedFrame) {
+func (ne *NonEncryptor) AppendFrame(src []byte, dst []byte, frame []byte) {
 	bufTail := ne.bufTail
-	srcLen := copy(bufTail, frame.srcPeer.NameByte)
+	srcLen := copy(bufTail, src)
 	bufTail = bufTail[srcLen:]
-	dstLen := copy(bufTail, frame.dstPeer.NameByte)
+	dstLen := copy(bufTail, dst)
 	bufTail = bufTail[dstLen:]
-	frameLen := len(frame.frame)
-	binary.BigEndian.PutUint16(bufTail, uint16(frameLen))
+	binary.BigEndian.PutUint16(bufTail, uint16(len(frame)))
 	bufTail = bufTail[2:]
-	copy(bufTail, frame.frame)
-	ne.bufTail = bufTail[frameLen:]
-	ne.buffered += srcLen + dstLen + 2 + frameLen
+	copy(bufTail, frame)
+	ne.bufTail = bufTail[len(frame):]
+	ne.buffered += srcLen + dstLen + 2 + len(frame)
 }
 
 func (ne *NonEncryptor) TotalLen() int {
 	return ne.buffered
 }
 
-func NewNaClEncryptor(prefix []byte, conn *LocalConnection, df bool) *NaClEncryptor {
+func NewNaClEncryptor(prefix []byte, sessionKey *[32]byte, outbound bool, df bool) *NaClEncryptor {
 	buf := make([]byte, MaxUDPPacketSize)
 	prefixLen := copy(buf, prefix)
-	flags := uint16(0)
-	if df {
-		flags = flags | (1 << 15)
-	}
-	return &NaClEncryptor{
+	ne := &NaClEncryptor{
 		NonEncryptor: *NewNonEncryptor([]byte{}),
 		buf:          buf,
-		offset:       0,
-		nonce:        nil,
-		nonceChan:    make(chan *[24]byte, ChannelSize),
-		flags:        flags,
 		prefixLen:    prefixLen,
-		conn:         conn,
+		sessionKey:   sessionKey,
 		df:           df}
+	if outbound {
+		ne.nonce[0] |= (1 << 7)
+	}
+	return ne
 }
 
-func (ne *NaClEncryptor) Bytes() []byte {
-	plaintext := ne.NonEncryptor.Bytes()
-	offsetFlags := ne.offset | ne.flags
+func (ne *NaClEncryptor) Bytes() ([]byte, error) {
+	plaintext, err := ne.NonEncryptor.Bytes()
+	if err != nil {
+		return nil, err
+	}
+	// We carry the DF flag in the (unencrypted portion of the)
+	// payload, rather than just extracting it from the packet headers
+	// at the receiving end, since we do not trust routers not to mess
+	// with headers. As we have different decryptors for non-DF and
+	// DF, that would result in hard to track down packet drops due to
+	// crypto errors.
+	seqNoAndDF := ne.seqNo
+	if ne.df {
+		seqNoAndDF |= (1 << 63)
+	}
 	ciphertext := ne.buf
-	binary.BigEndian.PutUint16(ciphertext[ne.prefixLen:], offsetFlags)
-	nonce := ne.nonce
-	if nonce == nil {
-		freshNonce, encodedNonce, err := EncodeNonce(ne.df)
-		if err = ne.conn.CheckFatal(err); err != nil {
-			return []byte{}
-		}
-		ne.conn.SendTCP(encodedNonce)
-		ne.nonce = freshNonce
-		nonce = freshNonce
-	}
-	offset := ne.offset
-	SetNonceLow15Bits(nonce, offset)
+	binary.BigEndian.PutUint64(ciphertext[ne.prefixLen:], seqNoAndDF)
+	binary.BigEndian.PutUint64(ne.nonce[16:24], seqNoAndDF)
 	// Seal *appends* to ciphertext
-	ciphertext = secretbox.Seal(ciphertext[:ne.prefixLen+2], plaintext, nonce, ne.conn.SessionKey)
-
-	offset = (offset + 1) & ((1 << 15) - 1)
-	if offset == 0 {
-		// need a new nonce please
-		ne.nonce = <-ne.nonceChan
-	} else if offset == 1<<14 { // half way through range, send new nonce
-		nonce, encodedNonce, err := EncodeNonce(ne.df)
-		if err = ne.conn.CheckFatal(err); err != nil {
-			return []byte{}
-		}
-		ne.nonceChan <- nonce
-		ne.conn.SendTCP(encodedNonce)
-	}
-	ne.offset = offset
-
-	return ciphertext
+	ciphertext = secretbox.Seal(ciphertext[:ne.prefixLen+8], plaintext, &ne.nonce, ne.sessionKey)
+	ne.seqNo++
+	return ciphertext, nil
 }
 
 func (ne *NaClEncryptor) PacketOverhead() int {
-	return ne.prefixLen + 2 + secretbox.Overhead + ne.NonEncryptor.PacketOverhead()
+	return ne.prefixLen + 8 + secretbox.Overhead + ne.NonEncryptor.PacketOverhead()
 }
 
 func (ne *NaClEncryptor) TotalLen() int {
@@ -212,170 +148,199 @@ func (ne *NaClEncryptor) TotalLen() int {
 
 // Frame Decryptors
 
-func NewNonDecryptor(conn *LocalConnection) *NonDecryptor {
-	return &NonDecryptor{conn: conn}
+type FrameConsumer func(src []byte, dst []byte, frame []byte)
+
+type Decryptor interface {
+	IterateFrames([]byte, FrameConsumer) error
 }
 
-func (nd *NonDecryptor) IterateFrames(fun FrameConsumer, packet *UDPPacket) error {
-	buf := packet.Packet
-	for len(buf) >= (2 + NameSize + NameSize) {
-		srcNameByte := buf[:NameSize]
-		buf = buf[NameSize:]
-		dstNameByte := buf[:NameSize]
-		buf = buf[NameSize:]
-		length := binary.BigEndian.Uint16(buf[:2])
-		buf = buf[2:]
-		if len(buf) < int(length) {
-			return PacketDecodingError{Desc: fmt.Sprintf("too short; expected frame of length %d, got %d", length, len(buf))}
-		}
-		frame := buf[:length]
-		buf = buf[length:]
-		err := fun(nd.conn, packet.Sender, srcNameByte, dstNameByte, length, frame)
-		if err != nil {
-			return err
-		}
+type NonDecryptor struct {
+}
+
+type NaClDecryptor struct {
+	NonDecryptor
+	sessionKey *[32]byte
+	instance   *NaClDecryptorInstance
+	instanceDF *NaClDecryptorInstance
+}
+
+type NaClDecryptorInstance struct {
+	nonce               [24]byte
+	currentWindow       uint64
+	usedOffsets         *bit.Set
+	previousUsedOffsets *bit.Set
+}
+
+func NewNaClDecryptorInstance(outbound bool) *NaClDecryptorInstance {
+	di := &NaClDecryptorInstance{usedOffsets: bit.New()}
+	if !outbound {
+		di.nonce[0] |= (1 << 7)
 	}
-	if len(buf) > 0 {
-		return PacketDecodingError{Desc: fmt.Sprintf("%d octets of trailing garbage", len(buf))}
+	return di
+}
+
+type PacketDecodingError struct {
+	Desc string
+}
+
+func NewNonDecryptor() *NonDecryptor {
+	return &NonDecryptor{}
+}
+
+func (nd *NonDecryptor) IterateFrames(packet []byte, consumer FrameConsumer) error {
+	for len(packet) >= (2 + NameSize + NameSize) {
+		srcNameByte := packet[:NameSize]
+		packet = packet[NameSize:]
+		dstNameByte := packet[:NameSize]
+		packet = packet[NameSize:]
+		length := binary.BigEndian.Uint16(packet[:2])
+		packet = packet[2:]
+		if len(packet) < int(length) {
+			return PacketDecodingError{Desc: fmt.Sprintf("too short; expected frame of length %d, got %d", length, len(packet))}
+		}
+		frame := packet[:length]
+		packet = packet[length:]
+		consumer(srcNameByte, dstNameByte, frame)
+	}
+	if len(packet) > 0 {
+		return PacketDecodingError{Desc: fmt.Sprintf("%d octets of trailing garbage", len(packet))}
 	}
 	return nil
 }
 
-func (nd *NonDecryptor) Shutdown() {
-}
-
-func (nd *NonDecryptor) ReceiveNonce(msg []byte) {
-	log.Println("Received Nonce on non-encrypted channel. Ignoring.")
-}
-
-func NewNaClDecryptor(conn *LocalConnection) *NaClDecryptor {
-	inst := NaClDecryptorInstance{
-		nonce:               nil,
-		previousNonce:       nil,
-		usedOffsets:         bit.New(),
-		previousUsedOffsets: nil,
-		highestOffsetSeen:   0,
-		nonceChan:           make(chan *[24]byte, ChannelSize)}
-	instDF := NaClDecryptorInstance{
-		nonce:               nil,
-		previousNonce:       nil,
-		usedOffsets:         bit.New(),
-		previousUsedOffsets: nil,
-		highestOffsetSeen:   0,
-		nonceChan:           make(chan *[24]byte, ChannelSize)}
+func NewNaClDecryptor(sessionKey *[32]byte, outbound bool) *NaClDecryptor {
 	return &NaClDecryptor{
-		NonDecryptor: *NewNonDecryptor(conn),
-		instance:     &inst,
-		instanceDF:   &instDF}
+		NonDecryptor: *NewNonDecryptor(),
+		sessionKey:   sessionKey,
+		instance:     NewNaClDecryptorInstance(outbound),
+		instanceDF:   NewNaClDecryptorInstance(outbound)}
 }
 
-func (nd *NaClDecryptor) Shutdown() {
-	close(nd.instance.nonceChan)
-	close(nd.instanceDF.nonceChan)
+func (nd *NaClDecryptor) IterateFrames(packet []byte, consumer FrameConsumer) error {
+	if len(packet) < 8 {
+		return PacketDecodingError{Desc: fmt.Sprintf("encrypted UDP packet too short; expected length >= 8, got %d", len(packet))}
+	}
+	buf, success := nd.decrypt(packet)
+	if !success {
+		return PacketDecodingError{Desc: fmt.Sprint("UDP packet decryption failed")}
+	}
+	return nd.NonDecryptor.IterateFrames(buf, consumer)
 }
 
-func (nd *NaClDecryptor) ReceiveNonce(msg []byte) {
-	df, nonce := DecodeNonce(msg)
+func (nd *NaClDecryptor) decrypt(buf []byte) ([]byte, bool) {
+	seqNoAndDF := binary.BigEndian.Uint64(buf[:8])
+	df := (seqNoAndDF & (1 << 63)) != 0
+	seqNo := seqNoAndDF & ((1 << 63) - 1)
+	var di *NaClDecryptorInstance
 	if df {
-		nd.instanceDF.nonceChan <- nonce
+		di = nd.instanceDF
 	} else {
-		nd.instance.nonceChan <- nonce
+		di = nd.instance
+	}
+	binary.BigEndian.PutUint64(di.nonce[16:24], seqNoAndDF)
+	result, success := secretbox.Open(nil, buf[8:], &di.nonce, nd.sessionKey)
+	if !success {
+		return nil, false
+	}
+	// Drop duplicates. We do this *after* decryption since we must
+	// not advance our state unless decryption succeeded. Doing so
+	// would open an easy attack vector where an adversary could
+	// inject a packet with a sequence number of (1 << 63) - 1,
+	// causing all subsequent genuine packets to get dropped.
+	offset, usedOffsets := di.advanceState(seqNo)
+	if usedOffsets == nil || usedOffsets.Contains(offset) {
+		// We have detected a possible replay attack, but it is
+		// possible we may have just received a very old packet, or
+		// duplication may have occurred in the network. So let's just
+		// drop the packet silently.
+		return nil, true
+	}
+	usedOffsets.Add(offset)
+	return result, success
+}
+
+// We record seen message sequence numbers in a sliding window of
+// 2*WindowSize which slides in WindowSize increments. This allows us
+// to process out-of-order delivery within the window, while
+// accurately discarding duplicates. By contrast any messages with
+// sequence numbers below the window are discarded as potential
+// duplicates.
+//
+// There are two sets, corresponding to the lower and upper half of
+// the window. We slide the window s.t. that 2nd set always contains
+// the highest seen sequence number. We do this regardless of how far
+// ahead of the current window that sequence number might be, so we
+// can cope with large gaps resulting from packet loss.
+
+const (
+	WindowSize = 20 // bits
+)
+
+func (di *NaClDecryptorInstance) advanceState(seqNo uint64) (int, *bit.Set) {
+	var (
+		offset = int(seqNo & ((1 << WindowSize) - 1))
+		window = seqNo >> WindowSize
+	)
+	switch delta := int64(window - di.currentWindow); {
+	case delta < -1:
+		return offset, nil
+	case delta == -1:
+		return offset, di.previousUsedOffsets
+	default:
+		return offset, di.usedOffsets
+	case delta == +1:
+		di.currentWindow = window
+		di.previousUsedOffsets = di.usedOffsets
+		di.usedOffsets = bit.New()
+		return offset, di.usedOffsets
+	case delta > +1:
+		di.currentWindow = window
+		di.previousUsedOffsets = bit.New()
+		di.usedOffsets = bit.New()
+		return offset, di.usedOffsets
 	}
 }
 
-func (nd *NaClDecryptor) IterateFrames(fun FrameConsumer, packet *UDPPacket) error {
-	buf, err := nd.decrypt(packet.Packet)
-	if err != nil {
-		return PacketDecodingError{Fatal: true, Desc: fmt.Sprint("decryption failed; ", err)}
-	}
-	packet.Packet = buf
-	return nd.NonDecryptor.IterateFrames(fun, packet)
+// TCP Senders/Receivers
+
+// The lowest 64 bits of the nonce contain the message sequence
+// number. The top most bit indicates the connection polarity at the
+// sender - '1' for outbound. The remaining 127 bits are zero. The
+// polarity is needed so that the two ends of a connection do not use
+// the same nonces. This is a requirement of the NaCl Security Model;
+// see http://nacl.cr.yp.to/box.html.
+type TCPCryptoState struct {
+	sessionKey *[32]byte
+	nonce      [24]byte
+	seqNo      uint64
 }
 
-func (nd *NaClDecryptor) decrypt(buf []byte) ([]byte, error) {
-	offset := binary.BigEndian.Uint16(buf[:2])
-	df := (offset & (1 << 15)) != 0
-	offsetNoFlags := offset & ((1 << 15) - 1)
-	var decState *NaClDecryptorInstance
-	if df {
-		decState = nd.instanceDF
-	} else {
-		decState = nd.instance
-	}
-	var nonce *[24]byte
-	var usedOffsets *bit.Set
-	var ok bool
-	if decState.nonce == nil {
-		if offsetNoFlags > (1 << 13) {
-			// offset is already beyond the first quarter and it's the
-			// first thing we've seen?! I don't think so.
-			return nil, fmt.Errorf("Unexpected offset when decrypting UDP packet")
-		}
-		decState.nonce, ok = <-decState.nonceChan
-		if !ok {
-			return nil, fmt.Errorf("Nonce chan closed")
-		}
-		decState.highestOffsetSeen = offsetNoFlags
-		nonce = decState.nonce
-		usedOffsets = decState.usedOffsets
-	} else {
-		highestOffsetSeen := decState.highestOffsetSeen
-		if offsetNoFlags < (1<<13) && highestOffsetSeen > ((1<<14)+(1<<13)) &&
-			(highestOffsetSeen-offsetNoFlags) > ((1<<14)+(1<<13)) {
-			// offset is in the first quarter, highestOffsetSeen is in
-			// the top quarter and under a quarter behind us. We
-			// interpret this as we need to move to the next nonce
-			decState.previousUsedOffsets = decState.usedOffsets
-			decState.usedOffsets = bit.New()
-			decState.previousNonce = decState.nonce
-			decState.nonce, ok = <-decState.nonceChan
-			if !ok {
-				return nil, fmt.Errorf("Nonce chan closed")
-			}
-			decState.highestOffsetSeen = offsetNoFlags
-			nonce = decState.nonce
-			usedOffsets = decState.usedOffsets
-		} else if offsetNoFlags > highestOffsetSeen &&
-			(offsetNoFlags-highestOffsetSeen) < (1<<13) {
-			// offset is under a quarter above highestOffsetSeen. This
-			// is ok - maybe some packet loss
-			decState.highestOffsetSeen = offsetNoFlags
-			nonce = decState.nonce
-			usedOffsets = decState.usedOffsets
-		} else if offsetNoFlags <= highestOffsetSeen &&
-			(highestOffsetSeen-offsetNoFlags) < (1<<13) {
-			// offset is within a quarter of the highest we've
-			// seen. This is ok - just assuming some out-of-order
-			// delivery.
-			nonce = decState.nonce
-			usedOffsets = decState.usedOffsets
-		} else if highestOffsetSeen < (1<<13) && offsetNoFlags > ((1<<14)+(1<<13)) &&
-			(offsetNoFlags-highestOffsetSeen) > ((1<<14)+(1<<13)) {
-			// offset is in the last quarter, highestOffsetSeen is in
-			// the first quarter, and offset is under a quarter behind
-			// us. This is ok - as above, just some out of order. But
-			// here it means we're dealing with the previous nonce
-			nonce = decState.previousNonce
-			usedOffsets = decState.previousUsedOffsets
-		} else {
-			return nil, fmt.Errorf("Unexpected offset when decrypting UDP packet")
-		}
-	}
-	offsetNoFlagsInt := int(offsetNoFlags)
-	if usedOffsets.Contains(offsetNoFlagsInt) {
-		return nil, fmt.Errorf("Suspected replay attack detected when decrypting UDP packet")
-	}
-	SetNonceLow15Bits(nonce, offsetNoFlags)
-	result, success := secretbox.Open(nil, buf[2:], nonce, nd.conn.SessionKey)
-	if success {
-		usedOffsets.Add(offsetNoFlagsInt)
-		return result, nil
-	} else {
-		return nil, fmt.Errorf("Unable to decrypt UDP packet")
+func NewTCPCryptoState(sessionKey *[32]byte, outbound bool) *TCPCryptoState {
+	s := &TCPCryptoState{sessionKey: sessionKey}
+	if outbound {
+		s.nonce[0] |= (1 << 7)
 	}
+	return s
 }
 
-// TCP Senders
+func (s *TCPCryptoState) advance() {
+	s.seqNo++
+	binary.BigEndian.PutUint64(s.nonce[16:24], s.seqNo)
+}
+
+type TCPSender interface {
+	Send([]byte) error
+}
+
+type SimpleTCPSender struct {
+	encoder *gob.Encoder
+}
+
+type EncryptedTCPSender struct {
+	SimpleTCPSender
+	sync.RWMutex
+	state *TCPCryptoState
+}
 
 func NewSimpleTCPSender(encoder *gob.Encoder) *SimpleTCPSender {
 	return &SimpleTCPSender{encoder: encoder}
@@ -385,36 +350,31 @@ func (sender *SimpleTCPSender) Send(msg []byte) error {
 	return sender.encoder.Encode(msg)
 }
 
-func NewEncryptedTCPSender(encoder *gob.Encoder, conn *LocalConnection) *EncryptedTCPSender {
-	buffer := new(bytes.Buffer)
+func NewEncryptedTCPSender(encoder *gob.Encoder, sessionKey *[32]byte, outbound bool) *EncryptedTCPSender {
 	return &EncryptedTCPSender{
-		outerEncoder: encoder,
-		innerEncoder: gob.NewEncoder(buffer),
-		buffer:       buffer,
-		conn:         conn,
-		msgCount:     0}
+		SimpleTCPSender: *NewSimpleTCPSender(encoder),
+		state:           NewTCPCryptoState(sessionKey, outbound)}
 }
 
 func (sender *EncryptedTCPSender) Send(msg []byte) error {
-	nonce, err := GenerateRandomNonce()
-	if err != nil {
-		return err
-	}
 	sender.Lock()
 	defer sender.Unlock()
-	wrappedMsg := EncryptedTCPMessage{Number: sender.msgCount, Body: msg}
-	buffer := sender.buffer
-	buffer.Reset()
-	err = sender.innerEncoder.Encode(wrappedMsg)
-	if err != nil {
-		return err
-	}
-	sender.msgCount = sender.msgCount + 1
-	return sender.outerEncoder.Encode(
-		EncryptPrefixNonce(buffer.Bytes(), &nonce, sender.conn.SessionKey))
+	encodedMsg := secretbox.Seal(nil, msg, &sender.state.nonce, sender.state.sessionKey)
+	sender.state.advance()
+	return sender.SimpleTCPSender.Send(encodedMsg)
 }
 
-// TCP Receivers
+type TCPReceiver interface {
+	Decode([]byte) ([]byte, error)
+}
+
+type SimpleTCPReceiver struct {
+}
+
+type EncryptedTCPReceiver struct {
+	SimpleTCPReceiver
+	state *TCPCryptoState
+}
 
 func NewSimpleTCPReceiver() *SimpleTCPReceiver {
 	return &SimpleTCPReceiver{}
@@ -424,33 +384,17 @@ func (receiver *SimpleTCPReceiver) Decode(msg []byte) ([]byte, error) {
 	return msg, nil
 }
 
-func NewEncryptedTCPReceiver(conn *LocalConnection) *EncryptedTCPReceiver {
-	buffer := new(bytes.Buffer)
+func NewEncryptedTCPReceiver(sessionKey *[32]byte, outbound bool) *EncryptedTCPReceiver {
 	return &EncryptedTCPReceiver{
-		conn:     conn,
-		decoder:  gob.NewDecoder(buffer),
-		buffer:   buffer,
-		msgCount: 0}
+		SimpleTCPReceiver: *NewSimpleTCPReceiver(),
+		state:             NewTCPCryptoState(sessionKey, !outbound)}
 }
 
 func (receiver *EncryptedTCPReceiver) Decode(msg []byte) ([]byte, error) {
-	plaintext, success := DecryptPrefixNonce(msg, receiver.conn.SessionKey)
+	decodedMsg, success := secretbox.Open(nil, msg, &receiver.state.nonce, receiver.state.sessionKey)
 	if !success {
-		return msg, fmt.Errorf("Unable to decrypt TCP msg")
+		return nil, fmt.Errorf("Unable to decrypt TCP msg")
 	}
-	receiver.buffer.Reset()
-	_, err := receiver.buffer.Write(plaintext)
-	if err != nil {
-		return msg, err
-	}
-	wrappedMsg := new(EncryptedTCPMessage)
-	err = receiver.decoder.Decode(wrappedMsg)
-	if err != nil {
-		return msg, err
-	}
-	if wrappedMsg.Number != receiver.msgCount {
-		return msg, fmt.Errorf("Received TCP message with wrong sequence number; possible replay attack")
-	}
-	receiver.msgCount = receiver.msgCount + 1
-	return wrappedMsg.Body, nil
+	receiver.state.advance()
+	return decodedMsg, nil
 }

router/ethernet_decoder.go

@@ -8,6 +8,13 @@ import (
 	"net"
 )
 
+type EthernetDecoder struct {
+	eth     layers.Ethernet
+	ip      layers.IPv4
+	decoded []gopacket.LayerType
+	parser  *gopacket.DecodingLayerParser
+}
+
 func NewEthernetDecoder() *EthernetDecoder {
 	dec := &EthernetDecoder{}
 	dec.parser = gopacket.NewDecodingLayerParser(layers.LayerTypeEthernet, &dec.eth, &dec.ip)
@@ -18,21 +25,7 @@ func (dec *EthernetDecoder) DecodeLayers(data []byte) error {
 	return dec.parser.DecodeLayers(data, &dec.decoded)
 }
 
-func (dec *EthernetDecoder) CheckFrameTooBig(err error, sendFrame func([]byte) error) error {
-	if ftbe, ok := err.(FrameTooBigError); ok {
-		// we know: 1. ip is valid, 2. it was ip and DF was set
-		icmpFrame, err := dec.formICMPMTUPacket(ftbe.EPMTU)
-		if err != nil {
-			return err
-		}
-		log.Printf("Sending ICMP 3,4 (%v -> %v): PMTU= %v\n", dec.ip.DstIP, dec.ip.SrcIP, ftbe.EPMTU)
-		return sendFrame(icmpFrame)
-	} else {
-		return err
-	}
-}
-
-func (dec *EthernetDecoder) formICMPMTUPacket(mtu int) ([]byte, error) {
+func (dec *EthernetDecoder) sendICMPFragNeeded(mtu int, sendFrame func([]byte) error) error {
 	buf := gopacket.NewSerializeBuffer()
 	opts := gopacket.SerializeOptions{
 		FixLengths:       true,
@@ -60,9 +53,11 @@ func (dec *EthernetDecoder) formICMPMTUPacket(mtu int) ([]byte, error) {
 			Seq:      uint16(mtu)},
 		&payload)
 	if err != nil {
-		return []byte{}, err
+		return err
 	}
-	return buf.Bytes(), nil
+
+	log.Printf("Sending ICMP 3,4 (%v -> %v): PMTU= %v\n", dec.ip.DstIP, dec.ip.SrcIP, mtu)
+	return sendFrame(buf.Bytes())
 }
 
 var (
@@ -75,7 +70,7 @@ func (dec *EthernetDecoder) DropFrame() bool {
 	return bytes.Equal(stpMACPrefix, dec.eth.DstMAC[:len(stpMACPrefix)])
 }
 
-func (dec *EthernetDecoder) IsPMTUVerify() bool {
-	return bytes.Equal(zeroMAC, dec.eth.SrcMAC) &&
-		bytes.Equal(zeroMAC, dec.eth.DstMAC)
+func (dec *EthernetDecoder) IsSpecial() bool {
+	return dec.eth.Length == 0 && dec.eth.EthernetType == layers.EthernetTypeLLC &&
+		bytes.Equal(zeroMAC, dec.eth.SrcMAC) && bytes.Equal(zeroMAC, dec.eth.DstMAC)
 }

router/forwarder.go

@@ -3,14 +3,22 @@ package router
 import (
 	"code.google.com/p/gopacket"
 	"code.google.com/p/gopacket/layers"
-	"log"
-	"net"
 	"syscall"
 	"time"
 )
 
+type ForwardedFrame struct {
+	srcPeer *Peer
+	dstPeer *Peer
+	frame   []byte
+}
+
+type FrameTooBigError struct {
+	EPMTU int // effective pmtu, i.e. what we tell packet senders
+}
+
 func (conn *LocalConnection) ensureForwarders() error {
-	if conn.forwardChan != nil || conn.forwardChanDF != nil {
+	if conn.forwarder != nil || conn.forwarderDF != nil {
 		return nil
 	}
 	udpSender := NewSimpleUDPSender(conn)
@@ -22,54 +30,36 @@ func (conn *LocalConnection) ensureForwarders() error {
 	usingPassword := conn.SessionKey != nil
 	var encryptor, encryptorDF Encryptor
 	if usingPassword {
-		encryptor = NewNaClEncryptor(conn.local.NameByte, conn, false)
-		encryptorDF = NewNaClEncryptor(conn.local.NameByte, conn, true)
+		encryptor = NewNaClEncryptor(conn.local.NameByte, conn.SessionKey, conn.outbound, false)
+		encryptorDF = NewNaClEncryptor(conn.local.NameByte, conn.SessionKey, conn.outbound, true)
 	} else {
 		encryptor = NewNonEncryptor(conn.local.NameByte)
 		encryptorDF = NewNonEncryptor(conn.local.NameByte)
 	}
 
-	var (
-		forwardChan   = make(chan *ForwardedFrame, ChannelSize)
-		forwardChanDF = make(chan *ForwardedFrame, ChannelSize)
-		stopForward   = make(chan interface{}, 0)
-		stopForwardDF = make(chan interface{}, 0)
-		verifyPMTU    = make(chan int, ChannelSize)
-	)
-	//NB: only forwarderDF can ever encounter EMSGSIZE errors, and
-	//thus perform PMTU verification
-	forwarder := NewForwarder(conn, forwardChan, stopForward, nil, encryptor, udpSender, DefaultPMTU)
-	forwarderDF := NewForwarder(conn, forwardChanDF, stopForwardDF, verifyPMTU, encryptorDF, udpSenderDF, DefaultPMTU)
+	forwarder := NewForwarder(conn, encryptor, udpSender, DefaultPMTU)
+	forwarderDF := NewForwarderDF(conn, encryptorDF, udpSenderDF, DefaultPMTU)
+	effectivePMTU := forwarderDF.unverifiedPMTU
+	forwarder.Start()
+	forwarderDF.Start()
 
 	// Various fields in the conn struct are read by other processes,
 	// so we have to use locks.
 	conn.Lock()
-	conn.forwardChan = forwardChan
-	conn.forwardChanDF = forwardChanDF
-	conn.stopForward = stopForward
-	conn.stopForwardDF = stopForwardDF
-	conn.verifyPMTU = verifyPMTU
-	conn.effectivePMTU = forwarder.unverifiedPMTU
+	conn.forwarder = forwarder
+	conn.forwarderDF = forwarderDF
+	conn.effectivePMTU = effectivePMTU
 	conn.Unlock()
 
-	go forwarder.Run()
-	go forwarderDF.Run()
-
 	return nil
 }
 
 func (conn *LocalConnection) stopForwarders() {
-	conn.Lock()
-	conn.forwardChan = nil
-	conn.forwardChanDF = nil
-	conn.Unlock()
-	// Now signal the forwarder loops to exit. They will drain the
-	// forwarder chans in order to unblock any router processes
-	// blocked on sending.
-	if conn.stopForward != nil {
-		conn.stopForward <- nil
-		conn.stopForwardDF <- nil
+	if conn.forwarder == nil || conn.forwarderDF == nil {
+		return
 	}
+	conn.forwarder.Shutdown()
+	conn.forwarderDF.Shutdown()
 }
 
 // Called from peer.Relay[Broadcast] which is itself invoked from
@@ -79,15 +69,15 @@ func (conn *LocalConnection) stopForwarders() {
 func (conn *LocalConnection) Forward(df bool, frame *ForwardedFrame, dec *EthernetDecoder) error {
 	conn.RLock()
 	var (
-		forwardChan   = conn.forwardChan
-		forwardChanDF = conn.forwardChanDF
+		forwarder     = conn.forwarder
+		forwarderDF   = conn.forwarderDF
 		effectivePMTU = conn.effectivePMTU
 		stackFrag     = conn.stackFrag
 	)
 	conn.RUnlock()
 
-	if forwardChan == nil || forwardChanDF == nil {
-		conn.log("Cannot forward frame yet - awaiting contact")
+	if forwarder == nil || forwarderDF == nil {
+		conn.Log("Cannot forward frame yet - awaiting contact")
 		return nil
 	}
 	// We could use non-blocking channel sends here, i.e. drop frames
@@ -102,29 +92,29 @@ func (conn *LocalConnection) Forward(df bool, frame *ForwardedFrame, dec *Ethern
 	// of our pipeline.
 	if df {
 		if !frameTooBig(frame, effectivePMTU) {
-			forwardChanDF <- frame
+			forwarderDF.Forward(frame)
 			return nil
 		}
 		return FrameTooBigError{EPMTU: effectivePMTU}
-	} else {
-		if stackFrag || dec == nil || len(dec.decoded) < 2 {
-			forwardChan <- frame
-			return nil
-		}
-		// Don't have trustworthy stack, so we're going to have to
-		// send it DF in any case.
-		if !frameTooBig(frame, effectivePMTU) {
-			forwardChanDF <- frame
-			return nil
-		}
-		conn.Router.LogFrame("Fragmenting", frame.frame, &dec.eth)
-		// We can't trust the stack to fragment, we have IP, and we
-		// have a frame that's too big for the MTU, so we have to
-		// fragment it ourself.
-		return fragment(dec.eth, dec.ip, effectivePMTU, frame, func(segFrame *ForwardedFrame) {
-			forwardChanDF <- segFrame
-		})
 	}
+
+	if stackFrag || dec == nil || len(dec.decoded) < 2 {
+		forwarder.Forward(frame)
+		return nil
+	}
+	// Don't have trustworthy stack, so we're going to have to
+	// send it DF in any case.
+	if !frameTooBig(frame, effectivePMTU) {
+		forwarderDF.Forward(frame)
+		return nil
+	}
+	conn.Router.LogFrame("Fragmenting", frame.frame, &dec.eth)
+	// We can't trust the stack to fragment, we have IP, and we
+	// have a frame that's too big for the MTU, so we have to
+	// fragment it ourself.
+	return fragment(dec.eth, dec.ip, effectivePMTU, frame, func(segFrame *ForwardedFrame) {
+		forwarderDF.Forward(segFrame)
+	})
 }
 
 func frameTooBig(frame *ForwardedFrame, effectivePMTU int) bool {
@@ -184,7 +174,7 @@ func fragment(eth layers.Ethernet, ip layers.IPv4, pmtu int, frame *ForwardedFra
 			return err
 		}
 		// make copies of the frame we received
-		var segFrame ForwardedFrame = *frame
+		segFrame := *frame
 		segFrame.frame = buf.Bytes()
 		forward(&segFrame)
 	}
@@ -193,33 +183,168 @@ func fragment(eth layers.Ethernet, ip layers.IPv4, pmtu int, frame *ForwardedFra
 
 // Forwarder
 
-func NewForwarder(conn *LocalConnection, ch <-chan *ForwardedFrame, stop <-chan interface{}, verifyPMTU <-chan int, enc Encryptor, udpSender UDPSender, pmtu int) *Forwarder {
-	fwd := &Forwarder{
-		conn:       conn,
-		ch:         ch,
-		stop:       stop,
-		verifyPMTU: verifyPMTU,
-		enc:        enc,
-		udpSender:  udpSender}
+type Forwarder struct {
+	conn             *LocalConnection
+	ch               chan<- *ForwardedFrame
+	finished         <-chan struct{}
+	enc              Encryptor
+	udpSender        UDPSender
+	maxPayload       int
+	processSendError func(error) error
+}
+
+func NewForwarder(conn *LocalConnection, enc Encryptor, udpSender UDPSender, pmtu int) *Forwarder {
+	return &Forwarder{
+		conn:             conn,
+		enc:              enc,
+		udpSender:        udpSender,
+		maxPayload:       pmtu - UDPOverhead,
+		processSendError: func(err error) error { return err }}
+}
+
+func (fwd *Forwarder) Start() {
+	ch := make(chan *ForwardedFrame, ChannelSize)
+	fwd.ch = ch
+	finished := make(chan struct{})
+	fwd.finished = finished
+	go fwd.run(ch, finished)
+}
+
+func (fwd *Forwarder) Shutdown() {
+	fwd.ch <- nil
+}
+
+func (fwd *Forwarder) Forward(frame *ForwardedFrame) {
+	select {
+	case fwd.ch <- frame:
+	case <-fwd.finished:
+	}
+}
+
+func (fwd *Forwarder) run(ch <-chan *ForwardedFrame, finished chan<- struct{}) {
+	defer fwd.udpSender.Shutdown()
+	for {
+		if !fwd.accumulateAndSendFrames(ch, <-ch) {
+			close(finished)
+			return
+		}
+	}
+}
+
+func (fwd *Forwarder) effectiveOverhead() int {
+	return UDPOverhead + fwd.enc.PacketOverhead() + fwd.enc.FrameOverhead() + EthernetOverhead
+}
+
+// Drain the inbound channel of frames, aggregating them into larger
+// packets for efficient transmission.
+//
+// FIXME Depending on the golang scheduler, and the rate at which
+// franes get sent to the forwarder, we can be going around this loop
+// forever. That is bad since there may be other stuff for us to do,
+// i.e. the other branches in the run loop.
+func (fwd *Forwarder) accumulateAndSendFrames(ch <-chan *ForwardedFrame, frame *ForwardedFrame) bool {
+	if frame == nil {
+		return false
+	}
+	if !fwd.appendFrame(frame) {
+		fwd.logDrop(frame)
+		// [1] The buffer is empty at this point and therefore we must
+		// not flush it. The easiest way to accomplish that is simply
+		// by returning to the surrounding run loop.
+		return true
+	}
+	for {
+		select {
+		case frame = <-ch:
+			if frame == nil {
+				return false
+			}
+			if !fwd.appendFrame(frame) {
+				fwd.flush()
+				if !fwd.appendFrame(frame) {
+					fwd.logDrop(frame)
+					return true // see [1]
+				}
+			}
+		default:
+			fwd.flush()
+			return true
+		}
+	}
+}
+
+func (fwd *Forwarder) logDrop(frame *ForwardedFrame) {
+	fwd.conn.Log("Dropping too big frame during forwarding: frame len:", len(frame.frame), "; effective PMTU:", fwd.maxPayload+UDPOverhead-fwd.effectiveOverhead())
+}
+
+func (fwd *Forwarder) appendFrame(frame *ForwardedFrame) bool {
+	frameLen := len(frame.frame)
+	if fwd.enc.TotalLen()+fwd.enc.FrameOverhead()+frameLen > fwd.maxPayload {
+		return false
+	}
+	fwd.enc.AppendFrame(frame.srcPeer.NameByte, frame.dstPeer.NameByte, frame.frame)
+	return true
+}
+
+func (fwd *Forwarder) flush() {
+	msg, err := fwd.enc.Bytes()
+	if err != nil {
+		fwd.conn.Shutdown(err)
+	}
+	err = fwd.processSendError(fwd.udpSender.Send(msg))
+	if err != nil && PosixError(err) != syscall.ENOBUFS {
+		fwd.conn.Shutdown(err)
+	}
+}
+
+type ForwarderDF struct {
+	Forwarder
+	verifyPMTUTick  <-chan time.Time
+	verifyPMTU      chan<- int
+	pmtuVerifyCount uint
+	pmtuVerified    bool
+	highestGoodPMTU int
+	unverifiedPMTU  int
+	lowestBadPMTU   int
+}
+
+func NewForwarderDF(conn *LocalConnection, enc Encryptor, udpSender UDPSender, pmtu int) *ForwarderDF {
+	fwd := &ForwarderDF{
+		Forwarder: Forwarder{
+			conn:       conn,
+			enc:        enc,
+			udpSender:  udpSender,
+			maxPayload: pmtu - UDPOverhead}}
+	fwd.Forwarder.processSendError = fwd.processSendError
 	fwd.unverifiedPMTU = pmtu - fwd.effectiveOverhead()
-	fwd.maxPayload = pmtu - UDPOverhead
 	return fwd
 }
 
-func (fwd *Forwarder) Run() {
+func (fwd *ForwarderDF) Start() {
+	ch := make(chan *ForwardedFrame, ChannelSize)
+	fwd.ch = ch
+	finished := make(chan struct{})
+	fwd.finished = finished
+	verifyPMTU := make(chan int, ChannelSize)
+	fwd.verifyPMTU = verifyPMTU
+	go fwd.run(ch, finished, verifyPMTU)
+}
+
+func (fwd *ForwarderDF) PMTUVerified(pmtu int) {
+	select {
+	case fwd.verifyPMTU <- pmtu:
+	case <-fwd.finished:
+	}
+}
+
+func (fwd *ForwarderDF) run(ch <-chan *ForwardedFrame, finished chan<- struct{}, verifyPMTU <-chan int) {
 	defer fwd.udpSender.Shutdown()
-	var flushed, ok bool
-	var frame *ForwardedFrame
 	for {
-		flushed = false
 		select {
-		case <-fwd.stop:
-			fwd.drain()
-			return
 		case <-fwd.verifyPMTUTick:
-			// We only do this case here when we know the buffers are
-			// all empty so that we don't risk appending verify-frames
-			// to other data.
+			// We only do this case here when we know the
+			// buffers are all empty so that we don't risk
+			// appending verify-frames to other data.
 			fwd.verifyPMTUTick = nil
 			if fwd.pmtuVerified {
 				continue
@@ -228,12 +353,12 @@ func (fwd *Forwarder) Run() {
 				fwd.pmtuVerifyCount--
 				fwd.attemptVerifyEffectivePMTU()
 			} else {
-				// we've exceeded the verification attempts of the
-				// unverifiedPMTU
+				// we've exceeded the verification
+				// attempts of the unverifiedPMTU
 				fwd.lowestBadPMTU = fwd.unverifiedPMTU
 				fwd.verifyEffectivePMTU((fwd.highestGoodPMTU + fwd.lowestBadPMTU) / 2)
 			}
-		case epmtu := <-fwd.verifyPMTU:
+		case epmtu := <-verifyPMTU:
 			if fwd.pmtuVerified || epmtu != fwd.unverifiedPMTU {
 				continue
 			}
@@ -244,210 +369,46 @@ func (fwd *Forwarder) Run() {
 				fwd.pmtuVerified = true
 				fwd.maxPayload = epmtu + fwd.effectiveOverhead() - UDPOverhead
 				fwd.conn.setEffectivePMTU(epmtu)
-				fwd.conn.log("Effective PMTU verified at", epmtu)
+				fwd.conn.Log("Effective PMTU verified at", epmtu)
 			}
-		case frame = <-fwd.ch:
-			if !fwd.appendFrame(frame) {
-				fwd.logDrop(frame)
-				continue
-			}
-			for !flushed {
-				select {
-				case frame, ok = <-fwd.ch:
-					if !ok {
-						return
-					}
-					if !fwd.appendFrame(frame) {
-						fwd.flush()
-						if !fwd.appendFrame(frame) {
-							fwd.logDrop(frame)
-							flushed = true
-						}
-					}
-				default:
-					fwd.flush()
-					flushed = true
-				}
+		case frame := <-ch:
+			if !fwd.accumulateAndSendFrames(ch, frame) {
+				close(finished)
+				return
 			}
 		}
 	}
 }
 
-func (fwd *Forwarder) effectiveOverhead() int {
-	return UDPOverhead + fwd.enc.PacketOverhead() + fwd.enc.FrameOverhead() + EthernetOverhead
-}
-
-func (fwd *Forwarder) verifyEffectivePMTU(newUnverifiedPMTU int) {
+func (fwd *ForwarderDF) verifyEffectivePMTU(newUnverifiedPMTU int) {
 	fwd.unverifiedPMTU = newUnverifiedPMTU
 	fwd.pmtuVerifyCount = PMTUVerifyAttempts
 	fwd.attemptVerifyEffectivePMTU()
 }
 
-func (fwd *Forwarder) attemptVerifyEffectivePMTU() {
-	pmtuVerifyFrame := &ForwardedFrame{
-		srcPeer: fwd.conn.local,
-		dstPeer: fwd.conn.remote,
-		frame:   make([]byte, fwd.unverifiedPMTU+EthernetOverhead)}
-	fwd.enc.AppendFrame(pmtuVerifyFrame)
+func (fwd *ForwarderDF) attemptVerifyEffectivePMTU() {
+	fwd.enc.AppendFrame(fwd.conn.local.NameByte, fwd.conn.remote.NameByte,
+		make([]byte, fwd.unverifiedPMTU+EthernetOverhead))
 	fwd.flush()
 	if fwd.verifyPMTUTick == nil {
 		fwd.verifyPMTUTick = time.After(PMTUVerifyTimeout << (PMTUVerifyAttempts - fwd.pmtuVerifyCount))
 	}
 }
 
-func (fwd *Forwarder) appendFrame(frame *ForwardedFrame) bool {
-	frameLen := len(frame.frame)
-	if fwd.enc.TotalLen()+fwd.enc.FrameOverhead()+frameLen > fwd.maxPayload {
-		return false
-	}
-	fwd.enc.AppendFrame(frame)
-	return true
-}
-
-func (fwd *Forwarder) flush() {
-	err := fwd.udpSender.Send(fwd.enc.Bytes())
-	if err != nil {
-		if mtbe, ok := err.(MsgTooBigError); ok {
-			newUnverifiedPMTU := mtbe.PMTU - fwd.effectiveOverhead()
-			if newUnverifiedPMTU >= fwd.unverifiedPMTU {
-				return
-			}
+func (fwd *ForwarderDF) processSendError(err error) error {
+	if mtbe, ok := err.(MsgTooBigError); ok {
+		newUnverifiedPMTU := mtbe.PMTU - fwd.effectiveOverhead()
+		if newUnverifiedPMTU < fwd.unverifiedPMTU {
 			fwd.pmtuVerified = false
 			fwd.maxPayload = mtbe.PMTU - UDPOverhead
 			fwd.highestGoodPMTU = 8
 			fwd.lowestBadPMTU = newUnverifiedPMTU + 1
 			fwd.conn.setEffectivePMTU(newUnverifiedPMTU)
 			fwd.verifyEffectivePMTU(newUnverifiedPMTU)
-		} else if PosixError(err) == syscall.ENOBUFS {
-			// TODO handle this better
-		} else {
-			fwd.conn.CheckFatal(err)
 		}
+
+		return nil
 	}
-}
 
-func (fwd *Forwarder) drain() {
-	// We want to drain before exiting otherwise we could get the
-	// packet sniffer or udp listener blocked on sending to a full
-	// chan
-	for {
-		select {
-		case <-fwd.ch:
-		default:
-			return
-		}
-	}
-}
-
-func (fwd *Forwarder) logDrop(frame *ForwardedFrame) {
-	fwd.conn.log("Dropping too big frame during forwarding: frame len:", len(frame.frame), "; effective PMTU:", fwd.maxPayload+UDPOverhead-fwd.effectiveOverhead())
-}
-
-// UDP Senders
-
-func NewSimpleUDPSender(conn *LocalConnection) *SimpleUDPSender {
-	return &SimpleUDPSender{udpConn: conn.Router.UDPListener, conn: conn}
-}
-
-func (sender *SimpleUDPSender) Send(msg []byte) error {
-	_, err := sender.udpConn.WriteToUDP(msg, sender.conn.RemoteUDPAddr())
 	return err
 }
-
-func (sender *SimpleUDPSender) Shutdown() error {
-	return nil
-}
-
-func NewRawUDPSender(conn *LocalConnection) (*RawUDPSender, error) {
-	ipSocket, err := dialIP(conn)
-	if err != nil {
-		return nil, err
-	}
-	udpHeader := &layers.UDP{SrcPort: layers.UDPPort(Port)}
-	ipBuf := gopacket.NewSerializeBuffer()
-	opts := gopacket.SerializeOptions{
-		FixLengths: true,
-		// UDP header is calculated with a phantom IP
-		// header. Yes, it's totally nuts. Thankfully, for UDP
-		// over IPv4, the checksum is optional. It's not
-		// optional for IPv6, but we'll ignore that for
-		// now. TODO
-		ComputeChecksums: false}
-
-	return &RawUDPSender{
-		ipBuf:     ipBuf,
-		opts:      opts,
-		udpHeader: udpHeader,
-		socket:    ipSocket,
-		conn:      conn}, nil
-}
-
-func (sender *RawUDPSender) Send(msg []byte) error {
-	payload := gopacket.Payload(msg)
-	sender.udpHeader.DstPort = layers.UDPPort(sender.conn.RemoteUDPAddr().Port)
-
-	err := gopacket.SerializeLayers(sender.ipBuf, sender.opts, sender.udpHeader, &payload)
-	if err != nil {
-		return err
-	}
-	packet := sender.ipBuf.Bytes()
-	_, err = sender.socket.Write(packet)
-	if err == nil || PosixError(err) != syscall.EMSGSIZE {
-		return err
-	}
-	f, err := sender.socket.File()
-	if err != nil {
-		return err
-	}
-	defer f.Close()
-	fd := int(f.Fd())
-	log.Println("EMSGSIZE on send, expecting PMTU update (IP packet was",
-		len(packet), "bytes, payload was", len(msg), "bytes)")
-	pmtu, err := syscall.GetsockoptInt(fd, syscall.IPPROTO_IP, syscall.IP_MTU)
-	if err != nil {
-		return err
-	}
-	return MsgTooBigError{PMTU: pmtu}
-}
-
-func (sender *RawUDPSender) Shutdown() error {
-	defer func() { sender.socket = nil }()
-	return sender.socket.Close()
-}
-
-func dialIP(conn *LocalConnection) (*net.IPConn, error) {
-	ipLocalAddr, err := ipAddr(conn.TCPConn.LocalAddr())
-	if err != nil {
-		return nil, err
-	}
-	ipRemoteAddr, err := ipAddr(conn.TCPConn.RemoteAddr())
-	if err != nil {
-		return nil, err
-	}
-	ipSocket, err := net.DialIP("ip4:UDP", ipLocalAddr, ipRemoteAddr)
-	if err != nil {
-		return nil, err
-	}
-	f, err := ipSocket.File()
-	if err != nil {
-		return nil, err
-	}
-	defer f.Close()
-	fd := int(f.Fd())
-	// This Makes sure all packets we send out have DF set on them.
-	err = syscall.SetsockoptInt(fd, syscall.IPPROTO_IP, syscall.IP_MTU_DISCOVER, syscall.IP_PMTUDISC_DO)
-	if err != nil {
-		return nil, err
-	}
-	return ipSocket, nil
-}
-
-func ipAddr(addr net.Addr) (*net.IPAddr, error) {
-	host, _, err := net.SplitHostPort(addr.String())
-	if err != nil {
-		return nil, err
-	}
-	return &net.IPAddr{
-		IP:   net.ParseIP(host),
-		Zone: ""}, nil
-}

router/gossip.go

@@ -0,0 +1,341 @@
+package router
+
+import (
+	"bytes"
+	"encoding/gob"
+	"fmt"
+	"log"
+	"sync"
+	"time"
+)
+
+const GossipInterval = 30 * time.Second
+
+type GossipData interface {
+	Encode() []byte
+	Merge(GossipData)
+}
+
+type Gossip interface {
+	// specific message from one peer to another
+	// intermediate peers relay it using unicast topology.
+	GossipUnicast(dstPeerName PeerName, msg []byte) error
+	// send gossip to every peer, relayed using broadcast topology.
+	GossipBroadcast(update GossipData) error
+}
+
+type Gossiper interface {
+	OnGossipUnicast(sender PeerName, msg []byte) error
+	// merge received data into state and return a representation of
+	// the received data, for further propagation
+	OnGossipBroadcast(update []byte) (GossipData, error)
+	// return state of everything we know; gets called periodically
+	Gossip() GossipData
+	// merge received data into state and return "everything new I've
+	// just learnt", or nil if nothing in the received data was new
+	OnGossip(update []byte) (GossipData, error)
+}
+
+// Accumulates GossipData that needs to be sent to one destination,
+// and sends it when possible.
+type GossipSender struct {
+	send func(GossipData)
+	cell chan GossipData
+}
+
+func NewGossipSender(send func(GossipData)) *GossipSender {
+	return &GossipSender{send: send}
+}
+
+func (sender *GossipSender) Start() {
+	sender.cell = make(chan GossipData, 1)
+	go sender.run()
+}
+
+func (sender *GossipSender) run() {
+	for {
+		if pending := <-sender.cell; pending == nil { // receive zero value when chan is closed
+			break
+		} else {
+			sender.send(pending)
+		}
+	}
+}
+
+func (sender *GossipSender) Send(data GossipData) {
+	// NB: this must not be invoked concurrently
+	select {
+	case pending := <-sender.cell:
+		pending.Merge(data)
+		sender.cell <- pending
+	default:
+		sender.cell <- data
+	}
+}
+
+func (sender *GossipSender) Stop() {
+	close(sender.cell)
+}
+
+type connectionSenders map[Connection]*GossipSender
+type peerSenders map[PeerName]*GossipSender
+
+type GossipChannel struct {
+	sync.Mutex
+	ourself      *LocalPeer
+	routes       *Routes
+	name         string
+	hash         uint32
+	gossiper     Gossiper
+	senders      connectionSenders
+	broadcasters peerSenders
+}
+
+func (router *Router) NewGossip(channelName string, g Gossiper) Gossip {
+	channelHash := hash(channelName)
+	channel := &GossipChannel{
+		ourself:      router.Ourself,
+		routes:       router.Routes,
+		name:         channelName,
+		hash:         channelHash,
+		gossiper:     g,
+		senders:      make(connectionSenders),
+		broadcasters: make(peerSenders)}
+	router.GossipChannels[channelHash] = channel
+	return channel
+}
+
+func (router *Router) SendAllGossip() {
+	for _, channel := range router.GossipChannels {
+		if gossip := channel.gossiper.Gossip(); gossip != nil {
+			channel.Send(router.Ourself.Name, gossip)
+		}
+	}
+}
+
+func (router *Router) SendAllGossipDown(conn Connection) {
+	for _, channel := range router.GossipChannels {
+		if gossip := channel.gossiper.Gossip(); gossip != nil {
+			channel.SendDown(conn, channel.gossiper.Gossip())
+		}
+	}
+}
+
+func (router *Router) handleGossip(tag ProtocolTag, payload []byte) error {
+	decoder := gob.NewDecoder(bytes.NewReader(payload))
+	var channelHash uint32
+	if err := decoder.Decode(&channelHash); err != nil {
+		return err
+	}
+	channel, found := router.GossipChannels[channelHash]
+	if !found {
+		return fmt.Errorf("[gossip] received unknown channel with hash %v", channelHash)
+	}
+	var srcName PeerName
+	if err := decoder.Decode(&srcName); err != nil {
+		return err
+	}
+	switch tag {
+	case ProtocolGossipUnicast:
+		return channel.deliverUnicast(srcName, payload, decoder)
+	case ProtocolGossipBroadcast:
+		return channel.deliverBroadcast(srcName, payload, decoder)
+	case ProtocolGossip:
+		return channel.deliver(srcName, payload, decoder)
+	}
+	return nil
+}
+
+func (c *GossipChannel) deliverUnicast(srcName PeerName, origPayload []byte, dec *gob.Decoder) error {
+	var destName PeerName
+	if err := dec.Decode(&destName); err != nil {
+		return err
+	}
+	if c.ourself.Name != destName {
+		return c.relayUnicast(destName, origPayload)
+	}
+	var payload []byte
+	if err := dec.Decode(&payload); err != nil {
+		return err
+	}
+	return c.gossiper.OnGossipUnicast(srcName, payload)
+}
+
+func (c *GossipChannel) deliverBroadcast(srcName PeerName, _ []byte, dec *gob.Decoder) error {
+	var payload []byte
+	if err := dec.Decode(&payload); err != nil {
+		return err
+	}
+	data, err := c.gossiper.OnGossipBroadcast(payload)
+	if err != nil || data == nil {
+		return err
+	}
+	return c.relayBroadcast(srcName, data)
+}
+
+func (c *GossipChannel) deliver(srcName PeerName, _ []byte, dec *gob.Decoder) error {
+	var payload []byte
+	if err := dec.Decode(&payload); err != nil {
+		return err
+	}
+	if data, err := c.gossiper.OnGossip(payload); err != nil {
+		return err
+	} else if data != nil {
+		c.Send(srcName, data)
+	}
+	return nil
+}
+
+func (c *GossipChannel) Send(srcName PeerName, data GossipData) {
+	// do this outside the lock below so we avoid lock nesting
+	c.routes.EnsureRecalculated()
+	selectedConnections := make(ConnectionSet)
+	for name := range c.routes.RandomNeighbours(srcName) {
+		if conn, found := c.ourself.ConnectionTo(name); found {
+			selectedConnections[conn] = void
+		}
+	}
+	if len(selectedConnections) == 0 {
+		return
+	}
+	connections := c.ourself.Connections()
+	c.Lock()
+	defer c.Unlock()
+	// GC - randomly (courtesy of go's map iterator) pick some
+	// existing entries and stop&remove them if the associated
+	// connection is no longer active.  We stop as soon as we
+	// encounter a valid entry; the idea being that when there is
+	// little or no garbage then this executes close to O(1)[1],
+	// whereas when there is lots of garbage we remove it quickly.
+	//
+	// [1] TODO Unfortunately, due to the desire to avoid nested
+	// locks, instead of simply invoking Peer.ConnectionTo(name)
+	// below, we have that Peer.Connections() invocation above. That
+	// is O(n_our_connections) at best.
+	for conn, sender := range c.senders {
+		if _, found := connections[conn]; !found {
+			delete(c.senders, conn)
+			sender.Stop()
+		} else {
+			break
+		}
+	}
+	for conn := range selectedConnections {
+		c.sendDown(conn, data)
+	}
+}
+
+func (c *GossipChannel) SendDown(conn Connection, data GossipData) {
+	c.Lock()
+	c.sendDown(conn, data)
+	c.Unlock()
+}
+
+func (c *GossipChannel) sendDown(conn Connection, data GossipData) {
+	sender, found := c.senders[conn]
+	if !found {
+		sender = NewGossipSender(func(pending GossipData) {
+			protocolMsg := ProtocolMsg{ProtocolGossip, GobEncode(c.hash, c.ourself.Name, pending.Encode())}
+			conn.(ProtocolSender).SendProtocolMsg(protocolMsg)
+		})
+		c.senders[conn] = sender
+		sender.Start()
+	}
+	sender.Send(data)
+}
+
+func (c *GossipChannel) GossipUnicast(dstPeerName PeerName, msg []byte) error {
+	return c.relayUnicast(dstPeerName, GobEncode(c.hash, c.ourself.Name, dstPeerName, msg))
+}
+
+func (c *GossipChannel) GossipBroadcast(update GossipData) error {
+	return c.relayBroadcast(c.ourself.Name, update)
+}
+
+func (c *GossipChannel) relayUnicast(dstPeerName PeerName, buf []byte) error {
+	if relayPeerName, found := c.routes.UnicastAll(dstPeerName); !found {
+		c.log("unknown relay destination:", dstPeerName)
+	} else if conn, found := c.ourself.ConnectionTo(relayPeerName); !found {
+		c.log("unable to find connection to relay peer", relayPeerName)
+	} else {
+		conn.(ProtocolSender).SendProtocolMsg(ProtocolMsg{ProtocolGossipUnicast, buf})
+	}
+	return nil
+}
+
+func (c *GossipChannel) relayBroadcast(srcName PeerName, update GossipData) error {
+	names := c.routes.PeerNames() // do this outside the lock so they don't nest
+	c.Lock()
+	defer c.Unlock()
+	// GC - randomly (courtesy of go's map iterator) pick some
+	// existing broadcasters and stop&remove them if their source peer
+	// is unknown. We stop as soon as we encounter a valid entry; the
+	// idea being that when there is little or no garbage then this
+	// executes close to O(1)[1], whereas when there is lots of
+	// garbage we remove it quickly.
+	//
+	// [1] TODO Unfortunately, due to the desire to avoid nested
+	// locks, instead of simply invoking Peers.Fetch(name) below, we
+	// have that Peers.Names() invocation above. That is O(n_peers) at
+	// best.
+	for name, broadcaster := range c.broadcasters {
+		if _, found := names[name]; !found {
+			delete(c.broadcasters, name)
+			broadcaster.Stop()
+		} else {
+			break
+		}
+	}
+	broadcaster, found := c.broadcasters[srcName]
+	if !found {
+		broadcaster = NewGossipSender(func(pending GossipData) { c.sendBroadcast(srcName, pending) })
+		c.broadcasters[srcName] = broadcaster
+		broadcaster.Start()
+	}
+	broadcaster.Send(update)
+	return nil
+}
+
+func (c *GossipChannel) sendBroadcast(srcName PeerName, update GossipData) {
+	c.routes.EnsureRecalculated()
+	nextHops := c.routes.BroadcastAll(srcName)
+	if len(nextHops) == 0 {
+		return
+	}
+	protocolMsg := ProtocolMsg{ProtocolGossipBroadcast, GobEncode(c.hash, srcName, update.Encode())}
+	// FIXME a single blocked connection can stall us
+	for _, conn := range c.ourself.ConnectionsTo(nextHops) {
+		conn.(ProtocolSender).SendProtocolMsg(protocolMsg)
+	}
+}
+
+func (c *GossipChannel) log(args ...interface{}) {
+	log.Println(append(append([]interface{}{}, "[gossip "+c.name+"]:"), args...)...)
+}
+
+// for testing
+
+// FIXME this doesn't actually guarantee everything has been sent
+// since a GossipSender may be in the process of sending and there is
+// no easy way for us to know when that has completed.
+func (router *Router) sendPendingGossip() {
+	for _, channel := range router.GossipChannels {
+		for _, sender := range channel.senders {
+			sender.flush()
+		}
+		for _, sender := range channel.broadcasters {
+			sender.flush()
+		}
+	}
+}
+
+func (sender *GossipSender) flush() {
+	for {
+		select {
+		case pending := <-sender.cell:
+			sender.send(pending)
+		default:
+			return
+		}
+	}
+}

router/gossip_test.go

@@ -0,0 +1,150 @@
+package router
+
+import (
+	wt "github.com/weaveworks/weave/testing"
+	"testing"
+	"time"
+)
+
+// TODO test gossip unicast and broadcast; atm we only test topology
+// gossip, which does not employ unicast or broadcast.
+
+type mockChannelConnection struct {
+	RemoteConnection
+	dest *Router
+}
+
+// Construct a "passive" Router, i.e. without any goroutines, except
+// for Routes and GossipSenders.
+func NewTestRouter(name PeerName) *Router {
+	router := NewRouter(RouterConfig{}, name, "")
+	// need to create a dummy channel otherwise tests hang on nil
+	// channels when the Router invoked ConnectionMaker.Refresh
+	router.ConnectionMaker.actionChan = make(chan ConnectionMakerAction, ChannelSize)
+	router.Routes.Start()
+	return router
+}
+
+func (conn *mockChannelConnection) SendProtocolMsg(protocolMsg ProtocolMsg) {
+	if err := conn.dest.handleGossip(protocolMsg.tag, protocolMsg.msg); err != nil {
+		panic(err)
+	}
+	conn.dest.sendPendingGossip()
+}
+
+func (router *Router) AddTestChannelConnection(r *Router) {
+	fromName := router.Ourself.Peer.Name
+	toName := r.Ourself.Peer.Name
+
+	fromPeer := NewPeer(fromName, "", router.Ourself.Peer.UID, 0)
+	toPeer := NewPeer(toName, "", r.Ourself.Peer.UID, 0)
+
+	r.Peers.FetchWithDefault(fromPeer)    // Has side-effect of incrementing refcount
+	router.Peers.FetchWithDefault(toPeer) //
+
+	conn := &mockChannelConnection{RemoteConnection{router.Ourself.Peer, toPeer, "", false, true}, r}
+	router.Ourself.handleAddConnection(conn)
+	router.Ourself.handleConnectionEstablished(conn)
+	router.sendPendingGossip()
+}
+
+func (router *Router) DeleteTestChannelConnection(r *Router) {
+	fromName := router.Ourself.Peer.Name
+	toName := r.Ourself.Peer.Name
+
+	fromPeer, _ := r.Peers.Fetch(fromName)
+	toPeer, _ := router.Peers.Fetch(toName)
+
+	r.Peers.Dereference(fromPeer)
+	router.Peers.Dereference(toPeer)
+
+	conn, _ := router.Ourself.ConnectionTo(toName)
+	router.Ourself.handleDeleteConnection(conn)
+	router.sendPendingGossip()
+}
+
+func TestGossipTopology(t *testing.T) {
+	wt.RunWithTimeout(t, 1*time.Second, func() {
+		implTestGossipTopology(t)
+	})
+}
+
+// Create a Peer representing the receiver router, with connections to
+// the routers supplied as arguments, carrying across all UID and
+// version information.
+func (router *Router) tp(routers ...*Router) *Peer {
+	peer := NewPeer(router.Ourself.Peer.Name, "", router.Ourself.Peer.UID, 0)
+	connections := make(map[PeerName]Connection)
+	for _, r := range routers {
+		p := NewPeer(r.Ourself.Peer.Name, "", r.Ourself.Peer.UID, r.Ourself.Peer.version)
+		connections[r.Ourself.Peer.Name] = newMockConnection(peer, p)
+	}
+	peer.version = router.Ourself.Peer.version
+	peer.connections = connections
+	return peer
+}
+
+// Check that the topology of router matches the peers and all of their connections
+func checkTopology(t *testing.T, router *Router, wantedPeers ...*Peer) {
+	checkTopologyPeers(t, true, router.Peers.allPeers(), wantedPeers...)
+}
+
+func implTestGossipTopology(t *testing.T) {
+	// Create some peers that will talk to each other
+	peer1Name, _ := PeerNameFromString("01:00:00:01:00:00")
+	peer2Name, _ := PeerNameFromString("02:00:00:02:00:00")
+	peer3Name, _ := PeerNameFromString("03:00:00:03:00:00")
+	r1 := NewTestRouter(peer1Name)
+	r2 := NewTestRouter(peer2Name)
+	r3 := NewTestRouter(peer3Name)
+
+	// Check state when they have no connections
+	checkTopology(t, r1, r1.tp())
+	checkTopology(t, r2, r2.tp())
+
+	// Now try adding some connections
+	r1.AddTestChannelConnection(r2)
+	checkTopology(t, r1, r1.tp(r2), r2.tp())
+	checkTopology(t, r2, r1.tp(r2), r2.tp())
+	r2.AddTestChannelConnection(r1)
+	checkTopology(t, r1, r1.tp(r2), r2.tp(r1))
+	checkTopology(t, r2, r1.tp(r2), r2.tp(r1))
+
+	// Currently, the connection from 2 to 3 is one-way only
+	r2.AddTestChannelConnection(r3)
+	checkTopology(t, r1, r1.tp(r2), r2.tp(r1, r3), r3.tp())
+	checkTopology(t, r2, r1.tp(r2), r2.tp(r1, r3), r3.tp())
+	// When r2 gossiped to r3, 1 was unreachable from r3 so it got removed from the
+	// list of peers, but remains referenced in the connection from 1 to 3.
+	checkTopology(t, r3, r2.tp(r1, r3), r3.tp())
+
+	// Add a connection from 3 to 1 and now r1 is reachable.
+	r3.AddTestChannelConnection(r1)
+	checkTopology(t, r1, r1.tp(r2), r2.tp(r1, r3), r3.tp(r1))
+	checkTopology(t, r2, r1.tp(r2), r2.tp(r1, r3), r3.tp(r1))
+	checkTopology(t, r3, r1.tp(), r2.tp(r1, r3), r3.tp(r1))
+
+	r1.AddTestChannelConnection(r3)
+	checkTopology(t, r1, r1.tp(r2, r3), r2.tp(r1, r3), r3.tp(r1))
+	checkTopology(t, r2, r1.tp(r2, r3), r2.tp(r1, r3), r3.tp(r1))
+	checkTopology(t, r3, r1.tp(r2, r3), r2.tp(r1, r3), r3.tp(r1))
+
+	// Drop the connection from 2 to 3
+	r2.DeleteTestChannelConnection(r3)
+	checkTopology(t, r1, r1.tp(r2, r3), r2.tp(r1), r3.tp(r1))
+	checkTopology(t, r2, r1.tp(r2, r3), r2.tp(r1))
+	checkTopology(t, r3, r1.tp(r2, r3), r2.tp(r1), r3.tp(r1))
+
+	// Drop the connection from 1 to 3
+	r1.DeleteTestChannelConnection(r3)
+	checkTopology(t, r1, r1.tp(r2), r2.tp(r1), r3.tp(r1))
+
+	checkTopology(t, r1, r1.tp(r2), r2.tp(r1), r3.tp(r1))
+	checkTopology(t, r2, r1.tp(r2), r2.tp(r1))
+	// r3 still thinks r1 has a connection to it
+	checkTopology(t, r3, r1.tp(r2, r3), r2.tp(r1), r3.tp(r1))
+
+	// On a timer, r3 will gossip to r1
+	r3.SendAllGossip()
+	checkTopology(t, r1, r1.tp(r2), r2.tp(r1), r3.tp(r1))
+}

router/handshake.go

@@ -0,0 +1,164 @@
+package router
+
+import (
+	"encoding/gob"
+	"encoding/hex"
+	"fmt"
+	"strconv"
+)
+
+type FieldValidator struct {
+	fields map[string]string
+	err    error
+}
+
+func NewFieldValidator(fields map[string]string) *FieldValidator {
+	return &FieldValidator{fields, nil}
+}
+
+func (fv *FieldValidator) Value(fieldName string) (string, error) {
+	if fv.err != nil {
+		return "", fv.err
+	}
+	val, found := fv.fields[fieldName]
+	if !found {
+		fv.err = fmt.Errorf("Field %s is missing", fieldName)
+		return "", fv.err
+	}
+	return val, nil
+}
+
+func (fv *FieldValidator) CheckEqual(fieldName, expectedValue string) error {
+	val, err := fv.Value(fieldName)
+	if err != nil {
+		return err
+	}
+	if val != expectedValue {
+		fv.err = fmt.Errorf("Field %s has wrong value; expected '%s', received '%s'", fieldName, expectedValue, val)
+		return fv.err
+	}
+	return nil
+}
+
+func (fv *FieldValidator) Err() error {
+	return fv.err
+}
+
+func (conn *LocalConnection) handshake(enc *gob.Encoder, dec *gob.Decoder, acceptNewPeer bool) error {
+	// We do not need to worry about locking in here as at this point
+	// the connection is not reachable by any go-routine other than
+	// ourself. Only when we add this connection to the conn.local
+	// peer will it be visible from multiple go-routines.
+
+	conn.extendReadDeadline()
+
+	localConnID := randUint64()
+	usingPassword := conn.Router.UsingPassword()
+	fv, private, err := conn.handshakeSendRecv(localConnID, usingPassword, enc, dec)
+	if err != nil {
+		return err
+	}
+	nameStr, _ := fv.Value("Name")
+	nickNameStr, _ := fv.Value("NickName")
+	uidStr, _ := fv.Value("UID")
+	remoteConnIDStr, _ := fv.Value("ConnID")
+	if err := fv.Err(); err != nil {
+		return err
+	}
+
+	name, err := PeerNameFromString(nameStr)
+	if err != nil {
+		return err
+	}
+	if !acceptNewPeer {
+		if _, found := conn.Router.Peers.Fetch(name); !found {
+			return fmt.Errorf("Found unknown remote name: %s at %s", name, conn.remoteTCPAddr)
+		}
+	}
+	if existingConn, found := conn.Router.Ourself.ConnectionTo(name); found && existingConn.Established() {
+		return fmt.Errorf("Already have connection to %s at %s", existingConn.Remote(), existingConn.RemoteTCPAddr())
+	}
+	uid, err := ParsePeerUID(uidStr)
+	if err != nil {
+		return err
+	}
+	remoteConnID, err := strconv.ParseUint(remoteConnIDStr, 10, 64)
+	if err != nil {
+		return err
+	}
+	conn.uid = localConnID ^ remoteConnID
+
+	remotePublicStr, rpErr := fv.Value("PublicKey")
+	if usingPassword {
+		if rpErr != nil {
+			return rpErr
+		}
+		remotePublicSlice, rpErr := hex.DecodeString(remotePublicStr)
+		if rpErr != nil {
+			return rpErr
+		}
+		remotePublic := [32]byte{}
+		for idx, elem := range remotePublicSlice {
+			remotePublic[idx] = elem
+		}
+		conn.SessionKey = FormSessionKey(&remotePublic, private, conn.Router.Password)
+		conn.tcpSender = NewEncryptedTCPSender(enc, conn.SessionKey, conn.outbound)
+		conn.Decryptor = NewNaClDecryptor(conn.SessionKey, conn.outbound)
+	} else {
+		if rpErr == nil {
+			return fmt.Errorf("Remote network is encrypted. Password required.")
+		}
+		conn.tcpSender = NewSimpleTCPSender(enc)
+		conn.Decryptor = NewNonDecryptor()
+	}
+
+	return conn.setRemote(NewPeer(name, nickNameStr, uid, 0))
+}
+
+func (conn *LocalConnection) handshakeSendRecv(localConnID uint64, usingPassword bool, enc *gob.Encoder, dec *gob.Decoder) (*FieldValidator, *[32]byte, error) {
+	versionStr := fmt.Sprint(ProtocolVersion)
+	handshakeSend := map[string]string{
+		"Protocol":        Protocol,
+		"ProtocolVersion": versionStr,
+		"PeerNameFlavour": PeerNameFlavour,
+		"Name":            conn.local.Name.String(),
+		"NickName":        conn.local.NickName,
+		"UID":             fmt.Sprint(conn.local.UID),
+		"ConnID":          fmt.Sprint(localConnID)}
+	handshakeRecv := map[string]string{}
+
+	var public, private *[32]byte
+	var err error
+	if usingPassword {
+		public, private, err = GenerateKeyPair()
+		if err != nil {
+			return nil, nil, err
+		}
+		handshakeSend["PublicKey"] = hex.EncodeToString(public[:])
+	}
+	enc.Encode(handshakeSend)
+
+	err = dec.Decode(&handshakeRecv)
+	if err != nil {
+		return nil, nil, err
+	}
+	fv := NewFieldValidator(handshakeRecv)
+	fv.CheckEqual("Protocol", Protocol)
+	fv.CheckEqual("ProtocolVersion", versionStr)
+	fv.CheckEqual("PeerNameFlavour", PeerNameFlavour)
+	return fv, private, nil
+}
+
+func (conn *LocalConnection) setRemote(toPeer *Peer) error {
+	toPeer = conn.Router.Peers.FetchWithDefault(toPeer)
+	switch toPeer {
+	case nil:
+		return fmt.Errorf("Connection appears to be with different version of a peer we already know of")
+	case conn.local:
+		conn.remote = toPeer // have to do assigment here to ensure Shutdown releases ref count
+		return fmt.Errorf("Cannot connect to ourself")
+	default:
+		conn.remote = toPeer
+		return nil
+	}
+}

router/handshake_test.go

@@ -0,0 +1,29 @@
+package router
+
+import (
+	wt "github.com/weaveworks/weave/testing"
+	"testing"
+)
+
+func TestFieldValidator(t *testing.T) {
+	testMap := map[string]string{"a": "a"}
+
+	fv := NewFieldValidator(testMap)
+	val, err := fv.Value("a")
+	wt.AssertNoErr(t, err)
+	wt.AssertNoErr(t, fv.Err())
+	wt.AssertEqualString(t, val, "a", "")
+	_, err = fv.Value("x")
+	wt.AssertFalse(t, err == nil || fv.Err() == nil, "Expected error")
+	_, err = fv.Value("a")
+	wt.AssertFalse(t, err == nil || fv.Err() == nil, "Previous error should be retained")
+
+	fv = NewFieldValidator(testMap)
+	err = fv.CheckEqual("a", "a")
+	wt.AssertNoErr(t, err)
+	wt.AssertNoErr(t, fv.Err())
+	err = fv.CheckEqual("a", "b")
+	wt.AssertFalse(t, err == nil || fv.Err() == nil, "Expected error")
+	err = fv.CheckEqual("a", "a")
+	wt.AssertFalse(t, err == nil || fv.Err() == nil, "Previous error should be retained")
+}

router/json.go

@@ -0,0 +1,98 @@
+package router
+
+import (
+	"encoding/json"
+	"fmt"
+	"time"
+)
+
+func (router *Router) StatusJSON(version, encryption string) ([]byte, error) {
+	return json.Marshal(struct {
+		Version    string
+		Encryption string
+		Name       string
+		NickName   string
+		Interface  string
+		Macs       *MacCache
+		Peers      *Peers
+		Routes     *Routes
+	}{version, encryption, router.Ourself.Name.String(), router.Ourself.NickName, fmt.Sprintf("%v", router.Iface), router.Macs, router.Peers, router.Routes})
+	// leaving out ConectionMaker due to async complexities
+}
+
+func (cache *MacCache) MarshalJSON() ([]byte, error) {
+	type cacheEntry struct {
+		Mac      string
+		Name     string
+		NickName string
+		LastSeen time.Time
+	}
+	var entries []*cacheEntry
+	for key, entry := range cache.table {
+		entries = append(entries, &cacheEntry{intmac(key).String(), entry.peer.Name.String(), entry.peer.NickName, entry.lastSeen})
+	}
+	return json.Marshal(entries)
+}
+
+func (peers *Peers) MarshalJSON() ([]byte, error) {
+	type p struct {
+		Name        string
+		NickName    string
+		UID         PeerUID
+		Version     uint64
+		Connections []Connection
+	}
+	var ps []*p
+	peers.ForEach(func(peer *Peer) {
+		var connections []Connection
+		if peer == peers.ourself.Peer {
+			for conn := range peers.ourself.Connections() {
+				connections = append(connections, conn)
+			}
+		} else {
+			// Modifying peer.connections requires a write lock on
+			// Peers, and since we are holding a read lock (due to the
+			// ForEach), access without locking the peer is safe.
+			for _, conn := range peer.connections {
+				connections = append(connections, conn)
+			}
+		}
+		ps = append(ps, &p{peer.Name.String(), peer.NickName, peer.UID, peer.version, connections})
+	})
+	return json.Marshal(ps)
+}
+
+func (routes *Routes) MarshalJSON() ([]byte, error) {
+	routes.RLock()
+	defer routes.RUnlock()
+	type uni struct {
+		Dest, Via PeerName
+	}
+	type broad struct {
+		Source PeerName
+		Via    []PeerName
+	}
+	var r struct {
+		Unicast   []*uni
+		Broadcast []*broad
+	}
+	for name, hop := range routes.unicast {
+		r.Unicast = append(r.Unicast, &uni{name, hop})
+	}
+	for name, hops := range routes.broadcast {
+		r.Broadcast = append(r.Broadcast, &broad{name, hops})
+	}
+	return json.Marshal(r)
+}
+
+func (conn *RemoteConnection) MarshalJSON() ([]byte, error) {
+	return json.Marshal(struct {
+		Name     string
+		NickName string
+		TCPAddr  string
+	}{conn.Remote().Name.String(), conn.Remote().NickName, conn.RemoteTCPAddr()})
+}
+
+func (name PeerName) MarshalJSON() ([]byte, error) {
+	return json.Marshal(name.String())
+}

router/local_peer.go

@@ -4,41 +4,43 @@ import (
 	"fmt"
 	"log"
 	"net"
+	"sync"
+	"time"
 )
 
 type LocalPeer struct {
+	sync.RWMutex
 	*Peer
-	Router    *Router
-	queryChan chan<- *PeerInteraction
+	router     *Router
+	actionChan chan<- LocalPeerAction
 }
 
-type PeerInteraction struct {
-	Interaction
-	payload interface{}
+type LocalPeerAction func()
+
+func NewLocalPeer(name PeerName, nickName string, router *Router) *LocalPeer {
+	return &LocalPeer{Peer: NewPeer(name, nickName, 0, 0), router: router}
 }
 
-func StartLocalPeer(name PeerName, router *Router) *LocalPeer {
-	peer := &LocalPeer{Peer: NewPeer(name, 0, 0), Router: router}
-	queryChan := make(chan *PeerInteraction, ChannelSize)
-	peer.queryChan = queryChan
-	go peer.queryLoop(queryChan)
-	return peer
+func (peer *LocalPeer) Start() {
+	actionChan := make(chan LocalPeerAction, ChannelSize)
+	peer.actionChan = actionChan
+	go peer.actorLoop(actionChan)
 }
 
 func (peer *LocalPeer) Forward(dstPeer *Peer, df bool, frame []byte, dec *EthernetDecoder) error {
 	return peer.Relay(peer.Peer, dstPeer, df, frame, dec)
 }
 
-func (peer *LocalPeer) Broadcast(df bool, frame []byte, dec *EthernetDecoder) error {
-	return peer.RelayBroadcast(peer.Peer, df, frame, dec)
+func (peer *LocalPeer) Broadcast(df bool, frame []byte, dec *EthernetDecoder) {
+	peer.RelayBroadcast(peer.Peer, df, frame, dec)
 }
 
 func (peer *LocalPeer) Relay(srcPeer, dstPeer *Peer, df bool, frame []byte, dec *EthernetDecoder) error {
-	relayPeerName, found := peer.Router.Routes.Unicast(dstPeer.Name)
+	relayPeerName, found := peer.router.Routes.Unicast(dstPeer.Name)
 	if !found {
 		// Not necessarily an error as there could be a race with the
 		// dst disappearing whilst the frame is in flight
-		log.Println("Received packet for unknown destination:", dstPeer.Name)
+		log.Println("Received packet for unknown destination:", dstPeer)
 		return nil
 	}
 	conn, found := peer.ConnectionTo(relayPeerName)
@@ -54,46 +56,70 @@ func (peer *LocalPeer) Relay(srcPeer, dstPeer *Peer, df bool, frame []byte, dec
 		dec)
 }
 
-func (peer *LocalPeer) RelayBroadcast(srcPeer *Peer, df bool, frame []byte, dec *EthernetDecoder) error {
-	nextHops := peer.Router.Routes.Broadcast(srcPeer.Name)
+func (peer *LocalPeer) RelayBroadcast(srcPeer *Peer, df bool, frame []byte, dec *EthernetDecoder) {
+	nextHops := peer.router.Routes.Broadcast(srcPeer.Name)
 	if len(nextHops) == 0 {
-		return nil
+		return
 	}
-	// We must not hold a read lock on peer during the conn.Forward
-	// below, since that is a potentially blocking operation (e.g. if
-	// the channel is full).
-	nextConns := make([]*LocalConnection, 0, len(nextHops))
-	peer.RLock()
-	for _, hopName := range nextHops {
-		conn, found := peer.connections[hopName]
-		// Again, !found could just be due to a race.
-		if found {
-			nextConns = append(nextConns, conn.(*LocalConnection))
-		}
-	}
-	peer.RUnlock()
-	var err error
-	for _, conn := range nextConns {
-		err = conn.Forward(df, &ForwardedFrame{
+	for _, conn := range peer.ConnectionsTo(nextHops) {
+		err := conn.(*LocalConnection).Forward(df, &ForwardedFrame{
 			srcPeer: srcPeer,
 			dstPeer: conn.Remote(),
 			frame:   frame},
 			dec)
 		if err != nil {
-			return err
+			if ftbe, ok := err.(FrameTooBigError); ok {
+				log.Printf("dropping too big DF broadcast frame (%v -> %v): PMTU= %v\n", dec.ip.DstIP, dec.ip.SrcIP, ftbe.EPMTU)
+			} else {
+				log.Println(err)
+			}
 		}
 	}
-	return nil
+}
+
+func (peer *LocalPeer) Info() string {
+	peer.RLock()
+	defer peer.RUnlock()
+	return peer.Peer.Info()
+}
+
+func (peer *LocalPeer) Connections() ConnectionSet {
+	connections := make(ConnectionSet)
+	peer.RLock()
+	defer peer.RUnlock()
+	for _, conn := range peer.connections {
+		connections[conn] = void
+	}
+	return connections
+}
+
+func (peer *LocalPeer) ConnectionTo(name PeerName) (Connection, bool) {
+	peer.RLock()
+	defer peer.RUnlock()
+	conn, found := peer.connections[name]
+	return conn, found // yes, you really can't inline that. FFS.
+}
+
+func (peer *LocalPeer) ConnectionsTo(names []PeerName) []Connection {
+	conns := make([]Connection, 0, len(names))
+	peer.RLock()
+	defer peer.RUnlock()
+	for _, name := range names {
+		conn, found := peer.connections[name]
+		// Again, !found could just be due to a race.
+		if found {
+			conns = append(conns, conn)
+		}
+	}
+	return conns
 }
 
 func (peer *LocalPeer) CreateConnection(peerAddr string, acceptNewPeer bool) error {
 	if err := peer.checkConnectionLimit(); err != nil {
 		return err
 	}
-	// We're dialing the remote so that means connections will come from random ports
-	addrStr := NormalisePeerAddr(peerAddr)
-	tcpAddr, tcpErr := net.ResolveTCPAddr("tcp4", addrStr)
-	udpAddr, udpErr := net.ResolveUDPAddr("udp4", addrStr)
+	tcpAddr, tcpErr := net.ResolveTCPAddr("tcp4", peerAddr)
+	udpAddr, udpErr := net.ResolveUDPAddr("udp4", peerAddr)
 	if tcpErr != nil || udpErr != nil {
 		// they really should have the same value, but just in case...
 		if tcpErr == nil {
@@ -105,74 +131,55 @@ func (peer *LocalPeer) CreateConnection(peerAddr string, acceptNewPeer bool) err
 	if err != nil {
 		return err
 	}
-	connRemote := NewRemoteConnection(peer.Peer, nil, tcpConn.RemoteAddr().String())
-	NewLocalConnection(connRemote, acceptNewPeer, tcpConn, udpAddr, peer.Router)
+	connRemote := NewRemoteConnection(peer.Peer, nil, tcpConn.RemoteAddr().String(), true, false)
+	connLocal := NewLocalConnection(connRemote, tcpConn, udpAddr, peer.router)
+	connLocal.Start(acceptNewPeer)
 	return nil
 }
 
 // ACTOR client API
 
-const (
-	PAddConnection         = iota
-	PBroadcastTCP          = iota
-	PDeleteConnection      = iota
-	PConnectionEstablished = iota
-)
+// Sync.
+func (peer *LocalPeer) AddConnection(conn *LocalConnection) error {
+	resultChan := make(chan error)
+	peer.actionChan <- func() {
+		resultChan <- peer.handleAddConnection(conn)
+	}
+	return <-resultChan
+}
 
-// Async: rely on the peer to shut us down if we shouldn't be adding
-// ourselves, so therefore this can be async
-func (peer *LocalPeer) AddConnection(conn *LocalConnection) {
-	peer.queryChan <- &PeerInteraction{
-		Interaction: Interaction{code: PAddConnection},
-		payload:     conn}
+// Async.
+func (peer *LocalPeer) ConnectionEstablished(conn *LocalConnection) {
+	peer.actionChan <- func() {
+		peer.handleConnectionEstablished(conn)
+	}
 }
 
 // Sync.
 func (peer *LocalPeer) DeleteConnection(conn *LocalConnection) {
 	resultChan := make(chan interface{})
-	peer.queryChan <- &PeerInteraction{
-		Interaction: Interaction{code: PDeleteConnection, resultChan: resultChan},
-		payload:     conn}
+	peer.actionChan <- func() {
+		peer.handleDeleteConnection(conn)
+		resultChan <- nil
+	}
 	<-resultChan
 }
 
-// Async.
-func (peer *LocalPeer) ConnectionEstablished(conn *LocalConnection) {
-	peer.queryChan <- &PeerInteraction{
-		Interaction: Interaction{code: PConnectionEstablished},
-		payload:     conn}
-}
-
-// Async.
-func (peer *LocalPeer) BroadcastTCP(msg []byte) {
-	peer.queryChan <- &PeerInteraction{
-		Interaction: Interaction{code: PBroadcastTCP},
-		payload:     msg}
-}
-
 // ACTOR server
 
-func (peer *LocalPeer) queryLoop(queryChan <-chan *PeerInteraction) {
+func (peer *LocalPeer) actorLoop(actionChan <-chan LocalPeerAction) {
+	gossipTimer := time.Tick(GossipInterval)
 	for {
-		query, ok := <-queryChan
-		if !ok {
-			return
-		}
-		switch query.code {
-		case PAddConnection:
-			peer.handleAddConnection(query.payload.(*LocalConnection))
-		case PDeleteConnection:
-			peer.handleDeleteConnection(query.payload.(*LocalConnection))
-			query.resultChan <- nil
-		case PConnectionEstablished:
-			peer.handleConnectionEstablished(query.payload.(*LocalConnection))
-		case PBroadcastTCP:
-			peer.handleBroadcastTCP(query.payload.([]byte))
+		select {
+		case action := <-actionChan:
+			action()
+		case <-gossipTimer:
+			peer.router.SendAllGossip()
 		}
 	}
 }
 
-func (peer *LocalPeer) handleAddConnection(conn *LocalConnection) {
+func (peer *LocalPeer) handleAddConnection(conn Connection) error {
 	if peer.Peer != conn.Local() {
 		log.Fatal("Attempt made to add connection to peer where peer is not the source of connection")
 	}
@@ -180,39 +187,54 @@ func (peer *LocalPeer) handleAddConnection(conn *LocalConnection) {
 		log.Fatal("Attempt made to add connection to peer with unknown remote peer")
 	}
 	toName := conn.Remote().Name
-	dupErr := fmt.Errorf("Multiple connections to %s added to %s", toName, peer.Name)
+	dupErr := fmt.Errorf("Multiple connections to %s added to %s", conn.Remote(), peer.String())
 	// deliberately non symmetrical
 	if dupConn, found := peer.connections[toName]; found {
 		if dupConn == conn {
-			return
+			return nil
 		}
-		// conn.UID is used as the tie breaker here, in the
-		// knowledge that both sides will make the same decision.
-		dupConnLocal := dupConn.(*LocalConnection)
-		if conn.UID == dupConnLocal.UID {
+		switch conn.BreakTie(dupConn) {
+		case TieBreakWon:
+			dupConn.Shutdown(dupErr)
+			peer.handleDeleteConnection(dupConn)
+		case TieBreakLost:
+			return dupErr
+		case TieBreakTied:
 			// oh good grief. Sod it, just kill both of them.
-			conn.CheckFatal(dupErr)
-			dupConnLocal.CheckFatal(dupErr)
-			peer.handleDeleteConnection(dupConnLocal)
-			return
-		} else if conn.UID < dupConnLocal.UID {
-			dupConnLocal.CheckFatal(dupErr)
-			peer.handleDeleteConnection(dupConnLocal)
-		} else {
-			conn.CheckFatal(dupErr)
-			return
+			dupConn.Shutdown(dupErr)
+			peer.handleDeleteConnection(dupConn)
+			return dupErr
 		}
 	}
 	if err := peer.checkConnectionLimit(); err != nil {
-		conn.CheckFatal(err)
-		return
+		return err
 	}
-	peer.Lock()
-	peer.connections[toName] = conn
-	peer.Unlock()
+	_, isConnectedPeer := peer.router.Routes.Unicast(toName)
+	peer.addConnection(conn)
+	if isConnectedPeer {
+		conn.Log("connection added")
+	} else {
+		conn.Log("connection added (new peer)")
+		peer.router.SendAllGossipDown(conn)
+	}
+	peer.broadcastPeerUpdate(conn.Remote())
+	return nil
 }
 
-func (peer *LocalPeer) handleDeleteConnection(conn *LocalConnection) {
+func (peer *LocalPeer) handleConnectionEstablished(conn Connection) {
+	if peer.Peer != conn.Local() {
+		log.Fatal("Peer informed of active connection where peer is not the source of connection")
+	}
+	if dupConn, found := peer.connections[conn.Remote().Name]; !found || conn != dupConn {
+		conn.Shutdown(fmt.Errorf("Cannot set unknown connection active"))
+		return
+	}
+	peer.connectionEstablished(conn)
+	conn.Log("connection fully established")
+	peer.broadcastPeerUpdate()
+}
+
+func (peer *LocalPeer) handleDeleteConnection(conn Connection) {
 	if peer.Peer != conn.Local() {
 		log.Fatal("Attempt made to delete connection from peer where peer is not the source of connection")
 	}
@@ -223,53 +245,51 @@ func (peer *LocalPeer) handleDeleteConnection(conn *LocalConnection) {
 	if connFound, found := peer.connections[toName]; !found || connFound != conn {
 		return
 	}
-	peer.Lock()
-	delete(peer.connections, toName)
-	peer.Unlock()
-	broadcast := false
-	if conn.Established() {
-		peer.Lock()
-		peer.version += 1
-		peer.Unlock()
-		broadcast = true
-	}
+	peer.deleteConnection(conn)
+	conn.Log("connection deleted")
 	// Must do garbage collection first to ensure we don't send out an
 	// update with unreachable peers (can cause looping)
-	peer.Router.Peers.GarbageCollect()
-	if broadcast {
-		peer.broadcastPeerUpdate()
-	}
+	peer.router.Peers.GarbageCollect()
+	peer.broadcastPeerUpdate()
 }
 
-func (peer *LocalPeer) handleConnectionEstablished(conn *LocalConnection) {
-	if peer.Peer != conn.Local() {
-		log.Fatal("Peer informed of active connection where peer is not the source of connection")
-	}
-	if dupConn, found := peer.connections[conn.Remote().Name]; !found || conn != dupConn {
-		conn.CheckFatal(fmt.Errorf("Cannot set unknown connection active"))
-		return
-	}
-	peer.Lock()
-	peer.version += 1
-	peer.Unlock()
-	log.Println("Peer", peer.Name, "established active connection to remote peer", conn.Remote().Name, "at", conn.RemoteTCPAddr())
-	peer.broadcastPeerUpdate(conn.Remote())
-}
-
-func (peer *LocalPeer) handleBroadcastTCP(msg []byte) {
-	peer.ForEachConnection(func(_ PeerName, conn Connection) {
-		conn.(*LocalConnection).SendTCP(msg)
-	})
-}
+// helpers
 
 func (peer *LocalPeer) broadcastPeerUpdate(peers ...*Peer) {
-	peer.Router.Routes.Recalculate()
-	peer.handleBroadcastTCP(Concat(ProtocolUpdateByte, EncodePeers(append(peers, peer.Peer)...)))
+	peer.router.Routes.Recalculate()
+	peer.router.TopologyGossip.GossipBroadcast(NewTopologyGossipData(peer.router.Peers, append(peers, peer.Peer)...))
 }
 
 func (peer *LocalPeer) checkConnectionLimit() error {
-	if 0 != peer.Router.ConnLimit && peer.ConnectionCount() >= peer.Router.ConnLimit {
-		return fmt.Errorf("Connection limit reached (%v)", peer.Router.ConnLimit)
+	limit := peer.router.ConnLimit
+	if 0 != limit && peer.connectionCount() >= limit {
+		return fmt.Errorf("Connection limit reached (%v)", limit)
 	}
 	return nil
 }
+
+func (peer *LocalPeer) addConnection(conn Connection) {
+	peer.Lock()
+	defer peer.Unlock()
+	peer.connections[conn.Remote().Name] = conn
+	peer.version++
+}
+
+func (peer *LocalPeer) deleteConnection(conn Connection) {
+	peer.Lock()
+	defer peer.Unlock()
+	delete(peer.connections, conn.Remote().Name)
+	peer.version++
+}
+
+func (peer *LocalPeer) connectionEstablished(conn Connection) {
+	peer.Lock()
+	defer peer.Unlock()
+	peer.version++
+}
+
+func (peer *LocalPeer) connectionCount() int {
+	peer.RLock()
+	defer peer.RUnlock()
+	return len(peer.connections)
+}

router/mac_cache.go

@@ -22,12 +22,14 @@ type MacCache struct {
 }
 
 func NewMacCache(maxAge time.Duration, onExpiry func(net.HardwareAddr, *Peer)) *MacCache {
-	res := &MacCache{
+	return &MacCache{
 		table:    make(map[uint64]*MacCacheEntry),
 		maxAge:   maxAge,
 		onExpiry: onExpiry}
-	res.setExpiryTimer()
-	return res
+}
+
+func (cache *MacCache) Start() {
+	cache.setExpiryTimer()
 }
 
 func (cache *MacCache) Enter(mac net.HardwareAddr, peer *Peer) bool {
@@ -38,9 +40,8 @@ func (cache *MacCache) Enter(mac net.HardwareAddr, peer *Peer) bool {
 	if found && entry.peer == peer && now.Before(entry.lastSeen.Add(cache.maxAge/10)) {
 		cache.RUnlock()
 		return false
-	} else {
-		cache.RUnlock()
 	}
+	cache.RUnlock()
 	cache.Lock()
 	defer cache.Unlock()
 	entry, found = cache.table[key]
@@ -88,7 +89,7 @@ func (cache *MacCache) String() string {
 	cache.RLock()
 	defer cache.RUnlock()
 	for key, entry := range cache.table {
-		buf.WriteString(fmt.Sprintf("%v -> %s (%v)\n", intmac(key), entry.peer.Name, entry.lastSeen))
+		fmt.Fprintf(&buf, "%v -> %s (%v)\n", intmac(key), entry.peer, entry.lastSeen)
 	}
 	return buf.String()
 }

router/mocks_test.go

@@ -0,0 +1,113 @@
+// No mocks are tested by this file.
+//
+// It supplies some mock implementations to other unit tests, and is
+// named "...test.go" so it is only compiled under `go test`.
+
+package router
+
+import (
+	wt "github.com/weaveworks/weave/testing"
+	"testing"
+)
+
+// Add to peers a connection from peers.ourself to p
+func (peers *Peers) AddTestConnection(p *Peer) {
+	toPeer := NewPeer(p.Name, "", p.UID, 0)
+	peers.FetchWithDefault(toPeer) // Has side-effect of incrementing refcount
+	conn := newMockConnection(peers.ourself.Peer, toPeer)
+	peers.ourself.addConnection(conn)
+	peers.ourself.connectionEstablished(conn)
+}
+
+// Add to peers a connection from p1 to p2
+func (peers *Peers) AddTestRemoteConnection(p1, p2 *Peer) {
+	fromName := p1.Name
+	fromPeer := NewPeer(fromName, "", p1.UID, 0)
+	fromPeer = peers.FetchWithDefault(fromPeer)
+	toName := p2.Name
+	toPeer := NewPeer(toName, "", p2.UID, 0)
+	toPeer = peers.FetchWithDefault(toPeer)
+	peers.ourself.addConnection(&RemoteConnection{fromPeer, toPeer, "", false, false})
+}
+
+func (peers *Peers) DeleteTestConnection(p *Peer) {
+	toName := p.Name
+	toPeer, _ := peers.Fetch(toName)
+	peers.Dereference(toPeer)
+	conn, _ := peers.ourself.ConnectionTo(toName)
+	peers.ourself.deleteConnection(conn)
+}
+
+// mockConnection used in testing is very similar to a
+// RemoteConnection, without the RemoteTCPAddr(). We are making it a
+// separate type in order to distinguish what is created by the test
+// from what is created by the real code.
+func newMockConnection(from, to *Peer) Connection {
+	type mockConnection struct{ RemoteConnection }
+	return &mockConnection{RemoteConnection{from, to, "", false, false}}
+}
+
+func checkEqualConns(t *testing.T, ourName PeerName, got, wanted map[PeerName]Connection) {
+	checkConns := make(PeerNameSet)
+	for _, conn := range wanted {
+		checkConns[conn.Remote().Name] = void
+	}
+	for _, conn := range got {
+		remoteName := conn.Remote().Name
+		if _, found := checkConns[remoteName]; found {
+			delete(checkConns, remoteName)
+		} else {
+			wt.Fatalf(t, "Unexpected connection from %s to %s", ourName, remoteName)
+		}
+	}
+	if len(checkConns) > 0 {
+		wt.Fatalf(t, "Expected connections not found: from %s to %v", ourName, checkConns)
+	}
+}
+
+// Get all the peers from a Peers in a slice
+func (peers *Peers) allPeers() []*Peer {
+	var res []*Peer
+	for _, peer := range peers.table {
+		res = append(res, peer)
+	}
+	return res
+}
+
+func (peers *Peers) allPeersExcept(excludeName PeerName) []*Peer {
+	res := peers.allPeers()
+	for i, peer := range res {
+		if peer.Name == excludeName {
+			return append(res[:i], res[i+1:]...)
+		}
+	}
+	return res
+}
+
+// Check that the peers slice matches the wanted peers
+func checkPeerArray(t *testing.T, peers []*Peer, wantedPeers ...*Peer) {
+	checkTopologyPeers(t, false, peers, wantedPeers...)
+}
+
+// Check that the peers slice matches the wanted peers and optionally
+// all of their connections
+func checkTopologyPeers(t *testing.T, checkConns bool, peers []*Peer, wantedPeers ...*Peer) {
+	check := make(map[PeerName]*Peer)
+	for _, peer := range wantedPeers {
+		check[peer.Name] = peer
+	}
+	for _, peer := range peers {
+		name := peer.Name
+		if wantedPeer, found := check[name]; found {
+			if checkConns {
+				checkEqualConns(t, name, peer.connections, wantedPeer.connections)
+			}
+			delete(check, name)
+		} else {
+			wt.Fatalf(t, "Unexpected peer: %s", name)
+		}
+	}
+	if len(check) > 0 {
+		wt.Fatalf(t, "Expected peers not found: %v", check)
+	}
+}

router/pcap.go

@@ -2,6 +2,7 @@ package router
 
 import (
 	"code.google.com/p/gopacket/pcap"
+	"fmt"
 )
 
 type PcapIO struct {
@@ -44,6 +45,16 @@ func newPcapIO(ifName string, promisc bool, snaplen int, bufSz int) (handle *Pca
 		return
 	}
 	if err = inactive.SetImmediateMode(true); err != nil {
+		// If gopacket is compiled against an older pcap.h that
+		// doesn't have pcap_set_immediate_mode, it supplies a dummy
+		// definition that always returns PCAP_ERROR.  That becomes
+		// "Generic error", which is not very helpful.  The real
+		// pcap_set_immediate_mode never returns PCAP_ERROR, so this
+		// turns it into a more informative message.
+		if fmt.Sprint(err) == "Generic error" {
+			err = fmt.Errorf("compiled against an old version of libpcap; please compile against libpcap-1.5.0 or later")
+		}
+
 		return
 	}
 	if err = inactive.SetBufferSize(bufSz); err != nil {

router/peer.go

@@ -3,87 +3,47 @@ package router
 import (
 	"fmt"
 	"sort"
-	"sync"
+	"strconv"
 )
 
+type PeerUID uint64
+
+func ParsePeerUID(s string) (PeerUID, error) {
+	uid, err := strconv.ParseUint(s, 10, 64)
+	return PeerUID(uid), err
+}
+
 type Peer struct {
-	sync.RWMutex
 	Name          PeerName
 	NameByte      []byte
-	UID           uint64
+	NickName      string
+	UID           PeerUID
 	version       uint64
-	localRefCount uint64
+	localRefCount uint64 // maintained by Peers
 	connections   map[PeerName]Connection
 }
 
-func NewPeer(name PeerName, uid uint64, version uint64) *Peer {
+type ConnectionSet map[Connection]struct{}
+
+func NewPeer(name PeerName, nickName string, uid PeerUID, version uint64) *Peer {
 	if uid == 0 {
-		uid = randUint64()
+		uid = PeerUID(randUint64())
 	}
 	return &Peer{
 		Name:        name,
 		NameByte:    name.Bin(),
+		NickName:    nickName,
 		UID:         uid,
 		version:     version,
 		connections: make(map[PeerName]Connection)}
 }
 
 func (peer *Peer) String() string {
-	peer.RLock()
-	defer peer.RUnlock()
-	return fmt.Sprint("Peer ", peer.Name, " (v", peer.version, ") (UID ", peer.UID, ")")
+	return fmt.Sprint(peer.Name, "(", peer.NickName, ")")
 }
 
-func (peer *Peer) Version() uint64 {
-	peer.RLock()
-	defer peer.RUnlock()
-	return peer.version
-}
-
-func (peer *Peer) IncrementLocalRefCount() {
-	peer.Lock()
-	defer peer.Unlock()
-	peer.localRefCount += 1
-}
-
-func (peer *Peer) DecrementLocalRefCount() {
-	peer.Lock()
-	defer peer.Unlock()
-	peer.localRefCount -= 1
-}
-
-func (peer *Peer) IsLocallyReferenced() bool {
-	peer.RLock()
-	defer peer.RUnlock()
-	return peer.localRefCount != 0
-}
-
-func (peer *Peer) ConnectionCount() int {
-	peer.RLock()
-	defer peer.RUnlock()
-	return len(peer.connections)
-}
-
-func (peer *Peer) ConnectionTo(name PeerName) (Connection, bool) {
-	peer.RLock()
-	defer peer.RUnlock()
-	conn, found := peer.connections[name]
-	return conn, found // yes, you really can't inline that. FFS.
-}
-
-func (peer *Peer) ForEachConnection(fun func(PeerName, Connection)) {
-	peer.RLock()
-	defer peer.RUnlock()
-	for name, conn := range peer.connections {
-		fun(name, conn)
-	}
-}
-
-func (peer *Peer) SetVersionAndConnections(version uint64, connections map[PeerName]Connection) {
-	peer.Lock()
-	defer peer.Unlock()
-	peer.version = version
-	peer.connections = connections
+func (peer *Peer) Info() string {
+	return fmt.Sprint(peer.String(), " (v", peer.version, ") (UID ", peer.UID, ")")
 }
 
 // Calculate the routing table from this peer to all peers reachable
@@ -99,21 +59,17 @@ func (peer *Peer) SetVersionAndConnections(version uint64, connections map[PeerN
 // same result. This is important since otherwise we risk message loss
 // or routing cycles.
 //
-// When the 'symmetric' flag is set, only symmetric connections are
-// considered, i.e. where both sides indicate they have a connection
-// to the other.
+// When the 'establishedAndSymmetric' flag is set, only connections
+// that are marked as 'established' and are symmetric (i.e. where both
+// sides indicate they have a connection to the other) are considered.
 //
 // When a non-nil stopAt peer is supplied, the widening stops when it
 // reaches that peer. The boolean return indicates whether that has
 // happened.
 //
-// We acquire read locks on peers as we encounter them during the
-// traversal. This prevents the connectivity graph from changing
-// underneath us in ways that would invalidate the result. Thus the
-// answer returned may be out of date, but never inconsistent.
-func (peer *Peer) Routes(stopAt *Peer, symmetric bool) (bool, map[PeerName]PeerName) {
-	peer.RLock()
-	defer peer.RUnlock()
+// NB: This function should generally be invoked while holding a read
+// lock on Peers and LocalPeer.
+func (peer *Peer) Routes(stopAt *Peer, establishedAndSymmetric bool) (bool, map[PeerName]PeerName) {
 	routes := make(map[PeerName]PeerName)
 	routes[peer.Name] = UnknownPeerName
 	nextWorklist := []*Peer{peer}
@@ -125,16 +81,10 @@ func (peer *Peer) Routes(stopAt *Peer, symmetric bool) (bool, map[PeerName]PeerN
 			if curPeer == stopAt {
 				return true, routes
 			}
-			curName := curPeer.Name
-			for remoteName, conn := range curPeer.connections {
-				if _, found := routes[remoteName]; found {
-					continue
-				}
-				remote := conn.Remote()
-				remote.RLock()
-				if _, found := remote.connections[curName]; !symmetric || found {
-					defer remote.RUnlock()
-					nextWorklist = append(nextWorklist, remote)
+			curPeer.ForEachConnectedPeer(establishedAndSymmetric, routes,
+				func(remotePeer *Peer) {
+					nextWorklist = append(nextWorklist, remotePeer)
+					remoteName := remotePeer.Name
 					// We now know how to get to remoteName: the same
 					// way we get to curPeer. Except, if curPeer is
 					// the starting peer in which case we know we can
@@ -142,13 +92,25 @@ func (peer *Peer) Routes(stopAt *Peer, symmetric bool) (bool, map[PeerName]PeerN
 					if curPeer == peer {
 						routes[remoteName] = remoteName
 					} else {
-						routes[remoteName] = routes[curName]
+						routes[remoteName] = routes[curPeer.Name]
 					}
-				} else {
-					remote.RUnlock()
-				}
-			}
+				})
 		}
 	}
 	return false, routes
 }
+
+func (peer *Peer) ForEachConnectedPeer(establishedAndSymmetric bool, exclude map[PeerName]PeerName, f func(*Peer)) {
+	for remoteName, conn := range peer.connections {
+		if establishedAndSymmetric && !conn.Established() {
+			continue
+		}
+		if _, found := exclude[remoteName]; found {
+			continue
+		}
+		remotePeer := conn.Remote()
+		if remoteConn, found := remotePeer.connections[peer.Name]; !establishedAndSymmetric || (found && remoteConn.Established()) {
+			f(remotePeer)
+		}
+	}
+}

router/peer_name_hash.go

@@ -2,6 +2,7 @@
 
 // Let peer names be sha256 hashes...of anything (as long as it's
 // unique).
+
 package router
 
 import (

router/peer_name_mac.go

@@ -14,6 +14,7 @@
 // Note that we do not mandate *what* MAC should be used as the peer
 // name. In particular it doesn't actually have to be the MAC of, say,
 // the network interface the peer is sniffing on.
+
 package router
 
 import (

router/peers.go

@@ -10,35 +10,54 @@ import (
 
 type Peers struct {
 	sync.RWMutex
-	ourself *Peer
-	macs    *MacCache
+	ourself *LocalPeer
 	table   map[PeerName]*Peer
 	onGC    func(*Peer)
 }
 
-func NewPeers(ourself *Peer, macs *MacCache, onGC func(*Peer)) *Peers {
+type UnknownPeerError struct {
+	Name PeerName
+}
+
+type NameCollisionError struct {
+	Name PeerName
+}
+
+type PeerNameSet map[PeerName]struct{}
+
+type PeerSummary struct {
+	NameByte []byte
+	NickName string
+	UID      PeerUID
+	Version  uint64
+}
+
+type ConnectionSummary struct {
+	NameByte      []byte
+	RemoteTCPAddr string
+	Outbound      bool
+	Established   bool
+}
+
+func NewPeers(ourself *LocalPeer, onGC func(*Peer)) *Peers {
 	return &Peers{
 		ourself: ourself,
-		macs:    macs,
 		table:   make(map[PeerName]*Peer),
 		onGC:    onGC}
 }
 
 func (peers *Peers) FetchWithDefault(peer *Peer) *Peer {
-	peers.RLock()
-	res, found := peers.fetchAlias(peer)
-	peers.RUnlock()
-	if found {
-		return res
-	}
 	peers.Lock()
 	defer peers.Unlock()
-	res, found = peers.fetchAlias(peer)
-	if found {
-		return res
+	if existingPeer, found := peers.table[peer.Name]; found {
+		if existingPeer.UID != peer.UID {
+			return nil
+		}
+		existingPeer.localRefCount++
+		return existingPeer
 	}
 	peers.table[peer.Name] = peer
-	peer.IncrementLocalRefCount()
+	peer.localRefCount++
 	return peer
 }
 
@@ -49,21 +68,33 @@ func (peers *Peers) Fetch(name PeerName) (*Peer, bool) {
 	return peer, found // GRRR, why can't I inline this!?
 }
 
-func (peers *Peers) ForEach(fun func(PeerName, *Peer)) {
+func (peers *Peers) Dereference(peer *Peer) {
+	peers.Lock()
+	defer peers.Unlock()
+	peer.localRefCount--
+}
+
+func (peers *Peers) ForEach(fun func(*Peer)) {
 	peers.RLock()
 	defer peers.RUnlock()
-	for name, peer := range peers.table {
-		fun(name, peer)
+	for _, peer := range peers.table {
+		fun(peer)
 	}
 }
 
-func (peers *Peers) ApplyUpdate(update []byte) ([]byte, error) {
+// Merge an incoming update with our own topology.
+//
+// We add peers hitherto unknown to us, and update peers for which the
+// update contains a more recent version than known to us. The return
+// value is a) a representation of the received update, and b) an
+// "improved" update containing just these new/updated elements.
+func (peers *Peers) ApplyUpdate(update []byte) (PeerNameSet, PeerNameSet, error) {
 	peers.Lock()
 
 	newPeers, decodedUpdate, decodedConns, err := peers.decodeUpdate(update)
 	if err != nil {
 		peers.Unlock()
-		return nil, err
+		return nil, nil, err
 	}
 
 	// By this point, we know the update doesn't refer to any peers we
@@ -83,20 +114,34 @@ func (peers *Peers) ApplyUpdate(update []byte) ([]byte, error) {
 	// Don't need to hold peers lock any longer
 	peers.Unlock()
 
-	return encodePeersMap(newUpdate), nil
+	updateNames := make(PeerNameSet)
+	for _, peer := range decodedUpdate {
+		updateNames[peer.Name] = void
+	}
+	return updateNames, setFromPeersMap(newUpdate), nil
 }
 
-func (peers *Peers) EncodeAllPeers() []byte {
+func (peers *Peers) Names() PeerNameSet {
 	peers.RLock()
 	defer peers.RUnlock()
-	return encodePeersMap(peers.table)
+	return setFromPeersMap(peers.table)
 }
 
-func EncodePeers(peers ...*Peer) []byte {
+func (peers *Peers) EncodePeers(names PeerNameSet) []byte {
 	buf := new(bytes.Buffer)
 	enc := gob.NewEncoder(buf)
-	for _, peer := range peers {
-		peer.encodePeer(enc)
+	peers.RLock()
+	defer peers.RUnlock()
+	peerList := make([]*Peer, 0, len(names))
+	for name := range names {
+		if peer, found := peers.table[name]; found {
+			if peer == peers.ourself.Peer {
+				peers.ourself.Peer.Encode(enc)
+			} else {
+				peer.Encode(enc)
+			}
+			peerList = append(peerList, peer)
+		}
 	}
 	return buf.Bytes()
 }
@@ -109,71 +154,76 @@ func (peers *Peers) GarbageCollect() []*Peer {
 
 func (peers *Peers) String() string {
 	var buf bytes.Buffer
-	peers.ForEach(func(name PeerName, peer *Peer) {
-		buf.WriteString(fmt.Sprint(peer, "\n"))
-		peer.ForEachConnection(func(remoteName PeerName, conn Connection) {
-			buf.WriteString(fmt.Sprintf("   -> %v [%v]\n", remoteName, conn.RemoteTCPAddr()))
-		})
+	printConnection := func(conn Connection) {
+		established := ""
+		if !conn.Established() {
+			established = " (unestablished)"
+		}
+		fmt.Fprintf(&buf, "   -> %s [%v%s]\n", conn.Remote(), conn.RemoteTCPAddr(), established)
+	}
+	peers.ForEach(func(peer *Peer) {
+		if peer == peers.ourself.Peer {
+			fmt.Fprintln(&buf, peers.ourself.Info())
+			for conn := range peers.ourself.Connections() {
+				printConnection(conn)
+			}
+		} else {
+			fmt.Fprintln(&buf, peer.Info())
+			// Modifying peer.connections requires a write lock on
+			// Peers, and since we are holding a read lock (due to the
+			// ForEach), access without locking the peer is safe.
+			for _, conn := range peer.connections {
+				printConnection(conn)
+			}
+		}
 	})
 	return buf.String()
 }
 
-func (peers *Peers) fetchAlias(peer *Peer) (*Peer, bool) {
-	if existingPeer, found := peers.table[peer.Name]; found {
-		if existingPeer.UID == peer.UID {
-			existingPeer.IncrementLocalRefCount()
-			return existingPeer, true
-		} else {
-			return nil, true
-		}
-	}
-	return nil, false
-}
-
 func (peers *Peers) garbageCollect() []*Peer {
 	removed := []*Peer{}
+	peers.ourself.RLock()
+	_, reached := peers.ourself.Routes(nil, false)
+	peers.ourself.RUnlock()
 	for name, peer := range peers.table {
-		found, _ := peers.ourself.Routes(peer, false)
-		if !found && !peer.IsLocallyReferenced() {
-			peers.onGC(peer)
+		if _, found := reached[peer.Name]; !found && peer.localRefCount == 0 {
 			delete(peers.table, name)
-			peers.macs.Delete(peer)
+			peers.onGC(peer)
 			removed = append(removed, peer)
 		}
 	}
 	return removed
 }
 
-func encodePeersMap(peers map[PeerName]*Peer) []byte {
-	buf := new(bytes.Buffer)
-	enc := gob.NewEncoder(buf)
-	for _, peer := range peers {
-		peer.encodePeer(enc)
+func setFromPeersMap(peers map[PeerName]*Peer) PeerNameSet {
+	names := make(PeerNameSet)
+	for name := range peers {
+		names[name] = void
 	}
-	return buf.Bytes()
+	return names
 }
 
-func (peers *Peers) decodeUpdate(update []byte) (newPeers map[PeerName]*Peer, decodedUpdate []*Peer, decodedConns [][]byte, err error) {
+func (peers *Peers) decodeUpdate(update []byte) (newPeers map[PeerName]*Peer, decodedUpdate []*Peer, decodedConns [][]ConnectionSummary, err error) {
 	newPeers = make(map[PeerName]*Peer)
 	decodedUpdate = []*Peer{}
-	decodedConns = [][]byte{}
+	decodedConns = [][]ConnectionSummary{}
 
 	updateBuf := new(bytes.Buffer)
 	updateBuf.Write(update)
 	decoder := gob.NewDecoder(updateBuf)
 
 	for {
-		nameByte, uid, version, connsBuf, decErr := decodePeerNoConns(decoder)
+		peerSummary, connSummaries, decErr := decodePeer(decoder)
 		if decErr == io.EOF {
 			break
 		} else if decErr != nil {
 			err = decErr
 			return
 		}
-		name := PeerNameFromBin(nameByte)
-		newPeer := NewPeer(name, uid, version)
+		name := PeerNameFromBin(peerSummary.NameByte)
+		newPeer := NewPeer(name, peerSummary.NickName, peerSummary.UID, peerSummary.Version)
 		decodedUpdate = append(decodedUpdate, newPeer)
-		decodedConns = append(decodedConns, connsBuf)
+		decodedConns = append(decodedConns, connSummaries)
 		existingPeer, found := peers.table[name]
 		if !found {
 			newPeers[name] = newPeer
@@ -183,52 +233,37 @@ func (peers *Peers) decodeUpdate(update []byte) (newPeers map[PeerName]*Peer, de
 		}
 	}
 
-	unknownPeers := false
-	for _, connsBuf := range decodedConns {
-		connsIterator(connsBuf, func(remoteNameByte []byte, _ string) {
-			remoteName := PeerNameFromBin(remoteNameByte)
+	for _, connSummaries := range decodedConns {
+		for _, connSummary := range connSummaries {
+			remoteName := PeerNameFromBin(connSummary.NameByte)
 			if _, found := newPeers[remoteName]; found {
-				return
+				continue
 			}
 			if _, found := peers.table[remoteName]; found {
-				return
+				continue
 			}
 			// Update refers to a peer which we have no knowledge
 			// of. Thus we can't apply the update. Abort.
-			unknownPeers = true
-		})
-	}
-	if unknownPeers {
-		err = UnknownPeersError{}
-		return
+			err = UnknownPeerError{remoteName}
+			return
+		}
 	}
 	return
 }
 
-func (peers *Peers) applyUpdate(decodedUpdate []*Peer, decodedConns [][]byte) map[PeerName]*Peer {
+func (peers *Peers) applyUpdate(decodedUpdate []*Peer, decodedConns [][]ConnectionSummary) map[PeerName]*Peer {
 	newUpdate := make(map[PeerName]*Peer)
 	for idx, newPeer := range decodedUpdate {
-		connsBuf := decodedConns[idx]
+		connSummaries := decodedConns[idx]
 		name := newPeer.Name
 		// guaranteed to find peer in the peers.table
 		peer := peers.table[name]
-		if peer != newPeer {
-			if peer.Version() > newPeer.Version() {
-				// we know more about this one than the update. If
-				// peer is ourself, this is slightly racey (further
-				// changes could occur to ourself in the mean
-				// time). But it doesn't matter as we know that we're
-				// already > newPeer.Version() and that's not going to
-				// change.
-				newUpdate[name] = peer
-				continue
-			} else if peer == peers.ourself {
-				// nobody but us updates us
-				continue
-			} else if peer.Version() == newPeer.Version() {
-				// implication is that connections are equal too
-				continue
-			}
+		if peer != newPeer &&
+			(peer == peers.ourself.Peer || peer.version >= newPeer.version) {
+			// Nobody but us updates us. And if we know more about a
+			// peer than what's in the the update, we ignore the
+			// latter.
+			continue
 		}
 		// If we're here, either it was a new peer, or the update has
 		// more info about the peer than we do. Either case, we need
@@ -240,78 +275,57 @@ func (peers *Peers) applyUpdate(decodedUpdate []*Peer, decodedConns [][]byte) ma
 		// for an update would be someone else calling
 		// router.Peers.ApplyUpdate. But ApplyUpdate takes the Lock on
 		// the router.Peers, so there can be no race here.
-		conns := readConnsMap(peer, connsBuf, peers.table)
-		peer.SetVersionAndConnections(newPeer.Version(), conns)
+		peer.version = newPeer.version
+		peer.connections = makeConnsMap(peer, connSummaries, peers.table)
 		newUpdate[name] = peer
 	}
 	return newUpdate
 }
 
-func (peer *Peer) encodePeer(enc *gob.Encoder) {
-	peer.RLock()
-	defer peer.RUnlock()
+func (peer *Peer) Encode(enc *gob.Encoder) {
+	checkFatal(enc.Encode(PeerSummary{
+		peer.NameByte,
+		peer.NickName,
+		peer.UID,
+		peer.version}))
 
-	checkFatal(enc.Encode(peer.NameByte))
-	checkFatal(enc.Encode(peer.UID))
-	checkFatal(enc.Encode(peer.version))
-
-	connsBuf := new(bytes.Buffer)
-	connsEnc := gob.NewEncoder(connsBuf)
+	connSummaries := []ConnectionSummary{}
 	for _, conn := range peer.connections {
-		// DANGER holding rlock on peer, going to take rlock on conn
-		if !conn.Established() {
-			continue
-		}
-		checkFatal(connsEnc.Encode(conn.Remote().NameByte))
-		checkFatal(connsEnc.Encode(conn.RemoteTCPAddr()))
+		connSummaries = append(connSummaries, ConnectionSummary{
+			conn.Remote().NameByte,
+			conn.RemoteTCPAddr(),
+			conn.Outbound(),
+			// DANGER holding rlock on peer, going to take rlock on conn
+			conn.Established(),
+		})
 	}
-	checkFatal(enc.Encode(connsBuf.Bytes()))
+
+	checkFatal(enc.Encode(connSummaries))
 }
 
-func decodePeerNoConns(dec *gob.Decoder) (nameByte []byte, uid uint64, version uint64, conns []byte, err error) {
-	err = dec.Decode(&nameByte)
-	if err != nil {
+func (peer *LocalPeer) Encode(enc *gob.Encoder) {
+	peer.RLock()
+	defer peer.RUnlock()
+	peer.Peer.Encode(enc)
+}
+
+func decodePeer(dec *gob.Decoder) (peerSummary PeerSummary, connSummaries []ConnectionSummary, err error) {
+	if err = dec.Decode(&peerSummary); err != nil {
 		return
 	}
-	err = dec.Decode(&uid)
-	if err != nil {
+	if err = dec.Decode(&connSummaries); err != nil {
 		return
 	}
-	err = dec.Decode(&version)
-	if err != nil {
-		return
-	}
-	err = dec.Decode(&conns)
-	if err == io.EOF {
-		err = nil
-	}
 	return
 }
 
-func connsIterator(input []byte, fun func([]byte, string)) {
-	buf := new(bytes.Buffer)
-	buf.Write(input)
-	dec := gob.NewDecoder(buf)
-	for {
-		var nameByte []byte
-		err := dec.Decode(&nameByte)
-		if err == io.EOF {
-			return
-		}
-		checkFatal(err)
-		var foundAt string
-		checkFatal(dec.Decode(&foundAt))
-		fun(nameByte, string(foundAt))
-	}
-}
-
-func readConnsMap(peer *Peer, buf []byte, table map[PeerName]*Peer) map[PeerName]Connection {
+func makeConnsMap(peer *Peer, connSummaries []ConnectionSummary, table map[PeerName]*Peer) map[PeerName]Connection {
 	conns := make(map[PeerName]Connection)
-	connsIterator(buf, func(nameByte []byte, remoteTCPAddr string) {
-		name := PeerNameFromBin(nameByte)
+	for _, connSummary := range connSummaries {
+		name := PeerNameFromBin(connSummary.NameByte)
 		remotePeer := table[name]
-		conn := NewRemoteConnection(peer, remotePeer, remoteTCPAddr)
+		conn := NewRemoteConnection(peer, remotePeer, connSummary.RemoteTCPAddr, connSummary.Outbound, connSummary.Established)
 		conns[name] = conn
-	})
+	}
 	return conns
 }

router/peers_test.go

@@ -0,0 +1,113 @@
+package router
+
+import (
+	"fmt"
+	wt "github.com/weaveworks/weave/testing"
+	"math/rand"
+	"testing"
+)
+
+// TODO we should also test:
+//
+// - applying an incremental update, including the case where that
+//   leads to an UnknownPeerError
+//
+// - the "improved update" calculation
+//
+// - non-gc of peers that are only referenced locally
+
+func newNode(name PeerName) (*Peer, *Peers) {
+	peer := NewLocalPeer(name, "", nil)
+	peers := NewPeers(peer, func(*Peer) {})
+	peers.FetchWithDefault(peer.Peer)
+	return peer.Peer, peers
+}
+
+// Check that ApplyUpdate copies the whole topology from peers
+func checkApplyUpdate(t *testing.T, peers *Peers) {
+	dummyName, _ := PeerNameFromString("99:00:00:01:00:00")
+	// We need a new node outside of the network, with a connection
+	// into it.
+	_, testBedPeers := newNode(dummyName)
+	testBedPeers.AddTestConnection(peers.ourself.Peer)
+	testBedPeers.ApplyUpdate(peers.EncodePeers(peers.Names()))
+
+	checkTopologyPeers(t, true, testBedPeers.allPeersExcept(dummyName), peers.allPeers()...)
+}
+
+func TestPeersEncoding(t *testing.T) {
+	const numNodes = 20
+	const numIters = 1000
+	var peer [numNodes]*Peer
+	var ps [numNodes]*Peers
+	for i := 0; i < numNodes; i++ {
+		name, _ := PeerNameFromString(fmt.Sprintf("%02d:00:00:01:00:00", i))
+		peer[i], ps[i] = newNode(name)
+	}
+
+	var conns []struct{ from, to int }
+	for i := 0; i < numIters; i++ {
+		oper := rand.Intn(2)
+		switch oper {
+		case 0:
+			from, to := rand.Intn(numNodes), rand.Intn(numNodes)
+			if from != to {
+				if _, found := peer[from].connections[peer[to].Name]; !found {
+					ps[from].AddTestConnection(peer[to])
+					conns = append(conns, struct{ from, to int }{from, to})
+					checkApplyUpdate(t, ps[from])
+				}
+			}
+		case 1:
+			if len(conns) > 0 {
+				n := rand.Intn(len(conns))
+				c := conns[n]
+				ps[c.from].DeleteTestConnection(peer[c.to])
+				ps[c.from].GarbageCollect()
+				checkApplyUpdate(t, ps[c.from])
+				conns = append(conns[:n], conns[n+1:]...)
+			}
+		}
+	}
+}
+
+func TestPeersGarbageCollection(t *testing.T) {
+	const (
+		peer1NameString = "01:00:00:01:00:00"
+		peer2NameString = "02:00:00:02:00:00"
+		peer3NameString = "03:00:00:03:00:00"
+	)
+	var (
+		peer1Name, _ = PeerNameFromString(peer1NameString)
+		peer2Name, _ = PeerNameFromString(peer2NameString)
+		peer3Name, _ = PeerNameFromString(peer3NameString)
+	)
+
+	// Create some peers with some connections to each other
+	p1, ps1 := newNode(peer1Name)
+	p2, ps2 := newNode(peer2Name)
+	p3, ps3 := newNode(peer3Name)
+	ps1.AddTestConnection(p2)
+	ps2.AddTestRemoteConnection(p1, p2)
+	ps2.AddTestConnection(p1)
+	ps2.AddTestConnection(p3)
+	ps3.AddTestConnection(p1)
+	ps1.AddTestConnection(p3)
+	ps2.AddTestRemoteConnection(p1, p3)
+	ps2.AddTestRemoteConnection(p3, p1)
+
+	// Every peer is referenced, so nothing should be dropped
+	wt.AssertEmpty(t, ps1.GarbageCollect(), "peers removed")
+	wt.AssertEmpty(t, ps2.GarbageCollect(), "peers removed")
+	wt.AssertEmpty(t, ps3.GarbageCollect(), "peers removed")
+
+	// Drop the connection from 2 to 3, and 3 isn't garbage-collected
+	// because 1 has a connection to 3
+	ps2.DeleteTestConnection(p3)
+	wt.AssertEmpty(t, ps2.GarbageCollect(), "peers removed")
+
+	// Drop the connection from 1 to 3, and 3 will get removed by
+	// garbage-collection
+	ps1.DeleteTestConnection(p3)
+	checkPeerArray(t, ps1.GarbageCollect(), p3)
+}

router/protocol.go

@@ -0,0 +1,28 @@
+package router
+
+const (
+	Protocol        = "weave"
+	ProtocolVersion = 17
+)
+
+type ProtocolTag byte
+
+const (
+	ProtocolHeartbeat ProtocolTag = iota
+	ProtocolConnectionEstablished
+	ProtocolFragmentationReceived
+	ProtocolStartFragmentationTest
+	ProtocolPMTUVerified
+	ProtocolGossip
+	ProtocolGossipUnicast
+	ProtocolGossipBroadcast
+)
+
+type ProtocolMsg struct {
+	tag ProtocolTag
+	msg []byte
+}
+
+type ProtocolSender interface {
+	SendProtocolMsg(m ProtocolMsg)
+}

router/router.go

@@ -17,55 +17,100 @@ const macMaxAge = 10 * time.Minute // [1]
 // [1] should be greater than typical ARP cache expiries, i.e. > 3/2 *
 // /proc/sys/net/ipv4_neigh/*/base_reachable_time_ms on Linux
 
-func NewRouter(iface *net.Interface, name PeerName, password []byte, connLimit int, bufSz int, logFrame func(string, []byte, *layers.Ethernet)) *Router {
+type LogFrameFunc func(string, []byte, *layers.Ethernet)
+
+type RouterConfig struct {
+	Port      int
+	Iface     *net.Interface
+	Password  []byte
+	ConnLimit int
+	BufSz     int
+	LogFrame  LogFrameFunc
+}
+
+type Router struct {
+	RouterConfig
+	Ourself         *LocalPeer
+	Macs            *MacCache
+	Peers           *Peers
+	Routes          *Routes
+	ConnectionMaker *ConnectionMaker
+	GossipChannels  map[uint32]*GossipChannel
+	TopologyGossip  Gossip
+	UDPListener     *net.UDPConn
+}
+
+type PacketSource interface {
+	ReadPacket() ([]byte, error)
+}
+
+type PacketSink interface {
+	WritePacket([]byte) error
+}
+
+type PacketSourceSink interface {
+	PacketSource
+	PacketSink
+}
+
+func NewRouter(config RouterConfig, name PeerName, nickName string) *Router {
+	router := &Router{RouterConfig: config, GossipChannels: make(map[uint32]*GossipChannel)}
 	onMacExpiry := func(mac net.HardwareAddr, peer *Peer) {
-		log.Println("Expired MAC", mac, "at", peer.Name)
+		log.Println("Expired MAC", mac, "at", peer)
 	}
 	onPeerGC := func(peer *Peer) {
-		log.Println("Removing unreachable", peer)
+		router.Macs.Delete(peer)
+		log.Println("Removed unreachable peer", peer)
 	}
-	router := &Router{
-		Iface:     iface,
-		Macs:      NewMacCache(macMaxAge, onMacExpiry),
-		ConnLimit: connLimit,
-		BufSz:     bufSz,
-		LogFrame:  logFrame}
-	if len(password) > 0 {
-		router.Password = &password
-	}
-	router.Ourself = StartLocalPeer(name, router)
-	router.Peers = NewPeers(router.Ourself.Peer, router.Macs, onPeerGC)
+	router.Ourself = NewLocalPeer(name, nickName, router)
+	router.Macs = NewMacCache(macMaxAge, onMacExpiry)
+	router.Peers = NewPeers(router.Ourself, onPeerGC)
 	router.Peers.FetchWithDefault(router.Ourself.Peer)
-	log.Println("Our name is", router.Ourself.Name)
-
+	router.Routes = NewRoutes(router.Ourself, router.Peers)
+	router.ConnectionMaker = NewConnectionMaker(router.Ourself, router.Peers, router.Port)
+	router.TopologyGossip = router.NewGossip("topology", router)
 	return router
 }
 
+func (router *Router) Start() {
+	// we need two pcap handles since they aren't thread-safe
+	var pio PacketSourceSink
+	var po PacketSink
+	var err error
+	if router.Iface != nil {
+		pio, err = NewPcapIO(router.Iface.Name, router.BufSz)
+		checkFatal(err)
+		po, err = NewPcapO(router.Iface.Name)
+		checkFatal(err)
+	}
+	router.Ourself.Start()
+	router.Macs.Start()
+	router.Routes.Start()
+	router.ConnectionMaker.Start()
+	router.UDPListener = router.listenUDP(router.Port, po)
+	router.listenTCP(router.Port)
+	if pio != nil {
+		router.sniff(pio)
+	}
+}
+
+func (router *Router) Stop() error {
+	// TODO: perform graceful shutdown...
+	return nil
+}
+
 func (router *Router) UsingPassword() bool {
 	return router.Password != nil
 }
 
-func (router *Router) Start() {
-	// we need two pcap handles since they aren't thread-safe
-	pio, err := NewPcapIO(router.Iface.Name, router.BufSz)
-	checkFatal(err)
-	po, err := NewPcapO(router.Iface.Name)
-	checkFatal(err)
-	router.Routes = StartRoutes(router.Ourself.Peer, router.Peers)
-	router.ConnectionMaker = StartConnectionMaker(router.Ourself, router.Peers)
-	router.UDPListener = router.listenUDP(Port, po)
-	router.listenTCP(Port)
-	router.sniff(pio)
-}
-
 func (router *Router) Status() string {
 	var buf bytes.Buffer
-	buf.WriteString(fmt.Sprintln("Our name is", router.Ourself.Name))
-	buf.WriteString(fmt.Sprintln("Sniffing traffic on", router.Iface))
-	buf.WriteString(fmt.Sprintf("MACs:\n%s", router.Macs))
-	buf.WriteString(fmt.Sprintf("Peers:\n%s", router.Peers))
-	buf.WriteString(fmt.Sprintf("Routes:\n%s", router.Routes))
-	buf.WriteString(fmt.Sprintf("Reconnects:\n%s", router.ConnectionMaker))
+	fmt.Fprintln(&buf, "Our name is", router.Ourself)
+	fmt.Fprintln(&buf, "Sniffing traffic on", router.Iface)
+	fmt.Fprintf(&buf, "MACs:\n%s", router.Macs)
+	fmt.Fprintf(&buf, "Peers:\n%s", router.Peers)
+	fmt.Fprintf(&buf, "Routes:\n%s", router.Routes)
+	fmt.Fprintf(&buf, "Reconnects:\n%s", router.ConnectionMaker)
 	return buf.String()
 }
 
@@ -73,8 +118,6 @@ func (router *Router) sniff(pio PacketSourceSink) {
 	log.Println("Sniffing traffic on", router.Iface)
 
 	dec := NewEthernetDecoder()
-	injectFrame := func(frame []byte) error { return pio.WritePacket(frame) }
-	checkFrameTooBig := func(err error) error { return dec.CheckFrameTooBig(err, injectFrame) }
 	mac := router.Iface.HardwareAddr
 	if router.Macs.Enter(mac, router.Ourself.Peer) {
 		log.Println("Discovered our MAC", mac)
@@ -84,16 +127,16 @@ func (router *Router) sniff(pio PacketSourceSink) {
 			pkt, err := pio.ReadPacket()
 			checkFatal(err)
 			router.LogFrame("Sniffed", pkt, nil)
-			checkWarn(router.handleCapturedPacket(pkt, dec, checkFrameTooBig))
+			router.handleCapturedPacket(pkt, dec, pio)
 		}
 	}()
 }
 
-func (router *Router) handleCapturedPacket(frameData []byte, dec *EthernetDecoder, checkFrameTooBig func(error) error) error {
+func (router *Router) handleCapturedPacket(frameData []byte, dec *EthernetDecoder, po PacketSink) {
 	dec.DecodeLayers(frameData)
 	decodedLen := len(dec.decoded)
 	if decodedLen == 0 {
-		return nil
+		return
 	}
 	srcMac := dec.eth.SrcMAC
 	srcPeer, found := router.Macs.Lookup(srcMac)
@@ -101,18 +144,18 @@ func (router *Router) handleCapturedPacket(frameData []byte, dec *EthernetDecode
 	// frames, the srcMAC will have been recorded as associated with a
 	// different peer.
 	if found && srcPeer != router.Ourself.Peer {
-		return nil
+		return
 	}
 	if router.Macs.Enter(srcMac, router.Ourself.Peer) {
 		log.Println("Discovered local MAC", srcMac)
 	}
 	if dec.DropFrame() {
-		return nil
+		return
 	}
 	dstMac := dec.eth.DstMAC
 	dstPeer, found := router.Macs.Lookup(dstMac)
 	if found && dstPeer == router.Ourself.Peer {
-		return nil
+		return
 	}
 	df := decodedLen == 2 && (dec.ip.Flags&layers.IPv4DontFragment != 0)
 	if df {
@@ -127,11 +170,18 @@ func (router *Router) handleCapturedPacket(frameData []byte, dec *EthernetDecode
 	frameCopy := make([]byte, frameLen, frameLen)
 	copy(frameCopy, frameData)
 
+	// If we don't know which peer corresponds to the dest MAC,
+	// broadcast it.
 	if !found {
-		return checkFrameTooBig(router.Ourself.Broadcast(df, frameCopy, dec))
-	} else {
-		return checkFrameTooBig(router.Ourself.Forward(dstPeer, df, frameCopy, dec))
+		router.Ourself.Broadcast(df, frameCopy, dec)
+		return
 	}
+
+	err := router.Ourself.Forward(dstPeer, df, frameCopy, dec)
+	if ftbe, ok := err.(FrameTooBigError); ok {
+		err = dec.sendICMPFragNeeded(ftbe.EPMTU, po.WritePacket)
+	}
+	checkWarn(err)
 }
 
 func (router *Router) listenTCP(localPort int) {
@@ -154,10 +204,13 @@ func (router *Router) listenTCP(localPort int) {
 
 func (router *Router) acceptTCP(tcpConn *net.TCPConn) {
 	// someone else is dialing us, so our udp sender is the conn
-	// on Port and we wait for them to send us something on UDP to
+	// on router.Port and we wait for them to send us something on UDP to
 	// start.
-	connRemote := NewRemoteConnection(router.Ourself.Peer, nil, tcpConn.RemoteAddr().String())
-	NewLocalConnection(connRemote, true, tcpConn, nil, router)
+	remoteAddrStr := tcpConn.RemoteAddr().String()
+	log.Printf("->[%s] connection accepted\n", remoteAddrStr)
+	connRemote := NewRemoteConnection(router.Ourself.Peer, nil, remoteAddrStr, false, false)
+	connLocal := NewLocalConnection(connRemote, tcpConn, nil, router)
+	connLocal.Start(true)
 }
 
 func (router *Router) listenUDP(localPort int, po PacketSink) *net.UDPConn {
@@ -179,7 +232,6 @@ func (router *Router) listenUDP(localPort int, po PacketSink) *net.UDPConn {
 func (router *Router) udpReader(conn *net.UDPConn, po PacketSink) {
 	defer conn.Close()
 	dec := NewEthernetDecoder()
-	handleUDPPacket := router.handleUDPPacketFunc(dec, po)
 	buf := make([]byte, MaxUDPPacketSize)
 	for {
 		n, sender, err := conn.ReadFromUDP(buf)
@@ -195,10 +247,6 @@ func (router *Router) udpReader(conn *net.UDPConn, po PacketSink) {
 		name := PeerNameFromBin(buf[:NameSize])
 		packet := make([]byte, n-NameSize)
 		copy(packet, buf[NameSize:n])
-		udpPacket := &UDPPacket{
-			Name:   name,
-			Packet: packet,
-			Sender: sender}
 		peerConn, found := router.Ourself.ConnectionTo(name)
 		if !found {
 			continue
@@ -207,94 +255,174 @@ func (router *Router) udpReader(conn *net.UDPConn, po PacketSink) {
 		if !ok {
 			continue
 		}
-		err = relayConn.Decryptor.IterateFrames(handleUDPPacket, udpPacket)
-		if pde, ok := err.(PacketDecodingError); ok {
-			if pde.Fatal {
-				relayConn.CheckFatal(pde)
-			} else {
-				relayConn.log(pde.Error())
-			}
-		} else {
-			checkWarn(err)
+		if err := relayConn.Decryptor.IterateFrames(packet, router.handleUDPPacketFunc(relayConn, dec, sender, po)); err != nil {
+			// Errors during UDP packet decoding / processing are
+			// non-fatal. One common cause is that we receive and
+			// attempt to decrypt a "stray" packet. This can actually
+			// happen quite easily if there is some connection churn
+			// between two peers. After all, UDP isn't a
+			// connection-oriented protocol, yet we pretend it is.
+			//
+			// If anything really is seriously, unrecoverably amiss
+			// with a connection, that will typically result in missed
+			// heartbeats and the connection getting shut down because
+			// of that.
+			relayConn.Log(err)
 		}
 	}
 }
 
-func (router *Router) handleUDPPacketFunc(dec *EthernetDecoder, po PacketSink) FrameConsumer {
-	checkFrameTooBig := func(err error, srcPeer *Peer) error {
-		if err == nil { // optimisation: avoid closure creation in common case
-			return nil
-		}
-		return dec.CheckFrameTooBig(err,
-			func(icmpFrame []byte) error {
-				return router.Ourself.Forward(srcPeer, false, icmpFrame, nil)
-			})
-	}
-
-	return func(relayConn *LocalConnection, sender *net.UDPAddr, srcNameByte, dstNameByte []byte, frameLen uint16, frame []byte) error {
-		srcName := PeerNameFromBin(srcNameByte)
-		dstName := PeerNameFromBin(dstNameByte)
-		srcPeer, found := router.Peers.Fetch(srcName)
+func (router *Router) handleUDPPacketFunc(relayConn *LocalConnection, dec *EthernetDecoder, sender *net.UDPAddr, po PacketSink) FrameConsumer {
+	return func(srcNameByte, dstNameByte []byte, frame []byte) {
+		srcPeer, found := router.Peers.Fetch(PeerNameFromBin(srcNameByte))
 		if !found {
-			return nil
+			return
 		}
-		dstPeer, found := router.Peers.Fetch(dstName)
+		dstPeer, found := router.Peers.Fetch(PeerNameFromBin(dstNameByte))
 		if !found {
-			return nil
+			return
 		}
 
 		dec.DecodeLayers(frame)
 		decodedLen := len(dec.decoded)
+		if decodedLen == 0 {
+			return
+		}
+		// Handle special frames produced internally (rather than
+		// captured/forwarded) by the remote router.
+		//
+		// We really shouldn't be decoding these above, since they are
+		// not genuine Ethernet frames. However, it is actually more
+		// efficient to do so, as we want to optimise for the common
+		// (i.e. non-special) frames. These always need decoding, and
+		// detecting special frames is cheaper post decoding than pre.
+		if decodedLen == 1 && dec.IsSpecial() {
+			if srcPeer == relayConn.Remote() && dstPeer == router.Ourself.Peer {
+				handleSpecialFrame(relayConn, sender, frame)
+			}
+			return
+		}
+
 		df := decodedLen == 2 && (dec.ip.Flags&layers.IPv4DontFragment != 0)
-		srcMac := dec.eth.SrcMAC
 
 		if dstPeer != router.Ourself.Peer {
 			// it's not for us, we're just relaying it
-			if decodedLen == 0 {
-				return nil
-			}
 			if df {
 				router.LogFrame("Relaying DF", frame, &dec.eth)
 			} else {
 				router.LogFrame("Relaying", frame, &dec.eth)
 			}
-			return checkFrameTooBig(router.Ourself.Relay(srcPeer, dstPeer, df, frame, dec), srcPeer)
-		}
 
-		if relayConn.Remote().Name == srcPeer.Name {
-			if frameLen == 0 {
-				relayConn.SetRemoteUDPAddr(sender)
-				return nil
-			} else if frameLen == FragTestSize && bytes.Equal(frame, FragTest) {
-				relayConn.SendTCP(ProtocolFragmentationReceivedByte)
-				return nil
-			} else if frameLen == PMTUDiscoverySize && bytes.Equal(frame, PMTUDiscovery) {
-				return nil
+			err := router.Ourself.Relay(srcPeer, dstPeer, df, frame, dec)
+			if ftbe, ok := err.(FrameTooBigError); ok {
+				err = dec.sendICMPFragNeeded(ftbe.EPMTU, func(icmpFrame []byte) error {
+					return router.Ourself.Forward(srcPeer, false, icmpFrame, nil)
+				})
 			}
+
+			checkWarn(err)
+			return
 		}
 
-		if decodedLen == 0 {
-			return nil
-		}
-
-		if dec.IsPMTUVerify() && relayConn.Remote().Name == srcPeer.Name {
-			frameLenBytes := []byte{0, 0}
-			binary.BigEndian.PutUint16(frameLenBytes, uint16(frameLen-EthernetOverhead))
-			relayConn.SendTCP(Concat(ProtocolPMTUVerifiedByte, frameLenBytes))
-			return nil
-		}
+		srcMac := dec.eth.SrcMAC
+		dstMac := dec.eth.DstMAC
 
 		if router.Macs.Enter(srcMac, srcPeer) {
-			log.Println("Discovered remote MAC", srcMac, "at", srcPeer.Name)
+			log.Println("Discovered remote MAC", srcMac, "at", srcPeer)
+		}
+		if po != nil {
+			router.LogFrame("Injecting", frame, &dec.eth)
+			checkWarn(po.WritePacket(frame))
 		}
-		router.LogFrame("Injecting", frame, &dec.eth)
-		checkWarn(po.WritePacket(frame))
 
-		dstPeer, found = router.Macs.Lookup(dec.eth.DstMAC)
+		dstPeer, found = router.Macs.Lookup(dstMac)
 		if !found || dstPeer != router.Ourself.Peer {
-			return checkFrameTooBig(router.Ourself.RelayBroadcast(srcPeer, df, frame, dec), srcPeer)
+			router.LogFrame("Relaying broadcast", frame, &dec.eth)
+			router.Ourself.RelayBroadcast(srcPeer, df, frame, dec)
 		}
-
-		return nil
 	}
 }
+
+func handleSpecialFrame(relayConn *LocalConnection, sender *net.UDPAddr, frame []byte) {
+	frameLen := len(frame)
+	switch {
+	case frameLen == EthernetOverhead+8:
+		relayConn.ReceivedHeartbeat(sender, binary.BigEndian.Uint64(frame[EthernetOverhead:]))
+	case frameLen == FragTestSize && bytes.Equal(frame, FragTest):
+		relayConn.SendProtocolMsg(ProtocolMsg{ProtocolFragmentationReceived, nil})
+	case frameLen == PMTUDiscoverySize && bytes.Equal(frame, PMTUDiscovery):
+	default:
+		frameLenBytes := []byte{0, 0}
+		binary.BigEndian.PutUint16(frameLenBytes, uint16(frameLen-EthernetOverhead))
+		relayConn.SendProtocolMsg(ProtocolMsg{ProtocolPMTUVerified, frameLenBytes})
+	}
+}
+
+// Gossiper methods - the Router is the topology Gossiper
+
+type TopologyGossipData struct {
+	peers  *Peers
+	update PeerNameSet
+}
+
+func NewTopologyGossipData(peers *Peers, update ...*Peer) *TopologyGossipData {
+	names := make(PeerNameSet)
+	for _, p := range update {
+		names[p.Name] = void
+	}
+	return &TopologyGossipData{peers: peers, update: names}
+}
+
+func (d *TopologyGossipData) Merge(other GossipData) {
+	for name := range other.(*TopologyGossipData).update {
+		d.update[name] = void
+	}
+}
+
+func (d *TopologyGossipData) Encode() []byte {
+	return d.peers.EncodePeers(d.update)
+}
+
+func (router *Router) OnGossipUnicast(sender PeerName, msg []byte) error {
+	return fmt.Errorf("unexpected topology gossip unicast: %v", msg)
+}
+
+func (router *Router) OnGossipBroadcast(update []byte) (GossipData, error) {
+	origUpdate, _, err := router.applyTopologyUpdate(update)
+	if err != nil || len(origUpdate) == 0 {
+		return nil, err
+	}
+	return &TopologyGossipData{peers: router.Peers, update: origUpdate}, nil
+}
+
+func (router *Router) Gossip() GossipData {
+	return &TopologyGossipData{peers: router.Peers, update: router.Peers.Names()}
+}
+
+func (router *Router) OnGossip(update []byte) (GossipData, error) {
+	_, newUpdate, err := router.applyTopologyUpdate(update)
+	if err != nil || len(newUpdate) == 0 {
+		return nil, err
+	}
+	return &TopologyGossipData{peers: router.Peers, update: newUpdate}, nil
+}
+
+func (router *Router) applyTopologyUpdate(update []byte) (PeerNameSet, PeerNameSet, error) {
+	origUpdate, newUpdate, err := router.Peers.ApplyUpdate(update)
+	if _, ok := err.(UnknownPeerError); err != nil && ok {
+		// That update contained a reference to a peer which wasn't
+		// itself included in the update, and we didn't know about
+		// already. We ignore this; eventually we should receive an
+		// update containing a complete topology.
+		log.Println("Topology gossip:", err)
+		return nil, nil, nil
+	}
+	if err != nil {
+		return nil, nil, err
+	}
+	if len(newUpdate) > 0 {
+		router.ConnectionMaker.Refresh()
+		router.Routes.Recalculate()
+	}
+	return origUpdate, newUpdate, nil
+}

router/routes.go

@@ -3,17 +3,49 @@ package router
 import (
 	"bytes"
 	"fmt"
-	"log"
+	"math"
 	"sync"
 )
 
 type Routes struct {
 	sync.RWMutex
-	ourself   *Peer
-	peers     *Peers
-	unicast   map[PeerName]PeerName
-	broadcast map[PeerName][]PeerName
-	queryChan chan<- *Interaction
+	ourself      *LocalPeer
+	peers        *Peers
+	unicast      map[PeerName]PeerName
+	unicastAll   map[PeerName]PeerName // [1]
+	broadcast    map[PeerName][]PeerName
+	broadcastAll map[PeerName][]PeerName // [1]
+	recalculate  chan<- *struct{}
+	wait         chan<- chan struct{}
+	// [1] based on *all* connections, not just established &
+	// symmetric ones
+}
+
+func NewRoutes(ourself *LocalPeer, peers *Peers) *Routes {
+	routes := &Routes{
+		ourself:      ourself,
+		peers:        peers,
+		unicast:      make(map[PeerName]PeerName),
+		unicastAll:   make(map[PeerName]PeerName),
+		broadcast:    make(map[PeerName][]PeerName),
+		broadcastAll: make(map[PeerName][]PeerName)}
+	routes.unicast[ourself.Name] = UnknownPeerName
+	routes.unicastAll[ourself.Name] = UnknownPeerName
+	routes.broadcast[ourself.Name] = []PeerName{}
+	routes.broadcastAll[ourself.Name] = []PeerName{}
+	return routes
+}
+
+func (routes *Routes) Start() {
+	recalculate := make(chan *struct{}, 1)
+	wait := make(chan chan struct{})
+	routes.recalculate = recalculate
+	routes.wait = wait
+	go routes.run(recalculate, wait)
+}
+
+func (routes *Routes) PeerNames() PeerNameSet {
+	return routes.peers.Names()
 }
 
 func (routes *Routes) Unicast(name PeerName) (PeerName, bool) {
@@ -23,6 +55,13 @@ func (routes *Routes) Unicast(name PeerName) (PeerName, bool) {
 	return hop, found
 }
 
+func (routes *Routes) UnicastAll(name PeerName) (PeerName, bool) {
+	routes.RLock()
+	defer routes.RUnlock()
+	hop, found := routes.unicastAll[name]
+	return hop, found
+}
+
 func (routes *Routes) Broadcast(name PeerName) []PeerName {
 	routes.RLock()
 	defer routes.RUnlock()
@@ -33,68 +72,119 @@ func (routes *Routes) Broadcast(name PeerName) []PeerName {
 	return hops
 }
 
+func (routes *Routes) BroadcastAll(name PeerName) []PeerName {
+	routes.RLock()
+	defer routes.RUnlock()
+	hops, found := routes.broadcastAll[name]
+	if !found {
+		return []PeerName{}
+	}
+	return hops
+}
+
+// Choose min(log2(n_peers), n_neighbouring_peers) neighbours, with a
+// random distribution that is topology-sensitive, favouring
+// neighbours at the end of "bottleneck links". We determine the
+// latter based on the unicast routing table. If a neighbour appears
+// as the value more frequently than others - meaning that we reach a
+// higher proportion of peers via that neighbour than other neighbours
+// - then it is chosen with a higher probability.
+//
+// Note that we choose log2(n_peers) *neighbours*, not
+// peers. Consequently, on sparsely connected peers this function
+// returns a higher proportion of neighbours than elsewhere. In
+// extremis, on peers with fewer than log2(n_peers) neighbours, all
+// neighbours are returned.
+func (routes *Routes) RandomNeighbours(except PeerName) PeerNameSet {
+	res := make(PeerNameSet)
+	routes.RLock()
+	defer routes.RUnlock()
+	count := int(math.Log2(float64(len(routes.unicastAll))))
+	// depends on go's random map iteration
+	for _, dst := range routes.unicastAll {
+		if dst != UnknownPeerName && dst != except {
+			res[dst] = void
+			if len(res) >= count {
+				break
+			}
+		}
+	}
+	return res
+}
+
 func (routes *Routes) String() string {
 	var buf bytes.Buffer
 	routes.RLock()
 	defer routes.RUnlock()
-	buf.WriteString(fmt.Sprintln("unicast:"))
+	fmt.Fprintln(&buf, "unicast:")
 	for name, hop := range routes.unicast {
-		buf.WriteString(fmt.Sprintf("%s -> %s\n", name, hop))
+		fmt.Fprintf(&buf, "%s -> %s\n", name, hop)
 	}
-	buf.WriteString(fmt.Sprintln("broadcast:"))
+	fmt.Fprintln(&buf, "broadcast:")
 	for name, hops := range routes.broadcast {
-		buf.WriteString(fmt.Sprintf("%s -> %v\n", name, hops))
+		fmt.Fprintf(&buf, "%s -> %v\n", name, hops)
 	}
+	// We don't include the 'all' routes here since they are of
+	// limited utility in troubleshooting
 	return buf.String()
 }
 
-func StartRoutes(ourself *Peer, peers *Peers) *Routes {
-	queryChan := make(chan *Interaction, ChannelSize)
-	state := &Routes{
-		ourself:   ourself,
-		peers:     peers,
-		unicast:   make(map[PeerName]PeerName),
-		broadcast: make(map[PeerName][]PeerName),
-		queryChan: queryChan}
-	state.unicast[ourself.Name] = UnknownPeerName
-	state.broadcast[ourself.Name] = []PeerName{}
-	go state.queryLoop(queryChan)
-	return state
-}
-
-// ACTOR client API
-
-const (
-	RRecalculate = iota
-)
-
-// Async.
+// Request recalculation of the routing table. This is async but can
+// effectively be made synchronous with a subsequent call to
+// EnsureRecalculated.
 func (routes *Routes) Recalculate() {
-	routes.queryChan <- &Interaction{code: RRecalculate}
+	// The use of a 1-capacity channel in combination with the
+	// non-blocking send is an optimisation that results in multiple
+	// requests being coalesced.
+	select {
+	case routes.recalculate <- nil:
+	default:
+	}
 }
 
-// ACTOR server
+// Wait for any preceding Recalculate requests to be processed.
+func (routes *Routes) EnsureRecalculated() {
+	done := make(chan struct{})
+	routes.wait <- done
+	<-done
+}
 
-func (routes *Routes) queryLoop(queryChan <-chan *Interaction) {
+func (routes *Routes) run(recalculate <-chan *struct{}, wait <-chan chan struct{}) {
 	for {
-		query, ok := <-queryChan
-		if !ok {
-			return
-		}
-		switch query.code {
-		case RRecalculate:
-			unicast := routes.calculateUnicast()
-			broadcast := routes.calculateBroadcast()
-			routes.Lock()
-			routes.unicast = unicast
-			routes.broadcast = broadcast
-			routes.Unlock()
-		default:
-			log.Fatal("Unexpected routes query:", query)
+		select {
+		case <-recalculate:
+			routes.calculate()
+		case done := <-wait:
+			select {
+			case <-recalculate:
+				routes.calculate()
+			default:
+			}
+			close(done)
 		}
 	}
 }
 
+func (routes *Routes) calculate() {
+	routes.peers.RLock()
+	routes.ourself.RLock()
+	var (
+		unicast      = routes.calculateUnicast(true)
+		unicastAll   = routes.calculateUnicast(false)
+		broadcast    = routes.calculateBroadcast(true)
+		broadcastAll = routes.calculateBroadcast(false)
+	)
+	routes.ourself.RUnlock()
+	routes.peers.RUnlock()
+
+	routes.Lock()
+	routes.unicast = unicast
+	routes.unicastAll = unicastAll
+	routes.broadcast = broadcast
+	routes.broadcastAll = broadcastAll
+	routes.Unlock()
+}
+
 // Calculate all the routes for the question: if *we* want to send a
 // packet to Peer X, what is the next hop?
 //
@@ -104,8 +194,8 @@ func (routes *Routes) queryLoop(queryChan <-chan *Interaction) {
 // any knowledge of the MAC address at all. Thus there's no need
 // to exchange knowledge of MAC addresses, nor any constraints on
 // the routes that we construct.
-func (routes *Routes) calculateUnicast() map[PeerName]PeerName {
-	_, unicast := routes.ourself.Routes(nil, true)
+func (routes *Routes) calculateUnicast(establishedAndSymmetric bool) map[PeerName]PeerName {
+	_, unicast := routes.ourself.Routes(nil, establishedAndSymmetric)
 	return unicast
 }
 
@@ -127,23 +217,15 @@ func (routes *Routes) calculateUnicast() map[PeerName]PeerName {
 //     Y =/= Z /\ X.Routes(Y) <= X.Routes(Z) =>
 //     X.Routes(Y) u [P | Y.HasSymmetricConnectionTo(P)] <= X.Routes(Z)
 // where <= is the subset relationship on keys of the returned map.
-func (routes *Routes) calculateBroadcast() map[PeerName][]PeerName {
+func (routes *Routes) calculateBroadcast(establishedAndSymmetric bool) map[PeerName][]PeerName {
 	broadcast := make(map[PeerName][]PeerName)
-	ourself := routes.ourself
-
-	routes.peers.ForEach(func(name PeerName, peer *Peer) {
+	for _, peer := range routes.peers.table {
 		hops := []PeerName{}
-		if found, reached := peer.Routes(ourself, true); found {
-			ourself.ForEachConnection(func(remoteName PeerName, conn Connection) {
-				if _, found := reached[remoteName]; found {
-					return
-				}
-				if _, found := conn.Remote().ConnectionTo(ourself.Name); found {
-					hops = append(hops, remoteName)
-				}
-			})
+		if found, reached := peer.Routes(routes.ourself.Peer, establishedAndSymmetric); found {
+			routes.ourself.ForEachConnectedPeer(establishedAndSymmetric, reached,
+				func(remotePeer *Peer) { hops = append(hops, remotePeer.Name) })
 		}
-		broadcast[name] = hops
-	})
+		broadcast[peer.Name] = hops
+	}
 	return broadcast
 }

router/types.go

@@ -1,269 +0,0 @@
-package router
-
-import (
-	"bytes"
-	"code.google.com/p/go-bit/bit"
-	"code.google.com/p/gopacket"
-	"code.google.com/p/gopacket/layers"
-	"encoding/gob"
-	"net"
-	"sync"
-	"time"
-)
-
-type Router struct {
-	Iface           *net.Interface
-	Ourself         *LocalPeer
-	Macs            *MacCache
-	Peers           *Peers
-	Routes          *Routes
-	ConnectionMaker *ConnectionMaker
-	UDPListener     *net.UDPConn
-	Password        *[]byte
-	ConnLimit       int
-	BufSz           int
-	LogFrame        func(string, []byte, *layers.Ethernet)
-}
-
-type Connection interface {
-	Local() *Peer
-	Remote() *Peer
-	RemoteTCPAddr() string
-	Established() bool
-	Shutdown()
-}
-
-type RemoteConnection struct {
-	local         *Peer
-	remote        *Peer
-	remoteTCPAddr string
-}
-
-type LocalConnection struct {
-	sync.RWMutex
-	RemoteConnection
-	TCPConn       *net.TCPConn
-	tcpSender     TCPSender
-	remoteUDPAddr *net.UDPAddr
-	established   bool
-	stackFrag     bool
-	effectivePMTU int
-	SessionKey    *[32]byte
-	heartbeat     *time.Ticker
-	fetchAll      *time.Ticker
-	fragTest      *time.Ticker
-	forwardChan   chan<- *ForwardedFrame
-	forwardChanDF chan<- *ForwardedFrame
-	stopForward   chan<- interface{}
-	stopForwardDF chan<- interface{}
-	verifyPMTU    chan<- int
-	Decryptor     Decryptor
-	Router        *Router
-	UID           uint64
-	shutdown      bool
-	queryChan     chan<- *ConnectionInteraction
-}
-
-type ConnectionInteraction struct {
-	Interaction
-	payload interface{}
-}
-
-type Forwarder struct {
-	conn            *LocalConnection
-	ch              <-chan *ForwardedFrame
-	stop            <-chan interface{}
-	verifyPMTUTick  <-chan time.Time
-	verifyPMTU      <-chan int
-	pmtuVerifyCount uint
-	enc             Encryptor
-	udpSender       UDPSender
-	maxPayload      int
-	pmtuVerified    bool
-	highestGoodPMTU int
-	unverifiedPMTU  int
-	lowestBadPMTU   int
-}
-
-type UDPPacket struct {
-	Name   PeerName
-	Packet []byte
-	Sender *net.UDPAddr
-}
-
-type EthernetDecoder struct {
-	eth     layers.Ethernet
-	ip      layers.IPv4
-	decoded []gopacket.LayerType
-	parser  *gopacket.DecodingLayerParser
-}
-
-type MsgTooBigError struct {
-	PMTU int // actual pmtu, i.e. what the kernel told us
-}
-
-type FrameTooBigError struct {
-	EPMTU int // effective pmtu, i.e. what we tell packet senders
-}
-
-type UnknownPeersError struct {
-}
-
-type NameCollisionError struct {
-	Name PeerName
-}
-
-type PacketDecodingError struct {
-	Fatal bool
-	Desc  string
-}
-
-type LocalAddress struct {
-	ip      net.IP
-	network *net.IPNet
-}
-
-type ForwardedFrame struct {
-	srcPeer *Peer
-	dstPeer *Peer
-	frame   []byte
-}
-
-type Interaction struct {
-	code       int
-	resultChan chan<- interface{}
-}
-
-// UDPSender interface and implementations
-
-type UDPSender interface {
-	Send([]byte) error
-	Shutdown() error
-}
-
-type SimpleUDPSender struct {
-	conn    *LocalConnection
-	udpConn *net.UDPConn
-}
-
-type RawUDPSender struct {
-	ipBuf     gopacket.SerializeBuffer
-	opts      gopacket.SerializeOptions
-	udpHeader *layers.UDP
-	socket    *net.IPConn
-	conn      *LocalConnection
-}
-
-// TCPSender interface and implementations
-
-type TCPSender interface {
-	Send([]byte) error
-}
-
-type SimpleTCPSender struct {
-	encoder *gob.Encoder
-}
-
-type EncryptedTCPSender struct {
-	sync.RWMutex
-	outerEncoder *gob.Encoder
-	innerEncoder *gob.Encoder
-	buffer       *bytes.Buffer
-	conn         *LocalConnection
-	msgCount     int
-}
-
-type EncryptedTCPMessage struct {
-	Number int
-	Body   []byte
-}
-
-// TCPReceiver interface and implementations
-
-type TCPReceiver interface {
-	Decode([]byte) ([]byte, error)
-}
-
-type SimpleTCPReceiver struct {
-}
-
-type EncryptedTCPReceiver struct {
-	conn     *LocalConnection
-	decoder  *gob.Decoder
-	buffer   *bytes.Buffer
-	msgCount int
-}
-
-// Encryptor interface and implementations
-
-type Encryptor interface {
-	FrameOverhead() int
-	PacketOverhead() int
-	IsEmpty() bool
-	Bytes() []byte
-	AppendFrame(*ForwardedFrame)
-	TotalLen() int
-}
-
-type NonEncryptor struct {
-	buf       []byte
-	bufTail   []byte
-	buffered  int
-	prefixLen int
-}
-
-type NaClEncryptor struct {
-	NonEncryptor
-	buf       []byte
-	offset    uint16
-	nonce     *[24]byte
-	nonceChan chan *[24]byte
-	flags     uint16
-	prefixLen int
-	conn      *LocalConnection
-	df        bool
-}
-
-// Decryptor interface and implementations
-
-type FrameConsumer func(*LocalConnection, *net.UDPAddr, []byte, []byte, uint16, []byte) error
-
-type Decryptor interface {
-	IterateFrames(FrameConsumer, *UDPPacket) error
-	ReceiveNonce([]byte)
-	Shutdown()
-}
-
-type NonDecryptor struct {
-	conn *LocalConnection
-}
-
-type NaClDecryptor struct {
-	NonDecryptor
-	instance   *NaClDecryptorInstance
-	instanceDF *NaClDecryptorInstance
-}
-
-type NaClDecryptorInstance struct {
-	nonce               *[24]byte
-	previousNonce       *[24]byte
-	usedOffsets         *bit.Set
-	previousUsedOffsets *bit.Set
-	highestOffsetSeen   uint16
-	nonceChan           chan *[24]byte
-}
-
-// Packet capture/inject interfaces
-
-type PacketSource interface {
-	ReadPacket() ([]byte, error)
-}
-
-type PacketSink interface {
-	WritePacket([]byte) error
-}
-
-type PacketSourceSink interface {
-	PacketSource
-	PacketSink
-}

router/udp_sender.go

@@ -0,0 +1,138 @@
+package router
+
+import (
+	"code.google.com/p/gopacket"
+	"code.google.com/p/gopacket/layers"
+	"log"
+	"net"
+	"syscall"
+)
+
+type UDPSender interface {
+	Send([]byte) error
+	Shutdown() error
+}
+
+type SimpleUDPSender struct {
+	conn    *LocalConnection
+	udpConn *net.UDPConn
+}
+
+type RawUDPSender struct {
+	ipBuf     gopacket.SerializeBuffer
+	opts      gopacket.SerializeOptions
+	udpHeader *layers.UDP
+	socket    *net.IPConn
+	conn      *LocalConnection
+}
+
+type MsgTooBigError struct {
+	PMTU int // actual pmtu, i.e. what the kernel told us
+}
+
+func NewSimpleUDPSender(conn *LocalConnection) *SimpleUDPSender {
+	return &SimpleUDPSender{udpConn: conn.Router.UDPListener, conn: conn}
+}
+
+func (sender *SimpleUDPSender) Send(msg []byte) error {
+	_, err := sender.udpConn.WriteToUDP(msg, sender.conn.RemoteUDPAddr())
+	return err
+}
+
+func (sender *SimpleUDPSender) Shutdown() error {
+	return nil
+}
+
+func NewRawUDPSender(conn *LocalConnection) (*RawUDPSender, error) {
+	ipSocket, err := dialIP(conn)
+	if err != nil {
+		return nil, err
+	}
+	udpHeader := &layers.UDP{SrcPort: layers.UDPPort(conn.Router.Port)}
+	ipBuf := gopacket.NewSerializeBuffer()
+	opts := gopacket.SerializeOptions{
+		FixLengths: true,
+		// UDP header is calculated with a phantom IP
+		// header. Yes, it's totally nuts. Thankfully, for UDP
+		// over IPv4, the checksum is optional. It's not
+		// optional for IPv6, but we'll ignore that for
+		// now. TODO
+		ComputeChecksums: false}
+
+	return &RawUDPSender{
+		ipBuf:     ipBuf,
+		opts:      opts,
+		udpHeader: udpHeader,
+		socket:    ipSocket,
+		conn:      conn}, nil
+}
+
+func (sender *RawUDPSender) Send(msg []byte) error {
+	payload := gopacket.Payload(msg)
+	sender.udpHeader.DstPort = layers.UDPPort(sender.conn.RemoteUDPAddr().Port)
+
+	err := gopacket.SerializeLayers(sender.ipBuf, sender.opts, sender.udpHeader, &payload)
+	if err != nil {
+		return err
+	}
+	packet := sender.ipBuf.Bytes()
+	_, err = sender.socket.Write(packet)
+	if err == nil || PosixError(err) != syscall.EMSGSIZE {
+		return err
+	}
+	f, err := sender.socket.File()
+	if err != nil {
+		return err
+	}
+	defer f.Close()
+	fd := int(f.Fd())
+	log.Println("EMSGSIZE on send, expecting PMTU update (IP packet was",
+		len(packet), "bytes, payload was", len(msg), "bytes)")
+	pmtu, err := syscall.GetsockoptInt(fd, syscall.IPPROTO_IP, syscall.IP_MTU)
+	if err != nil {
+		return err
+	}
+	return MsgTooBigError{PMTU: pmtu}
+}
+
+func (sender *RawUDPSender) Shutdown() error {
+	defer func() { sender.socket = nil }()
+	return sender.socket.Close()
+}
+
+func dialIP(conn *LocalConnection) (*net.IPConn, error) {
+	ipLocalAddr, err := ipAddr(conn.TCPConn.LocalAddr())
+	if err != nil {
+		return nil, err
+	}
+	ipRemoteAddr, err := ipAddr(conn.TCPConn.RemoteAddr())
+	if err != nil {
+		return nil, err
+	}
+	ipSocket, err := net.DialIP("ip4:UDP", ipLocalAddr, ipRemoteAddr)
+	if err != nil {
+		return nil, err
+	}
+	f, err := ipSocket.File()
+	if err != nil {
+		return nil, err
+	}
+	defer f.Close()
+	fd := int(f.Fd())
+	// This Makes sure all packets we send out have DF set on them.
+	err = syscall.SetsockoptInt(fd, syscall.IPPROTO_IP, syscall.IP_MTU_DISCOVER, syscall.IP_PMTUDISC_DO)
+	if err != nil {
+		return nil, err
+	}
+	return ipSocket, nil
+}
+
+func ipAddr(addr net.Addr) (*net.IPAddr, error) {
+	host, _, err := net.SplitHostPort(addr.String())
+	if err != nil {
+		return nil, err
+	}
+	return &net.IPAddr{
+		IP:   net.ParseIP(host),
+		Zone: ""}, nil
+}

router/utils.go

@@ -1,12 +1,17 @@
 package router
 
 import (
+	"bytes"
 	"crypto/rand"
+	"encoding/gob"
 	"fmt"
+	"hash/fnv"
 	"log"
 	"net"
 )
 
+var void = struct{}{}
+
 func checkFatal(e error) {
 	if e != nil {
 		log.Fatal(e)
@@ -38,8 +43,8 @@ func (ftbe FrameTooBigError) Error() string {
 	return fmt.Sprint("Frame too big error. Effective PMTU is ", ftbe.EPMTU)
 }
 
-func (upe UnknownPeersError) Error() string {
-	return fmt.Sprint("Reference to unknown peers")
+func (upe UnknownPeerError) Error() string {
+	return fmt.Sprint("Reference to unknown peer ", upe.Name)
 }
 
 func (nce NameCollisionError) Error() string {
@@ -50,10 +55,6 @@ func (pde PacketDecodingError) Error() string {
 	return fmt.Sprint("Failed to decode packet: ", pde.Desc)
 }
 
-func (packet UDPPacket) String() string {
-	return fmt.Sprintf("UDP Packet\n name: %s\n sender: %v\n payload: % X", packet.Name, packet.Sender, packet.Packet)
-}
-
 func Concat(elems ...[]byte) []byte {
 	res := []byte{}
 	for _, e := range elems {
@@ -73,6 +74,21 @@ func randUint64() (r uint64) {
 	return
 }
 
+func hash(s string) uint32 {
+	h := fnv.New32a()
+	h.Write([]byte(s))
+	return h.Sum32()
+}
+
+func GobEncode(items ...interface{}) []byte {
+	buf := new(bytes.Buffer)
+	enc := gob.NewEncoder(buf)
+	for _, i := range items {
+		checkFatal(enc.Encode(i))
+	}
+	return buf.Bytes()
+}
+
 func macint(mac net.HardwareAddr) (r uint64) {
 	for _, b := range mac {
 		r <<= 8
@@ -101,14 +117,3 @@ func (lop ListOfPeers) Swap(i, j int) {
 func (lop ListOfPeers) Less(i, j int) bool {
 	return lop[i].Name < lop[j].Name
 }
-
-// given an address like '1.2.3.4:567', return the address if it has a port,
-// otherwise return the address with weave's standard port number
-func NormalisePeerAddr(peerAddr string) string {
-	_, _, err := net.SplitHostPort(peerAddr)
-	if err == nil {
-		return peerAddr
-	} else {
-		return fmt.Sprintf("%s:%d", peerAddr, Port)
-	}
-}

sigproxy/main.go

@@ -0,0 +1,50 @@
+package main
+
+import (
+	"log"
+	"os"
+	"os/exec"
+	"os/signal"
+	"syscall"
+)
+
+func main() {
+	if len(os.Args) == 1 {
+		log.Fatal("USAGE: sigproxy <command> [arguments ...]")
+	}
+
+	// Install signal handler as soon as possible - channel is buffered so
+	// we'll catch signals that arrive whilst child process is starting
+	sc := make(chan os.Signal, 1)
+	signal.Notify(sc, syscall.SIGINT, syscall.SIGTERM)
+
+	cmd := exec.Command(os.Args[1], os.Args[2:]...)
+
+	// These default to /dev/null, so set them explicitly to ours
+	cmd.Stdin = os.Stdin
+	cmd.Stdout = os.Stdout
+	cmd.Stderr = os.Stderr
+
+	if err := cmd.Start(); err != nil {
+		log.Fatal(err)
+	}
+
+	// Only begin delivering signals after the child has started
+	go func() {
+		for {
+			// Signalling PID 0 delivers to our process group
+			syscall.Kill(0, (<-sc).(syscall.Signal))
+		}
+	}()
+
+	if err := cmd.Wait(); err != nil {
+		// Exit status is platform specific so not directly accessible - casts
+		// required to access system-dependent exit information
+		if exitErr, ok := err.(*exec.ExitError); ok {
+			waitStatus := exitErr.Sys().(syscall.WaitStatus)
+			os.Exit(waitStatus.ExitStatus())
+		}
+		os.Exit(1)
+	}
+	os.Exit(0)
+}

site/Gemfile

@@ -0,0 +1,3 @@
+source 'https://rubygems.org'
+
+gem 'jekyll'

site/_config.yml

@@ -2,3 +2,9 @@ markdown: redcarpet
 
 redcarpet:
   extensions: ['fenced_code_blocks', 'smart']
+
+exclude:
+  - 'Gemfile'
+  - 'Gemfile.lock'
+  - 'params.json'
+  - 'build-and-publish.sh'

site/_layouts/default.html

@@ -15,7 +15,7 @@
     <!-- HEADER -->
     <div id="header_wrap" class="outer">
         <header class="inner">
-          <a id="forkme_banner" href="https://github.com/zettio/weave">View on GitHub</a>
+          <a id="forkme_banner" href="https://github.com/weaveworks/weave">View on GitHub</a>
 
           <h1 id="project_title">weave</h1>
           <h2 id="project_tagline">The Docker Network</h2>

site/boot2docker.md

@@ -23,7 +23,7 @@ example above, the 'launch' command would be run like this:
 
 and the 'run' command like this:
 
-    host1# C=$(boot2docker ssh "sudo ./weave run 10.0.1.1/24 -t -i ubuntu")
+    host1$ C=$(boot2docker ssh "sudo ./weave run 10.2.1.1/24 -t -i ubuntu")
 
 For more information about how to access services running in
 Boot2Docker from the host or other machines (i.e. outside of the weave

site/build-and-publish.sh

@@ -0,0 +1,36 @@
+#!/bin/sh -ex
+
+cd $(dirname $0)/
+
+## Use GCS until we set up Route53/CloudFront/S3
+
+BRANCH=${TRAVIS_BRANCH:-`git rev-parse --abbrev-ref @`}
+COMMIT=${TRAVIS_COMMIT:-`git rev-parse @`}
+OUTPUT="weave/${BRANCH}/${COMMIT}"
+
+if [ -n "${TRAVIS_TAG}" ]; then
+  ## Travis will run separate build for commit and tag pushes,
+  OUTPUT="weave/${TRAVIS_TAG}"
+fi
+
+if [ "${BRANCH}" = "latest_release_doc_updates" ]; then
+  OUTPUT="weave/latest_release"
+fi
+
+export BRANCH COMMIT OUTPUT
+
+bundle install --path=.bundle
+bundle exec jekyll build --verbose
+
+gsutil -m rsync -r -d _site "gs://docs.weave.works/${OUTPUT}"
+
+echo "Published at http://docs.weave.works/${OUTPUT}"
+
+if [ -z "${TRAVIS_TAG}" -a ! "${BRANCH}" = "latest_release_doc_updates" ]; then
+  echo "<meta http-equiv=\"refresh\" content=\"0; url=http://docs.weave.works/${OUTPUT}\" />" \
+    | gsutil \
+      -h "Content-Type:text/html" \
+      -h "Cache-Control:private, max-age=0, no-transform" \
+      cp -a "public-read" - "gs://docs.weave.works/weave/${BRANCH}/index.html"
+  gsutil -m rsync -r -d -p gs://docs.weave.works/${OUTPUT} gs://docs.weave.works/weave/${BRANCH}/head
+fi