228 lines
9.0 KiB
Markdown
228 lines
9.0 KiB
Markdown
---
|
|
title: Automatic IP Address Management
|
|
layout: default
|
|
---
|
|
|
|
# Automatic IP Address Management
|
|
|
|
Weave automatically assigns containers an IP address that is unique
|
|
across the network, and releases that address when a container
|
|
exits. This happens for all invocations of the `run`, `start`,
|
|
`attach`, `detach`, `expose`, and `hide` commands, unless
|
|
the user explictly specified an address. Weave can also assign
|
|
addresses in multiple subnets.
|
|
|
|
* [Initialisation](#initialisation)
|
|
* [Choosing an allocation range](#range)
|
|
* [Automatic allocation across multiple subnets](#subnets)
|
|
* [Mixing automatic and manual allocation](#manual)
|
|
* [Stopping and removing peers](#stop)
|
|
* [Troubleshooting](#troubleshooting)
|
|
|
|
## <a name="initialisation"></a>Initialisation
|
|
|
|
Just once, when the first automatic IP address allocation is requested
|
|
in the whole network, weave needs a majority of peers to be present in
|
|
order to avoid formation of isolated groups, which could lead to
|
|
inconsistency, i.e. the same IP address being allocated to two
|
|
different containers. Therefore, you must either supply the list of
|
|
all peers in the network to `weave launch` or add the
|
|
`--init-peer-count` flag to specify how many peers there will be.
|
|
|
|
To illustrate, suppose you have three hosts, accessible to each other
|
|
as `$HOST1`, `$HOST2` and `$HOST3`. You can start weave on those three
|
|
hosts with these three commands:
|
|
|
|
host1$ weave launch $HOST2 $HOST3
|
|
|
|
host2$ weave launch $HOST1 $HOST3
|
|
|
|
host3$ weave launch $HOST1 $HOST2
|
|
|
|
Or, if it is not convenient to name all the other hosts at launch
|
|
time, you can give the number of peers like this:
|
|
|
|
host1$ weave launch --init-peer-count 3
|
|
|
|
host2$ weave launch --init-peer-count 3 $HOST3
|
|
|
|
host3$ weave launch --init-peer-count 3 $HOST2
|
|
|
|
The consensus mechanism used to determine a majority transitions
|
|
through three states: 'deferred', 'waiting' and 'achieved':
|
|
|
|
* 'deferred' - no allocation requests or claims have been made yet;
|
|
consensus is deferred until then
|
|
* 'waiting' - an attempt to achieve consensus is ongoing, triggered by
|
|
an allocation or claim request; allocations will block. This state
|
|
persists until a quorum of peers are able to communicate amongst
|
|
themselves successfully
|
|
* 'achieved' - consensus achieved; allocations proceed normally
|
|
|
|
### More on `--init-peer-count`
|
|
|
|
TL;DR: it isn't a problem to over-estimate by a bit, but if you supply
|
|
a number that is too small then multiple independent groups may form.
|
|
|
|
Weave uses the estimate of the number of peers at initialization to
|
|
compute a majority or quorum number - specifically floor(n/2) + 1. So,
|
|
if the actual number of peers is less than half the number stated then
|
|
they will keep waiting for someone else to join to reach a quorum. On
|
|
the other hand, if the actual number is more than twice the quorum
|
|
number then you could have two sets of peers each reach a quorum and
|
|
initialize independent data structures. You'd have to be quite unlucky
|
|
for that to happen in practice, as they have to go through the whole
|
|
agreement process without learning about each other, but it's
|
|
definitely possible.
|
|
|
|
The quorum number is only used once at start-up (specifically, the
|
|
first time someone tries to allocate or claim an IP address), so once
|
|
a set of peers is initialized you can add more and they will join on
|
|
to the data structure used by the existing set. The one thing you
|
|
have to watch is if the early ones get restarted, you must restart
|
|
them with the current number of peers - if they use the smaller number
|
|
that was correct when they first started then they could form an
|
|
independent set again.
|
|
|
|
To illustrate this last point, the following sequence of operations
|
|
would be safe wrt weave's startup quorum:
|
|
|
|
host1$ weave launch
|
|
...time passes...
|
|
host2$ weave launch $HOST1
|
|
...time passes...
|
|
host3$ weave launch $HOST1 $HOST2
|
|
...time passes...
|
|
...host1 is rebooted...
|
|
host1$ weave launch $HOST2 $HOST3
|
|
|
|
## <a name="range"></a>Choosing an allocation range
|
|
|
|
By default, weave will allocate IP addresses in the 10.32.0.0/12
|
|
range. This can be overridden with the `--ipalloc-range` option, e.g.
|
|
|
|
host1$ weave launch --ipalloc-range 10.2.0.0/16
|
|
|
|
and must be the same on every host.
|
|
|
|
The range parameter is written in
|
|
[CIDR notation](http://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing) -
|
|
in this example "/16" means the first 16 bits of the address form the
|
|
network address and the allocator is to allocate container addresses
|
|
that all start 10.2.
|
|
|
|
Weave shares the IP address range across all peers, dynamically
|
|
according to their needs. If a group of peers becomes isolated from
|
|
the rest (a partition), they can continue to work with the address
|
|
ranges they had before isolation, and can subsequently be re-connected
|
|
to the rest of the network without any conflicts arising.
|
|
|
|
## <a name="subnets"></a>Automatic allocation across multiple subnets
|
|
|
|
IP subnets are used to define or restrict routing. By default, weave
|
|
puts all containers in a subnet that spans the entire allocation
|
|
range, so every weave-attached container can talk to every other
|
|
weave-attached container.
|
|
|
|
If you want some [isolation](features.html#application-isolation), you
|
|
can choose to run containers on different subnets. To request the
|
|
allocation of an address from a particular subnet, set the
|
|
`WEAVE_CIDR` environment variable to `net:<subnet>` when creating the
|
|
container, e.g.
|
|
|
|
host1$ docker run -e WEAVE_CIDR=net:10.2.7.0/24 -ti ubuntu
|
|
|
|
You can ask for multiple addresses in different subnets and add in
|
|
manually-assigned addresses (outside the automatic allocation range),
|
|
for instance:
|
|
|
|
host1$ docker run -e WEAVE_CIDR="net:10.2.7.0/24 net:10.2.8.0/24 ip:10.3.9.1/24" -ti ubuntu
|
|
|
|
(Note the ".0" and ".-1" addresses in a subnet are not used, as required by
|
|
[RFC 1122](https://tools.ietf.org/html/rfc1122#page-29)).
|
|
|
|
When working with multiple subnets in this way, it is usually
|
|
desirable to constrain the default subnet - i.e. the one chosen by the
|
|
allocator when no subnet is supplied - so that it does not overlap
|
|
with others. One can specify that with `--ipalloc-default-subnet`:
|
|
|
|
host1$ weave launch --ipalloc-range 10.2.0.0/16 --ipalloc-default-subnet 10.2.3.0/24
|
|
|
|
`--ipalloc-range` should cover the entire range that you will ever use
|
|
for allocation, and `--ipalloc-default-subnet` is the subnet that will
|
|
be used when you don't explicitly specify one.
|
|
|
|
When specifying addresses, the default subnet can be denoted
|
|
symbolically with `net:default`.
|
|
|
|
## <a name="manual"></a>Mixing automatic and manual allocation
|
|
|
|
You can start containers with a mixture of automatically-allocated
|
|
addresses and manually-chosen addresses in the same range, but you may
|
|
find that the automatic allocator has already reserved a specific
|
|
address that you wanted.
|
|
|
|
To reserve a range for manual allocation in the same subnet as the
|
|
automatic allocator, you can specify an
|
|
`--ipalloc-range` that is smaller than `--ip-default-subnet`, For
|
|
example, if you launch weave with:
|
|
|
|
host1$ weave launch --ipalloc-range 10.9.0.0/17 --ipalloc-default-subnet 10.9.0.0/16
|
|
|
|
then you can run all containers in the 10.9.0.0/16 subnet, with
|
|
automatic allocation using the lower half, leaving the upper half free
|
|
for manual allocation.
|
|
|
|
## <a name="stop"></a>Stopping and removing peers
|
|
|
|
You may wish to `weave stop` and re-launch to change some config or to
|
|
upgrade to a new version; provided the underlying protocol hasn't
|
|
changed it will pick up where it left off and learn from peers in the
|
|
network which address ranges it was previously using. If, however, you
|
|
run `weave reset` this will remove the peer from the network so
|
|
if Weave is run again on that node it will start from scratch.
|
|
|
|
For failed peers, the `weave rmpeer` command can be used to
|
|
permanently remove the ranges allocated to said peer. This will allow
|
|
other peers to allocate IPs in the ranges previously owner by the rm'd
|
|
peer, and as such should be used with extreme caution - if the rm'd
|
|
peer had transferred some range of IP addresses to another peer but
|
|
this is not known to the whole network, or if it later rejoins
|
|
the Weave network, the same IP address may be allocated twice.
|
|
|
|
Assuming we had started the three peers in the example earlier, and
|
|
host3 has caught fire, we can go to one of the other hosts and run:
|
|
|
|
host1$ weave rmpeer host3
|
|
|
|
Weave will take all the IP address ranges owned by host3 and transfer
|
|
them to be owned by host1. The name "host3" is resolved via the
|
|
'nickname' feature of weave, which defaults to the local host
|
|
name. Alternatively, one can supply a peer name as shown in `weave
|
|
status`.
|
|
|
|
## <a name="troubleshooting"></a>Troubleshooting
|
|
|
|
The command
|
|
|
|
weave status
|
|
|
|
reports on the current status of the weave router and IP allocator:
|
|
|
|
````
|
|
...
|
|
|
|
Service: ipam
|
|
Consensus: waiting(quorum: 2, known: 0)
|
|
Range: 10.32.0.0-10.47.255.255
|
|
DefaultSubnet: 10.32.0.0/12
|
|
|
|
...
|
|
````
|
|
|
|
The first section covers the router; see the [troubleshooting
|
|
guide](troubleshooting.html#weave-status) for full details.
|
|
|
|
The 'Service: ipam' section displays the consensus state as well as
|
|
the total allocation range and default subnet.
|