reweave/weave
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Ops edits (#2371)

Browse Source
This commit is contained in:
Anita Buehrle authored and Matthias Radestock committed 2016-06-20 03:04:53 +01:00
1 parent ca7220b775
commit 6356fbc882
6 files changed
+173 -138
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

site/operational-guide/autoscaling.md
+14 -12
View File
@@ -3,7 +3,7 @@ title: Autoscaling
menu_order: 40
---
### Bootstrap
### Bootstrapping
An autoscaling configuration begins with a fixed cluster:
@@ -21,30 +21,32 @@ changes to the configuration of the fixed cluster. As with the fixed
cluster, dynamically added nodes recover automatically from reboots
and partitions.
### Scale-out
### Scaling Out
On additional dynamic peer, at boot, via
On the additional dynamic peer, at boot, via
[systemd](/site/installing-weave/systemd.md) or equivalent:
weave launch --no-restart --ipalloc-init=observer $PEERS
In this case `$PEERS` means all peers in the _fixed cluster_, initial
Where,
* `$PEERS` means all peers in the _fixed cluster_, initial
and subsequently added, which have not been explicitly removed. It
should include fixed peers which are temporarily offline or stopped.
Note that you do not have to keep track of and specify the addresses
You do not have to keep track of and specify the addresses
of other dynamic peers in `$PEERS` - they will discover and connect to
each other via the fixed cluster.
> The use of `--ipalloc-init=observer` prevents dynamic peers from
> coming to a consensus on their own - this is important to stop a
> clique forming amongst a group of dynamically added peers if they
> become partitioned from the fixed cluster after having learned about
> each other via discovery.
>>**Note:** The use of `--ipalloc-init=observer` prevents dynamic peers from
coming to a consensus on their own - this is important to stop a
clique forming amongst a group of dynamically added peers if they
become partitioned from the fixed cluster after having learned about
each other via discovery.
### Scale-in
### Scaling In
On dynamic peer to be removed:
On the dynamic peer to be removed:
weave reset
site/operational-guide/concepts.md
+28 -27
View File
@@ -2,15 +2,15 @@
title: Concepts
menu_order: 10
---
This section describes some essential concepts with which you will
need to be familiar before moving on to the example deployment
This section describes some of the essential concepts with which you will
need to be familiar before continuing to the example deployment
scenarios.
## Host
For the purposes of this documentation we consider a host to be an
installation of the Linux operating system which is running an
instance of the Docker Engine. It may be executing directly on bare
For the purposes of this documentation a host is an
installation of the Linux operating system that is running an
instance of the Docker Engine. The host may be executing directly on bare
hardware or inside a virtual machine.
## Peer
@@ -19,22 +19,23 @@ A peer is a running instance of Weave Net, typically one per host.
## Peer Name
Peers in the weave network are identified by a 48-bit value formatted
like an ethernet MAC address e.g. `01:23:45:67:89:ab`. This 'peer
Weave Net peers are identified by a 48-bit value formatted like an
ethernet MAC address, for example, `01:23:45:67:89:ab`. The 'peer
name' is used for various purposes:
* Routing of packets between containers on the overlay network
* Recording the origin peer of DNS entries
* Recording ownership of IP address ranges
Whilst it is desirable for the peer name to remain stable across
restarts, it is essential that it is unique - if two or more peers
share the same name chaos will ensue, including but not limited to
double allocation of addresses and inability to route packets on the
overlay network. Consequently when the router is launched on a host it
derives its peer name in order of preference:
While it is desirable for the peer name to remain stable across
restarts, it is essential that it is unique. If two or more peers
share the same name chaos will ensue, which includes but is not limited to
double allocation of addresses and the inability to route packets on the
overlay network.
* From the command line; user is responsible for uniqueness and
When the router is launched on a host, it derives its peer name in order of preference:
* From the command line, where the user is responsible for uniqueness and
stability
* From the BIOS product UUID, which is generally stable across
restarts and unique across different physical hardware and certain
@@ -51,8 +52,8 @@ detail below.
## Peer Discovery
Peer discovery is a mechanism which allows peers to learn about new
weave hosts from existing peers without being explicitly told. Peer
Peer discovery is a mechanism that allows peers to learn about new
Weave Net hosts from existing peers without being explicitly told. Peer
discovery is
[enabled by default](/site/using-weave/finding-adding-hosts-dynamically.md).
@@ -61,7 +62,7 @@ discovery is
A network partition is a transient condition whereby some arbitrary
subsets of peers are unable to communicate with each other for the
duration - perhaps because a network switch has failed, or a fibre
optic line severed. Weave is designed to allow peers and their
optic line severed. Weave Net is designed to allow peers and their
containers to make maximum safe progress under conditions of
partition, healing automatically once the partition is over.
@@ -84,14 +85,14 @@ constraints and consequences:
* Every peer added to the network _must_ receive the same seed list,
for all time, or they will not be able to join together to form a
single cohesive whole
single cohesive whole.
* Because the 'product UUID' and 'random value' methods of peer name
assignment are unpredictable, the end user must by necessity also
specify peer names
specify peer names.
* Even though every peer _must_ receive the same seed, that seed does
_not_ have to include every peer in the network, nor does it have to
be updated when new peers are added (in fact due to the first
constraint above it may not be)
constraint above it may not be).
Example configurations are given in the section on deployment
@@ -101,12 +102,12 @@ scenarios:
### Consensus
Alternatively, when a new network is formed for the first time peers
Alternatively, when a new network is formed for the first time, peers
can be configured to co-ordinate amongst themselves to automatically
divide up the IP allocation range. This process is known as consensus
and requires each peer to be told the total number of expected peers
(the 'initial peer count') in order to prevent formation of disjoint
groups of peers which would, ultimately, result in duplicate IP
and it requires each peer to be told the total number of expected peers
(the 'initial peer count') in order to prevent the formation of disjoint
peer groups which would, ultimately, result in duplicate IP
addresses.
Example configurations are given in the section on deployment
@@ -118,10 +119,10 @@ scenarios:
### Observers
Finally, an option is provided to start a peer as an _observer_. Such
peers do not require either a seed peer name list nor an initial peer
peers do not require a seed peer name list or an initial peer
count; instead they rely on the existence of other peers in the
network which have been so configured. When an observer needs address
space, it will ask for it from one of the peers which partook of the
space, it asks for it from one of the peers which partook of the
initial division, triggering consensus if necessary.
Example configurations are given in the section on deployment
@@ -131,7 +132,7 @@ scenarios:
## Persistence
Certain information is remembered between launches of weave (for
Certain information is remembered between launches of Weave Net (for
example across reboots):
* The division of the IP allocation range amongst peers
site/operational-guide/interactive.md
+25 -23
View File
@@ -2,55 +2,57 @@
title: Interactive Deployment
menu_order: 20
---
Weave Net can be launched interactively on the command line, and as
long as Docker is configured to start on boot, the network will survive
host reboots without the use of a systemd. However, since launching
Weave Net commands in interactive mode is not amenable to automation
and configuration management, it is recommended that deploying Weave Net
in this mode be reserved for exploration and evaluation only.
This pattern is recommended for exploration and evaluation only, as
the commands described herein are interactive and not readily amenable
to automation and configuration management. Nevertheless, the
resulting weave network will survive host reboots without the use of a
systemd unit as long as Docker is configured to start on boot.
### Bootstrapping
### Bootstrap
On initial peer:
On the initial peer:
weave launch
### Add a Peer
### Adding a Peer
On new peer:
On the new peer:
weave launch <extant peers>
Where `<extant peers>` means all peers in the network, initial and
subsequently added, which have not been explicitly removed. It should
include peers which are temporarily offline or stopped.
Where,
You must then execute:
* `<extant peers>` indicates all peers in the network, initial and
subsequently added, which have not been explicitly removed. It should
include peers that are temporarily offline or stopped.
To ensure that the new peer has joined the existing network,
execute the following:
weave prime
to ensure that the new peer has joined to the existing network; you
_must_ wait for this to complete successfully before moving on to add
further new peers. If this command blocks it means that there is some
Before adding any new peers, you _must_ wait for this to complete.
If this command waits and does not exit, it means that there is some
issue (such as a network partition or failed peers) that is preventing
a quorum from being reached - you will need to [address
that](/site/troubleshooting.md) before moving on.
### Stop a Peer
### Stopping a Peer
A peer can be stopped temporarily with the following command:
weave stop
Such a peer will remember IP address allocation information on the
A temporarily stopped peer will remember IP address allocation information on the
next `weave launch` but will forget any discovered peers or
modifications to the initial peer list that were made with `weave
connect` or `weave forget`. Note that if the host reboots, Docker
will restart the peer automatically.
automatically restarts the peer.
### Remove a Peer
### Removing a Peer
On peer to be removed:
On the peer to be removed:
weave reset
@@ -60,4 +62,4 @@ Then optionally on each remaining peer:
This step is not mandatory, but it will eliminate log noise and
spurious network traffic by stopping reconnection attempts and
preventing further connection attempts after restart.
preventing further connection attempts after a restart.
site/operational-guide/tasks.md
+60 -40
View File
@@ -2,26 +2,38 @@
title: Administrative Tasks
menu_order: 60
---
##<a name="start-on-boot"></a>Configure Weave to Start Automatically on Boot
`weave launch` runs all weave's containers with a Docker restart
policy of `always`, so as long as you have launched weave manually
once and your system is configured to start Docker on boot then weave
will be started automatically on system restarts.
The following administrative tasks are discussed:
If you're aiming for a non-interactive installation, you can use
systemd to launch weave after Docker - see [systemd
docs](/site/installing-weave/systemd.md) for details.
* [Configuring Weave Net to Start Automatically on Boot](#start-on-boot)
* [Detecting and Reclaiming Lost IP Address Space](#detect-reclaim-ipam)
*[Manually Reclaiming Lost Address Space](#reclaim-address-space)
* [Upgrading a Cluster](#cluster-upgrade)
* [Resetting Persisted Data](#reset)
##<a name="detect-reclaim-ipam"></a>Detect and Reclaim Lost IP Address Space
The recommended way of removing a peer is to run `weave reset` on that
peer before the underlying host is decommissioned or repurposed - this
##<a name="start-on-boot"></a>Configuring Weave Net to Start Automatically on Boot
`weave launch` runs all of Weave Net's containers with a Docker restart
policy set to `always`. If you have launched Weave Net manually at least
once and your system is configured to start Docker on boot, then Weave Net
will start automatically on system restarts.
If you are aiming for a non-interactive installation, use
[systemd]((/site/installing-weave/systemd.md) or a similar init system to
launch Weave using the `--no-restart` flag after Docker has been started.
##<a name="detect-reclaim-ipam"></a>Detecting and Reclaiming Lost IP Address Space
The recommended method of removing a peer is to run `weave reset` on that
peer before the underlying host is decommissioned or repurposed. This
ensures that the portion of the IPAM allocation range assigned to the
peer is released for reuse. Under certain circumstances this operation
may not be successful, or indeed possible:
peer is released for reuse.
* If the peer in question is partitioned from the rest of the network
Under certain circumstances this operation may not be successful,
or possible:
* If the peer is partitioned from the rest of the network
when `weave reset` is executed on it
* If the underlying host is no longer available to execute `weave
reset` due to a hardware failure or other unplanned termination (for
@@ -30,8 +42,8 @@ may not be successful, or indeed possible:
In some cases you may already be aware of the problem, as you were
unable to execute `weave reset` successfully or because you know
through other channels that the host has died - in these cases you can
proceed straight to the section on reclaiming lost space.
through other channels that the host has died. In these cases you can
proceed straight to [Manually Reclaiming Lost Address Space](#reclaim-address-space).
However in some scenarios it may not be obvious that space has been
lost, in which case you can check for it periodically with the
@@ -39,56 +51,58 @@ following command on any peer:
weave status ipam
This will list the names of unreachable peers; if you are satisifed
that they are truly gone, rather than temporarily unreachable due to a
This command displays the peer names and nicknames, absolute quantity/percentage of allocation
range managed by peer and also identifies the names of unreachable peers. If you are satisifed
that the peer is truly gone, rather than temporarily unreachable due to a
partition, you can reclaim their space manually.
###<a name="manually-reclaim-address-space"></a>Manually Reclaiming Address Space
When a peer dies unexpectedly the remaining peers will consider its
address space to be unavailable even after it has remained unreachable
for prolonged periods; there is no universally applicable time limit
for prolonged periods. There is no universally applicable time limit
after which one of the remaining peers could decide unilaterally that
it is safe to appropriate the space for itself, and so an
administrative action is required to reclaim it.
The `weave rmpeer` command is provided to perform this task, and must
be executed on _one_ of the remaining peers. That peer will take
The [`weave rmpeer`](/site/ipam/stop-remove-peers-ipam.md)
command is provided to perform this task, and must
be executed on _one_ of the remaining peers. That peer will then take
ownership of the freed address space.
##<a name="cluster-upgrade"></a>Upgrade a Cluster
##<a name="cluster-upgrade"></a>Upgrading a Cluster
Protocol versioning and feature negotiation are employed in Weave Net
to enable incremental rolling upgrades - each major release maintains
to enable incremental rolling upgrades. Each major maintains
the ability to speak to the preceding major release at a minimum, and
connected peers only utilise features which both support. The general
upgrade procedure is as follows:
connected peers only utilize features which both support.
On each peer in turn:
The general upgrade procedure is as follows:
* Stop the old weave with `weave stop` (or `systemctl stop weave` if
On each peer:
* Stop the old Weave Net with `weave stop` (or `systemctl stop weave` if
you're using a systemd unit file)
* Download the new weave script and replace the existing one
* Start the new weave with `weave launch <existing peer list>` (or
* Download the new Weave Net script and replace the existing one
* Start the new Weave with `weave launch <existing peer list>` (or
`systemctl start weave` if you're using a systemd unit file)
This will result in some downtime as the first launch with the new
script has to pull the new container images; if you wish to minimise
downtime you can download the new script to a temporary location
first:
To minimize downtime while the new script is pulling the new container images:
* Download the new weave script to a temporary location e.g.
* Download the new Weave Net script to a temporary location, for example,
`/path/to/new/weave`
* Pull the new images with `/path/to/new/weave setup`
* Stop the old weave with `weave stop` (or `systemctl stop weave` if
* Stop the old Weave Net with `weave stop` (or `systemctl stop weave` if
you're using a systemd unit file)
* Replace the existing script with the new one
* Start the new weave with `weave launch <existing peer list>` (or
* Start the new Weave Net with `weave launch <existing peer list>` (or
`systemctl start weave` if you're using a systemd unit file)
> NB Always check the release notes for specific versions in case
> there are any special caveats or deviations from the standard
> procedure.
>>**Note:** Always check the Release Notes for specific versions in case
there are any special caveats or deviations from the standard
procedure.
##<a name="reset"></a>Reset Persisted Data
##<a name="reset"></a>Resetting Persisted Data
Weave Net persists information in a data volume container named
`weavedb`. If you wish to start from a completely clean slate (for
@@ -96,4 +110,10 @@ example to withdraw a peer from one network and join it to another)
you can issue the following command:
weave reset
**See Also**
* [Allocating IP Addresses](/site/ipam.md)
* [Troubleshooting the IP Allocator](/site/ipam/troubleshooting-ipam.md)
site/operational-guide/uniform-dynamic-cluster.md
+19 -16
View File
@@ -6,7 +6,7 @@ menu_order: 50
A uniform dynamic cluster has the following characteristics:
* Recovers automatically after reboots and partitions.
* All peers have identical configuration.
* Identical configuration for all of its peers.
* Initial cluster peers can make progress from the outset even if
bootstrapping occurs under conditions of partition.
* Once an initial cluster has been bootstrapped, arbitrary numbers of
@@ -14,16 +14,18 @@ A uniform dynamic cluster has the following characteristics:
it ideally suited for use with convergent provisioning tools that
operate across multiple hosts in an asynchronous fashion.
## Bootstrap
## Bootstrapping
On each initial peer N, at boot, via
[systemd](/site/installing-weave/systemd.md):
hostN$ weave launch --no-restart --name ::N --ipalloc-init seed=$SEED $PEERS
Where `$SEED` is a comma separated list of the _names_ of the initial
peers, e.g. `::1,::2,::3` and `$PEERS` is obtained from
`/etc/sysconfig/weave` as described in the linked systemd
Where,
* `$SEED` is a comma separated list of the _names_ of the initial
peers, for example, `::1,::2,::3` and,
* `$PEERS` is obtained from `/etc/sysconfig/weave` as described in the linked systemd
documentation, and includes the complete set of initial peer
_addresses_.
@@ -39,50 +41,51 @@ Where
SEED="::1,::2,::3"
PEERS="host1 host2 host3"
## Add a Peer
## Adding a Peer
On each new peer, at boot, via
[systemd](/site/installing-weave/systemd.md):
hostN$ weave launch --no-restart --name ::N --ipam-seed $SEED $PEERS
* `--no-restart` disables the Docker restart policy as this will be
Where,
* `--no-restart` disables the Docker restart policy, since this will be
handled by systemd.
* `--name` specifies a unique name for this new peer.
* `--ipam-seed` specifies the names of only those peers that were
involved in the initial cluster bootstrap - even if they have been
subsequently removed from the cluster. You can view this as a kind
of 'cluster identity' - peers may only interoperate in the same
of 'cluster identity',where peers may only interoperate in the same
cluster if they share the same seed.
* `$PEERS` is obtained from `/etc/sysconfig/weave` as described in the
linked systemd documentation. For convenience, this may contain the
address of the peer which is being launched, so that you don't have
address of the peer that is being launched, so that you don't have
to compute separate lists of 'other' peers tailored to each peer -
just supply the same complete list of peer addresses to every peer.
Note that unlike [Interactive](/site/operational-guide/interactive.md)
and [Uniform Fixed
Cluster](/site/operational-guide/uniform-fixed-cluster.md) deployments
there is no `weave prime` step; you can add as many new peers in
and [Uniform Fixed Cluster](/site/operational-guide/uniform-fixed-cluster.md) deployments
there is no `weave prime` step. You can add as many new peers in
parallel as you like, even under conditions of partition, and they
will all (eventually) join safely. This is ideal for use in
conjunction with asynchronous provisioning systems such as puppet or
chef.
For maximum robustness, you should then distribute an updated
For maximum robustness, you should distribute an updated
`/etc/sysconfig/weave` file including the new peer to all existing
peers.
### Remove a Peer
### Removing a Peer
On peer to be removed:
On the peer to be removed:
weave reset
You may remove a seed peer, as long as there is at least one other
seed peer left in the network.
Then distribute an updated `/etc/sysconfig/weave` to the remaining
Then, distribute an updated `/etc/sysconfig/weave` to the remaining
peers, omitting the removed peer from `$PEERS`.
On each remaining peer:
site/operational-guide/uniform-fixed-cluster.md
+27 -20
View File
@@ -4,29 +4,32 @@ menu_order: 30
---
This scenario describes a production deployment of a fixed number of N
nodes (N=1 in the simplest case). A uniform fixed cluster has the
following characteristics:
nodes (N=1 in the simplest case).
A uniform fixed cluster has the following characteristics:
* Recovers automatically from reboots and partitions.
* All peers have identical configuration.
* There is a controlled process for adding or removing nodes, however
the end user is responsible for ensuring that only one instance of
the process is in-flight at a time. Whilst it is possible to
the process is in-flight at a time. While it is possible to
automate, the potentially-blocking `weave prime` operation and the
need for global serialisation make it non-trivial; it is however
need for global serialization make it non-trivial. It is however
relatively straightforward for a human to provide the necessary
guarantees and exception handling manually, so this scenario is best
guarantees and exception handling manually, and so this scenario is best
suited to deployments which change size infrequently as a planned
maintenance event.
### Bootstrap
### Bootstrapping
On each initial peer, at boot, via
[systemd](/site/installing-weave/systemd.md):
weave launch --no-restart $PEERS
* `--no-restart` disables the Docker restart policy as this will be
Where,
* `--no-restart` disables the Docker restart policy, since this will be
handled by systemd.
* `$PEERS` is obtained from `/etc/sysconfig/weave` as described in the
linked systemd documentation. For convenience, this may contain the
@@ -34,38 +37,42 @@ On each initial peer, at boot, via
to compute separate lists of 'other' peers tailored to each peer -
just supply the same complete list of peer addresses to every peer.
Then on _any_ peer (note you can run this safely on more than one or
even all peers, but it's only strictly necessary to run it on one of
them):
Then on _any_ peer run the following to force consensus:
weave prime
to force consensus - once this completes successfully, IP address
>>**Note:** You can run this safely on more than one or
even all peers, but it's only strictly necessary to run it on one of
them.
Once this command completes successfully, IP address
allocations can proceed under partition and it is safe to add new
peers. If this command blocks it means that there is some issue (such
peers. If this command waits without exiting, it means that there is an issue (such
as a network partition or failed peers) that is preventing a quorum
from being reached – you will need to [address
that](/site/troubleshooting.md) before moving on.
### Add a Peer
### Adding a Peer
On new peer, at boot, via
[systemd](/site/installing-weave/systemd.md):
On the new peer, at boot, via
[systemd](/site/installing-weave/systemd.md) run:
weave launch --no-restart $PEERS
Where `$PEERS` is the new peer plus all other peers in the network,
Where,
* `$PEERS` is the new peer plus all other peers in the network,
initial and subsequently added, which have not been explicitly
removed. It should include peers which are temporarily offline or
stopped.
For maximum robustness, you should then distribute an updated
For maximum robustness, distribute an updated
`/etc/sysconfig/weave` file including the new peer to all existing
peers.
### Remove a Peer
### Removing a Peer
On peer to be removed:
On the peer to be removed:
weave reset
@@ -77,4 +84,4 @@ On each remaining peer:
weave forget <removed peer>
This final step is not mandatory, but it will eliminate log noise and
spurious network traffic by stopping reconnection attempts.
spurious network traffic by stopping any reconnection attempts.