reweave/weave
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Operational guide

Browse Source
This commit is contained in:
Adam Harrison committed 2016-06-03 17:12:29 +01:00
1 parent dbc8b4fe1a
commit e092a517aa
8 files changed
+539 -1
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

site/installing-weave/systemd.md
+1 -1
View File
@@ -19,7 +19,7 @@ normally placed in `/etc/systemd/system/weave.service`.
After=docker.service
[Service]
EnvironmentFile=-/etc/sysconfig/weave
ExecStartPre=/usr/local/bin/weave launch $PEERS
ExecStartPre=/usr/local/bin/weave launch --no-restart $PEERS
ExecStart=/usr/bin/docker attach weave
ExecStop=/usr/local/bin/weave stop
[Install]
site/operational-guide.md
+5
View File
@@ -0,0 +1,5 @@
---
title: Operational Guide
menu_order: 35
---
Operations manual.
site/operational-guide/autoscaling.md
+55
View File
@@ -0,0 +1,55 @@
---
title: Autoscaling
menu_order: 40
---
### Bootstrap
An autoscaling configuration begins with a fixed cluster:
* Configured as per the [Uniform Fixed Cluster](/site/operational-guide/uniform-fixed-cluster.md)
scenario.
* Hosted on reserved or protected instances to ensure long-term
stability.
* Ideally sized at a minimum of three or five nodes (you can make your
fixed cluster bigger to accommodate base load as required; a minimum
is specified here in the interests of resilience only.)
Building on this foundation, arbitrary numbers of dynamic peers can be
added or removed concurrently as desired, without requiring any
changes to the configuration of the fixed cluster. As with the fixed
cluster, dynamically added nodes recover automatically from reboots
and partitions.
### Scale-out
On additional dynamic peer, at boot, via
[systemd](/site/installing-weave/systemd.md) or equivalent:
weave launch --no-restart --ipalloc-init=observer $PEERS
In this case `$PEERS` means all peers in the _fixed cluster_, initial
and subsequently added, which have not been explicitly removed. It
should include fixed peers which are temporarily offline or stopped.
Note that you do not have to keep track of and specify the addresses
of other dynamic peers in `$PEERS` - they will discover and connect to
each other via the fixed cluster.
> The use of `--ipalloc-init=observer` prevents dynamic peers from
> coming to a consensus on their own - this is important to stop a
> clique forming amongst a group of dynamically added peers if they
> become partitioned from the fixed cluster after having learned about
> each other via discovery.
### Scale-in
On dynamic peer to be removed:
weave reset
If for any reason you cannot arrange for `weave reset` to be run on
the peer before the underlying host is destroyed (for example when
using spot instances that can be destroyed without notice), you will
need an asynchronous process to [reclaim lost IP address
space](/site/operational-guide/tasks.md#detect-reclaim-ipam).
site/operational-guide/concepts.md
+143
View File
@@ -0,0 +1,143 @@
---
title: Concepts
menu_order: 10
---
This section describes some essential concepts with which you will
need to be familiar before moving on to the example deployment
scenarios.
## Host
For the purposes of this documentation we consider a host to be an
installation of the Linux operating system which is running an
instance of the Docker Engine. It may be executing directly on bare
hardware or inside a virtual machine.
## Peer
A peer is a running instance of Weave Net, typically one per host.
## Peer Name
Peers in the weave network are identified by a 48-bit value formatted
like an ethernet MAC address e.g. `01:23:45:67:89:ab`. This 'peer
name' is used for various purposes:
* Routing of packets between containers on the overlay network
* Recording the origin peer of DNS entries
* Recording ownership of IP address ranges
Whilst it is desirable for the peer name to remain stable across
restarts, it is essential that it is unique - if two or more peers
share the same name chaos will ensue, including but not limited to
double allocation of addresses and inability to route packets on the
overlay network. Consequently when the router is launched on a host it
derives its peer name in order of preference:
* From the command line; user is responsible for uniqueness and
stability
* From the BIOS product UUID, which is generally stable across
restarts and unique across different physical hardware and certain
cloned VMs
* From the hypervisor UUID, which is generally stable across restarts
and unique across VMs which do not provide access to a BIOS product
UUID
* From a random value, practically unique across different physical
hardware and cloned VMs but not stable across restarts
The appropriate strategy for assigning peer names depends on the type
and method of your particular deployment and is discussed in more
detail below.
## Peer Discovery
Peer discovery is a mechanism which allows peers to learn about new
weave hosts from existing peers without being explicitly told. Peer
discovery is
[enabled by default](/site/using-weave/finding-adding-hosts-dynamically.md).
## Network Partition
A network partition is a transient condition whereby some arbitrary
subsets of peers are unable to communicate with each other for the
duration - perhaps because a network switch has failed, or a fibre
optic line severed. Weave is designed to allow peers and their
containers to make maximum safe progress under conditions of
partition, healing automatically once the partition is over.
## IP Address Manager (IPAM)
[IPAM](/site/ipam.md) is the subsystem responsible for dividing up a
large contiguous block of IP addresses (known as the IP allocation
range) amongst peers so that individual addresses may be uniquely
assigned to containers anywhere on the overlay network.
When a new network is formed an initial division of the IP allocation
range must be made. Two (mutually exclusive) mechanisms with different
tradeoffs are provided to perform this task: seeding and consensus.
### Seeding
Seeding requires each peer to be told the list of peer names amongst
which the address space is to be divided initially. There are some
constraints and consequences:
* Every peer added to the network _must_ receive the same seed list,
for all time, or they will not be able to join together to form a
single cohesive whole
* Because the 'product UUID' and 'random value' methods of peer name
assignment are unpredictable, the end user must by necessity also
specify peer names
* Even though every peer _must_ receive the same seed, that seed does
_not_ have to include every peer in the network, nor does it have to
be updated when new peers are added (in fact due to the first
constraint above it may not be)
Example configurations are given in the section on deployment
scenarios:
* [Uniform Dynamic Cluster](/site/operational-guide/uniform-dynamic-cluster.md)
### Consensus
Alternatively, when a new network is formed for the first time peers
can be configured to co-ordinate amongst themselves to automatically
divide up the IP allocation range. This process is known as consensus
and requires each peer to be told the total number of expected peers
(the 'initial peer count') in order to prevent formation of disjoint
groups of peers which would, ultimately, result in duplicate IP
addresses.
Example configurations are given in the section on deployment
scenarios:
* [Interactive Deployment](/site/operational-guide/interactive.md)
* [Uniform Fixed Cluster](/site/operational-guide/uniform-fixed-cluster.md)
### Observers
Finally, an option is provided to start a peer as an _observer_. Such
peers do not require either a seed peer name list nor an initial peer
count; instead they rely on the existence of other peers in the
network which have been so configured. When an observer needs address
space, it will ask for it from one of the peers which partook of the
initial division, triggering consensus if necessary.
Example configurations are given in the section on deployment
scenarios:
* [Autoscaling](/site/operational-guide/autoscaling.md)
## Persistence
Certain information is remembered between launches of weave (for
example across reboots):
* The division of the IP allocation range amongst peers
* Allocation of addresses to containers
The persistence of this information is managed transparently in a
volume container but can be
[destroyed explicitly](/site/operational-guide/tasks.md#reset)
if necessary.
site/operational-guide/interactive.md
+63
View File
@@ -0,0 +1,63 @@
---
title: Interactive Deployment
menu_order: 20
---
This pattern is recommended for exploration and evaluation only, as
the commands described herein are interactive and not readily amenable
to automation and configuration management. Nevertheless, the
resulting weave network will survive host reboots without the use of a
systemd unit as long as Docker is configured to start on boot.
### Bootstrap
On initial peer:
weave launch
### Add a Peer
On new peer:
weave launch <extant peers>
Where `<extant peers>` means all peers in the network, initial and
subsequently added, which have not been explicitly removed. It should
include peers which are temporarily offline or stopped.
You must then execute:
weave prime
to ensure that the new peer has joined to the existing network; you
_must_ wait for this to complete successfully before moving on to add
further new peers. If this command blocks it means that there is some
issue (such as a network partition or failed peers) that is preventing
a quorum from being reached - you will need to [address
that](/site/troubleshooting.md) before moving on.
### Stop a Peer
A peer can be stopped temporarily with the following command:
weave stop
Such a peer will remember IP address allocation information on the
next `weave launch` but will forget any discovered peers or
modifications to the initial peer list that were made with `weave
connect` or `weave forget`. Note that if the host reboots, Docker
will restart the peer automatically.
### Remove a Peer
On peer to be removed:
weave reset
Then optionally on each remaining peer:
weave forget <removed peer>
This step is not mandatory, but it will eliminate log noise and
spurious network traffic by stopping reconnection attempts and
preventing further connection attempts after restart.
site/operational-guide/tasks.md
+99
View File
@@ -0,0 +1,99 @@
---
title: Administrative Tasks
menu_order: 60
---
##<a name="start-on-boot"></a>Configure Weave to Start Automatically on Boot
`weave launch` runs all weave's containers with a Docker restart
policy of `always`, so as long as you have launched weave manually
once and your system is configured to start Docker on boot then weave
will be started automatically on system restarts.
If you're aiming for a non-interactive installation, you can use
systemd to launch weave after Docker - see [systemd
docs](/site/installing-weave/systemd.md) for details.
##<a name="detect-reclaim-ipam"></a>Detect and Reclaim Lost IP Address Space
The recommended way of removing a peer is to run `weave reset` on that
peer before the underlying host is decommissioned or repurposed - this
ensures that the portion of the IPAM allocation range assigned to the
peer is released for reuse. Under certain circumstances this operation
may not be successful, or indeed possible:
* If the peer in question is partitioned from the rest of the network
when `weave reset` is executed on it
* If the underlying host is no longer available to execute `weave
reset` due to a hardware failure or other unplanned termination (for
example when using autoscaling with spot-instances that can be
destroyed without notice)
In some cases you may already be aware of the problem, as you were
unable to execute `weave reset` successfully or because you know
through other channels that the host has died - in these cases you can
proceed straight to the section on reclaiming lost space.
However in some scenarios it may not be obvious that space has been
lost, in which case you can check for it periodically with the
following command on any peer:
weave status ipam
This will list the names of unreachable peers; if you are satisifed
that they are truly gone, rather than temporarily unreachable due to a
partition, you can reclaim their space manually.
When a peer dies unexpectedly the remaining peers will consider its
address space to be unavailable even after it has remained unreachable
for prolonged periods; there is no universally applicable time limit
after which one of the remaining peers could decide unilaterally that
it is safe to appropriate the space for itself, and so an
administrative action is required to reclaim it.
The `weave rmpeer` command is provided to perform this task, and must
be executed on _one_ of the remaining peers. That peer will take
ownership of the freed address space.
##<a name="cluster-upgrade"></a>Upgrade a Cluster
Protocol versioning and feature negotiation are employed in Weave Net
to enable incremental rolling upgrades - each major release maintains
the ability to speak to the preceding major release at a minimum, and
connected peers only utilise features which both support. The general
upgrade procedure is as follows:
On each peer in turn:
* Stop the old weave with `weave stop` (or `systemctl stop weave` if
you're using a systemd unit file)
* Download the new weave script and replace the existing one
* Start the new weave with `weave launch <existing peer list>` (or
`systemctl start weave` if you're using a systemd unit file)
This will result in some downtime as the first launch with the new
script has to pull the new container images; if you wish to minimise
downtime you can download the new script to a temporary location
first:
* Download the new weave script to a temporary location e.g.
`/path/to/new/weave`
* Pull the new images with `/path/to/new/weave setup`
* Stop the old weave with `weave stop` (or `systemctl stop weave` if
you're using a systemd unit file)
* Replace the existing script with the new one
* Start the new weave with `weave launch <existing peer list>` (or
`systemctl start weave` if you're using a systemd unit file)
> NB Always check the release notes for specific versions in case
> there are any special caveats or deviations from the standard
> procedure.
##<a name="reset"></a>Reset Persisted Data
Weave Net persists information in a data volume container named
`weavedb`. If you wish to start from a completely clean slate (for
example to withdraw a peer from one network and join it to another)
you can issue the following command:
weave reset
site/operational-guide/uniform-dynamic-cluster.md
+93
View File
@@ -0,0 +1,93 @@
---
title: Uniform Dynamic Clusters
menu_order: 50
---
A uniform dynamic cluster has the following characteristics:
* Recovers automatically after reboots and partitions.
* All peers have identical configuration.
* Initial cluster peers can make progress from the outset even if
bootstrapping occurs under conditions of partition.
* Once an initial cluster has been bootstrapped, arbitrary numbers of
new peers can be added in parallel without coordination. This makes
it ideally suited for use with convergent provisioning tools that
operate across multiple hosts in an asynchronous fashion.
## Bootstrap
On each initial peer N, at boot, via
[systemd](/site/installing-weave/systemd.md):
hostN$ weave launch --no-restart --name ::N --ipalloc-init seed=$SEED $PEERS
Where `$SEED` is a comma separated list of the _names_ of the initial
peers, e.g. `::1,::2,::3` and `$PEERS` is obtained from
`/etc/sysconfig/weave` as described in the linked systemd
documentation, and includes the complete set of initial peer
_addresses_.
For example, if you have three initial peers you would specify the
following:
host1$ weave launch --no-restart --name ::1 --ipalloc-init seed=$SEED $PEERS
host2$ weave launch --no-restart --name ::2 --ipalloc-init seed=$SEED $PEERS
host3$ weave launch --no-restart --name ::3 --ipalloc-init seed=$SEED $PEERS
Where
SEED="::1,::2,::3"
PEERS="host1 host2 host3"
## Add a Peer
On each new peer, at boot, via
[systemd](/site/installing-weave/systemd.md):
hostN$ weave launch --no-restart --name ::N --ipam-seed $SEED $PEERS
* `--no-restart` disables the Docker restart policy as this will be
handled by systemd.
* `--name` specifies a unique name for this new peer.
* `--ipam-seed` specifies the names of only those peers that were
involved in the initial cluster bootstrap - even if they have been
subsequently removed from the cluster. You can view this as a kind
of 'cluster identity' - peers may only interoperate in the same
cluster if they share the same seed.
* `$PEERS` is obtained from `/etc/sysconfig/weave` as described in the
linked systemd documentation. For convenience, this may contain the
address of the peer which is being launched, so that you don't have
to compute separate lists of 'other' peers tailored to each peer -
just supply the same complete list of peer addresses to every peer.
Note that unlike [Interactive](/site/operational-guide/interactive.md)
and [Uniform Fixed
Cluster](/site/operational-guide/uniform-fixed-cluster.md) deployments
there is no `weave prime` step; you can add as many new peers in
parallel as you like, even under conditions of partition, and they
will all (eventually) join safely. This is ideal for use in
conjunction with asynchronous provisioning systems such as puppet or
chef.
For maximum robustness, you should then distribute an updated
`/etc/sysconfig/weave` file including the new peer to all existing
peers.
### Remove a Peer
On peer to be removed:
weave reset
You may remove a seed peer, as long as there is at least one other
seed peer left in the network.
Then distribute an updated `/etc/sysconfig/weave` to the remaining
peers, omitting the removed peer from `$PEERS`.
On each remaining peer:
weave forget <removed peer>
This step is not mandatory, but it will eliminate log noise and
spurious network traffic by stopping reconnection attempts.
site/operational-guide/uniform-fixed-cluster.md
+80
View File
@@ -0,0 +1,80 @@
---
title: Uniform Fixed Clusters
menu_order: 30
---
This scenario describes a production deployment of a fixed number of N
nodes (N=1 in the simplest case). A uniform fixed cluster has the
following characteristics:
* Recovers automatically from reboots and partitions.
* All peers have identical configuration.
* There is a controlled process for adding or removing nodes, however
the end user is responsible for ensuring that only one instance of
the process is in-flight at a time. Whilst it is possible to
automate, the potentially-blocking `weave prime` operation and the
need for global serialisation make it non-trivial; it is however
relatively straightforward for a human to provide the necessary
guarantees and exception handling manually, so this scenario is best
suited to deployments which change size infrequently as a planned
maintenance event.
### Bootstrap
On each initial peer, at boot, via
[systemd](/site/installing-weave/systemd.md):
weave launch --no-restart $PEERS
* `--no-restart` disables the Docker restart policy as this will be
handled by systemd.
* `$PEERS` is obtained from `/etc/sysconfig/weave` as described in the
linked systemd documentation. For convenience, this may contain the
address of the peer which is being launched, so that you don't have
to compute separate lists of 'other' peers tailored to each peer -
just supply the same complete list of peer addresses to every peer.
Then on _any_ peer (note you can run this safely on more than one or
even all peers, but it's only strictly necessary to run it on one of
them):
weave prime
to force consensus - once this completes successfully, IP address
allocations can proceed under partition and it is safe to add new
peers. If this command blocks it means that there is some issue (such
as a network partition or failed peers) that is preventing a quorum
from being reached – you will need to [address
that](/site/troubleshooting.md) before moving on.
### Add a Peer
On new peer, at boot, via
[systemd](/site/installing-weave/systemd.md):
weave launch --no-restart $PEERS
Where `$PEERS` is the new peer plus all other peers in the network,
initial and subsequently added, which have not been explicitly
removed. It should include peers which are temporarily offline or
stopped.
For maximum robustness, you should then distribute an updated
`/etc/sysconfig/weave` file including the new peer to all existing
peers.
### Remove a Peer
On peer to be removed:
weave reset
Then distribute an updated `/etc/sysconfig/weave` to the remaining
peers, omitting the removed peer from `$PEERS`.
On each remaining peer:
weave forget <removed peer>
This final step is not mandatory, but it will eliminate log noise and
spurious network traffic by stopping reconnection attempts.