diff --git a/site/operational-guide/autoscaling.md b/site/operational-guide/autoscaling.md index 96f2ef13..59dcbf04 100644 --- a/site/operational-guide/autoscaling.md +++ b/site/operational-guide/autoscaling.md @@ -3,7 +3,7 @@ title: Autoscaling menu_order: 40 --- -### Bootstrap +### Bootstrapping An autoscaling configuration begins with a fixed cluster: @@ -21,30 +21,32 @@ changes to the configuration of the fixed cluster. As with the fixed cluster, dynamically added nodes recover automatically from reboots and partitions. -### Scale-out +### Scaling Out -On additional dynamic peer, at boot, via +On the additional dynamic peer, at boot, via [systemd](/site/installing-weave/systemd.md) or equivalent: weave launch --no-restart --ipalloc-init=observer $PEERS -In this case `$PEERS` means all peers in the _fixed cluster_, initial +Where, + + * `$PEERS` means all peers in the _fixed cluster_, initial and subsequently added, which have not been explicitly removed. It should include fixed peers which are temporarily offline or stopped. -Note that you do not have to keep track of and specify the addresses +You do not have to keep track of and specify the addresses of other dynamic peers in `$PEERS` - they will discover and connect to each other via the fixed cluster. -> The use of `--ipalloc-init=observer` prevents dynamic peers from -> coming to a consensus on their own - this is important to stop a -> clique forming amongst a group of dynamically added peers if they -> become partitioned from the fixed cluster after having learned about -> each other via discovery. +>>**Note:** The use of `--ipalloc-init=observer` prevents dynamic peers from +coming to a consensus on their own - this is important to stop a +clique forming amongst a group of dynamically added peers if they +become partitioned from the fixed cluster after having learned about +each other via discovery. -### Scale-in +### Scaling In -On dynamic peer to be removed: +On the dynamic peer to be removed: weave reset diff --git a/site/operational-guide/concepts.md b/site/operational-guide/concepts.md index edf33d73..e087fe59 100644 --- a/site/operational-guide/concepts.md +++ b/site/operational-guide/concepts.md @@ -2,15 +2,15 @@ title: Concepts menu_order: 10 --- -This section describes some essential concepts with which you will -need to be familiar before moving on to the example deployment +This section describes some of the essential concepts with which you will +need to be familiar before continuing to the example deployment scenarios. ## Host -For the purposes of this documentation we consider a host to be an -installation of the Linux operating system which is running an -instance of the Docker Engine. It may be executing directly on bare +For the purposes of this documentation a host is an +installation of the Linux operating system that is running an +instance of the Docker Engine. The host may be executing directly on bare hardware or inside a virtual machine. ## Peer @@ -19,22 +19,23 @@ A peer is a running instance of Weave Net, typically one per host. ## Peer Name -Peers in the weave network are identified by a 48-bit value formatted -like an ethernet MAC address e.g. `01:23:45:67:89:ab`. This 'peer +Weave Net peers are identified by a 48-bit value formatted like an +ethernet MAC address, for example, `01:23:45:67:89:ab`. The 'peer name' is used for various purposes: * Routing of packets between containers on the overlay network * Recording the origin peer of DNS entries * Recording ownership of IP address ranges -Whilst it is desirable for the peer name to remain stable across -restarts, it is essential that it is unique - if two or more peers -share the same name chaos will ensue, including but not limited to -double allocation of addresses and inability to route packets on the -overlay network. Consequently when the router is launched on a host it -derives its peer name in order of preference: +While it is desirable for the peer name to remain stable across +restarts, it is essential that it is unique. If two or more peers +share the same name chaos will ensue, which includes but is not limited to +double allocation of addresses and the inability to route packets on the +overlay network. -* From the command line; user is responsible for uniqueness and +When the router is launched on a host, it derives its peer name in order of preference: + +* From the command line, where the user is responsible for uniqueness and stability * From the BIOS product UUID, which is generally stable across restarts and unique across different physical hardware and certain @@ -51,8 +52,8 @@ detail below. ## Peer Discovery -Peer discovery is a mechanism which allows peers to learn about new -weave hosts from existing peers without being explicitly told. Peer +Peer discovery is a mechanism that allows peers to learn about new +Weave Net hosts from existing peers without being explicitly told. Peer discovery is [enabled by default](/site/using-weave/finding-adding-hosts-dynamically.md). @@ -61,7 +62,7 @@ discovery is A network partition is a transient condition whereby some arbitrary subsets of peers are unable to communicate with each other for the duration - perhaps because a network switch has failed, or a fibre -optic line severed. Weave is designed to allow peers and their +optic line severed. Weave Net is designed to allow peers and their containers to make maximum safe progress under conditions of partition, healing automatically once the partition is over. @@ -84,14 +85,14 @@ constraints and consequences: * Every peer added to the network _must_ receive the same seed list, for all time, or they will not be able to join together to form a - single cohesive whole + single cohesive whole. * Because the 'product UUID' and 'random value' methods of peer name assignment are unpredictable, the end user must by necessity also - specify peer names + specify peer names. * Even though every peer _must_ receive the same seed, that seed does _not_ have to include every peer in the network, nor does it have to be updated when new peers are added (in fact due to the first - constraint above it may not be) + constraint above it may not be). Example configurations are given in the section on deployment @@ -101,12 +102,12 @@ scenarios: ### Consensus -Alternatively, when a new network is formed for the first time peers +Alternatively, when a new network is formed for the first time, peers can be configured to co-ordinate amongst themselves to automatically divide up the IP allocation range. This process is known as consensus -and requires each peer to be told the total number of expected peers -(the 'initial peer count') in order to prevent formation of disjoint -groups of peers which would, ultimately, result in duplicate IP +and it requires each peer to be told the total number of expected peers +(the 'initial peer count') in order to prevent the formation of disjoint +peer groups which would, ultimately, result in duplicate IP addresses. Example configurations are given in the section on deployment @@ -118,10 +119,10 @@ scenarios: ### Observers Finally, an option is provided to start a peer as an _observer_. Such -peers do not require either a seed peer name list nor an initial peer +peers do not require a seed peer name list or an initial peer count; instead they rely on the existence of other peers in the network which have been so configured. When an observer needs address -space, it will ask for it from one of the peers which partook of the +space, it asks for it from one of the peers which partook of the initial division, triggering consensus if necessary. Example configurations are given in the section on deployment @@ -131,7 +132,7 @@ scenarios: ## Persistence -Certain information is remembered between launches of weave (for +Certain information is remembered between launches of Weave Net (for example across reboots): * The division of the IP allocation range amongst peers diff --git a/site/operational-guide/interactive.md b/site/operational-guide/interactive.md index 10fd6cb7..8a8feae8 100644 --- a/site/operational-guide/interactive.md +++ b/site/operational-guide/interactive.md @@ -2,55 +2,57 @@ title: Interactive Deployment menu_order: 20 --- +Weave Net can be launched interactively on the command line, and as +long as Docker is configured to start on boot, the network will survive +host reboots without the use of a systemd. However, since launching +Weave Net commands in interactive mode is not amenable to automation +and configuration management, it is recommended that deploying Weave Net +in this mode be reserved for exploration and evaluation only. -This pattern is recommended for exploration and evaluation only, as -the commands described herein are interactive and not readily amenable -to automation and configuration management. Nevertheless, the -resulting weave network will survive host reboots without the use of a -systemd unit as long as Docker is configured to start on boot. +### Bootstrapping -### Bootstrap - -On initial peer: +On the initial peer: weave launch -### Add a Peer +### Adding a Peer -On new peer: +On the new peer: weave launch -Where `` means all peers in the network, initial and -subsequently added, which have not been explicitly removed. It should -include peers which are temporarily offline or stopped. +Where, -You must then execute: +* `` indicates all peers in the network, initial and +subsequently added, which have not been explicitly removed. It should +include peers that are temporarily offline or stopped. + +To ensure that the new peer has joined the existing network, +execute the following: weave prime -to ensure that the new peer has joined to the existing network; you -_must_ wait for this to complete successfully before moving on to add -further new peers. If this command blocks it means that there is some +Before adding any new peers, you _must_ wait for this to complete. +If this command waits and does not exit, it means that there is some issue (such as a network partition or failed peers) that is preventing a quorum from being reached - you will need to [address that](/site/troubleshooting.md) before moving on. -### Stop a Peer +### Stopping a Peer A peer can be stopped temporarily with the following command: weave stop -Such a peer will remember IP address allocation information on the +A temporarily stopped peer will remember IP address allocation information on the next `weave launch` but will forget any discovered peers or modifications to the initial peer list that were made with `weave connect` or `weave forget`. Note that if the host reboots, Docker -will restart the peer automatically. +automatically restarts the peer. -### Remove a Peer +### Removing a Peer -On peer to be removed: +On the peer to be removed: weave reset @@ -60,4 +62,4 @@ Then optionally on each remaining peer: This step is not mandatory, but it will eliminate log noise and spurious network traffic by stopping reconnection attempts and -preventing further connection attempts after restart. +preventing further connection attempts after a restart. diff --git a/site/operational-guide/tasks.md b/site/operational-guide/tasks.md index 82f61bf0..d2ad985c 100644 --- a/site/operational-guide/tasks.md +++ b/site/operational-guide/tasks.md @@ -2,26 +2,38 @@ title: Administrative Tasks menu_order: 60 --- -##Configure Weave to Start Automatically on Boot -`weave launch` runs all weave's containers with a Docker restart -policy of `always`, so as long as you have launched weave manually -once and your system is configured to start Docker on boot then weave -will be started automatically on system restarts. +The following administrative tasks are discussed: -If you're aiming for a non-interactive installation, you can use -systemd to launch weave after Docker - see [systemd -docs](/site/installing-weave/systemd.md) for details. +* [Configuring Weave Net to Start Automatically on Boot](#start-on-boot) +* [Detecting and Reclaiming Lost IP Address Space](#detect-reclaim-ipam) + *[Manually Reclaiming Lost Address Space](#reclaim-address-space) +* [Upgrading a Cluster](#cluster-upgrade) +* [Resetting Persisted Data](#reset) -##Detect and Reclaim Lost IP Address Space -The recommended way of removing a peer is to run `weave reset` on that -peer before the underlying host is decommissioned or repurposed - this +##Configuring Weave Net to Start Automatically on Boot + +`weave launch` runs all of Weave Net's containers with a Docker restart +policy set to `always`. If you have launched Weave Net manually at least +once and your system is configured to start Docker on boot, then Weave Net +will start automatically on system restarts. + +If you are aiming for a non-interactive installation, use +[systemd]((/site/installing-weave/systemd.md) or a similar init system to +launch Weave using the `--no-restart` flag after Docker has been started. + +##Detecting and Reclaiming Lost IP Address Space + +The recommended method of removing a peer is to run `weave reset` on that +peer before the underlying host is decommissioned or repurposed. This ensures that the portion of the IPAM allocation range assigned to the -peer is released for reuse. Under certain circumstances this operation -may not be successful, or indeed possible: +peer is released for reuse. -* If the peer in question is partitioned from the rest of the network +Under certain circumstances this operation may not be successful, +or possible: + +* If the peer is partitioned from the rest of the network when `weave reset` is executed on it * If the underlying host is no longer available to execute `weave reset` due to a hardware failure or other unplanned termination (for @@ -30,8 +42,8 @@ may not be successful, or indeed possible: In some cases you may already be aware of the problem, as you were unable to execute `weave reset` successfully or because you know -through other channels that the host has died - in these cases you can -proceed straight to the section on reclaiming lost space. +through other channels that the host has died. In these cases you can +proceed straight to [Manually Reclaiming Lost Address Space](#reclaim-address-space). However in some scenarios it may not be obvious that space has been lost, in which case you can check for it periodically with the @@ -39,56 +51,58 @@ following command on any peer: weave status ipam -This will list the names of unreachable peers; if you are satisifed -that they are truly gone, rather than temporarily unreachable due to a +This command displays the peer names and nicknames, absolute quantity/percentage of allocation +range managed by peer and also identifies the names of unreachable peers. If you are satisifed +that the peer is truly gone, rather than temporarily unreachable due to a partition, you can reclaim their space manually. +###Manually Reclaiming Address Space + When a peer dies unexpectedly the remaining peers will consider its address space to be unavailable even after it has remained unreachable -for prolonged periods; there is no universally applicable time limit +for prolonged periods. There is no universally applicable time limit after which one of the remaining peers could decide unilaterally that it is safe to appropriate the space for itself, and so an administrative action is required to reclaim it. -The `weave rmpeer` command is provided to perform this task, and must -be executed on _one_ of the remaining peers. That peer will take +The [`weave rmpeer`](/site/ipam/stop-remove-peers-ipam.md) +command is provided to perform this task, and must +be executed on _one_ of the remaining peers. That peer will then take ownership of the freed address space. -##Upgrade a Cluster +##Upgrading a Cluster Protocol versioning and feature negotiation are employed in Weave Net -to enable incremental rolling upgrades - each major release maintains +to enable incremental rolling upgrades. Each major maintains the ability to speak to the preceding major release at a minimum, and -connected peers only utilise features which both support. The general -upgrade procedure is as follows: +connected peers only utilize features which both support. -On each peer in turn: +The general upgrade procedure is as follows: -* Stop the old weave with `weave stop` (or `systemctl stop weave` if +On each peer: + +* Stop the old Weave Net with `weave stop` (or `systemctl stop weave` if you're using a systemd unit file) -* Download the new weave script and replace the existing one -* Start the new weave with `weave launch ` (or +* Download the new Weave Net script and replace the existing one +* Start the new Weave with `weave launch ` (or `systemctl start weave` if you're using a systemd unit file) -This will result in some downtime as the first launch with the new -script has to pull the new container images; if you wish to minimise -downtime you can download the new script to a temporary location -first: +To minimize downtime while the new script is pulling the new container images: -* Download the new weave script to a temporary location e.g. +* Download the new Weave Net script to a temporary location, for example, `/path/to/new/weave` * Pull the new images with `/path/to/new/weave setup` -* Stop the old weave with `weave stop` (or `systemctl stop weave` if +* Stop the old Weave Net with `weave stop` (or `systemctl stop weave` if you're using a systemd unit file) * Replace the existing script with the new one -* Start the new weave with `weave launch ` (or +* Start the new Weave Net with `weave launch ` (or `systemctl start weave` if you're using a systemd unit file) -> NB Always check the release notes for specific versions in case -> there are any special caveats or deviations from the standard -> procedure. +>>**Note:** Always check the Release Notes for specific versions in case +there are any special caveats or deviations from the standard +procedure. -##Reset Persisted Data +##Resetting Persisted Data Weave Net persists information in a data volume container named `weavedb`. If you wish to start from a completely clean slate (for @@ -96,4 +110,10 @@ example to withdraw a peer from one network and join it to another) you can issue the following command: weave reset + + +**See Also** + + * [Allocating IP Addresses](/site/ipam.md) + * [Troubleshooting the IP Allocator](/site/ipam/troubleshooting-ipam.md) diff --git a/site/operational-guide/uniform-dynamic-cluster.md b/site/operational-guide/uniform-dynamic-cluster.md index 2ddbefb1..8a73022f 100644 --- a/site/operational-guide/uniform-dynamic-cluster.md +++ b/site/operational-guide/uniform-dynamic-cluster.md @@ -6,7 +6,7 @@ menu_order: 50 A uniform dynamic cluster has the following characteristics: * Recovers automatically after reboots and partitions. -* All peers have identical configuration. +* Identical configuration for all of its peers. * Initial cluster peers can make progress from the outset even if bootstrapping occurs under conditions of partition. * Once an initial cluster has been bootstrapped, arbitrary numbers of @@ -14,16 +14,18 @@ A uniform dynamic cluster has the following characteristics: it ideally suited for use with convergent provisioning tools that operate across multiple hosts in an asynchronous fashion. -## Bootstrap +## Bootstrapping On each initial peer N, at boot, via [systemd](/site/installing-weave/systemd.md): hostN$ weave launch --no-restart --name ::N --ipalloc-init seed=$SEED $PEERS -Where `$SEED` is a comma separated list of the _names_ of the initial -peers, e.g. `::1,::2,::3` and `$PEERS` is obtained from -`/etc/sysconfig/weave` as described in the linked systemd +Where, + +* `$SEED` is a comma separated list of the _names_ of the initial +peers, for example, `::1,::2,::3` and, +* `$PEERS` is obtained from `/etc/sysconfig/weave` as described in the linked systemd documentation, and includes the complete set of initial peer _addresses_. @@ -39,50 +41,51 @@ Where SEED="::1,::2,::3" PEERS="host1 host2 host3" -## Add a Peer +## Adding a Peer On each new peer, at boot, via [systemd](/site/installing-weave/systemd.md): hostN$ weave launch --no-restart --name ::N --ipam-seed $SEED $PEERS -* `--no-restart` disables the Docker restart policy as this will be +Where, + +* `--no-restart` disables the Docker restart policy, since this will be handled by systemd. * `--name` specifies a unique name for this new peer. * `--ipam-seed` specifies the names of only those peers that were involved in the initial cluster bootstrap - even if they have been subsequently removed from the cluster. You can view this as a kind - of 'cluster identity' - peers may only interoperate in the same + of 'cluster identity',where peers may only interoperate in the same cluster if they share the same seed. * `$PEERS` is obtained from `/etc/sysconfig/weave` as described in the linked systemd documentation. For convenience, this may contain the - address of the peer which is being launched, so that you don't have + address of the peer that is being launched, so that you don't have to compute separate lists of 'other' peers tailored to each peer - just supply the same complete list of peer addresses to every peer. Note that unlike [Interactive](/site/operational-guide/interactive.md) -and [Uniform Fixed -Cluster](/site/operational-guide/uniform-fixed-cluster.md) deployments -there is no `weave prime` step; you can add as many new peers in +and [Uniform Fixed Cluster](/site/operational-guide/uniform-fixed-cluster.md) deployments +there is no `weave prime` step. You can add as many new peers in parallel as you like, even under conditions of partition, and they will all (eventually) join safely. This is ideal for use in conjunction with asynchronous provisioning systems such as puppet or chef. -For maximum robustness, you should then distribute an updated +For maximum robustness, you should distribute an updated `/etc/sysconfig/weave` file including the new peer to all existing peers. -### Remove a Peer +### Removing a Peer -On peer to be removed: +On the peer to be removed: weave reset You may remove a seed peer, as long as there is at least one other seed peer left in the network. -Then distribute an updated `/etc/sysconfig/weave` to the remaining +Then, distribute an updated `/etc/sysconfig/weave` to the remaining peers, omitting the removed peer from `$PEERS`. On each remaining peer: diff --git a/site/operational-guide/uniform-fixed-cluster.md b/site/operational-guide/uniform-fixed-cluster.md index f46ec56c..eed3aa0f 100644 --- a/site/operational-guide/uniform-fixed-cluster.md +++ b/site/operational-guide/uniform-fixed-cluster.md @@ -4,29 +4,32 @@ menu_order: 30 --- This scenario describes a production deployment of a fixed number of N -nodes (N=1 in the simplest case). A uniform fixed cluster has the -following characteristics: +nodes (N=1 in the simplest case). + +A uniform fixed cluster has the following characteristics: * Recovers automatically from reboots and partitions. * All peers have identical configuration. * There is a controlled process for adding or removing nodes, however the end user is responsible for ensuring that only one instance of - the process is in-flight at a time. Whilst it is possible to + the process is in-flight at a time. While it is possible to automate, the potentially-blocking `weave prime` operation and the - need for global serialisation make it non-trivial; it is however + need for global serialization make it non-trivial. It is however relatively straightforward for a human to provide the necessary - guarantees and exception handling manually, so this scenario is best + guarantees and exception handling manually, and so this scenario is best suited to deployments which change size infrequently as a planned maintenance event. -### Bootstrap +### Bootstrapping On each initial peer, at boot, via [systemd](/site/installing-weave/systemd.md): weave launch --no-restart $PEERS -* `--no-restart` disables the Docker restart policy as this will be +Where, + +* `--no-restart` disables the Docker restart policy, since this will be handled by systemd. * `$PEERS` is obtained from `/etc/sysconfig/weave` as described in the linked systemd documentation. For convenience, this may contain the @@ -34,38 +37,42 @@ On each initial peer, at boot, via to compute separate lists of 'other' peers tailored to each peer - just supply the same complete list of peer addresses to every peer. -Then on _any_ peer (note you can run this safely on more than one or -even all peers, but it's only strictly necessary to run it on one of -them): +Then on _any_ peer run the following to force consensus: weave prime -to force consensus - once this completes successfully, IP address +>>**Note:** You can run this safely on more than one or +even all peers, but it's only strictly necessary to run it on one of +them. + +Once this command completes successfully, IP address allocations can proceed under partition and it is safe to add new -peers. If this command blocks it means that there is some issue (such +peers. If this command waits without exiting, it means that there is an issue (such as a network partition or failed peers) that is preventing a quorum from being reached – you will need to [address that](/site/troubleshooting.md) before moving on. -### Add a Peer +### Adding a Peer -On new peer, at boot, via -[systemd](/site/installing-weave/systemd.md): +On the new peer, at boot, via +[systemd](/site/installing-weave/systemd.md) run: weave launch --no-restart $PEERS -Where `$PEERS` is the new peer plus all other peers in the network, +Where, + +* `$PEERS` is the new peer plus all other peers in the network, initial and subsequently added, which have not been explicitly removed. It should include peers which are temporarily offline or stopped. -For maximum robustness, you should then distribute an updated +For maximum robustness, distribute an updated `/etc/sysconfig/weave` file including the new peer to all existing peers. -### Remove a Peer +### Removing a Peer -On peer to be removed: +On the peer to be removed: weave reset @@ -77,4 +84,4 @@ On each remaining peer: weave forget This final step is not mandatory, but it will eliminate log noise and -spurious network traffic by stopping reconnection attempts. +spurious network traffic by stopping any reconnection attempts.