From 60c5e11871b0b01fd73c637949a5c37957ed168c Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Wed, 5 Apr 2017 13:54:18 +0000 Subject: [PATCH 1/3] Add pod resources and eviction to weave-kube doc --- site/kube-addon.md | 45 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/site/kube-addon.md b/site/kube-addon.md index bfd5f4de..e9cab421 100644 --- a/site/kube-addon.md +++ b/site/kube-addon.md @@ -8,6 +8,8 @@ The following topics are discussed: * [Installation](#install) * [Upgrading Kubernetes to version 1.6](#kube-1.6-upgrade) * [Upgrading the Daemon Sets](#daemon-sets) + * [CPU and Memory Requirements](#resources) + * [Pod Eviction](#eviction) * [Network Policy Controller](#npc) * [Troubleshooting Blocked Connections](#blocked-connections) * [Changing Configuration Options](#configuration-options) @@ -91,6 +93,49 @@ and so you will need to perform the procedure manually: lose track of IP address range ownership, possibly leading to duplicate IP addresses if you then start a new copy of Weave Net. +## CPU and Memory Requirements + +Kubernetes manages +[resources](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/) +on each node, and will only schedule pods to run on nodes which have +enough free resources. A typical Kubernetes install will have +components (etcd, scheduler, api-server, ...) totalling 95% of one +CPU, leaving very little room to run anything else. Yet Weave Net +must run on every node, including master, for all features to work. + +The best way to resolve this is to use machines with at least two CPU +cores. However, people installing for the first time will probably not +know this, so we write into the DaemonSet specification that Weave Net +only needs 1% CPU for each container, and thus it will start up on a +single-CPU node. + +Weave Net will, depending on workload, need more than 1% of CPU, but +the number in the DaemonSet is a minimum request, not a limit, so +Weave Net can "burst" above that level. + +## Pod Eviction + +If a node runs out of CPU, memory or disk, Kubernetes may [decide to +evict](https://kubernetes.io/docs/concepts/cluster-administration/out-of-resource/) +one or more pods. It may choose to evict the Weave Net pod, which will +disrupt pod network operations. + +You can reduce the chance of eviction by changing the DaemonSet to +have a much bigger request, and a limit of the same value. This will +cause Kubernetes to apply ["guaranteed" rather than "burstable" +policy](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-qos.md). +However there is no similar request for disk space, so you should be +aware of this issue and monitor your resources to stay below 100%. + +Kubernetes will give a notification in the event of an eviction, like this: + +``` +pod weave-net-4ozht_kube-system(546acee0-ee25-11e6-8965-068f417b4097) evicted successfully +``` + +If you see this in your cluster, consider some of the above steps to +reduce disruption. + ## Network Policy Controller The addon also supports the [Kubernetes policy From 109e249aa6dcb35c6b6e167bd3e97901ad9750d8 Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Tue, 25 Apr 2017 14:08:25 +0000 Subject: [PATCH 2/3] Review feedback --- site/kube-addon.md | 48 +++++++++++++++++++++++++++------------------- 1 file changed, 28 insertions(+), 20 deletions(-) diff --git a/site/kube-addon.md b/site/kube-addon.md index e9cab421..2d9fb814 100644 --- a/site/kube-addon.md +++ b/site/kube-addon.md @@ -36,7 +36,8 @@ After a few seconds, a Weave Net pod should be running on each Node and any further pods you create will be automatically attached to the Weave network. -**Note:** This command requires Kubernetes 1.4 or later. +**Note:** This command requires Kubernetes 1.4 or later, and we +recommend your master node has at least two CPU cores. > CNI, the [_Container Network Interface_](https://github.com/containernetworking/cni), > is a proposed standard for configuring network interfaces for Linux @@ -97,21 +98,26 @@ and so you will need to perform the procedure manually: Kubernetes manages [resources](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/) -on each node, and will only schedule pods to run on nodes which have -enough free resources. A typical Kubernetes install will have -components (etcd, scheduler, api-server, ...) totalling 95% of one -CPU, leaving very little room to run anything else. Yet Weave Net -must run on every node, including master, for all features to work. +on each node, and only schedules pods to run on nodes that have enough +free resources. -The best way to resolve this is to use machines with at least two CPU -cores. However, people installing for the first time will probably not -know this, so we write into the DaemonSet specification that Weave Net -only needs 1% CPU for each container, and thus it will start up on a -single-CPU node. +The components of a typical Kubernetes installation (with the master +node running etcd, scheduler, api-server, etc.) take up about 95% of a +CPU, which leaves little room to run anything else. For all of Weave +Net's features to work, it must run on every node, including the +master. -Weave Net will, depending on workload, need more than 1% of CPU, but -the number in the DaemonSet is a minimum request, not a limit, so -Weave Net can "burst" above that level. +The best way to resolve this issue is to use machines with at least +two CPU cores. However if you are installing Kubernetes and Weave Net +for the first time, you may not be aware of this requirement. For this +reason, Weave Net launches as a DaemonSet with a specification that +reserves at least 1% CPU for each container. This enables Weave Net to +start up seamlessly on a single-CPU node. + +Depending on the workload, Weave Net may need more than 1% of the +CPU. The percentage set in the DaemonSet is the minimum and not a +limit. This minimum setting allows Weave Net to take advantage of +available CPU and "burst" above that limit if it needs to. ## Pod Eviction @@ -121,13 +127,15 @@ one or more pods. It may choose to evict the Weave Net pod, which will disrupt pod network operations. You can reduce the chance of eviction by changing the DaemonSet to -have a much bigger request, and a limit of the same value. This will -cause Kubernetes to apply ["guaranteed" rather than "burstable" -policy](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-qos.md). -However there is no similar request for disk space, so you should be -aware of this issue and monitor your resources to stay below 100%. +have a much bigger request, and a limit of the same value. -Kubernetes will give a notification in the event of an eviction, like this: +This causes Kubernetes to apply a ["guaranteed" rather than a +"burstable" policy](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-qos.md). +However a similar request for disk space can not +be made, and so please be aware of this issue and monitor your +resources to ensure that they stay below 100%. + +Kubernetes displays a notification in the event of an eviction similar to this: ``` pod weave-net-4ozht_kube-system(546acee0-ee25-11e6-8965-068f417b4097) evicted successfully From 909b751d6da8e2a879da87d3801241e08b372250 Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Mon, 8 May 2017 13:10:15 +0000 Subject: [PATCH 3/3] Explain how to see Evicted pods --- site/kube-addon.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/site/kube-addon.md b/site/kube-addon.md index 2d9fb814..87ef2fe2 100644 --- a/site/kube-addon.md +++ b/site/kube-addon.md @@ -135,10 +135,18 @@ However a similar request for disk space can not be made, and so please be aware of this issue and monitor your resources to ensure that they stay below 100%. -Kubernetes displays a notification in the event of an eviction similar to this: +You can see when pods have been evicted via the `kubectl get events` command ``` -pod weave-net-4ozht_kube-system(546acee0-ee25-11e6-8965-068f417b4097) evicted successfully +LASTSEEN COUNT NAME KIND TYPE REASON SOURCE MESSAGE +1m 1 mypod-09vkd Pod Warning Evicted kubelet, node-1 The node was low on resource: memory. +``` + +or `kubectl get pods` + +``` +NAME READY STATUS RESTARTS AGE IP NODE +mypod-09vkd 0/1 Evicted 0 1h node-1 ``` If you see this in your cluster, consider some of the above steps to