From 9a5e1e333dc26af731e03d84c70914c0ec26921f Mon Sep 17 00:00:00 2001 From: Raj Chaudhuri Date: Wed, 8 Jun 2022 10:49:39 +0530 Subject: [PATCH 1/2] Update docs to include possible containerd issue. If a kubernetes cluster uses the containerd cri, and containerd is at version 1.6.0 through 1.6.4, weave does not work. This doc update highlights this and provides a solution, which is to upgrade containerd. --- site/kubernetes/kube-addon.md | 44 +++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/site/kubernetes/kube-addon.md b/site/kubernetes/kube-addon.md index 40ea4c2f..213d5045 100644 --- a/site/kubernetes/kube-addon.md +++ b/site/kubernetes/kube-addon.md @@ -19,6 +19,7 @@ The following topics are discussed: * [Reading the logs](#logs) * [Troubleshooting Blocked Connections](#blocked-connections) * [Things to watch out for](#key-points) + * [Troubleshooting FailedCreatePodSandBox errors](#failedcreatepodsandbox) * [Changing Configuration Options](#configuration-options) * [Securing The Setup](#securing-the-setup) @@ -370,6 +371,49 @@ UDP connection from 10.32.0.7:56648 to 10.32.0.11:80 blocked by Weave NPC. - Weave Net can be run on minikube v0.28 or later with the default CNI config shipped with minikube being disabled. See [#3124](https://github.com/weaveworks/weave/issues/3124#issuecomment-397820940) for more details. +- Weave Net has a problem with containerd versions 1.6.0 through 1.6.4. See [Troubleshooting FailedCreatePodSandBox errors](#failedcreatepodsandbox) below. + +### Troubleshooting FailedCreatePodSandBox errors + +If your Kubernetes cluster uses the `containerd` runtime (versions 1.6.0 through 1.6.4), Weave Net will not be able to allocate IP addresses to pods. Your pods, execept the ones that use HostNetworking, will be stuck at `ContainerCreating` status. + +You can examine any pod so affected by running `kubectl describe`, for example: + +``` +$ kubectl describe pod -n kube-system coredns-78fcd69978-dbxs9 +``` + +The events section will repeatedly show errors like the following: + +``` + Warning FailedCreatePodSandBox 3m6s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "09a23f79c96333b9f54e12df54e817837c8021cbaa32bdfeefbe2a1fb215d9ef": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/09a23f79c96333b9f54e12df54e817837c8021cbaa32bdfeefbe2a1fb215d9ef": dial tcp 127.0.0.1:6784: connect: connection refused +``` + +You can verify that you are running an affected version of containerd by using the following: + +``` +$ kubectl get nodes -o wide + +NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME +host-1 Ready control-plane,master 13m v1.22.10 172.21.107.129 Debian GNU/Linux 11 (bullseye) 5.10.0-14-amd64 containerd://1.6.4 +``` + +The last column shows the container runtime and version. + +Alternatively, you can run: + +``` +$ containerd -v +containerd containerd.io 1.6.4 212e8b6fa2f44b9c21b2798135fc6fb7c53efc16 +``` + +The problem can be solved by upgrading containerd to v1.6.5 or above. For example, on Debian Linux, using the docker official repositories, you can use: + +``` +sudo apt install containerd.io=1.6.6-1 +``` + +The problem occurs because of a behaviour change in cni v1.1.0, which caused a regression issue in Weave. It was corrected in cni v1.1.1. Containerd 1.6.5 onwards uses cni1.1.1. ## Changing Configuration Options From 1686ded5d52c87c4fa6d89c6522011c205a32013 Mon Sep 17 00:00:00 2001 From: Raj Chaudhuri Date: Thu, 9 Jun 2022 17:16:06 +0530 Subject: [PATCH 2/2] Apply suggestions from code review Co-authored-by: Kingdon Barrett --- site/kubernetes/kube-addon.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/site/kubernetes/kube-addon.md b/site/kubernetes/kube-addon.md index 213d5045..bbe8ba75 100644 --- a/site/kubernetes/kube-addon.md +++ b/site/kubernetes/kube-addon.md @@ -375,7 +375,7 @@ UDP connection from 10.32.0.7:56648 to 10.32.0.11:80 blocked by Weave NPC. ### Troubleshooting FailedCreatePodSandBox errors -If your Kubernetes cluster uses the `containerd` runtime (versions 1.6.0 through 1.6.4), Weave Net will not be able to allocate IP addresses to pods. Your pods, execept the ones that use HostNetworking, will be stuck at `ContainerCreating` status. +If your Kubernetes cluster uses the `containerd` runtime (versions 1.6.0 through 1.6.4), Weave Net will not be able to allocate IP addresses to pods. Your pods, except the ones that use HostNetworking, will be stuck at `ContainerCreating` status. You can examine any pod so affected by running `kubectl describe`, for example: @@ -383,7 +383,7 @@ You can examine any pod so affected by running `kubectl describe`, for example: $ kubectl describe pod -n kube-system coredns-78fcd69978-dbxs9 ``` -The events section will repeatedly show errors like the following: +The events section will show repeated errors like the following: ``` Warning FailedCreatePodSandBox 3m6s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "09a23f79c96333b9f54e12df54e817837c8021cbaa32bdfeefbe2a1fb215d9ef": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/09a23f79c96333b9f54e12df54e817837c8021cbaa32bdfeefbe2a1fb215d9ef": dial tcp 127.0.0.1:6784: connect: connection refused @@ -413,7 +413,7 @@ The problem can be solved by upgrading containerd to v1.6.5 or above. For exampl sudo apt install containerd.io=1.6.6-1 ``` -The problem occurs because of a behaviour change in cni v1.1.0, which caused a regression issue in Weave. It was corrected in cni v1.1.1. Containerd 1.6.5 onwards uses cni1.1.1. +The problem occurs because of a behaviour change in cni v1.1.0, which caused a regression issue in Weave. It was corrected in cni v1.1.1. Containerd 1.6.5 onwards uses cni 1.1.1 and above. ## Changing Configuration Options