From 8d44f49edd1b9f7c38dad8e43931d5c9853cebde Mon Sep 17 00:00:00 2001
From: Damien Lespiau <damien@weave.works>
Date: Fri, 28 Sep 2018 15:07:21 +0100
Subject: [PATCH] kubernetes: Morph the livenessProbe into readinessProbe

The liveness probe has the not-so-great side effect of killing the weave pods
if anything goes wrong, making debugging really difficult.

Not only that, we also bumped into the livenessProbe killing pods before they
had a chance to start (see below for the full story).

The readinessProbe is only used for Kubernetes Services: no traffic will be
routed to non-Ready pods when using the Service IP. We can use to to signal the
user something in wrong with Weave Net while not having Kubernetes delete the
pods behind our backs.

I removed the ininitialDelaySeconds field as Kurbernetes won't kill the pod if
not ready.

Notes from a debugging session:

Some nodes may have a vast amount of iptables rules, making each iptables
command take a few seconds (~5-10s). At startup, when the weave container
creates its chains, it runs quite a few iptables commands and the total time of
execution was > 30s.

It's important to note that, at this point, the weave process hasn't started as
those iptables commands are run by an initial script.

Kubernetes was trying to check for the container liveness, was failing, and
then proceeded to kill the weave pods before they had a chance to fully start.
---
 prog/weave-kube/weave-daemonset-k8s-1.6.yaml | 3 +--
 prog/weave-kube/weave-daemonset-k8s-1.7.yaml | 3 +--
 prog/weave-kube/weave-daemonset-k8s-1.8.yaml | 3 +--
 prog/weave-kube/weave-daemonset.yaml         | 3 +--
 4 files changed, 4 insertions(+), 8 deletions(-)

diff --git a/prog/weave-kube/weave-daemonset-k8s-1.6.yaml b/prog/weave-kube/weave-daemonset-k8s-1.6.yaml
index e7b0e18d..58c5fa4b 100644
--- a/prog/weave-kube/weave-daemonset-k8s-1.6.yaml
+++ b/prog/weave-kube/weave-daemonset-k8s-1.6.yaml
@@ -110,12 +110,11 @@ items:
                       fieldPath: spec.nodeName
               image: 'weaveworks/weave-kube:latest'
               imagePullPolicy: Always
-              livenessProbe:
+              readinessProbe:
                 httpGet:
                   host: 127.0.0.1
                   path: /status
                   port: 6784
-                initialDelaySeconds: 30
               resources:
                 requests:
                   cpu: 10m
diff --git a/prog/weave-kube/weave-daemonset-k8s-1.7.yaml b/prog/weave-kube/weave-daemonset-k8s-1.7.yaml
index 6f5ba5fc..4b0e4c53 100644
--- a/prog/weave-kube/weave-daemonset-k8s-1.7.yaml
+++ b/prog/weave-kube/weave-daemonset-k8s-1.7.yaml
@@ -119,12 +119,11 @@ items:
                       fieldPath: spec.nodeName
               image: 'weaveworks/weave-kube:latest'
               imagePullPolicy: Always
-              livenessProbe:
+              readinessProbe:
                 httpGet:
                   host: 127.0.0.1
                   path: /status
                   port: 6784
-                initialDelaySeconds: 30
               resources:
                 requests:
                   cpu: 10m
diff --git a/prog/weave-kube/weave-daemonset-k8s-1.8.yaml b/prog/weave-kube/weave-daemonset-k8s-1.8.yaml
index 87da2c8a..27c75145 100644
--- a/prog/weave-kube/weave-daemonset-k8s-1.8.yaml
+++ b/prog/weave-kube/weave-daemonset-k8s-1.8.yaml
@@ -127,12 +127,11 @@ items:
                       fieldPath: spec.nodeName
               image: 'weaveworks/weave-kube:latest'
               imagePullPolicy: Always
-              livenessProbe:
+              readinessProbe:
                 httpGet:
                   host: 127.0.0.1
                   path: /status
                   port: 6784
-                initialDelaySeconds: 30
               resources:
                 requests:
                   cpu: 10m
diff --git a/prog/weave-kube/weave-daemonset.yaml b/prog/weave-kube/weave-daemonset.yaml
index 57f0eac2..1eaf4adb 100644
--- a/prog/weave-kube/weave-daemonset.yaml
+++ b/prog/weave-kube/weave-daemonset.yaml
@@ -36,12 +36,11 @@ items:
                       fieldPath: spec.nodeName
               image: 'weaveworks/weave-kube:latest'
               imagePullPolicy: Always
-              livenessProbe:
+              readinessProbe:
                 httpGet:
                   host: 127.0.0.1
                   path: /status
                   port: 6784
-                initialDelaySeconds: 30
               resources:
                 requests:
                   cpu: 10m