Commit Graph
6924 Commits
Author SHA1 Message Date
Murali Reddy bc9939fcdc add delay before checking ipam status 2018-11-01 14:51:19 +05:30
Murali Reddy c48c746791 replace peerlist with current set of Kubernetes nodes 2018-10-31 18:06:48 +05:30
Murali Reddy f16d4561e0 remove realunch weave pod to test the unreachable_ip_addresses_count 2018-10-30 12:34:28 +05:30
Murali Reddy f334575e69 perform weave connect replace on node delete with empty peer list 2018-10-30 11:54:59 +05:30
Murali Reddy ea423755cc review comment: removing unnessary verbose code 2018-10-30 10:56:05 +05:30
Murali Reddy 4a40380d0f addressing review comments 2018-10-24 13:45:35 +05:30
Murali Reddy df467be26e watch for node delete events and call reclaimRemovedPeers 2018-10-24 13:45:35 +05:30
Murali Reddy edba711283 update client-go to include informeres, listers 2018-10-24 13:45:35 +05:30
Bryan Boreham 1b01da45f5 Merge pull request #3430 from weaveworks/bump-go-odp
Bump go-odp dependency
2018-10-17 18:03:44 +01:00
Bryan Boreham bf9985b0c8 Merge pull request #3423 from weaveworks/issue-3344
Replace weave-local-pods ipset with physdev iptables rule
2018-10-17 18:03:26 +01:00
Murali Reddy 408e5f7bdd make error message more appropriate 2018-10-17 21:16:52 +05:30
Martynas Pumputis 46177657b2 Bump go-odp dependency
To include the fix https://github.com/weaveworks/go-odp/pull/7.
2018-10-17 15:23:38 +03:00
Murali Reddy d351b9f020 remove weave-local-pods ipset which no longer used by weave-npc 2018-10-17 15:34:46 +05:30
Bryan Boreham 727fe46b2a Merge pull request #3385 from weaveworks/retry_forwarder
Add logic to re-try chooseBest() (which will select best forwarder)
2018-10-16 16:37:31 +01:00
Bryan Boreham b10af5a6cb Fix golint path - it moved upstream 2018-10-16 14:04:33 +00:00
Bryan Boreham 965b5c627e Merge pull request #3421 from weaveworks/2018-09-28-moar-initialDelaySeconds
kubernetes: Morph the livenessProbe into readinessProbe
2018-10-16 14:46:34 +01:00
Murali Reddy d00e7335bb dont maintain weave-local-pods ipset on pod add/delete/update 2018-10-16 14:02:23 +05:30
Murali Reddy e3eb16a1f9 addressing review comments 2018-10-09 11:21:47 +05:30
Bryan Boreham 078b76f3f1 Merge pull request #3416 from weaveworks/issue-3386
own action to reclaim when peer does not exist
2018-10-05 09:27:46 +01:00
Murali Reddy 6b2631e378 addressing review comment. keep forwarder running forever even on heart beat timeouts 2018-10-05 09:47:23 +05:30
Murali Reddy 59c1abe9b9 addressing review comments 2018-10-04 14:08:56 +05:30
Murali Reddy 4f9f0dfc3e Replace weave-local-pods ipset with physdev iptables rule
by-pass ingress network policy chains and accept the packet if
destination is not one of the local pods

return the packets so the egress network policy chains are skipped
in case packet's source is not one of the local pods

Fixes #3344
2018-10-02 11:02:36 +05:30
Damien Lespiau 8d44f49edd kubernetes: Morph the livenessProbe into readinessProbe
The liveness probe has the not-so-great side effect of killing the weave pods
if anything goes wrong, making debugging really difficult.

Not only that, we also bumped into the livenessProbe killing pods before they
had a chance to start (see below for the full story).

The readinessProbe is only used for Kubernetes Services: no traffic will be
routed to non-Ready pods when using the Service IP. We can use to to signal the
user something in wrong with Weave Net while not having Kubernetes delete the
pods behind our backs.

I removed the ininitialDelaySeconds field as Kurbernetes won't kill the pod if
not ready.

Notes from a debugging session:

Some nodes may have a vast amount of iptables rules, making each iptables
command take a few seconds (~5-10s). At startup, when the weave container
creates its chains, it runs quite a few iptables commands and the total time of
execution was > 30s.

It's important to note that, at this point, the weave process hasn't started as
those iptables commands are run by an initial script.

Kubernetes was trying to check for the container liveness, was failing, and
then proceeded to kill the weave pods before they had a chance to fully start.
2018-09-28 15:50:39 +01:00
Murali Reddy 7a959df648 handles an edge case where peer claimed to own the action (to reclaim) but no longer
exists hence lock persists foever, fix makes a peer own the reclaim when nonexistent node found

Fixes #3386
2018-09-27 16:00:44 +05:30
Bryan Boreham a73330f752 Merge branch '2.4' 2018-09-26 10:20:04 +01:00
Bryan Boreham 427e7a6868 Merge pull request #3408 from weaveworks/issues/3405-tracker-nil-deref
ipam: Do not deref nil in GET /ipinfo/tracker if tracker doesn't exist
2018-09-25 12:29:03 +01:00
Murali Reddy b3cb38eda4 addressing review comments 2018-09-24 17:53:28 +05:30
Murali Reddy 62758ca337 log the heartbeat send/recieve/miss 2018-09-21 16:45:47 +05:30
Murali Reddy 3c67c3c5f1 changes differentiate beteween fatal errors (for e.g. ipsec init)
from transient errors (heartbeast misses). In case of transient
errors OverlayForwarder is marked to be unhealty. When overlay forwarder
is unhealty its hept on hold so chooseBest() will skip selecting the forwrder

Once OverlayForwarder is considered healthy run chooseBest() again so the
forwarder can be selected if its the best forwarder

Fixes #1737
2018-09-21 16:45:47 +05:30
Martynas Pumputis 79f77f94f4 ipam: Do not deref nil in GET /ipinfo/tracker if tracker doesn't exist
In some cases, we do not create any IPAM tracker, which leads
to a panic when GET'ing /ipinfo/tracker.
2018-09-21 13:29:03 +03:00
Bryan Boreham fc759e2df9 Merge branch '2.4' 2018-09-13 16:59:03 +00:00
Bryan Boreham 1fcd521f67 Add release v2.4.1 v2.4.1 2018-09-13 15:25:47 +00:00
Martynas Pumputis 66bd443183 Merge remote-tracking branch 'origin/2.4' 2018-09-12 18:25:30 +03:00
Martynas Pumputis fa74e9f290 Merge pull request #3400 from weaveworks/issues/3384-peerlist-race
Fix two race conditions in Kubernetes IPAM reclaim

Fix #3384
2018-09-12 18:13:56 +03:00
Bryan Boreham 008eeb03ad Improve comments and error checking 2018-09-12 14:11:09 +00:00
Bryan Boreham 41c48dc678 Update test to call reclaimPeer() 2018-09-12 14:11:09 +00:00
Bryan Boreham 09be35dee0 Fetch stored peer list every time round the retry loop
This ensures we take account of other peers' updates and closes a race
condition whereby another peer's changes can be overwritten.
2018-09-12 14:11:09 +00:00
Bryan Boreham c3ad641d94 Refactor: extract reclaimPeer() function 2018-09-12 14:11:09 +00:00
Bryan Boreham bacb3952a3 Fetch Kubernetes node list every time round reclaim loop
If we retain the Kubernetes list for a long time we might see a peer
appear in the Weave list and conclude it has been deleted. We must
always fetch the Kubernetes list after fetching the Weave list, during
deletion.
2018-09-12 14:10:59 +00:00
Bryan Boreham 70257a6c19 Add test for kube-utils peer list
- In parallel on several goroutines, randomly add and remove peers.
 - Maintain the test's own idea of current state.
 - Check that the test's state and kube-utils' state are consistent.
2018-09-12 12:37:16 +00:00
Bryan Boreham bfe9f00e55 Fake kubernetes API which implements optimistic locking
Mostly copied from kubernetes/client-go files
2018-09-12 12:37:16 +00:00
Bryan Boreham 591be2d08f Refactor: Introduce an interface so weave client can be mocked 2018-09-12 12:37:16 +00:00
Bryan Boreham c3ce052cc5 Refactor: use Kubernetes.Interface instead of *Clientset
So we can call with a mock for testing
2018-09-12 12:35:34 +00:00
Bryan Boreham 7dcd4bdf47 Refactor: extract peerlist fns to new file 2018-09-12 12:35:34 +00:00
Bryan Boreham 2d8373249a Vendor mock Kubernetes APIs for testing 2018-09-12 12:35:33 +00:00
Bryan Boreham 4979ce1f5f Update to dep v5
This adds more metadata to Gopkg.lock, and seems to have deleted a few
more files.
2018-09-12 12:35:33 +00:00
Martynas Pumputis 79d812a9c2 Merge remote-tracking branch 'origin/2.4' 2018-09-12 13:58:08 +03:00
Martynas Pumputis 78e00f9a3a Merge pull request #3397 from weaveworks/issues/3394-delete-old-me
Reclaim IPs when node name is re-used
2018-09-12 13:57:12 +03:00
Bryan Boreham 2ff0bbe150 Reclaim IPs when node name is re-used
Change the map used to reclaim peers so it is indexed by PeerName
which should be unique over time, rather than NodeName which can be
re-used.  Create an auxilliary set of NodeNames to help.

This way, if we can see that a different peer was previously running
with the same node name, it can be included for reclaim.
2018-09-10 15:10:51 +00:00
Martynas Pumputis a182bd4514 Merge pull request #3388 from myENA/feature/skip-nat-via-weave-expose
add --without-masquerade option to `weave expose`
2018-09-10 15:45:52 +03:00