lb: EgressService - a LoadBalancer Service's pods leave from its address
images / images (push) Successful in 18m34s
images / images (push) Successful in 18m34s
EgressService (reweave.levk.dev/v1alpha1, named after a LoadBalancer Service of class reweave.levk.dev/node-address) makes every pod the Service selects (by pod IP, never readiness) leave the cluster from the Service's address, or not at all. spec.emit is explicit: Holder (the only mode so far) tunnels members' packets to the node holding the address, which emits them. The holder (S1's: the one node with the address on an interface) is the status' sole writer: host, address, and a 512-port block per node in 61000-65535 (above the default ephemeral range), stable across churn. A member's node SNATs it to address:its-block before it leaves, so the holder forwards it unchanged and routes replies back by port alone: ip rules (to address, ipproto tcp/udp, dport block) ahead of the local table, which moves from 0 to 100, since the address is local on the holder. Members' packets travel both ways in reweave-eg, an ipip tunnel between weave bridges (NPC drops new routed packets into weave), with accept_local and loose rp_filter on that device only. Members' new connections are classed at the tail of nat PREROUTING, after kube-proxy's DNAT has ended the nat walk for Services: the pod network, multicast and the nodes' addresses are not outside. Marked packets follow a per-member rule to the holder's table (or to an unreachable one), are SNATed into the tunnel, and a FORWARD guard drops anything else a member sends outside (other protocols, other ways out). The tunnel is accepted only from the nodes' bridges, a holder accepts from it only the blocks' ports, and TCP MSS is clamped into it. Fail closed throughout: until the holder has published the status, without a block, with no single holder, or for a pod two EgressServices select, a member's outside traffic is dropped. The CNI plugin asks weave-npc to admit every pod (unix socket /run/weave-npc/egress.sock, conflist "egressAdmission": true, now rewritten when it differs) after IPAM and before the pod has an interface, so a member's first packet already leaves from the address; no answer, no pod. weave-npc applies nothing about egress from caches that are not synced (unless the API is absent): a restart never strips members' rules. reweave/test/egress/run.sh checks the design's assertions against a live cluster; both end-to-end scripts now re-evaluate each check on every retry (it was expanded once, before the loop). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UHXzGrsxfkGk8LmwrBbHNZ
This commit is contained in:
1 parent
8018ea4bf9
commit
305ac4f3d3
15 files changed
+1866
-64
No files matched your search
@@ -1,6 +1,68 @@
|
||||
apiVersion: v1
|
||||
kind: List
|
||||
items:
|
||||
# EgressService (reweave.levk.dev/v1alpha1), served by weave-npc's lb agent: a LoadBalancer Service's pods
|
||||
# leave the cluster from the Service's address (named after the Service, same namespace)
|
||||
- apiVersion: apiextensions.k8s.io/v1
|
||||
kind: CustomResourceDefinition
|
||||
metadata:
|
||||
name: egressservices.reweave.levk.dev
|
||||
labels:
|
||||
name: weave-net
|
||||
spec:
|
||||
group: reweave.levk.dev
|
||||
scope: Namespaced
|
||||
names:
|
||||
plural: egressservices
|
||||
singular: egressservice
|
||||
kind: EgressService
|
||||
listKind: EgressServiceList
|
||||
versions:
|
||||
- name: v1alpha1
|
||||
served: true
|
||||
storage: true
|
||||
subresources:
|
||||
status: {}
|
||||
additionalPrinterColumns:
|
||||
- {name: Emit, type: string, jsonPath: .spec.emit}
|
||||
- {name: Host, type: string, jsonPath: .status.host}
|
||||
- {name: Address, type: string, jsonPath: .status.address}
|
||||
- {name: Age, type: date, jsonPath: .metadata.creationTimestamp}
|
||||
schema:
|
||||
openAPIV3Schema:
|
||||
type: object
|
||||
description: >-
|
||||
Makes every pod the same-named LoadBalancer Service (class reweave.levk.dev/node-address)
|
||||
selects leave the cluster from that Service's address, or not at all.
|
||||
required: [spec]
|
||||
properties:
|
||||
spec:
|
||||
type: object
|
||||
required: [emit]
|
||||
properties:
|
||||
sourceIPBy:
|
||||
type: string
|
||||
description: LoadBalancerIP - members leave from the Service's load-balancer address.
|
||||
enum: [LoadBalancerIP]
|
||||
default: LoadBalancerIP
|
||||
emit:
|
||||
type: string
|
||||
description: Holder - members' packets are tunnelled to the node holding the address, which emits them.
|
||||
enum: [Holder]
|
||||
status:
|
||||
type: object
|
||||
properties:
|
||||
host: {type: string}
|
||||
address: {type: string}
|
||||
blocks:
|
||||
type: array
|
||||
items:
|
||||
type: object
|
||||
required: [node, from, to]
|
||||
properties:
|
||||
node: {type: string}
|
||||
from: {type: integer, format: int32}
|
||||
to: {type: integer, format: int32}
|
||||
- apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
@@ -78,6 +140,20 @@ items:
|
||||
- nodes
|
||||
verbs:
|
||||
- patch
|
||||
- apiGroups:
|
||||
- reweave.levk.dev
|
||||
resources:
|
||||
- egressservices
|
||||
verbs:
|
||||
- get
|
||||
- list
|
||||
- watch
|
||||
- apiGroups:
|
||||
- reweave.levk.dev
|
||||
resources:
|
||||
- egressservices/status
|
||||
verbs:
|
||||
- patch
|
||||
- apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
@@ -226,6 +302,8 @@ items:
|
||||
- name: xtables-lock
|
||||
mountPath: /run/xtables.lock
|
||||
readOnly: false
|
||||
- name: weave-npc-run
|
||||
mountPath: /run/weave-npc
|
||||
hostNetwork: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
hostPID: false
|
||||
@@ -264,6 +342,10 @@ items:
|
||||
hostPath:
|
||||
path: /run/xtables.lock
|
||||
type: FileOrCreate
|
||||
- name: weave-npc-run
|
||||
hostPath:
|
||||
path: /run/weave-npc
|
||||
type: DirectoryOrCreate
|
||||
priorityClassName: system-node-critical
|
||||
updateStrategy:
|
||||
type: RollingUpdate
|
||||
@@ -340,8 +340,11 @@ func root(cmd *cobra.Command, args []string) {
|
||||
go npController.Run(wait.NeverStop)
|
||||
|
||||
if serveLB {
|
||||
// LoadBalancer Services of class lb.Class: addresses this node holds, client address kept
|
||||
go lb.NewController(nodeName, client, lb.NewDatapath(ipt)).Run(context.Background())
|
||||
// LoadBalancer Services of class lb.Class: addresses this node holds, client address kept; and
|
||||
// EgressServices: those Services' pods leave the cluster from that address (or not at all)
|
||||
egress, err := lb.NewEgressClient(config)
|
||||
handleFatal(err)
|
||||
go lb.NewController(nodeName, client, egress, lb.NewDatapath(ipt)).Run(context.Background())
|
||||
}
|
||||
|
||||
signals := make(chan os.Signal, 1)
|
||||
|
||||
Reference in new issue
Block a user