lb: EgressService - a LoadBalancer Service's pods leave from its address
images / images (push) Successful in 18m34s

EgressService (reweave.levk.dev/v1alpha1, named after a LoadBalancer
Service of class reweave.levk.dev/node-address) makes every pod the
Service selects (by pod IP, never readiness) leave the cluster from the
Service's address, or not at all. spec.emit is explicit: Holder (the only
mode so far) tunnels members' packets to the node holding the address,
which emits them.

The holder (S1's: the one node with the address on an interface) is the
status' sole writer: host, address, and a 512-port block per node in
61000-65535 (above the default ephemeral range), stable across churn.
A member's node SNATs it to address:its-block before it leaves, so the
holder forwards it unchanged and routes replies back by port alone: ip
rules (to address, ipproto tcp/udp, dport block) ahead of the local
table, which moves from 0 to 100, since the address is local on the
holder. Members' packets travel both ways in reweave-eg, an ipip tunnel
between weave bridges (NPC drops new routed packets into weave), with
accept_local and loose rp_filter on that device only.

Members' new connections are classed at the tail of nat PREROUTING,
after kube-proxy's DNAT has ended the nat walk for Services: the pod
network, multicast and the nodes' addresses are not outside. Marked
packets follow a per-member rule to the holder's table (or to an
unreachable one), are SNATed into the tunnel, and a FORWARD guard drops
anything else a member sends outside (other protocols, other ways out).
The tunnel is accepted only from the nodes' bridges, a holder accepts
from it only the blocks' ports, and TCP MSS is clamped into it.

Fail closed throughout: until the holder has published the status,
without a block, with no single holder, or for a pod two EgressServices
select, a member's outside traffic is dropped. The CNI plugin asks
weave-npc to admit every pod (unix socket /run/weave-npc/egress.sock,
conflist "egressAdmission": true, now rewritten when it differs) after
IPAM and before the pod has an interface, so a member's first packet
already leaves from the address; no answer, no pod. weave-npc applies
nothing about egress from caches that are not synced (unless the API is
absent): a restart never strips members' rules.

reweave/test/egress/run.sh checks the design's assertions against a
live cluster; both end-to-end scripts now re-evaluate each check on
every retry (it was expanded once, before the loop).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHXzGrsxfkGk8LmwrBbHNZ
This commit is contained in:
levaliciousandClaude Opus 5.5 committed 2026-10-13 14:39:44 -04:00
1 parent 8018ea4bf9
commit 305ac4f3d3
15 files changed
+1866 -64

No files matched your search

@@ -1,6 +1,68 @@
apiVersion: v1
kind: List
items:
# EgressService (reweave.levk.dev/v1alpha1), served by weave-npc's lb agent: a LoadBalancer Service's pods
# leave the cluster from the Service's address (named after the Service, same namespace)
- apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: egressservices.reweave.levk.dev
labels:
name: weave-net
spec:
group: reweave.levk.dev
scope: Namespaced
names:
plural: egressservices
singular: egressservice
kind: EgressService
listKind: EgressServiceList
versions:
- name: v1alpha1
served: true
storage: true
subresources:
status: {}
additionalPrinterColumns:
- {name: Emit, type: string, jsonPath: .spec.emit}
- {name: Host, type: string, jsonPath: .status.host}
- {name: Address, type: string, jsonPath: .status.address}
- {name: Age, type: date, jsonPath: .metadata.creationTimestamp}
schema:
openAPIV3Schema:
type: object
description: >-
Makes every pod the same-named LoadBalancer Service (class reweave.levk.dev/node-address)
selects leave the cluster from that Service's address, or not at all.
required: [spec]
properties:
spec:
type: object
required: [emit]
properties:
sourceIPBy:
type: string
description: LoadBalancerIP - members leave from the Service's load-balancer address.
enum: [LoadBalancerIP]
default: LoadBalancerIP
emit:
type: string
description: Holder - members' packets are tunnelled to the node holding the address, which emits them.
enum: [Holder]
status:
type: object
properties:
host: {type: string}
address: {type: string}
blocks:
type: array
items:
type: object
required: [node, from, to]
properties:
node: {type: string}
from: {type: integer, format: int32}
to: {type: integer, format: int32}
- apiVersion: v1
kind: ServiceAccount
metadata:
@@ -78,6 +140,20 @@ items:
- nodes
verbs:
- patch
- apiGroups:
- reweave.levk.dev
resources:
- egressservices
verbs:
- get
- list
- watch
- apiGroups:
- reweave.levk.dev
resources:
- egressservices/status
verbs:
- patch
- apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
@@ -226,6 +302,8 @@ items:
- name: xtables-lock
mountPath: /run/xtables.lock
readOnly: false
- name: weave-npc-run
mountPath: /run/weave-npc
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
hostPID: false
@@ -264,6 +342,10 @@ items:
hostPath:
path: /run/xtables.lock
type: FileOrCreate
- name: weave-npc-run
hostPath:
path: /run/weave-npc
type: DirectoryOrCreate
priorityClassName: system-node-critical
updateStrategy:
type: RollingUpdate
+5 -2
View File
@@ -340,8 +340,11 @@ func root(cmd *cobra.Command, args []string) {
go npController.Run(wait.NeverStop)
if serveLB {
// LoadBalancer Services of class lb.Class: addresses this node holds, client address kept
go lb.NewController(nodeName, client, lb.NewDatapath(ipt)).Run(context.Background())
// LoadBalancer Services of class lb.Class: addresses this node holds, client address kept; and
// EgressServices: those Services' pods leave the cluster from that address (or not at all)
egress, err := lb.NewEgressClient(config)
handleFatal(err)
go lb.NewController(nodeName, client, egress, lb.NewDatapath(ipt)).Run(context.Background())
}
signals := make(chan os.Signal, 1)