Gateways of a GatewayClass whose controllerName is reweave.levk.dev/gateway
are served like LoadBalancer Services: a Gateway names one IPv4 address in
spec.addresses, and the node holding that address serves all of it and
writes its status.
- TCP listeners forward the oldest TCPRoute attached to them in the kernel,
through the same DNAT and reply steering as LoadBalancer Services, so the
backend sees the client's own address.
- HTTP and HTTPS listeners are served by the agent itself on the holder:
TLS terminated with the listener's certificate (chosen by SNI, swapped in
place when its Secret changes, HTTP/2 offered), requests routed by the
attached HTTPRoutes in the order the spec prescribes (hostname, exact
path, longest prefix, method, headers, query, age, name, rule), and passed
to a ready endpoint of the backend with X-Forwarded-For built from this
hop only. RequestRedirect, URLRewrite and both header modifiers are
implemented; other filters and regular expressions are refused
(Accepted False, UnsupportedValue). Bodies are streamed, never limited.
Unresolved backends answer 500, backends without ready endpoints 503,
requests nothing matches 404. Those ports are let in ahead of the host
firewall (REWEAVE-LB-IN).
Listeners are checked for protocol, route kinds, certificates (Secrets in
the Gateway's namespace or granted by a ReferenceGrant, and only in the
namespaces the agent may read: --gateway-namespaces, default
gateway-system), conflicts on a port, and ports LoadBalancer Services
already forward. Routes attach by parentRef (sectionName, port),
allowedRoutes (namespaces Same, All or Selector; kinds) and hostname
intersection, with the spec's conditions and reasons.
One writer per status: the holder writes its Gateway's (the first node by
name when nobody holds the address, so failures are reported once), the
first node by name writes GatewayClass and route statuses, keeping other
controllers' route parents. Writes are conditional on the resourceVersion
they were computed from, and a condition keeps its transition time while
its status stands, so an unchanged status is never rewritten.
The types are plain structs for the subset implemented (the upstream module
needs a newer client-go than the fork's); the CRDs are installed apart from
this manifest, which grants the agent read access to the Gateway API kinds,
patch on their status, and Secrets in gateway-system only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHXzGrsxfkGk8LmwrBbHNZ
A pod could put any source MAC or address on the wire: weave has no layer-2
anti-spoofing, so a pod with NET_RAW could pose as another pod or as a node's
bridge, on its own node (local bridging never reaches the router) and, through
weave, on others. That undercut every rule that trusts a source MAC or address,
including the egress holder's check of which node a tunnel packet came from.
The CNI plugin now locks the host side of each pod's veth with tc ingress
filters: IPv4 from the pod's MAC and address, ARP speaking for exactly those
(or probing from 0.0.0.0) pass; everything else is dropped before the bridge
sees it. The u32 keys follow iproute2's layout for "ether src" (-8, -4) and
"ip src" (12). ADD fails if the port cannot be locked, so an unlocked pod never
starts. The filters go away with the veth.
Pods attached before this plugin get locked by weave-init: "weaveutil
lock-ports" walks libcni's result cache (each attachment's netns, interface and
address) and locks their ports, leaving locked ones alone. weave-init mounts
the cache and /var/run/netns read-only for it.
TestPortLockInKernel (root) builds a bridge and two pods in throwaway
namespaces and checks what reaches the other pod; reweave/test/portlock runs
the same against a live cluster, across nodes, after checking every port is
locked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHXzGrsxfkGk8LmwrBbHNZ
EgressService (reweave.levk.dev/v1alpha1, named after a LoadBalancer
Service of class reweave.levk.dev/node-address) makes every pod the
Service selects (by pod IP, never readiness) leave the cluster from the
Service's address, or not at all. spec.emit is explicit: Holder (the only
mode so far) tunnels members' packets to the node holding the address,
which emits them.
The holder (S1's: the one node with the address on an interface) is the
status' sole writer: host, address, and a 512-port block per node in
61000-65535 (above the default ephemeral range), stable across churn.
A member's node SNATs it to address:its-block before it leaves, so the
holder forwards it unchanged and routes replies back by port alone: ip
rules (to address, ipproto tcp/udp, dport block) ahead of the local
table, which moves from 0 to 100, since the address is local on the
holder. Members' packets travel both ways in reweave-eg, an ipip tunnel
between weave bridges (NPC drops new routed packets into weave), with
accept_local and loose rp_filter on that device only.
Members' new connections are classed at the tail of nat PREROUTING,
after kube-proxy's DNAT has ended the nat walk for Services: the pod
network, multicast and the nodes' addresses are not outside. Marked
packets follow a per-member rule to the holder's table (or to an
unreachable one), are SNATed into the tunnel, and a FORWARD guard drops
anything else a member sends outside (other protocols, other ways out).
The tunnel is accepted only from the nodes' bridges, a holder accepts
from it only the blocks' ports, and TCP MSS is clamped into it.
Fail closed throughout: until the holder has published the status,
without a block, with no single holder, or for a pod two EgressServices
select, a member's outside traffic is dropped. The CNI plugin asks
weave-npc to admit every pod (unix socket /run/weave-npc/egress.sock,
conflist "egressAdmission": true, now rewritten when it differs) after
IPAM and before the pod has an interface, so a member's first packet
already leaves from the address; no answer, no pod. weave-npc applies
nothing about egress from caches that are not synced (unless the API is
absent): a restart never strips members' rules.
reweave/test/egress/run.sh checks the design's assertions against a
live cluster; both end-to-end scripts now re-evaluate each check on
every retry (it was expanded once, before the loop).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHXzGrsxfkGk8LmwrBbHNZ
A Service of type LoadBalancer with loadBalancerClass
reweave.levk.dev/node-address names its address in spec.loadBalancerIP.
Each node's weave-npc publishes the IPv4 addresses on its host interfaces
and its weave bridge (Node annotations reweave.levk.dev/addresses and
reweave.levk.dev/weave). The one node that holds a Service's address is
its holder: it alone writes status.loadBalancer (ipMode Proxy, so
kube-proxy installs nothing for the address) and DNATs the address's ports
to ready endpoints (EndpointSlices), without SNAT, from nat PREROUTING
ahead of kube-proxy. Every other node marks the connections the holder
forwards to its pods (new, from the holder's bridge MAC, from outside the
pod network) and routes their replies back to the holder's bridge through
a per-holder policy rule and table, so the holder's conntrack undoes the
DNAT and the reply leaves from the address the client connected to.
Fail closed: an address no node holds, or that several nodes hold, is
served by no node, and its status is cleared (by its holder, or by the
first node by name if none holds it). Compute is the policy as a pure
function; Datapath replaces exactly the chains it owns with
iptables-restore --noflush and keeps its rules and tables exact.
weave-npc runs it unless --lb=false; the ClusterRole gains services
(watch), services/status (patch), endpointslices (watch), nodes (patch).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHXzGrsxfkGk8LmwrBbHNZ
* Module name changed to `github.com/rajch/weave`
* Added environment variable `CHECKPOINT_DISABLE=1` to default manifests, to bypass weaveworks telemetry
This allows the main container to run without write access to the host
filesystem, except for two specific areas, thus trimming the attack
surface.
We lose the behaviour that the CNI plugin is only installed if Weave Net
gets up and running at least once, so the error reports from kubelet
will be slightly different in the case that it never manages to run.
This setting is passed by Kubelet to the Linux scheduler to ask for a share of
available CPU, so we should ask for a bit more than 1% of 1 CPU. Otherwise if a
CPU is contended Weave Net will get throttled frequently.
Weave Net in fastdp mode shouldn't need a lot of CPU most of the time, but at
startup there is quite a lot to do.
The setting is also used by the Kubernetes scheduler to decide how much will fit
on a node, so setting it much higher will reserve CPU that we shouldn't need most
of the time.
* fixes#3691
* `weave-daemonset-k8s-1.11.yaml` is `weave-daemonset-k8s-1.9.yaml` plus `priorityClassName: system-node-critical`
* notice that a Kubernetes cluster created with `kubeadm` has its `kube-proxy`s marked as `system-node-critical` (and several other pods, such as `kube-apiserver` marked as `system-cluster-critical`)
The liveness probe has the not-so-great side effect of killing the weave pods
if anything goes wrong, making debugging really difficult.
Not only that, we also bumped into the livenessProbe killing pods before they
had a chance to start (see below for the full story).
The readinessProbe is only used for Kubernetes Services: no traffic will be
routed to non-Ready pods when using the Service IP. We can use to to signal the
user something in wrong with Weave Net while not having Kubernetes delete the
pods behind our backs.
I removed the ininitialDelaySeconds field as Kurbernetes won't kill the pod if
not ready.
Notes from a debugging session:
Some nodes may have a vast amount of iptables rules, making each iptables
command take a few seconds (~5-10s). At startup, when the weave container
creates its chains, it runs quite a few iptables commands and the total time of
execution was > 30s.
It's important to note that, at this point, the weave process hasn't started as
those iptables commands are run by an initial script.
Kubernetes was trying to check for the container liveness, was failing, and
then proceeded to kill the weave pods before they had a chance to fully start.