The images go to the registry with --push, but each build also left its
tagged images in the runner's local store and an unbounded build cache:
after a day of builds the runner's 10Gi Docker volume filled up and run 21
died mid-build (no space left on device). A final step, run even when the
build fails, removes our pushed tags and caps the build cache at 4GB
(recent layers stay, so builds stay cached).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHXzGrsxfkGk8LmwrBbHNZ
The holder could tell a member's node only by source port block, so
replies without ports never got back: ICMP errors (path-MTU "too big",
unreachable, time exceeded), ping, and port-less protocols. Now every
node pair has its own ipip tunnel (rweg-<hash of the peer>, between the
two weave bridges), so the holder knows the sending node by the device a
packet arrives on:
- it tags each connection arriving from a peer's tunnel with that peer
(connmark: peer index in the top byte, plus 0x200000), once;
- replies and related ICMP errors restore the tag, and a policy rule per
peer (ahead of the local table, like the port rules) routes them into
that peer's tunnel, where the member node's conntrack translates them
back to the member;
- an arrival on one peer's tunnel whose connection another peer already
holds (a port-less protocol to the same remote from two nodes) is
dropped, never delivered across;
- the stateless port-block rules stay, now for TCP, UDP and SCTP.
Members' SNAT keeps TCP, UDP and SCTP within the node's block and puts
everything else (ping, GRE, ...) on the address alone; the guard lets any
protocol into the holder's tunnel and nothing anywhere else. The holder
only accepts, per tunnel, the address with that node's ports.
Routes in the reply tables are now read with one dump per pass instead of
one per candidate table.
reweave/test/egress/run.sh also checks ping from every node, that ICMP
errors from routers beyond the holder reach a member (traceroute), and
that ping is dropped while nobody holds the address.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHXzGrsxfkGk8LmwrBbHNZ
RouteList returns a default route without Dst, and netlink's RouteDel
refuses a route with no Dst, Src or Gw. The egress unreachable table's
"unreachable default" could not be removed once installed: every later
pass failed there, before updating rules, so members stayed dropped after
their EgressService got a holder again (fail closed, but stuck). S1's
stale reply tables had the same latent fault. Both now delete through
delRoute, which spells the default destination out; the unreachable route
is compared by type alone, so it is not replaced on every pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHXzGrsxfkGk8LmwrBbHNZ
- The tunnel needs an address. Reverse-path filtering (loose, as on our
nodes) rejects what arrives on a device without an IPv4 address unless
the reverse route leaves by that very device: the holder dropped every
member packet after decapsulation (ip route get: EXDEV), and a member's
node would drop the replies. reweave-eg now carries the node's bridge
address as a /32 (it never shows on the wire: the tunnel sets the outer
source itself, and what crosses it is forwarded).
- The kernel leaves out the priority attribute of a rule at 0, so the
default local rule lists as unset (-1) and moveLocalRule never found
it: it stayed at 0 beside the new one at 100, and the holder delivered
replies to its own address to itself (answering with RSTs). Unset now
counts as 0.
- install_cni_plugin copied the plugin only when weave-plugin-$VERSION was
missing, and every build of ours is "latest": nodes kept a plugin that
ignores egressAdmission, so a new member's first connection left from
its node. It is now replaced whenever its content differs (sha256),
atomically.
Also: the end-to-end manifest's non-member label is a string ("none", not
YAML's boolean no).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHXzGrsxfkGk8LmwrBbHNZ
With no EgressService and no member rules installed, an admission has
nothing to protect or clean up, so a failing pass (a datapath bug, a
bridge not yet addressed) answers "not a member" instead of stopping
every pod on the node from starting. Wherever EgressServices exist,
admission still fails closed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHXzGrsxfkGk8LmwrBbHNZ
EgressService (reweave.levk.dev/v1alpha1, named after a LoadBalancer
Service of class reweave.levk.dev/node-address) makes every pod the
Service selects (by pod IP, never readiness) leave the cluster from the
Service's address, or not at all. spec.emit is explicit: Holder (the only
mode so far) tunnels members' packets to the node holding the address,
which emits them.
The holder (S1's: the one node with the address on an interface) is the
status' sole writer: host, address, and a 512-port block per node in
61000-65535 (above the default ephemeral range), stable across churn.
A member's node SNATs it to address:its-block before it leaves, so the
holder forwards it unchanged and routes replies back by port alone: ip
rules (to address, ipproto tcp/udp, dport block) ahead of the local
table, which moves from 0 to 100, since the address is local on the
holder. Members' packets travel both ways in reweave-eg, an ipip tunnel
between weave bridges (NPC drops new routed packets into weave), with
accept_local and loose rp_filter on that device only.
Members' new connections are classed at the tail of nat PREROUTING,
after kube-proxy's DNAT has ended the nat walk for Services: the pod
network, multicast and the nodes' addresses are not outside. Marked
packets follow a per-member rule to the holder's table (or to an
unreachable one), are SNATed into the tunnel, and a FORWARD guard drops
anything else a member sends outside (other protocols, other ways out).
The tunnel is accepted only from the nodes' bridges, a holder accepts
from it only the blocks' ports, and TCP MSS is clamped into it.
Fail closed throughout: until the holder has published the status,
without a block, with no single holder, or for a pod two EgressServices
select, a member's outside traffic is dropped. The CNI plugin asks
weave-npc to admit every pod (unix socket /run/weave-npc/egress.sock,
conflist "egressAdmission": true, now rewritten when it differs) after
IPAM and before the pod has an interface, so a member's first packet
already leaves from the address; no answer, no pod. weave-npc applies
nothing about egress from caches that are not synced (unless the API is
absent): a restart never strips members' rules.
reweave/test/egress/run.sh checks the design's assertions against a
live cluster; both end-to-end scripts now re-evaluate each check on
every retry (it was expanded once, before the loop).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHXzGrsxfkGk8LmwrBbHNZ
Weave's expose rule (nat WEAVE: ! -s <alloc> -d <alloc> -j MASQUERADE)
masquerades everything entering the pod network from outside unless it is
for a local pod (weaver-no-masq-local). Live on the SG cluster a pod on
another node than the holder saw the holder's bridge address instead of
the client's. A new chain, REWEAVE-LB-POST, hooked first in nat
POSTROUTING, lets a connection we DNATed to a held address leave the nat
table with its source untouched; one that comes from a pod (hairpin) is
masqueraded to the holder instead, so its reply returns through the
holder's conntrack.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHXzGrsxfkGk8LmwrBbHNZ
iptables refuses a DNAT to ip:port unless the same rule names the
protocol, so every per-port chain's DNAT now has -p. Live on the SG
cluster the first sync (no endpoints yet) applied; every later one failed
with "Need TCP, UDP, SCTP or DCCP with port specification" and left the
chains empty. The unit test now rejects any DNAT line without its
protocol; the rendered nat and mangle payloads were also checked with the
nodes' own iptables-restore --test (nf_tables).
reweave/test/lb/run.sh is the end-to-end test: from outside, both a pod on
the holder and one on another node see the client's own address; status
is [{ip, ipMode: Proxy}] and kube-proxy has nothing for the address; a pod
on a third node reaches it; an unheld address clears the status and stops
the forward; deleting the Services removes the holder's chains.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHXzGrsxfkGk8LmwrBbHNZ
A Service of type LoadBalancer with loadBalancerClass
reweave.levk.dev/node-address names its address in spec.loadBalancerIP.
Each node's weave-npc publishes the IPv4 addresses on its host interfaces
and its weave bridge (Node annotations reweave.levk.dev/addresses and
reweave.levk.dev/weave). The one node that holds a Service's address is
its holder: it alone writes status.loadBalancer (ipMode Proxy, so
kube-proxy installs nothing for the address) and DNATs the address's ports
to ready endpoints (EndpointSlices), without SNAT, from nat PREROUTING
ahead of kube-proxy. Every other node marks the connections the holder
forwards to its pods (new, from the holder's bridge MAC, from outside the
pod network) and routes their replies back to the holder's bridge through
a per-holder policy rule and table, so the holder's conntrack undoes the
DNAT and the reply leaves from the address the client connected to.
Fail closed: an address no node holds, or that several nodes hold, is
served by no node, and its status is cleared (by its holder, or by the
first node by name if none holds it). Compute is the policy as a pure
function; Datapath replaces exactly the chains it owns with
iptables-restore --noflush and keeps its rules and tables exact.
weave-npc runs it unless --lb=false; the ClusterRole gains services
(watch), services/status (patch), endpointslices (watch), nodes (patch).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHXzGrsxfkGk8LmwrBbHNZ
reweave/build's `make all` also builds tools/runner from the tools
submodule (weaveworks/build-tools); run 7 compiled every weave binary and
then stopped there. Fetched from upstream for now, like the build's other
fetches.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHXzGrsxfkGk8LmwrBbHNZ
Debian has archived both: buster is gone from deb.debian.org and
bullseye's security pool there now 404s on packages its index still
lists (run 6: xz-utils, dpkg-dev, libdpkg-perl). archive.debian.org
serves both releases, including libpcap 1.8.1-6+deb10u1 for the static
link.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHXzGrsxfkGk8LmwrBbHNZ
Gitea's registry refuses the job token (no package scope at login), so log
in as reweave-ci with the org secret REGISTRY_TOKEN, and drop the
packages permission that had no effect.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHXzGrsxfkGk8LmwrBbHNZ
A Gitea Actions workflow that builds the two images the cluster runs from
reweave/build/Dockerfile (linux/amd64 only) and pushes them to
git.levk.dev/reweave/{weave-kube,weave-npc}, tagged with the full commit
SHA so a tag never changes meaning. It stops at the push; rollout is the
CD side's job. Build-time fetches (base images, apt/apk, Go modules) stay
external for now and are to be peeled out once self-deploy works.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHXzGrsxfkGk8LmwrBbHNZ
levaliciousandClaude Opus 5.5