lb: EgressService - a LoadBalancer Service's pods leave from its address

EgressService (reweave.levk.dev/v1alpha1, named after a LoadBalancer
Service of class reweave.levk.dev/node-address) makes every pod the
Service selects (by pod IP, never readiness) leave the cluster from the
Service's address, or not at all. spec.emit is explicit: Holder (the only
mode so far) tunnels members' packets to the node holding the address,
which emits them.

The holder (S1's: the one node with the address on an interface) is the
status' sole writer: host, address, and a 512-port block per node in
61000-65535 (above the default ephemeral range), stable across churn.
A member's node SNATs it to address:its-block before it leaves, so the
holder forwards it unchanged and routes replies back by port alone: ip
rules (to address, ipproto tcp/udp, dport block) ahead of the local
table, which moves from 0 to 100, since the address is local on the
holder. Members' packets travel both ways in reweave-eg, an ipip tunnel
between weave bridges (NPC drops new routed packets into weave), with
accept_local and loose rp_filter on that device only.

Members' new connections are classed at the tail of nat PREROUTING,
after kube-proxy's DNAT has ended the nat walk for Services: the pod
network, multicast and the nodes' addresses are not outside. Marked
packets follow a per-member rule to the holder's table (or to an
unreachable one), are SNATed into the tunnel, and a FORWARD guard drops
anything else a member sends outside (other protocols, other ways out).
The tunnel is accepted only from the nodes' bridges, a holder accepts
from it only the blocks' ports, and TCP MSS is clamped into it.

Fail closed throughout: until the holder has published the status,
without a block, with no single holder, or for a pod two EgressServices
select, a member's outside traffic is dropped. The CNI plugin asks
weave-npc to admit every pod (unix socket /run/weave-npc/egress.sock,
conflist "egressAdmission": true, now rewritten when it differs) after
IPAM and before the pod has an interface, so a member's first packet
already leaves from the address; no answer, no pod. weave-npc applies
nothing about egress from caches that are not synced (unless the API is
absent): a restart never strips members' rules.

reweave/test/egress/run.sh checks the design's assertions against a
live cluster; both end-to-end scripts now re-evaluate each check on
every retry (it was expanded once, before the loop).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UHXzGrsxfkGk8LmwrBbHNZ

plugin/net/cni.go

@@ -1,6 +1,7 @@
 package plugin
 
 import (
+	"context"
 	"crypto/rand"
 	"encoding/json"
 	"errors"
@@ -14,6 +15,7 @@ import (
 	current "github.com/containernetworking/cni/pkg/types/100"
 	"github.com/containernetworking/plugins/pkg/ipam"
 	weaveapi "github.com/rajch/weave/api"
+	"github.com/rajch/weave/lb/admit"
 	weavenet "github.com/rajch/weave/net"
 	ipamplugin "github.com/rajch/weave/plugin/ipam"
 	"github.com/vishvananda/netlink"
@@ -86,6 +88,18 @@ func (c *CNIPlugin) CmdAdd(args *skel.CmdArgs) error {
 	// Only expecting one address
 	ip := result.IPs[0]
 
+	// An EgressService member's first packet must already leave from its Service's address (or not at all):
+	// weave-npc installs the pod's egress rules before the pod has an interface. No answer, no pod (the
+	// runtime calls DEL, which releases the address, and kubelet retries).
+	if conf.EgressAdmission {
+		if ns, name := admit.PodFromArgs(args.Args); ns != "" && name != "" {
+			if _, err := admit.Admit(context.Background(), admit.SocketPath,
+				admit.Request{Namespace: ns, Name: name, IP: ip.Address.IP.String()}); err != nil {
+				return err
+			}
+		}
+	}
+
 	// If config says nothing about routes or gateway, default one will be via the bridge
 	if len(result.Routes) == 0 && ip.Gateway == nil {
 		bridgeIP, err := weavenet.FindBridgeIP(conf.BrName, &ip.Address)
@@ -241,4 +255,6 @@ type NetConf struct {
 	IPMasq      bool   `json:"ipMasq"`
 	MTU         int    `json:"mtu"`
 	HairpinMode bool   `json:"hairpinMode"`
+	// EgressAdmission: weave-npc must admit every pod before it gets its interface (EgressService members)
+	EgressAdmission bool `json:"egressAdmission"`
 }