I wanted to play with custom MCP servers. I spent the night at the physical link layer.

The plan for the evening was fun. I wanted a clean VM to experiment on. Somewhere to stand up my own MCP servers, wire them into a model, and generally poke at the custom-tooling side of the AI stack without cluttering a machine I actually use for anything. New sandbox, no baggage, break whatever I want. That was the whole idea.

The idea did not survive contact with networking. The subnet behind my Proxmox host's management bridge had run out of usable addresses. It's a /29, which gives you six usable hosts, and I had cheerfully filled all six. The sensible fix is a bigger subnet. What I actually did, and I'd like it on the record that I found this funny at the time and did it anyway, was carve out a brand new network that is also a /29. Six usable addresses again. I will run out of this one too, and I will somehow be surprised when I do.

Renumbering the live management network was off the table, so the "quick" prerequisite was: grab a second physical port on the server, patch it into that new network, make a new bridge, drop the VM on it, done. Then I'd get to the actual fun part.

Reader, I did not get to the fun part. I got four networking failures stacked on top of each other, each one wearing the previous one's symptoms, and by the time the VM could reach the internet it was late and I had written exactly zero lines of MCP. This is the writeup of the prerequisite, because the prerequisite turned out to be the whole story.

Here's the design I was aiming for. vmbr0 was the existing management bridge on eno8, carrying the Proxmox host, its default gateway, and all the existing VMs on 192.168.10.0/29. The new bridge, vmbr1, would sit on eno5, patched into 192.168.20.0/29, for the new AI sandbox VM.

192.168.10.0/29            192.168.20.0/29
   gateway .1                 gateway .1
      |                          |
    eno8                       eno5
      |                          |
   vmbr0                       vmbr1
      |                          |
 Proxmox mgmt +            New VMs
 existing VMs

Two independent bridges, two physical ports, two networks, one server. Clean separation, the management interface never gets touched, and my AI sandbox lives on its own subnet where it can't hurt anything. On paper it's two boxes and an arrow. Every one of those addresses is a private LAN example, by the way, so swap in whatever your own network uses.

A bridge is not a network

Before the failures, the thing worth saying plainly, because it's the mental model that everything else hangs off: a Proxmox Linux bridge is just a software switch. That's it. It connects virtual NICs to whatever physical interface you've made its bridge port, at Layer 2, and it does nothing else.

It does not give you DHCP. It does not give you DNS. It does not do NAT, it does not route, it does not provide a gateway, it does not know anything about internet access. If you attach eno5 to vmbr1 and drop a VM on it, all you've done is wire that VM's virtual NIC to the physical cable coming out of eno5. Everything past the cable, the switch, the router, the firewall, the route back, still has to be told about your new subnet by somebody. I knew this in the abstract. I still managed to spend an evening rediscovering it, because when connectivity fails it never announces which layer it failed at.

Failure one: there can be only one default gateway

I went into the Proxmox web UI, node → System → Network → Create → Linux Bridge, and started filling in the new bridge the way you'd expect: name vmbr1, bridge port eno5, and, because I was thinking of it as "the .20 network", I started to give it an IP and the 192.168.20.1 gateway. Proxmox stopped me cold:

There can be only one single default gateway

Which is correct, and I should have expected it. vmbr0 already owns the host's default route. A Proxmox host is one Linux machine, and one Linux machine gets one default gateway unless you go out of your way to set up policy routing, which I had no reason to do. The gateway field on a bridge configures routing for the Proxmox host itself, not for the VMs hanging off that bridge. That distinction is the whole trick, and it's easy to miss because the field sits right there on the bridge config looking like it belongs to the bridge.

So the correct config is almost aggressively empty:

Name:        vmbr1
Bridge ports: eno5
IPv4/CIDR:    (blank)
Gateway:      (blank)
Autostart:    yes
VLAN aware:   no (unless you actually need it)

No IP, no gateway. vmbr1 is a pure Layer 2 pipe. The VMs on it get their own gateway (192.168.20.1) configured inside the guest OS, and that never touches the host's routing table. vmbr0 keeps its management IP and its default route, untouched, which is exactly what you want.

One more trap in that same spot: don't give vmbr1 the network address either. With 192.168.20.0/29, the address 192.168.20.0 is the network. It identifies the subnet, it is not a host address, and assigning it to the bridge is a different flavour of wrong from the gateway error. Blank is the answer. The bridge doesn't need an address to move frames.

Failure two: bridge 'vmbr1' does not exist

Bridge created, gateway field left empty, I went to start the VM I'd already pointed at vmbr1. It refused:

bridge 'vmbr1' does not exist
network script /var/lib/qemu-server/pve-bridge failed
TASK ERROR: start failed

This one confused me for a minute because I had just made the bridge. It was right there in the Network screen. The detail I'd walked straight past is that Proxmox stages network changes rather than applying them live, which is sensible, because applying a bad network change to a remote host live is how you lock yourself out of your own server. New lines in the Network screen show up with a + next to them, meaning pending, not yet applied. My vmbr1 existed in the config file. It did not yet exist in the running kernel, so the VM's network script couldn't find it.

The fix is to actually apply the staged config:

Node → System → Network
Confirm vmbr1 looks right
Click Apply Configuration
Wait for the + markers to clear
Confirm vmbr1 now shows up as real
Start the VM

A reboot also flushes staged changes, but Apply Configuration is the right tool. No reason to bounce the whole node for this. Once I applied it, the bridge went from a line in a file to an actual interface, and the "does not exist" error went away.

Failure three: NO-CARRIER

Now the VM started. I gave it a static config during install (address 192.168.20.2/29, gateway 192.168.20.1, a couple of DNS servers) and tried to reach the gateway. Nothing:

Destination Host Unreachable

The VM's IP config was fine. The bridge existed. And yet it couldn't reach a gateway one hop away on its own subnet. This is the point where it's tempting to start second-guessing the netmask, or the VM's route, or the DNS, and burn an hour in the guest. The thing that saved me was looking at the physical interface on the host instead:

eno5  DOWN  <NO-CARRIER,BROADCAST,MULTICAST,UP>

I'll be honest, that string didn't mean much to me at first glance. <NO-CARRIER,BROADCAST,MULTICAST,UP> reads like a pile of flags that are either all fine or all broken, and it took me a good while of digging around the internet, reading man pages and half-relevant forum threads, before it clicked that two of those words are actively contradicting each other and that the contradiction is the diagnosis.

Once it clicked it was obvious. UP means Linux has administratively enabled the interface. The OS side is fine, the driver's happy, the port is switched on. NO-CARRIER means there is no Ethernet link signal on the wire. The port is on, but it's not seeing anything on the other end of the cable. BROADCAST and MULTICAST are just capability flags and total noise for this problem; the whole story is UP sitting right next to NO-CARRIER.

That single line ruled out basically everything I'd been about to go debug. It wasn't the guest's IP, wasn't the subnet mask, wasn't DNS, wasn't the bridge definition, wasn't the VM's default route. It was the physical layer. There was nothing plugged into a live port on the other end.

The commands I leaned on here, none of which need anything installed:

ip -br link          # every interface and its state, one line each
ip -br address       # same, with IPs
watch -n 1 'ip -br link'   # watch states change live

That last one is the genuinely useful one. Run it, then plug and unplug cables, and the interface that flips state is the one you're touching. It's the fastest way to map "which physical port is eno5" when you're standing in front of a server with eight identical ports.

You can administratively bring an interface up:

ip link set eno5 up

but that does nothing for NO-CARRIER, because the flag was already UP. If it says UP and still says NO-CARRIER, the OS has done its whole job and the problem is downstream of the OS: cable unplugged, dead cable, wrong port on the server, a disabled switch port, a patch-panel mapped somewhere you didn't expect, a bad NIC port, a transceiver the card doesn't like, or an auto-negotiation mismatch.

ethtool confirms it from the hardware's side:

apt install -y ethtool
ethtool eno5

Speed:         Unknown!
Duplex:        Unknown!
Link detected: no

Link detected: no is the hardware agreeing with the kernel: the driver sees the card, the card sees no link. And if you're not sure which physical socket eno5 even is:

ethtool -p eno5 15

That blinks the port's LED for 15 seconds, if the card supports it, which is much nicer than tracing cables by hand. You can also read the carrier state straight out of sysfs without ethtool at all:

cat /sys/class/net/eno5/carrier    # 1 = link, 0 = no link
cat /sys/class/net/eno5/operstate

Same card, different port, different reality

The part that genuinely tripped me up: eno5 and eno8 are two ports on the same multiport NIC. eno8 was up, negotiated, running a happy gigabit full-duplex link. So my instinct was that the card was obviously fine, therefore eno5 should be fine, therefore the problem must be software.

That instinct is wrong, and dmesg spelled out why:

dmesg | grep -Ei 'eno5|link|firmware|nic'

The kernel log showed the Intel igb driver detecting eno5, eno5 getting attached to vmbr1, and the interface entering promiscuous mode, all the normal bridge-attach messages:

vmbr1: port 1(eno5) entered blocking state
vmbr1: port 1(eno5) entered disabled state
device eno5 entered promiscuous mode

But those messages only prove Proxmox wired eno5 into the bridge. They say nothing about carrier. What was missing was the one line I wanted:

eno5 NIC Link is Up 1000 Mbps Full Duplex

eno8 had logged exactly that. eno5 never did. Every physical port has its own cable, its own PHY, its own link state, its own negotiation, its own switch port. A working port next to a dead one proves the driver and the card are fine in general. It proves nothing about the dead port's cable. That's the lesson I'll actually remember from this: "the card works" and "this port has carrier" are completely different claims.

None of which I could fix from an SSH session. Once I understood it was a carrier problem there was nothing left to type; I had to physically walk over to the server, and the fix lived entirely in the real world: checking which port eno5 actually was, reseating the cable, and sorting out the switch side so the port was live. ethtool -p eno5 15 blinking the LED earned its keep here, standing at the back of the rack trying to match a name to a socket. Once the cabling was right, eno5 came up, and I could confirm it properly:

ethtool eno5   # now: Link detected: yes

And from the VM, the gateway finally answered:

ping -c 4 192.168.20.1

That proved the whole Layer 2 and Layer 3 path was intact. Traffic went from the VM's ens18 through vmbr1, eno5, the switch, and finally 192.168.20.1. Progress. Not done.

Failure four: the one that wasn't mine

Gateway reachable, and immediately the next wall:

Temporary failure in name resolution

DNS. The VM could reach its gateway but couldn't resolve a hostname. On Ubuntu the resolver looked completely normal:

nameserver 127.0.0.53
options edns0 trust-ad
search .

127.0.0.53 is systemd-resolved's local stub, which is expected, not a bug. It just forwards queries to whatever upstream servers the interface is configured with. I checked the actual state:

resolvectl status
resolvectl query google.com

and the query hung, then died:

resolve call failed: Connection timed out

So the local resolver was alive and forwarding, but nothing was coming back from upstream. This is exactly the moment where you can waste an enormous amount of time reconfiguring DNS on a machine whose DNS config is already correct. The discipline that gets you out of it is to test in strict layers and not skip ahead:

ping -c 4 192.168.20.1   # local gateway, Layer 2/3 on my side
ping -c 4 1.1.1.1        # external IP, routing past the gateway
resolvectl query google.com   # DNS, only meaningful if the above works

The gateway answered. 1.1.1.1 did not. And that's the tell: if you can reach your gateway but not a single external IP address, the problem is not DNS. DNS hadn't even entered the picture yet. An external IP failing while the gateway works means the packets are leaving my subnet and never coming back: no return route, outbound not permitted, the new interface not in the firewall policy, NAT missing, that class of thing. Testing DNS at that point would have told me nothing except "also broken," which I already knew.

I took it to the network admin, and that's where the real cause was: the upstream master firewall hadn't been configured for the new interface and subnet at all. It didn't recognise 192.168.20.0/29, so it wasn't routing it, wasn't allowing it outbound, wasn't permitting its DNS, wasn't returning traffic to it, and wasn't NATing it. Every symptom, no external IP, no DNS, no internet, collapsed into that one missing piece. My Proxmox bridge was fine. My Ubuntu config was fine. The thing standing between my AI sandbox and every model API and package registry it was ever going to need lived on a device I don't administer, and no amount of debugging the VM was going to change that.

Once the firewall knew about the subnet, the whole chain lit up in order:

ip -br address                 # interface + IP
ip route                       # default via 192.168.20.1 dev ens18
ping -c 4 192.168.20.1         # gateway
ping -c 4 1.1.1.1              # external IP  → firewall routing works
resolvectl query google.com    # DNS  → port 53 works
curl -I https://www.google.com # HTTPS → application layer works

Each line proves one layer, in order, and each one only means something if the line above it already passed. That ladder is the single most useful thing I took away from the whole exercise.

The permanent config

For the record, the netplan I ended up with on the VM, so it survives reboots:

network:
  version: 2
  ethernets:
    ens18:
      dhcp4: false
      addresses:
        - 192.168.20.2/29
      routes:
        - to: default
          via: 192.168.20.1
      nameservers:
        search: [example.local]
        addresses: [1.1.1.1, 8.8.8.8]

Test before you commit (sudo netplan try rolls back automatically if you lock yourself out), then sudo netplan apply, and ip route should show the default via 192.168.20.1 dev ens18 alongside the on-link 192.168.20.0/29.

What I'd tell myself before starting

A few things worth pinning up before you do this on a host you care about.

The biggest operational risk isn't vmbr1 at all. It's fat-fingering vmbr0, eno8, the management IP, or the default gateway while you're in there, and dropping yourself off the box. Have out-of-band access ready (IPMI, iLO, iDRAC, a physical console, something) before you touch network config, not after you've locked yourself out. And leave the existing bridge strictly alone.

Watch out for the second-default-gateway temptation. Adding a default route to vmbr1 gives you asymmetric routing and connectivity that fails in ways that don't make sense. The host gets one default route, on vmbr0. Guests carry their own gateway internally.

Size the subnet for where the thing is going, not where it is today. I say this as a man who ran out of a /29, learned nothing, and provisioned another /29. Six usable hosts fills up faster than you think the moment you're spinning VMs up and down, and growing a subnet later is exactly the kind of live-network surgery this whole exercise existed to avoid.

If the upstream network uses VLANs, the Proxmox side and the switch side have to agree, because a VLAN mismatch is nasty: you can have physical carrier and still get no traffic, which looks like a working link that mysteriously does nothing. And in a cluster, every node that might run a given VM needs a bridge of the same name, or the VM fails to start after a migration because vmbr1 isn't there on the far side.

Finally, everything on vmbr1 now hangs off a single thread: eno5, its cable, its switch port, the firewall, the route. Any one of those failing takes the whole bridge's VMs down with it. Single-link dependencies are fine until they aren't.

The actual takeaway

The thing I keep coming back to is that virtualization networking doesn't end at the bridge, and almost every mistake I made was some version of forgetting that. Creating vmbr1 connected the VM to a cable and nothing more. UP and NO-CARRIER are not the same claim. Reaching your gateway proves nothing about reaching the internet. DNS is worth testing only after an external IP works. And the final blocker wasn't in Proxmox or Ubuntu at all. It was a firewall policy on a device I don't own, which is a very humbling place for an evening of debugging to end.

Four failures, four different layers, each one wearing the previous one's symptoms. Test in order, from the wire up, and don't let a broken layer above convince you the problem is a layer below.

I did eventually get to the MCP servers. The VM boots, it's on its own subnet, it can reach the model APIs and pull packages, and it's a genuinely nice place to experiment now that nothing on it can take down anything I care about. But that's a different post. This one is a reminder that the interesting work often sits behind an hour of deeply uninteresting work, and the uninteresting work does not care that you had other plans.