Let me be honest about the requirements first, because the rest of this only makes sense if I am. I do not need BGP at home. Nobody needs BGP at home. I have one internet connection, a handful of VLANs, and a dozen-ish machines that all sit within a single building roughly forty feet across. Static routes would do. Static routes did, for years. What I have now is a small interior BGP fabric speaking between my router and a couple of nodes, and the reason is somewhere between "it solved a real problem" and "I wanted to know how it felt." I am going to defend both halves.
the real problem
The honest version is this. I run a few services that I want reachable on stable addresses regardless of which host they happen to be living on this week. Containers move. A node goes down for a kernel update and its workloads land elsewhere. I had been doing this with a pile of static routes and a NAT rule per service, and every time something migrated I was hand-editing routes on the router and forgetting one, which is its own special category of two-hours-of-debugging-for-a-typo.
What I actually wanted was for a service to advertise "I am here" from wherever it currently runs, and for the router to believe it. That is a routing protocol. You can do it with OSPF, and OSPF would arguably have been the saner choice for a single flat estate. But I reach for BGP at work, and I wanted the at-home version to teach me the bits the at-work version hides behind a vendor abstraction.
the setup
The router is running a build that can speak BGP natively, and on the host side I am running FRR. Each node that wants to advertise a service IP brings up that address on a dummy/loopback interface and FRR redistributes it. The router peers with each node over the management VLAN. Private ASNs throughout, one per node, the router as the route reflector so I am not building a full mesh of sessions for no reason.
A trimmed version of the FRR config on a node looks like this:
router bgp 64512
bgp router-id 10.10.0.11
neighbor 10.10.0.1 remote-as 64500
neighbor 10.10.0.1 description router-rr
!
address-family ipv4 unicast
network 10.20.5.10/32
redistribute connected route-map LOOPBACKS
exit-address-family
!
route-map LOOPBACKS permit 10
match interface dummy0
The shape of it is: bring the service address up on dummy0, FRR sees it, advertises a /32 to the router, the router installs it. When the workload moves to another node, the address comes up there, the old session withdraws it, the new one announces it. The router converges in a second or two and traffic follows. No hand-edited routes. No forgotten NAT rule. I migrate a container and the network simply catches up.
the bit that bites you
Failover is the obvious win and also the obvious trap. If a node simply dies, hard, its BGP session times out and the route is withdrawn, which is exactly what you want, but the default hold timer is ninety seconds and ninety seconds is a long time to black-hole a service. I tuned the timers down and added BFD so a dead link is detected in well under a second rather than waiting on the BGP keepalives. That single change is the difference between "huh, it self-healed" and "why was the thing down for a minute and a half."
The other trap is the one everyone hits: you can advertise a route to an address that the host is not actually serving. BGP only knows the address is up on an interface, not that the thing behind it is healthy. So a service can crash whilst its loopback stays merrily advertised, and you have built yourself a confident path to a closed port. The fix is to tie the advertisement to a health check, so the address only comes up when the service is genuinely answering. I do this with a small watcher that adds and removes the loopback address based on a check passing. It is the unglamorous glue that makes the elegant bit actually trustworthy.
There is a subtler version of the same mistake that took me longer to spot. When two nodes can both serve a given service, they can both end up advertising the same /32, and the router will happily load-balance across them or pick whichever it heard first. Sometimes that is exactly what you want, anycast at home, which is a faintly ridiculous sentence to type. But for a stateful service it is a recipe for confusion, because requests scatter across instances that do not share state. So I am careful about which addresses are allowed to be advertised from more than one place, and I lean on local-preference to make one node clearly primary and the other a standby that only wins when the primary withdraws. Once I had that straight the failover stopped being exciting, which is the highest praise I can give a networking change.
A few things I would tell past-me:
- Turn on BFD before you do anything else, or your failover demo will be embarrassingly slow.
- Filter what you accept. Even at home, even with kit you trust, an
importpolicy that only accepts the /32s you expect will save you from the day a misconfigured node tries to announce a default route into your network. I have made that exact mistake. - Log the session state somewhere you will actually look. A flapping peer at home is silent right up until it is your whole evening.
was it worth it
For the practical problem, yes, but a smaller tool would have served. The migrations-just-work behaviour is genuinely lovely and I would not give it back. But OSPF, or honestly a decent service mesh, would have got me most of the way with less ceremony.
For the other half, the wanting-to-know half, it was worth it twice over. I now understand route reflectors and BFD and import policies in my hands, not just in a diagram, and that has already made me better at the work version where the blast radius is somewhat larger than my garage. The homelab is the only place I get to break BGP and have the only affected user be me, sitting eight feet away, swearing gently. That is a rare and valuable thing.
So: did I need BGP at home? No. Am I keeping it? Absolutely. The best lab projects are the ones that are slightly too much for the problem, because the slightly-too-much is where the learning lives.