Tagged “ops”

The Afternoon A Full /var Took The Service Down

The Bug That Refused To Exist Under strace

The Bug That Only Existed When I Was Not Looking

A Systemd Unit That Refused To Stay Dead

The Day It Was, In Fact, DNS

The systemd Unit That Refused to Stay Dead

Cache Misses, and Why the Fast Version Was Slow

A Memory Leak That Was a Map I Never Cleared

Moving Root Onto ZFS

The Service That Wouldn't Stay Stopped

A Kernel Panic I Could Actually Reproduce

Three Days Lost to a Race I Couldn't Reproduce

A Runaway Process, And The cgroup That Caught It

The Off-by-One That Three of Us Approved

When logrotate Wins the Rotation but Loses the Logs

cgroups v2 and a Runaway Process

The Bug Was in My Head, Not the Function

It Was the MTU. It's Always the MTU.

tcpdump Saved Me Again

The Off-By-One That Three of Us Approved

When the Code Was Right and I Was the Bug

The Bug Was in My Assumptions, Not the Code

the leak was a map that only ever grew

the query that was fine until it wasn't

when in doubt, watch the wire

the fast version was slower, and the cache was why

what a syscall actually costs, with numbers

the cron job that fired twice and told no one

how cgroups v2 caught a runaway before it took the box down

making peace with rootless podman

how much swap, and why i stopped arguing about it

a kernel panic that finally held still long enough to catch

three days lost to a gap between two lines

when journald quietly ate forty gigabytes

moving root onto zfs and not regretting it

the day journald quietly ate forty gigabytes

when journald quietly ate twelve gigs

the cron job that ran twice and told no one

how much swap, finally answered for my own boxes

moving root onto zfs without losing my nerve

the cron job that fired twice and told no one

it was the mtu, it's always the mtu

perf top on a box that had no business being busy

the year i stopped writing iptables rules by hand

what a syscall actually costs you, with numbers

how much does crossing into the kernel actually cost

the leak was a cache that only ever grew

when the writeback storm hits and everything stalls

the average is lying to you, look at p99

watching syscalls without a debugger holding the door open

the query that wasn't slow, it was just run a million times

the process that ate a box and the cgroup that didn't

stopping journald from eating the disk

the service that wouldn't die, and the restart loop that hid it

chasing a race condition for three days

the fast version was slower, and the cache told me why

three days for a missing lock

the average response time that hid a fire

the query that was quietly killing us

three days chasing a bug that only existed sometimes

the cost of a syscall, measured

the day dns took down everything

poking at io_uring for an afternoon

moving root onto zfs without losing my nerve

the query that was fine until it wasn't

the rollback that turned a bad upgrade into a non-event

the cron job that ran twice and told no one

taming journald disk usage

the backup that overlapped itself and nobody noticed

how much swap, and the answer nobody on the internet agrees on

running containers rootless without it ruining my week

the leak was a map, and the map was me

putting a runaway process back in its box with cgroups v2

chasing a race condition for three days

the bug that only existed when nobody was watching

the code was fine, i was the bug

the fast version was slower, and the cache told me why

small packets fine, large packets gone: it was the mtu

it was never the database, it was dns

when the box freezes for a second, look at dirty_ratio

the leak was a map, and the map was me

when the writeback stalls everything

the cron job that ran twice and told no one

a kernel panic i could actually reproduce

cache misses, and why the fast version was slower

perf top on a box that should have been idle

the leak was a map i forgot to empty

it was never the network, it was always dns

the outage was a full disk, the disk was full of logs about the outage

the code was fine, my mental model was the bug

the backup that ran on two boxes and nobody noticed

when journald ate half my root partition

a kernel panic i could actually reproduce

the flamegraph that pointed at the wrong hero

the query that wasn't slow, it was just always running

io_uring, and the joy of not making a syscall per read

the leak was a map i forgot to delete from

the bug that only happened when nobody was watching

rootless containers, and making peace with subuid

it's always dns, and this time it really was

how a single full filesystem took down a perfectly healthy service

a database query that was quietly killing us

perf top on a box that shouldn't be busy

the cron job that ran twice and told nobody

it was the mtu, it's always the mtu

the process that ate the box, and the cgroup that fenced it in

when the logs lie, the wire doesn't

the query that was fine until it wasn't

the app that kept writing to a deleted log file

nftables instead of iptables, finally

when the app, the logs and the dashboards all lied, tcpdump didn't

the cron job that ran twice and told no one

the average latency was fine, which is why everyone was angry

yes i still run swap, even with plenty of ram

when the page cache fights back

the outage that was just a disk quietly filling up

how btrfs snapshots gave me my weekend back

btrfs snapshots saved my weekend

rewriting the firewall in nftables

the box that was busy doing nothing

when a busy box stalls every thirty seconds, look at dirty_ratio

the bug that only existed when nobody was watching

when the app won't reopen its logs

the idle box that wasn't

i finally moved the firewall to nftables

the runaway process that cgroups v2 quietly contained

the day journald ate the root partition

the systemd unit that would not stay dead

the map that ate all the memory

the off-by-one three of us missed

the loop that processed every day except the last one

the off-by-one three people signed off on

the flame graph that pointed at a function i'd forgotten existed

perf top, and an idle box that wasn't

putting root on zfs, and why i would do it again

the service that would not die

putting a leash on a process that ate the box

the night a logfile took the service down

the rare gift of a kernel panic that reproduces on demand

the connection that hung at exactly the wrong size

when logrotate quietly stops working because the app won't reopen

rootless podman, and the uid map that finally clicked

when logrotate and your app disagree about SIGHUP

what a syscall actually costs you

how much swap does a homelab actually need

the leak that was just a map nobody ever emptied

when logrotate and a deaf daemon disagree

i was certain it was a race condition

a btrfs snapshot bought back my saturday

when the write cache fights back: tuning dirty_ratio

the flamegraph that pointed at logging, of all things

stop letting journald eat your root partition

how much does a syscall actually cost?

the service that would not die

flamegraphs and a hot path i never suspected

the cost of a syscall, measured

the average latency that lied to my face

the bug that only existed when nobody was watching

rootless podman without the rage

the flamegraph that pointed at the wrong file entirely

the optimisation that made everything slower

the leak was a map nobody ever deleted from

stopping journald from eating the disk

when one process ate the box, and cgroups v2 finally fenced it in

the off-by-one that three of us missed

the flamegraph pointed somewhere stupid

btrfs snapshots saved my weekend

a fence round the process that kept eating the box

it was the mtu, it's always the mtu

moving root onto zfs and not regretting it

the service that resurrected itself every ten seconds

the connection that worked until the payload got big

the log file that grew forever because nobody told the app

moving root onto zfs and not regretting it

i finally moved off iptables

small packets fine, big packets gone

the code was correct, my mental model wasn't

the afternoon one PHP worker tried to eat the box

the average was fine and the customers were furious

logrotate did its job, the app kept writing to a deleted file

the query that was fine until it wasn't

when in doubt, watch the wire

i finally rewrote my firewall in nftables

rootless containers without losing my mind

the runaway process that cgroups v2 caught for me

the optimisation that made everything slower

the average latency is lying to you

the query that only got slow once it mattered

the log that wouldn't rotate

the journal grew until the disk noticed

the outage that was just a full /var

the rollback that turned a disaster into a five minute job

stop journald from eating the disk

the cost of a syscall, measured

rootless containers, and the few things that bite

ebpf, or how i stopped guessing and watched the kernel

taming journald disk usage

the swap debate, settled for my homelab

getting rootless containers working without losing my mind

the outage that was just /var filling up

i finally moved off iptables, and it was overdue

latency, p99, and the averages that lie to you

when the box freezes for a second every minute

three days for a bug that only happened when nobody was looking

it wasn't the database, it wasn't the network, it was dns again

caging a runaway process with cgroups v2

ebpf, or finally being able to ask the kernel a question

how much swap, and why i stopped arguing about it

a btrfs snapshot turned a wrecked upgrade into a five-second rollback

the app that politely ignored logrotate

the night /var filled and took everything with it

the flamegraph pointed at the one function i'd ruled out

the off-by-one we all signed off on

tcpdump saved me again

taming writeback with dirty_ratio

the rare luxury of a kernel panic that came back on demand

the outage was just /var being full

the off-by-one we all read and nobody saw

the bits of root-on-zfs nobody warns you about

moving root onto zfs and not regretting it

when the application logs lie, the wire doesn't

the code was fine, i was wrong

the off-by-one four of us read and none of us saw

the cron job that ran twice and told nobody

three days for a bug that lived in a missing word

the rewrite that was meant to be faster, and was not

how much swap, and why i stopped arguing about it

how much swap, and why i finally stopped arguing about it

when logrotate works but the app keeps writing to the deleted file

when the logs lie, tcpdump tells the truth

the query that was quietly killing us

the fast version was slower, and the cache told me why

bpftrace, and finally being able to ask the kernel a question

the long version: why your write-heavy box stalls, and how to stop it

the page cache was lying to me about disk writes

the average is fine, the p99 is on fire

the cron job that ran twice and told no one

small packets fine, big packets gone

the bug that only happened when nobody was watching

the query that was fine until it wasn't

when a snapshot turned a ruined saturday into a five-minute rollback

putting root on zfs, and what i actually gained

the service that wouldn't stay stopped

the idle box that was pegging a core

when journald quietly ate a third of the disk

i finally made my peace with swap

a fencepost error nobody saw because it looked right

the off-by-one that three of us signed off on

perf top on a box that should have been idle

it was dns, it is always dns

the runaway process, revisited: cpu and io weights in cgroups v2

cgroups v2 and a runaway process

the fencepost that three of us read and none of us saw

small packets fine, big packets gone, and a tunnel in the middle

the flamegraph that pointed at the one function i trusted

when nothing was down except the names

the bug that only existed when nobody was looking

an idle box at forty percent cpu, and what perf top told me

when the writeback stalls everything

the cron job that ran twice and told nobody

half the requests worked, which is how i knew it was the mtu

a runaway process and the cgroup that caught it

what does a syscall actually cost?

the disk wasn't full, /var was

capping a runaway with one cgroups v2 line

watching syscalls without a debugger in sight

a kernel panic that did me the courtesy of being repeatable

chasing a race condition for three days

i finally moved my firewall to nftables

the query that ate a node every afternoon

the rollback that turned a disaster into a footnote

the "optimised" loop that ran slower than the naive one

the log that kept growing after rotation

how much swap, and the answer i finally stopped arguing about

three days for a bug that only existed when I wasn't looking

when logrotate wins and the app keeps writing to the old file

three days for a missing mutex

the average is lying to you

when in doubt, put it on the wire and watch

it's always dns, and this time it really was

the disk wasn't full, only the partition that mattered

the bug that only existed when nobody was watching

io_uring, first impressions

when nothing made sense, the wire did

the code was fine, i wasn't

the panic that turned up on demand

the outage caused by a full /var

the query that was bleeding us for months

stop the kernel hoarding dirty pages

rootless containers, and the subuid rabbit hole

the night a build job ate the whole machine

when nobody believes the network, run tcpdump

three days, one race, and a log line that lied to me

perf top on a box that shouldn't be busy

a kernel panic I could actually reproduce

perf top on a box that shouldn't be busy

watching the kernel work, finally, with ebpf

the off-by-one that three of us approved

perf top on a box that had no business being busy

it's never dns, until it is

moving the firewall to nftables, at last

when in doubt, put it on the wire

the night a forgotten log file took down the lot

three days hunting a race condition that only existed under load

the unit that would not die

the day /var filled up and took everything with it

a database query that was quietly killing us

a systemd unit that refused to stay dead

the day dns took down everything

the query that worked fine until it didn't

the average is fine, which is exactly the problem

who is eating all the cpu on an idle server

the query that wasn't slow, just slow ten thousand times

the outage caused by a full /var

the leak was a map i forgot to ever delete from

the orm that hid a thousand queries

how much does a syscall actually cost

i finally moved off iptables, and i'm not going back

when a box stalls every thirty seconds, look at dirty_ratio

the idle server that wasn't idle

when in doubt, look at the wire

the flamegraph that pointed at the wrong thing, then the right one

the off-by-one three of us read and approved

putting root on zfs without regretting it

yes, i still give my servers swap

the day i could finally watch the kernel work

putting root on zfs, and why i finally bothered

three days hunting a bug that only happened when i wasn't looking

the one panic that didn't waste my week

when /var fills up and everything gets weird

the panic that only showed up under load

a first poke at io_uring

the average that hid the outage

the service that kept coming back

watching the kernel without a debugger

how much swap, then? what i actually do on my own boxes

the systemd unit that would not stay dead

the outage that was just a full /var

the average is lying to you, look at p99

the report that doubled and nobody noticed

eBPF, or finally being able to ask the kernel what it's doing

the bug that only existed when nobody was looking

why the average latency was lying to me

perf top on a box that should have been asleep

when a write-heavy box stalls every thirty seconds

ebpf, and finally seeing what the kernel sees

io_uring, first impressions from the bleeding edge

three days lost to a bug that only happened when i wasn't looking

poking at io_uring on a fresh kernel

the rare gift of a kernel panic you can reproduce

how much swap, and the answer i stopped arguing about

the cron job that ran twice and said nothing

switching to nftables, at last

the leak was a map i kept adding to and never deleting from

i moved a firewall to nftables and stopped flinching

perf top on a box that was meant to be idle

when logrotate wins the battle and the app keeps writing to the old file

it's never dns, until it's the only resolver in the house

the swap question, and why zram quietly won me over

the bug that fixed itself the moment i looked at it

yes i still run swap, and here is the one setting that matters

the log file that grew forever no matter what logrotate did

the slow leak that was just a map nobody deleted from

the rare gift of a kernel panic that happened on demand

moving the firewall to nftables, at long last

the unit that came back from the dead, repeatedly

eBPF, or finally seeing what the kernel sees

journald ate the disk again, so i bounded it

i finally moved a box from iptables to nftables

the flamegraph that pointed at the wrong thing, which was the right thing

a day lost to packets that almost made it

when the journal quietly ate the disk

the off-by-one four of us nodded straight past

how much swap, and why i stopped arguing about it

when a busy box stalls every thirty seconds, look at dirty_ratio

ebpf, and finally seeing what the kernel sees

what a syscall actually costs you

moving root onto zfs, and the boot snag nobody warns you about

cgroups v2 caught the thing cgroups v1 kept letting through

the bug that only existed when nobody was watching

io_uring is the async i/o interface linux always needed

the idle server that was burning a whole core

the service i could not convince systemd to stop restarting

moving my root filesystem onto zfs without losing my nerve

the night /var filled up and took the whole box with it

when too much page cache becomes a problem

the query that was bleeding us a hundred milliseconds at a time

the service that wouldn't stay stopped

the off-by-one four of us read and none of us saw

it was the mtu. it's always the mtu

the bug that only existed when nobody was watching

the outage that wasn't down, it was just lying

the service that came back from the dead every ten seconds

three days for one missing lock

when the write-back stalls: tuning dirty_ratio on a busy box

the slow leak that was just a map nobody ever emptied

the flamegraph that pointed at the one function i'd never have profiled

the outage caused by a full /var

the bug that only existed when nobody was watching

when logrotate and a stubborn daemon disagree

a kernel panic with the decency to be repeatable

the flamegraph that pointed at the last function I'd have guessed

journald ate the disk, here is the one line that stops it

how much swap, asked for the hundredth time

a botched upgrade, and the btrfs snapshot that undid it in seconds

when the box freezes for five seconds and writeback is to blame

the N+1 that hid behind a fast endpoint

it was the mtu, it's always the mtu

the outage that was just a full /var

the backup that ran twice and corrupted itself

a vpn that throttled itself, and yes it was the mtu

three days lost to a goroutine that started too early

a botched upgrade, and the snapshot that undid it

penning in a runaway process with cgroups v2

the join that got expensive when nobody was looking

stopping journald from eating the disk

tuning dirty_ratio on a box that writes a lot

a kernel panic i could actually reproduce

a map with no exit, and the eviction i should have written first

the hot path was in the logging

the unbounded map, and how i finally went looking for it

how much does crossing into the kernel actually cost

it wasn't the network, it was the names

the cache that grew until the box fell over

ebpf, and finally seeing what the kernel sees

the overlapping cron job that ate its own tail

when a job fired twice because two clocks disagreed

switching root onto zfs, carefully

the outage caused by a full /var

ebpf, or how i stopped guessing about a slow box

when writeback stalls everything, look at dirty_ratio

the slow leak that was a cache i forgot to evict

it was dns, it is always dns

the outage where the disk was full and nobody had noticed

the day dns took down everything, again

the outage nobody saw coming, because /var was full

the outage caused by a full /var

instrumenting the kernel without rebooting it

it was never the network, it was the resolver

the flamegraph that pointed at the one line i'd never have guessed

the outage that was just a full /var

moving the firewall to nftables, finally

the night /var filled up and took the app with it

small packets fine, big packets gone: the mtu strikes again

the outage that wasn't the database, it was dns again

what a syscall actually costs, with numbers

it was the mtu, it's always the mtu

the cron job that ran twice and never told me

a systemd unit that refused to stay dead

when logrotate and a stubborn daemon disagree

moving a small firewall from iptables to nftables

three days inside a race i couldn't reproduce

when logrotate rotates the file and the app keeps writing to nowhere

one rollback and the weekend was mine again

it was never the app, it was always dns

moving root onto zfs, and why i'd do it again

your average latency is lying to you

two boxes, one cron line, and a backup that ran in stereo

the map that grew until the process didn't

how much swap, and why I stopped arguing about it

the average latency is lying to you

the cron job that ran twice and told no one

what a syscall actually costs you

the day /var quietly filled up

the btrfs snapshot that turned a bad upgrade into a five-minute fix

your average latency is lying to you

getting the systemd journal to stop eating the disk

the job that fired twice because two boxes thought they were in charge

the cache that only ever grew

the service that kept coming back from the dead

when logrotate and a stubborn daemon disagree

how much swap, and why i stopped arguing about it

putting root on zfs and not regretting it

the leak was a map i kept adding to and never pruned

everything broke because /var was full

the log file that grew to forty gigabytes

the leak was a map, and the map was me

moving a box from iptables to nftables at last

the query that was quietly killing us

a syscall is about 80 nanoseconds, so stop making so many

the night /var filled up and took the lot with it

the flamegraph found the thing I'd sworn was fine

how much does a syscall actually cost?

stopping journald from eating the disk

the leak was a map i forgot to empty

moving my root filesystem onto zfs

it was dns, it is always dns

the cron job that ran twice and told no one

moving root onto zfs without losing my nerve

the day journald quietly ate the disk

the flamegraph that pointed at the one function i'd never have guessed

the hot path was in the logging

flamegraphs and a hot path i never suspected

logrotate vs the app that wouldn't let go

the flamegraph that pointed at the wrong file entirely

the leak was a map i forgot to empty

the weekend btrfs snapshots earned their keep

moving a root filesystem onto zfs on linux