Tagged “debugging”

The Afternoon A Full /var Took The Service Down

The Bug That Refused To Exist Under strace

The Bug That Only Existed When I Was Not Looking

The Day It Was, In Fact, DNS

A Memory Leak That Was a Map I Never Cleared

Three Days Lost to a Race I Couldn't Reproduce

The Off-by-One That Three of Us Approved

The Bug Was in My Head, Not the Function

It Was the MTU. It's Always the MTU.

tcpdump Saved Me Again

The Off-By-One That Three of Us Approved

When the Code Was Right and I Was the Bug

The Bug Was in My Assumptions, Not the Code

the leak was a map that only ever grew

when in doubt, watch the wire

the cron job that fired twice and told no one

three days lost to a gap between two lines

the cron job that ran twice and told no one

the cron job that fired twice and told no one

it was the mtu, it's always the mtu

the leak was a cache that only ever grew

chasing a race condition for three days

three days for a missing lock

three days chasing a bug that only existed sometimes

the day dns took down everything

the cron job that ran twice and told no one

the backup that overlapped itself and nobody noticed

the leak was a map, and the map was me

chasing a race condition for three days

the bug that only existed when nobody was watching

the code was fine, i was the bug

small packets fine, large packets gone: it was the mtu

it was never the database, it was dns

the leak was a map, and the map was me

the cron job that ran twice and told no one

the leak was a map i forgot to empty

it was never the network, it was always dns

the outage was a full disk, the disk was full of logs about the outage

the code was fine, my mental model was the bug

the backup that ran on two boxes and nobody noticed

the leak was a map i forgot to delete from

the bug that only happened when nobody was watching

it's always dns, and this time it really was

how a single full filesystem took down a perfectly healthy service

the cron job that ran twice and told nobody

it was the mtu, it's always the mtu

when the logs lie, the wire doesn't

when the app, the logs and the dashboards all lied, tcpdump didn't

the cron job that ran twice and told no one

the outage that was just a disk quietly filling up

the bug that only existed when nobody was watching

the map that ate all the memory

the off-by-one three of us missed

the loop that processed every day except the last one

the off-by-one three people signed off on

the night a logfile took the service down

the connection that hung at exactly the wrong size

the leak that was just a map nobody ever emptied

i was certain it was a race condition

the bug that only existed when nobody was watching

the leak was a map nobody ever deleted from

the off-by-one that three of us missed

it was the mtu, it's always the mtu

the connection that worked until the payload got big

small packets fine, big packets gone

the code was correct, my mental model wasn't

when in doubt, watch the wire

the outage that was just a full /var

the outage that was just /var filling up

three days for a bug that only happened when nobody was looking

it wasn't the database, it wasn't the network, it was dns again

the night /var filled and took everything with it

the off-by-one we all signed off on

tcpdump saved me again

the outage was just /var being full

the off-by-one we all read and nobody saw

when the application logs lie, the wire doesn't

the code was fine, i was wrong

the off-by-one four of us read and none of us saw

the cron job that ran twice and told nobody

three days for a bug that lived in a missing word

when the logs lie, tcpdump tells the truth

the cron job that ran twice and told no one

small packets fine, big packets gone

the bug that only happened when nobody was watching

a fencepost error nobody saw because it looked right

the off-by-one that three of us signed off on

it was dns, it is always dns

the fencepost that three of us read and none of us saw

small packets fine, big packets gone, and a tunnel in the middle

when nothing was down except the names

the bug that only existed when nobody was looking

the cron job that ran twice and told nobody

half the requests worked, which is how i knew it was the mtu

the disk wasn't full, /var was

chasing a race condition for three days

three days for a bug that only existed when I wasn't looking

three days for a missing mutex

when in doubt, put it on the wire and watch

it's always dns, and this time it really was

the disk wasn't full, only the partition that mattered

the bug that only existed when nobody was watching

when nothing made sense, the wire did

the code was fine, i wasn't

the outage caused by a full /var

when nobody believes the network, run tcpdump

three days, one race, and a log line that lied to me

the off-by-one that three of us approved

it's never dns, until it is

when in doubt, put it on the wire

the night a forgotten log file took down the lot

three days hunting a race condition that only existed under load

the day /var filled up and took everything with it

the day dns took down everything

the outage caused by a full /var

the leak was a map i forgot to ever delete from

when in doubt, look at the wire

the off-by-one three of us read and approved

three days hunting a bug that only happened when i wasn't looking

when /var fills up and everything gets weird

the outage that was just a full /var

the report that doubled and nobody noticed

the bug that only existed when nobody was looking

three days lost to a bug that only happened when i wasn't looking

the cron job that ran twice and said nothing

the leak was a map i kept adding to and never deleting from

it's never dns, until it's the only resolver in the house

the bug that fixed itself the moment i looked at it

the slow leak that was just a map nobody deleted from

a day lost to packets that almost made it

the off-by-one four of us nodded straight past

the bug that only existed when nobody was watching

the night /var filled up and took the whole box with it

the off-by-one four of us read and none of us saw

it was the mtu. it's always the mtu

the bug that only existed when nobody was watching

the outage that wasn't down, it was just lying

three days for one missing lock

the slow leak that was just a map nobody ever emptied

the outage caused by a full /var

the bug that only existed when nobody was watching

it was the mtu, it's always the mtu

the outage that was just a full /var

the backup that ran twice and corrupted itself

a vpn that throttled itself, and yes it was the mtu

three days lost to a goroutine that started too early

a map with no exit, and the eviction i should have written first

the unbounded map, and how i finally went looking for it

it wasn't the network, it was the names

the cache that grew until the box fell over

the overlapping cron job that ate its own tail

when a job fired twice because two clocks disagreed

the outage caused by a full /var

the slow leak that was a cache i forgot to evict

it was dns, it is always dns

the outage where the disk was full and nobody had noticed

the day dns took down everything, again

the outage nobody saw coming, because /var was full

the outage caused by a full /var

it was never the network, it was the resolver

the outage that was just a full /var

the night /var filled up and took the app with it

small packets fine, big packets gone: the mtu strikes again

the outage that wasn't the database, it was dns again

it was the mtu, it's always the mtu

the cron job that ran twice and never told me

three days inside a race i couldn't reproduce

it was never the app, it was always dns

two boxes, one cron line, and a backup that ran in stereo

the map that grew until the process didn't

the cron job that ran twice and told no one

the day /var quietly filled up

the job that fired twice because two boxes thought they were in charge

the cache that only ever grew

the leak was a map i kept adding to and never pruned

everything broke because /var was full

the leak was a map, and the map was me

the night /var filled up and took the lot with it

the leak was a map i forgot to empty

it was dns, it is always dns

the cron job that ran twice and told no one

the leak was a map i forgot to empty