In year 2000, the group that is responsible for the JPEG file format, called the JPEG group, decided to come up with a new version of the JPEG format. It was called JPEG2000. It had some really cool ideas. One of them was that it supported streaming of images. One JP2 image contained multiple resolutions of the same image, and the lower resolutions were kept upfront. So, when you are downloading the image, you will get a lower resolution immediately. The advantage was that on low speed connections, the browser could show a low-res image pretty quickly. Also, devices that were low-res could simply stop downloading the image when they had the resolution they wanted

At the time, JPEG2000 was hoping that the JP2 standard would make its way into browsers (spoiler: it still hasn’t). We wanted to use it to build mapping applications. We had encoded aerial photos into the JP2 format, and we had a server that returned SVG maps. Since there was no browser support, I built an ActiveX control that would stream the JP2 image and overlay it with the SVG maps. It was very cool. Our resolution was like 10 times better than Google Maps (at the time).

I was using a library called Kakadu library. It was an open-source library that could stream and parse JP2 images. Kakadu performed really really well. It was fast! Except that once in a while it would get stuck. Randomly. No pattern. So, they pulled me in and I suspected a problem that arises because of thread contention. So, I got to debugging it. At the time, I was young, and I understood multi-threading pretty well, but hadn’t really fixed a thread-contention problem. I was excited.

So, I dug into their code and started debugging it. The first thing I figured is when I debug, the problem goes away. Crap! Essentially, the debugger itself acts as a synchronization mechanism, and it changes the timing of how the instructions in the thread execute.

So, I started adding logs. The next thing I figured out is that when I add logs, the problem goes away! Crap again! Again, since the logging goes to the file system, the file system acts as a synchronization mechanism and throws the timing off.

So, I can’t debug, and I can’t log. It’s a fairly complicated piece of engineering, that I haven’t written. So, before I solve the real problem, I need to figure out how to troubleshoot in a multi-threaded environment. Crap!

So, I started thinking that it’s the synchronization by things outside the code that throws the timing off, right? So, as long as I keep the code inside of the external synchronization very tight, I might prevent the timing from going off. So, I started minimizing my log statements. Eventually, I figured out that if I put one character logs, I am fine. And I can’t put too many 1-character logs.

So, the first thing I had to figure out was if the code takes a path that is different when the problem occurs versus when the problem doesn’t occur. Remember, I could put only single character logs, and not too many of them. So, I started reading through the code without trying to understand it. Whenever I reached a decision point, I would put 2 logs in the 2 branches. One branch logged “\”, the other branch logged “/”. When I saw a loop, I logged “|” inside the loops. When I would see the logs, I would see the log as

\|||//|||||||||\/\/\\\ 

I would run the app and note down these strings of characters when the code ran fine, then note down the characters when it didn’t. Next I compared the strings of characters to find the deviation. Finally, I would trace back through the code to find the spot at which the log message deviated.

Since I couldn’t put too many logs, I had to be judicious. Luckily, the Kakadu code was structured very very nicely. I want to kiss the Kakadu developers (even though they caused the bug). All their code was layered very nicely. They had high-level functions that called lower-level functions, that called even-lower-level functions. So, I picked the topmost layer and put my magic single-character logs there. When I found the deviation, I would understand the code to figure out why the deviation happened. Usually, it was because a lower-level routine behaved differently. So, I had to remove all my logs, and then add similar logs in a lower-level routine. I did this layer by layer till I found the bug.

This entire process took about three weeks. It was a one character fix. There was a busy-wait loop in the rendering thread that waited for data to be loaded by a producer thread. It checked a counter using < instead of <=. Usually, the counter would go from counter < expected to counter > expected, and it would work fine. In the rare condition that the = condition was satisfied, the rendering thread would prematurely parse the data, get an exception and cleanly exit. This would stop all rendering.

Three weeks, one character. I should get a T-shirt that says that.

Fixing this bug really showed me the value of building your code in layers.

I was the supporting our multi-platform program (LabVIEW) for Windows 3.1

Second most requested feature (hundreds of reports from the users) was to fix the print functions to work on all Windows printers. The problem was that the string font would change size arbitrarily and it would become either too small or too large.
Our code was working fine on MacOS, SunOS, HPUX, Windows printers that supported postscript and high end windows printers .
But the average user did not used these fancy,expensive printers. No, they were using the cheap, 50$ printers that you bought during sales at Best Buy or Circuit City.
It looks like 50% of these cheap printers were working fine but the other 50% was failing. Of course, Microsoft Word and Excel were working fine on all these printers.
But it was impossible to figure out the problem. The same code that was drawing the data on screen, was drawing the data on the printer. Same code, same picture, some printer worked, some don’t.
I was baffled. I ask around. I went to Microsoft (we had some special business support ). Nothing helped.

Five years (yes not a typo, 5 years) later, I was still trying to debug the issue when in a act of desperation, I’ve decided to calculate my own bounding box of a string one character at a time and add them together instead on relying on the build-in function that you pass the whole string and it returns you the bounding box (based of course on font size).
And that is when I discovered that the build-in function in windows is actually implemented by the device driver. And the device driver for cheap printers were buggy as hell. They could calculate the size of one character at a time, but give the driver a string and the answer is a random bounding box. Basically cheap printers has cheap printer drivers that had multiple bugs.
Switching to my calculations instead of the printer driver calculation fixed the problem. My code was right from the beginning.

Just because an API exist, it does not mean that it does the right thing under all circumstances ;-)

What's the hardest bug you've debugged?

It was not a hard bug for a developer, but it was for me. Let me lay out the background.

I was a network administrator at the Superior Court of Justice in Brazil (STJ) from 2000 to 2008 responsible for supporting and performance monitoring of Tomcat application servers, in addition to managing the entire local and remote data network. Our developers had full access to the production servers, they could deploy a new application or update an existing one anytime. Yes, I know, big mistake, but it was long ago, before the IT governance era on STJ. IT was really improvised, our “servers” were overglorified desktops spread across two rows of stands.

I was no java developer. I have developed in C++ and Pascal a lot during my university years and kept creating shell and perl scripts to support my role at STJ. One of my greatest skills was application troubleshooting at network level using a packet sniffer, Sniffer Pro at first and, later, the almighty Wireshark. When I was idle at work, I liked to mirror some busy server port and spend hours understanding the cascade of Ethernet frames.

It was like being Cypher staring at the Nebuchadnezzar´s Matrix console:

“You get used to it, though. Your brain does the translating. I don't even see the code. All I see is blonde, brunette, redhead.”

Unfortunately I didn´t see the beautiful women that Cypher saw, but I knew the traffic baseline as no one. It was before this cryptography-everywhere era. HTTPS was an unnecessary burden to our deskt… err, servers. Bottom line, I could see everything: requests, responses, database queries, measure response times and correlate them.

Modesty aside, I was good. I was so good that other IT departments lined at my door for a troubleshooting analysis. DB, authentication, file servers, I debugged them all. They came with the problem: My application is slow, it fails at random times, and the most common:

“- I tested it thoroughly at my local development environment (application, files and database at a local desktop) and it worked fine, now that I moved to production, my performance is crap. Your network sucks!”

9 out of 10 times I could pinpoint the bottleneck: Chattiness, huge file accesses or even some left out code that queried non-existent servers until a timeout. After some months they learned to stop blaming my top-notch data network and just came asking for help, which I gladly provided every time.

STJ had even given me a bonus for teaching a network troubleshooting course for other network administrators, where I taught every trick I had on my sleeve: packet filtering, reordering, time anchoring, tcp stream following, windowing, data reconstruction, HTTP request-database query correlation… Everything. Unfortunately no other employee had reached my troubleshooting skill level, maybe because I am a lousy teacher, or maybe there was not enough engagement from their part, or both.

Network was my kingdom and I was the king.

Enough blowing my own trumpet. Now with the bug: it was just another day at work when suddenly my Tomcat server crashes. First response: restart the service. The server runs fine again.

A couple of days later, another inexplicable crash in the middle of the afternoon. My boss complains. I ask the java developer team if they had made any change on the applications recently. They say the code is months old and no change had been made.

Some days later, another crash. I had enough. Our users are complaining, my boss is blaming my servers and the java developers swear that they did not change anything.

I enable every java management extension I could see and start logging. Now I am just looking forward for the next crash, which, of course, happens some time after.

I go through the logs, and what I found was a quick surge on the memory usage of the java process. It suddenly spiked to the maximum heap size and crashed the server, jumping from a healthy 50% to 100% in a matter of minutes. No clue of what application or class misbehaved.

I ask the developers again: negative responses and some frowns upon me. Ok, maybe it is my server, let´s see…

Out of management extensions left, I decided to call in the big guns on this bug:

RELEASE THE KRAKEN!

• Wireshark laptop deploy: Check!
• Server port mirror: Check!
• Promiscuous mode: Check!
• Spurious traffic filters: Check!
• Circular buffers: Check!

Now that wireshark is online, I just wait for the incident to happen again, and it surely happens. I think: “Let´s get down to business, shall we…”

I look at the java management extension logs and I pinpoint the approximate time the server starts misbehaving based on the memory usage logs. I deep dive on the packets of that approximate time, and to my dismay, there are zillions of packets. I saw blondes, brunettes, redheads, dwarfs, zebras, giants, E.T.s…. But not a single defective request. The server was on a very busy time, the capture was simply too crowded. It was a needle in a haystack.

Think, McFly, think! How a misbehaving request is different from a well behaved one?

OF COURSE!!! There is no spoon!

There is no RESPONSE from a defective application!!!

I crack my knuckles and start coding some filters. First, only HTTP requests. Then, I add the responses. I separate the individual streams, fiddle with the time counting and ordering and VOILA! I locate the only HTTP request with absolutely no response. Just a TCP ack and radio silence !! And it occurs just minutes before the crash, right on the start of the memory usage climbing.

Now I remove some filters and analyse what packets appear around this problem. Surely enough, there is a database query immediately after the HTTP request being received by the server. And another. And another. The request caused an infinite loop of repeated database queries until the server crashes. I extract the offending SQL request text.

Time to locate the faulty code. I do a grep on my server files searching for the SQL query I have just discovered. I found the java file containing the query, and of course, it is inside a loop.

I march to the java developers office, kick the door in and, without warning, yell only the name of the defective java file. A guy on the corner of the office stares at me with big startled eyes and, in an almost inaudible squeak, moans:

“- This change shouldn´t have caused a problem…”

Long story short: Developers were kicked out of production servers. They now had to open a formal request to update applications outside of business hours. We have started a long due IT governance project.

Epilogue

Two months after I left STJ in 2008, I receive a call from the new network administrator asking for help debugging an application misbehavior. I instruct him to mirror the correct server port, capture some data during the issue and send me the file.

Surely enough, I found the problem.

I recently heard that there are still tales around STJ about a legendary guy that used to solve all IT software problems armed only with Wireshark.

These were really fun times!

Edit: As this answer had quite a view count and was even read by one of the STJ developers that witnessed my adventure(thanks for dropping me a line in the comment section, Flavio Borges Botelho), I dived into my digital archives for the actual bug report I wrote after kicking the door in. And I found not only the almost 12 year old report, but even the actual capture file with the offending HTTP request packet number (63352) conveniently written in its file name.

Nicely done, young Rafael McFly. :)

So I think it would be entertaining to my nerd-readers see the actual NMIS graphs, Wireshark capture and the demoniac java code snippet. Enjoy.

Clue 1: Abnormal data traffic increase minutes before each crash (I had two on this day) :

Clue 2: JVMStat Old Generation memory pool usage during the same time frame:

Clue 3: A HTTP request without an answer buried deep in a 74 seconds capture file with 100.000 packets, aka, a needle in a haystack:

Clue 4: A barrage of identical SQL queries just after the never answered HTTP request:

A grep later: VOILA

And remember, kids…

Never trust a database query to stop a loop!

It's kind of painful to re-live this one. As a programmer, you learn to blame your code first, second, and third... and somewhere around 10,000th you blame the compiler. Well down the list after that, you blame the hardware.

This is my hardware bug story.

Among other things, I wrote the memory card (load/save) code for Crash Bandicoot. For a swaggering game coder, this is like a walk in the park; I expected it would take a few days. I ended up debugging that code for six weeks. I did other stuff during that time, but I kept coming back to this bug -- a few hours every few days. It was agonizing.

The symptom was that you'd go to save your progress and it would access the memory card, and almost all the time, it worked normally... But every once in a while the write or read would time out... for no obvious reason. A short write would often corrupt the memory card. The player would go to save, and not only would we not save, we'd wipe their memory card. D'Oh.

After a while, our producer at Sony, Connie Booth, began to panic. We obviously couldn't ship the game with that bug, and after six weeks I still had no clue what the problem was. Via Connie we put the word out to other PlayStation 1 developers -- had anybody seen anything like this? Nope. Absolutely nobody had any problems with the memory card system.

About the only thing you can do when you run out of ideas debugging is divide and conquer: keep removing more and more of the errant program's code until you're left with something relatively small that still exhibits the problem. You keep carving parts away until the only stuff left is where the bug is.

The challenge with this in the context of, say, a video game is that it's very hard to remove pieces. How do you still run the game if you remove the code that simulates gravity in the game? Or renders the characters?

What you have to do is replace entire modules with stubs that pretend to do the real thing, but actually do something completely trivial that can't be buggy. You have to write new scaffolding code just to keep things working at all. It is a slow, painful process.

Long story short: I did this. I kept removing more and more hunks of code until I ended up, pretty much, with nothing but the startup code -- just the code that set up the system to run the game, initialized the rendering hardware, etc. Of course, I couldn't put up the load/save menu at that point because I'd stubbed out all the graphics code. But I could pretend the user used the (invisible) load/save screen and asked to save, then write to the card.

I ultimately ended up with a pretty small amount of code that exhibited the problem -- but still randomly! Most of the time, it would work, but every once in a while, it would fail. Almost all of the actual Crash Bandicoot code had been removed, but it still happened. This was really baffling: the code that remained wasn't really doing anything.

At some moment -- it was probably 3 am -- a thought entered my mind. Reading and writing (I/O) involves precise timing. Whether you're dealing with a hard drive, a compact flash card, a Bluetooth transmitter -- whatever -- the low-level code that reads and writes has to do so according to a clock.

The clock lets the hardware device -- which isn't directly connected to the CPU -- stay in sync with the code the CPU is running. The clock determines the baud rate -- the rate at which data is sent from one side to the other. If the timing gets messed up, the hardware or the software -- or both -- get confused. This is really, really bad, and usually results in data corruption.

What if something in our setup code was messing up the timing somehow? I looked again at the code in the test program for timing-related stuff, and noticed that we set the programmable timer on the PlayStation 1 to 1 kHz (1000 ticks/second). This is relatively fast; it was running at something like 100 Hz in its default state when the PlayStation 1 started up. Most games, therefore, would have this timer running at 100 Hz.

Andy, the lead (and only other) developer on the game, set the timer to 1 kHz so that the motion calculations in Crash Bandicoot would be more accurate. Andy likes overkill, and if we were going to simulate gravity, we ought to do it as high-precision as possible!

But what if increasing this timer somehow interfered with the overall timing of the program, and therefore with the clock used to set the baud rate for the memory card?

I commented the timer code out. I couldn't make the error happen again. But this didn't mean it was fixed; the problem only happened randomly. What if I was just getting lucky?

As more days went on, I kept playing with my test program. The bug never happened again. I went back to the full Crash Bandicoot code base, and modified the load/save code to reset the programmable timer to its default setting (100 Hz) before accessing the memory card, then put it back to 1 kHz afterwards. We never saw the read/write problems again.

But why?

I returned repeatedly to the test program, trying to detect some pattern to the errors that occurred when the timer was set to 1 kHz. Eventually, I noticed that the errors happened when someone was playing with the PlayStation 1 controller. Since I would rarely do this myself -- why would I play with the controller when testing the load/save code? -- I hadn't noticed it. But one day one of the artists was waiting for me to finish testing -- I'm sure I was cursing at the time -- and he was nervously fiddling with the controller. It failed. "Wait, what? Hey, do that again!"

Once I had the insight that the two things were correlated, it was easy to reproduce: start writing to memory card, wiggle controller, corrupt memory card. Sure looked like a hardware bug to me.

I went back to Connie and told her what I'd found. She relayed this to one of the hardware engineers who had designed the PlayStation 1. "Impossible," she was told. "This cannot be a hardware problem." I told her to ask if I could speak with him.

He called me and, in his broken English and my (extremely) broken Japanese, we argued. I finally said, "just let me send you a 30-line test program that makes it happen when you wiggle the controller." He relented. This would be a waste of time, he assured me, and he was extremely busy with a new project, but he would oblige because we were a very important developer for Sony. I cleaned up my little test program and sent it over.

The next evening (we were in LA and he was in Tokyo, so it was evening for me when he came in the next day) he called me and sheepishly apologized. It was a hardware problem.

I've never been totally clear on what the exact problem was, but my impression from what I heard back from Sony HQ was that setting the programmable timer to a sufficiently high clock rate would interfere with things on the motherboard near the timer crystal. One of these things was the baud rate controller for the memory card, which also set the baud rate for the controllers. I'm not a hardware guy, so I'm pretty fuzzy on the details.

But the gist of it was that crosstalk between individual parts on the motherboard, and the combination of sending data over both the controller port and the memory card port while running the timer at 1 kHz would cause bits to get dropped... and the data lost... and the card corrupted.

This is the only time in my entire programming life that I've debugged a problem caused by quantum mechanics.

Footnotes for posterity:

A few people have pointed out that this bug really wasn't a product of quantum mechanical effects, any more than any other bug is. Of course I was being hyperbolic mentioning quantum mechanics. But this bug did feel different to me, in that the behavior was -- at least at the level of the source code -- non-deterministic.

Some people have said I should have taken more electronics classes. That is absolutely true; I consider myself a "full stack" programmer, but my stack really only goes down to hand-writing assembly code, not to playing with transistors. Perhaps some day I will learn more about the "bare metal"...

Finally, a few have questioned whether a better development methodology would have prevented this kind of bug in the first place. I don't think so, but it's possible. I use test-driven development for some coding tasks these days, but it's doubtful we could have usefully applied these techniques given the constraints of the systems and tools we were using.

I get motion sick. Always have. Joined the Navy to kill or cure it. Did both. Learned to use Dramamine/Gravol to fend off the nausea. Basically I start the day before, and during the actual flight take a few more.

So anyway, I’m running an avionics shop in Calgary and trying to get new customers. My best ploy is to ask to get a chance to fix some difficult avionics snag they have in their fleet. So that’s what I did to Time Air back in the 80’s. Time Air had a few older Convair 580’s with first gen Sperry flight control systems.

And sure enough, they had such a problem. They had a 580 where the pilot’s flight director would topple once in a while, ever since they acquired the plane and no one could fix it. They had sent out for testing, every single component in the system with nothing discovered.

So I jumped at the chance and said that’s exactly the sort of snag I loved, and when could I test-fly the aircraft, for that had to be where the problem was? Well an engine change was due a week later so they scheduled me to trouble shoot on that test flight. And that gave me a week to memorize the system.

I arrive on the appointed day, and prepared for my testing. I had flashlights and oscilloscopes and meters and service manuals and blueprints ready. I even took the lid off the vertical gyro so I could see the relation between the actual gyro and the pilot’s flight director.

But first the pilots have to flight test the engine. Well when you take an airliner and have no freight or passengers and very little fuel, you basically have a lively overpowered aircraft pretty much with the same power weight ratios as a fighter aircraft! And that’s what the pilot’s thought they were flying. Having a fine time with fast climbs and sharp turns!

Oops - the bottom fell out. I got so sick that I had to write notes to people because I couldn’t speak, and crawl on my hands and knees because I couldn’t stand. And my thinking became ponderous, slow and heavy.

But I found the bug.

All aircraft gyro systems have two rates of alligning the gyro with the earth: fast and slow. Fast erection is called for during startup as the gyro mass is spooling up to its full speed for maximum inertial stability, and it is fast erection that ensures that it’s ready to use. Slow erection is called for after the startup and continues as the equipment is in use. It is the reduced earth’s gravity correction that maintains the correct attitude correlation to the earth while the aircraft bounces around the sky.

While the pilots were re-enacting the Battle of Britain, I saw the fault develop. It was the gyro itself toppling and the instrumentation was faithfully displaying the fault. There were no slow erect commands getting to the gyro. As long as the flights were gentle and coordinated the gyro would stay true to the earth, but if not, the precessional forces would not be corrected and the gyro would topple.

So we had the evidence we needed and returned to base. There I found there was no wire connecting the control box to the gyro. Then comparing the aircraft blueprints with the Sperry drawings I found the actual bug. There was an error on the Sperry system wiring drawing - the needed slow erect wire was not drawn. And Allegheny Airlines, the first owner, installing the system in the 70’s, followed the Sperry plan (mfr in 60’s). A 30 year old blooper. But because Allegheny’s hangar burned down in the 80’s the maintenance records were lost.

What a humungously horrifying and yet gladifyingly glorious experience! This was the aircraft:

When I was in grad school, I was the lone programmer in a lab full of engineers, so I got to write basically all the code for this new robot we were building:

I had decided to use a new single board computer that used a compact PCI bus, which at the time was a brand-new standard. It was very expensive - $25,000 - which was a whole lot of money for a university lab, but the computer had specs that we just couldn't beat with other existing single-board computers at that time.

There were no available compact PCI motor controller boards, so we had to use a motor controller board that was build for a different bus standard, and then convert from the Compact PCI system to the other board using a bridge chip. The particular motor controller board we chose was based on an 8-bit motor controller IC, the LM629. This particular chip uses memory-mapped 8-bit registers, and in order to communicate with it you have to write and read the registers in a very specific order. If you do anything in the wrong order, or you try to write to a read-only register, or vice-versa, the chip generates an error.

I was a really good C programmer at that time, so I was able to crank out the code in two days. But it didn't work. Whenever we tried to communicate with the chip, it threw an error. I went over the code with a fine-toothed comb, and I was absolutely certain it was all correct. I had no idea what was wrong. I was looking pretty bad to my advisor; I was the C stud, and I couldn't even write this simple device driver. And worse, I had recommended that we use this particular computer system, which cost $25,000, far more expensive than any other SBC we had ever bought, and now I couldn't make the thing work.

Finally, after banging my head against it for a week and making no progress, we threw up our hands and asked the motor controller board vendor if we could bring our system to their facility and get their help debugging it.

We arrived at the vendor and set up. Their programmer checked my code, and he couldn't find anything wrong with it either. After two days the owner took pity on us and asked his best engineer, a digital logic expert, to help us. He carted in a $20,000 digital logic analyzer and hooked it up and had me run my code. What he discovered was that when I had issued an eight-bit read, the chip saw a 16-bit read, which it wasn't expecting, so it threw an error, because the high-order byte was getting read from a write-only register. But the code was clearly issuing an 8-bit read. So where was the 16-bit read coming from?

It turned out the bridge chip had a bug. When it saw an incoming 8-bit read request on one bus, it translated it into a 16-bit read on the other, then threw away the most significant byte. We called the manufacturer, and were told "that's known, documented behavior - it's clearly spelled out in the manual." And when we checked, sure enough, it was - it was mentioned on page 48 in the third footnote, in 8-point type.

The solution we eventually came up with was to cut all of the memory address lines on the motor controller board and shift them to the right by one, and then take the lowest bit line and connect it to highest line on the chip. That way, access requests to any odd 16-bit memory address would map into unmapped register space so the chip wouldn't see them. Worked like a charm, as long as you remembered to only use even memory addresses. But I still feel sorry for the grad students who had that robot after we graduated. There was no way they ever figured out what we had done.

The One That Wasn't There...

Mid-80s, I'm working as a consultant for a medical devices company working on a new generation of Positron emission tomography scanners. Hush-hush, race to beat the big guys to market with a new technique for which the paper hasn't even been refereed yet.

I'm the real-time UNIX guy, doing an embedded system. I've done this a zillion times before for other types of control systems, this is my third medical device.

I think about it, doodle, prototype a little, then in a three day spree sprint code the whole thing. Type "make install" and watch it compile, build, construct a download package, download to the device, and reboot.

And it worked. Perfectly.

I. Could. Not. Believe. It.

You NEVER write code that actually runs the first time, it's just a stub to kick off the debugging. Everyone knows that.

So I ripped it apart, stuck in debugging statements, hooked up logic analyzers out the wazoo, and spent literally a month on it.

IT HAS TO BE IN THERE SOMEWHERE

I knew it, the project manager knew it. We even brought in a colleague to provide a second set of eyes.

Nope. After 6 weeks, we declared defeat. Or victory, as you prefer. There was no bug. It really did work, perfectly, from the first time.

And no, I've never replicated that since.

Here’s a hardware one; not particularly hard but it sticks in my mind.

I’d designed a data acquisition module for a physics experiment. The production run was not that large so we assembled them in-house, using people not robots.

One of the modules failed commissioning tests, so I was debugging it - sending commands via the instrument bus and tracing signals with a logic analyzer and oscilloscope. The initial inspection showed nothing obviously wrong - no missing parts, no solder bridges. I found a missing clock signal - there was supposed to be a pulse on a particular circuit trace to clock data into a register, but there wasn’t. The signals going to the chip that generated the pulse were OK, but it wasn’t there on the output. It looked like a short-circuit, so I checked with a meter. No, high-impedance as it ought to be.

After some head scratching and looking at the circuit board some more, I finally spotted a decoupling capacitor in the wrong place.

Not my circuit board, but a capacitor like C7, though rather smaller.

In digital circuits, a large instantaneous current can flow when a register changes from 0x0000 to 0xFFFF - all zeros to all ones. So we use low-inductance power and ground planes, and capacitors placed close to chips to provide the required electric charge within a few picoseconds. Electricity travels about a foot per nanosecond, and waiting for current to arrive from the power supply would take too long. This particular capacitor was in the wrong place, between a through-hole connection to a ground plane (as it ought to be) and a signal via - one of the small holes that connect traces on one side of the board to the other On my board, the vias were bigger than the ones in the picture, and the small diameter leads on the capacitor fit. The capacitor had a sufficiently large value to absorb the narrow clock pulse. It was effectively open to DC, but a short-circuit to AC.

This one didn't take long to figure out, but it was one of my weirdest debugging experiences of all time.

So, while studying electrical engineering at school, we had a class where you were supposed to do a project related to embedded systems. Two classmates and I really enjoyed the course and decided to build an autonomous RC helicopter. We attached an MCU to a helicopter to control the servos with some input from an accelerometer. The project was a little too ambitious for the 3 month class, especially since we were all new to embedded systems. But it was very fun, and we worked hard, so things were moving along fine.

One late evening we sort of had all the different parts of the system working and were ready to mount it all on the helicopter to start doing trial flights. The only problem was that once we started the system, the servos went bananas every now and then. We went over all the code several times, removed more and more pieces from the system but still couldn't get rid of this behavior.

After a long night of debugging where fewer and fewer things were making sense we didn't really know what to do. One of the team members got so tired of everything that he leaned back, put his shoes up on the table and closed his eyes for a while. Suddenly, the bug didn't appear any more. Tired and out of ideas we started joking around about his shoes maybe being a magical cure for the bug. He played along and started taking his shoes up an down from the table. The freaky thing was that the bug actually wouldn't appear when his shoes were on the table, but did appear when they weren't. After a while we actually started considering that there could be a correlation. Half laughing, half crying of exhaustion we actually did 10-20 runs with feet on table / feet off table and the bug happened exclusively when his feet were off the table. I think this probably was one of my most confusing moments in life, at least related to technology.

That's when it hit us. Common ground! Turns out we forgot to connect ground between two parts of the system which led to communication between them being extremely unstable and sensitive to pretty much anything. When my teammate put his feet on the table, he connected ground between the two parts of the system with his shoe and the table where the other part of the system was located. Even though this connection was probably extremely weak, it was enough to make the communication a little more stable. As soon as we realized what happened, we connected the missing wire and everything was running perfectly (well, at least in regards to that problem).

I guess we were lucky to stumble on the solution by accident relatively quickly so that it got to be more of a fun than a painful memory.

A couple years ago, there was a crash in the Flash Player that was reported to us by both Mozilla and Microsoft. None of us could reproduce the crash, we'd know where the crash was due to a crash log, but it'd make no sense. In fact, there were several crash logs pointing to different lines of code that were due to the same bug (as we later realized).

Finally, an awesome Quality Engineer on our team was able to hunt down a machine on which it did crash, and was able to come up with fairly reliable repro steps. It turned out that it only occurred when using slow hard drives.

The crash would occur during the Flash Player's destruction sequence (like when you navigate to another web page in some cases), when a video was being deallocated. The video file stream wouldn't clear out in time, and exposed a thread synchronization issue.

The thing that made this bug so hard was that it was so hard to find a system to reproduce in-house, and the fact that there was some nasty multithreading going on around where the crash was.

I fixed that crash, and it turned out to be quite popular. It probably prevented tens of millions of crashes -- in a time when browsers would crash along with the plugin.

We felt like heroes :-)

While working at the largest backend telecommunication equipment company in the world, on my checkin, 10 different types of backbone routers stopped communicating to their control servers.

That is, on 10 different platforms, there was no way to transfer the new code to the router. (It was actually more than 10 platforms, probably like 25-40, but since I can't remember I put in 10. Yes, 25-40 - all one code base in C ... #IFDEF hell :) )

So the whole company's build failed on my checkin. And I get a "nice" email from the CTO. Of course the bug wasn't in my code. :) ...

What had happened was that on my checkin, the size of the image on all these platforms had become an exact multiple of 1024 bytes. And I discovered that when the size of the image was an exact multiple of 1024 bytes a bug in the FTP code would make the transfer hang, because it would drop the second last packet. Insidious! If I remember correctly, it was one of those "off by 1" bugs.

I still can't believe I found that bug as I had never ventured near that code ever before, and it was a huge code base.

After finding the bug I got a congratulatory letter from the CTO and two sleepless nights :). The engineer assigned to fix the bug calls me up and with the magic of xkibitz I coded the fix in his repository which he then checked in and tested.

Yes, these were the days of single-threaded embedded systems.

It's not my hardest bug but it's one of the most fun to tell about. I was working on Midnight Club: Los Angeles for PS3 and 360, past first, second, maybe event third submission (Rockstar had a habit of leaning pretty hard on the console first party QA teams, probably one of the reasons why Microsoft now charges a ton of money for multiple submissions).

One particular race our designers are reporting their tires get kicked up a bit too high into the air when they drive over a single curb in a hairpin turn, causing them to go a bit wide and occasionally slam into a building. They can reproduce it 80% of the time, but only by running the first few minutes of the race, and the effect is pretty subtle.

They're playing the release build we submitted, so obviously I ask them to try to reproduce it in a debug build where we log the hell out of everything. In fact I owned much of the logging system because we used the system to detect divergences in the deterministic race replay logic and I had been responsible for instant replays.

No dice. 20 races later we still haven't seen it happen in a debug build. Maybe we imagined it. Fire up release again, and it's pretty clear the car is handling a bit different over this particular curb. Only one curb. In the entire freaking city of Los Angeles we modeled.

Needless to say, when we're past third submission and we have to ship for the holidays or we're all doomed, we don't want to waste any time. I start bisecting possible changes between our release and debug configurations. Eventually we get it down to "logging enabled" = "car drives fine," "logging disabled" = "tires bounce off the curb too much." Well shit.

We enabled the logging function calls in release builds but left the functions themselves as empty stubs. Never did figure it out. Game shipped. Onto the next project. I figure it was uninitialized stack variables but we'll never know.

• Error Pattern recognition.

Well if there is a bug in the code we’ve written somehow by spending some extra time we can fix that out. But what if the bug is in Code Editor. It’s gonna freak you out.

I’ve recently faced this issue with VSCode and almost spent 2 days time debugging it. Below is the problem statement. Read it till the end, I guarantee it thrills you.

Problem Statement :

When our GO microservice started in 2016 there was no official client library for Apache Kafka in Golang. So we planned to use Sarama a open source Kafka Client for Golang. But recently Confluent the company that maintains Kafka had released a official Version for Go client called Confluentic-kafka-go.

Here comes the problem. The Confluent Kakfa client internally imports functions written in C. So for every Go file written in Kafka, the underlying operations are completely based out of C language.

If you see this code, this is for Kafka Consumer and at line number 20 you see there is an import called ‘C’.

The problem with this package is that, you cant import it and go compile time handles this package. All the code written as comments will also be interpreted by the Go Compile time.

But Poor VSCode don’t know this and as usual if a package is not found in GOPATH, it stops auto suggestions and recommendations.

This became a hell of task for me because if I was not able to see the underlying structs and their properties without which I can't write my Kafka client to produce or consume.

I started my debugging, read tons of references in VSCode Github issues but none of them had an answer. But after 2 days, I found a pattern in the way VSCode works.

So this is what I’ve did.

I went into each go file of confluent-kafka-go package and flipped the non importable C package to the top and importable packages to the bottom.

And kaboooom, it started working and started showing me all the recommendations.

With VSCode the pattern is that, if the last imported package is not found it won’t show you any suggestions. But if the same package is on the top and all the bottom packages are importable then it works as it had to be.

The result is I have to change the same in these many files for the auto recommendations to work again. Hope this helps for anyone who is setting up Confluent Kafka client with VSCode.

Teaching “smart” people to read the manual. It is the biggest bug in most systems.

Yep they have PhDs, MS government positions, funding and all sorts of fancy credentials but wont read the manual.

Why wont the imu in my fancy machine stabilize? its broken. We need a new $15000 imu. Because The manual says to turn it on and let it run for 5 minutes parked on the ground so it can stabilize. It cant stabilize in flight when its moving with the airplane.

This gps is supposed to give 50 foot accuracy why is it off by 500 feet? Because the satellite system cant give 50 foot accuracy in this area without a correction signal and there is no correction signal. That's in the faa manual on GPS.

Why is my Lazer gas chromatography giving false readings? I calibrate it before use every time. Because the manual says you are supposed to calibrate it with distilled water not that stupid store bought flavoured water which has different chemicals in every bottle and flavour.

This fancy school attendance program cant print out daily class attendance between periods. We need to get a whole new system to catch students ditching one class. Try entering ‘xvxvxvxv’ like the manual says on page 36. See how easy that was!

Read the damn manuals. I solved all these problems and more for the offending specialists. By reading the manuals.

For me, the hardest debug challenges are those where most of the effort is focused on writing the correct debugging facilities. Here is my story...

The year was 2005 and a customer had asked me to debug a machine that was running Windows 2000 (SP4). The machine was crashing, displaying a Blue-screen (back then, Blue-screens were common) and wasn’t creating any dump files. To give you a better idea of what I was seeing, here is a snapshot from Welcome to Flickr - Photo Sharing which displays similar symptoms:

(Taken from fogindex @ Welcome to Flickr - Photo Sharing)

When debugging, it’s always a good idea to understand the history of the bug as well as the current scenario, so I asked the customer to give me more details. That’s where things started to get complicated. It turns out that the bug was presenting itself ONLY when the machine was tested in the field - and when I say field, I mean it literally: a field. The crash was occurring only after the PC was being driven through rough terrain, sometimes a day after and sometimes a week after. To further complicate the situation, when the customer added 4G of memory, things worked much better - the crash appeared every couple of weeks. When I asked what the heck they were running, I received the following reply: a 2GB Visual Basic application(!!).

The fact that the bug was rare made it more difficult to catch and frustrating. For this reason, I generally advise users not to attempt their own work-around without first understanding the underlying cause of the problem.

Ok, now it was time to get to work. So what did I know so far? There was a standard PC machine, running an unmodified Windows 2000 kernel with a 2GB Visual Basic application, that was crashing every 2-4 weeks while driving in a field. Well, it wasn’t exactly a good starting point. Debugging kernel panic with only four numbers…

Looking at the Blue-screen images from previous crashes I saw two types of crashes: KERNEL_STACK_INPAGE_ERROR and KERNEL_DATA_INPAGE_ERROR. In both crashes the second parameter was STATUS_NO_SUCH_DEVICE. Reversing the Windows kernel, it was clear that the crash could have originated from only eight places, and in all of them the kernel tried to do a page-in (load a page from a cache to the memory) and failed. So the main challenge was to debug a crash that was happening once every two weeks and wasn’t leaving behind a dump file or any other debug information.

How to debug such a crash? Well, a lot of debugging infrastructure needed to be written. One of the challenges was to display the logging information - since for unknown reasons the machine didn’t generate any dump file. I worked around this problem by replacing the Microsoft Blue-screen window with a “green” screen that displayed dedicated information. Here is a (real) example:

Whenever a Blue-screen displayed, I redrew a “green” screen with a stack trace of the crash (Enumerate stack frame using EBP register chain), and displayed information on the active devices to see why I got “STATUS_NO_SUCH_DEVICE”.

For the kernel hacker out there, the way I did this was by patching KeBugCheckEx (the kernel function which is invoked for every kernel-panic) using “code patching” techniques. Replacing the assembly bytes of the KeBugCheckEx from their normal function header:

   0x55,                                  // push ebp 
    0x8B, 0xEC,                            // mov  ebp, esp 
    0x81, 0xEC, 0x74, 0x03, 0x00, 0x00     // sub  esp, 374h 

into a jump call to my function:

0xE9, <Relative address> // jmp MyBugCheckHandler

The new jump invoked my function - “MyBugCheckHandler” - which displayed a green-screen using boot display API (e.g. InbvIsBootDriverInstalled). The function cleared the interrupt flag, avoiding any unwelcome context-switch. Since the function never exited, I could later take a digital camera and photograph my messages.

Since “MyBugCheckHandler” is just a function, it can call other kernel API’s. One of the API’s that I have used is Plug&Play API, in order to scan all devices (of FILE_DEVICE_DISK type) and see which device has been removed.

The result of the test was that for some unknown reason the hard drive (ATAPI device) was removed from the system, but I couldn’t understand when and why. It seemed like the drive was removed very early but the machine kept running. Only later did I connect the dots: since the customer had increased the memory to 4GB, there hadn’t been any paging activity until much later...

So, I modified my humble tool and created “Atamon”. Atamon is a kernel debugger that runs inside the kernel and places breakpoints in strategic places within the atapi.sys driver and logs them for future use:

The main purpose of the Atamon was to be able to read the ATAPI registers and display them. There wasn’t any other way to try to understand why the device was essentially committing suicide besides code-patching the atapi.sys driver.

Digging in, using the Atamon, I saw that at some point the disk decided to lock and remain locked. No matter what atamon.sys tried to do, and no matter the amount of resets to the controller, the BUSY bit of the ATAPI simply never went down. Changing Atamon to toggle the power line GPIO and forcing HW reset to the controller was the only thing that solved the issue. Furthermore, the Atamon tool could identify, in the field, the exact time of the ATAPI disk crash and helped understand the physical conditions leading to the crash.

And that, my fellow programmers, is the hardest bug I have ever come across. It totalled about one month of writing NT kernel-mode debugging infrastructure. Since then, I have fallen in love with my kernel debugger. I love it so much that I use it as a debugging tool for myself.

Instead of using WinDbg (which stalls the entire system and can’t gather information on runtime) to solve the problem I have just described to you, using this tool to count/remember to invoke API while debugging proved to be useful. The only problem is that it’s not a generic tool and only I can use it. Hopefully one day I’ll gather the time and energy to release it as an open-source product.

An error in Java TreeSet that caused an object to be in the set while not being in the set at the same time.

Essentially, if you asked tree_set.contains(obj) it returned false, but if you iterated over the set for (o in tree_set) { if (o==obj) return true; }, then it was found. My workaround was to add a facade to the TreeSet and replace the contains method with the slower iteration-based implementation.

The problem was that the fault occurred very infrequently and I had to set conditional breakpoints in the solution and narrow down the search in steps. To check if the fault occurred took about 5–10 minutes. There were lliterally thousands of sets, since it was an implementation of incremental and local anomaly detection.

It took me between 300–400 hours to track it down and once I found it I went WTF? I could not believe my eyes.

Then I have a hardware error in a Sun Sparc Ultra 2 that gave a memory exception while pointing at an machine instruction that did not memory access what so ever. That fault disappeared.

Sporadic Network file transfer corruption, only happened during business hours, never repeatable by every method we could figure might cause such a thing….

Until we realized the operations staff had wrapped Christmas lights around the network drop poles… and in a couple instances near enough where the network wiring emerged from the poles that the “blinking” of the holiday lights would, on occasion if the timing was just right, mess up a packet in a way the error correction codes wouldn’t catch

Only happened when the lights were the blinking kind, solid illumination lights didn’t create the problem.

You can’t make this stuff up.

A bug for getting a bounty from google. It took me 40 hours but eventually, I found one.

Basically, I had received some acknowledgements from companies but I really wanted a name on Google Hall of Fame. I knew that it would be hard to find a bug. I tried a lot but was unable to find one. I went on to university, came back, ate supper, started finding bug. This routine went on for almost 4 days and I spent almost 30–40 hours on finding just a single bug. I had to surf more than 100 google sub-domains and some acquisitions to find it. Although the bounty was less, I felt utter happiness when I received it. I had just started bug bounties a few months ago and getting acknowledgement from Google was what every bug hunter dreamed of.

PS: The vulnerability was in an outdated plugin used in one of the acquisition’ website. A public exploit for that plugin was available.

I just posted this under another question, but it's also the hardest bug we ever tackled:

Years ago at eBay, we hit a number of versioning bugs during our transition to Java technology on our servers.

Some of the versioning problems were quite obvious. For example, I don't think anyone would be surprised to learn that you can't compile code against IBM's JVM 1.5 and then run the bytecode against Microsoft's JVM 1.4.

Later, we had similar problems when a project was transitioning to JVM 1.6. Code compiled for IBM's JVM 1.5 was compatible about 99.9% of the time with JVM 1.6, but that remaining 0.1% was enough to cause serious headaches. We had to carefully assure that all compiled code was compiled for the appropriate JVM targets. Most Java developers are aware of these potential issues as well.

But then we hit the ultimate versioning problem: Different builds of IBM JVM 1.6 were incompatible with each other. Same major and minor version of the JVM, same manufacturer, just different builds. The problem manifested as a memory leak, though. Further confounding a solution, it was impossible to reproduce outside of production, because engineers were compiling and running on consistent versions of the JVM installed on their boxes.

That one took a significant amount of digging to figure out and caused intermittent outages for a period of about 4 weeks(*), because we just couldn't nail down the cause of the problem. We eventually brought in an IBM rep to help us diagnose the problem. No one remotely expected that different builds of the JVM with everything else equal, could itself be the cause of such an obscure problem.

The final solution was just to install the server JVM on every developer box, so that we could compile against the JVM server target. We also worked with IBM to nail down the root cause of the memory leak, when JVM builds were inconsistent.

The moral of the story, always compile against the very same version of your Java JVM and JRE that you intend to run against.

No pain, no gain: This incident had a silver lining, because it resulted in eBay building out a much more rigorous production debugging framework, and a production sandbox environment where engineers could more easily get access to running production servers for the purposes of testing bugs directly against production traffic.

(*) Fortunately eBay had enough redundancy, that there was no customer impact.

My favourite bug of all time was uncovered because I was just too impatient!

I was working on a fairly large and complex embedded project. A microcontroller was interfacing with some custom hardware. It would configure, control, and monitor the hardware. The microprocessor was running an RTOS with a lightweight TCP/IP stack, so remote clients could connect to the device via Ethernet and perform remote configuration. We also used this for diagnostics and testing during development. We had successfully used a similar design on several products, with great results. We had also designed a protocol for monitoring and control that was used in earlier models, and we extended it here - though this product was significantly larger and more complex than its predecessors. What could possibly go wrong?

The first thing the microprocessor does when you connect is to dump the status
of the entire system. With smaller devices, this was very fast. But this new
model was so large and had many more components and subsystems, it was taking a very long time every time you connected. This device can also serve many concurrent clients, so this slow connection startup would be repeated for every client. The processing and time cost was not trivial, and this delay would
limit how quickly clients could start updating their own settings.

Slow and inefficient code bothers me. Eventually I got sick of waiting a good
10-20 seconds or so every time I connected before I could start working with the hardware to test my latest changes, so I decided to track down the performance problem. As a baseline, I measured the system performance before changing anything, and then got to work. It had to be something simple, after all - we've had this same code working on other products for many months in the field.

I figured a few hours would probably be enough to solve it. The debugging
process was made somewhat more difficult by the fact that there was no keyboard or monitor on this hardware; all debugging was either via JTAG and a serial USB connection, or by the Ethernet port - which was part of the problem.

The actual time to solve the mystery would be more like a week, with several long, late, pizza-fuelled nights, drilling through many layers and thousands of lines of code to find the very surprising root cause.

First layer: application protocol

The most obvious point to start with was the application level buffering. A
protocol layer abstracted away the TCP/IP networking (which also allowed for
serial interfaces) and had its own read and write buffers. I guessed it might
help to increase the buffer size a little, to have fewer networking calls. A
little increase didn't help much overall, nor did a big increase. Ok, so it's
not the protocol buffering.

The status dump on connection has to traverse many data structures, build
strings and create structured text for output. I reviewed the algorithms,
memory management and improved the code by reducing reallocations and copies, tidied things up and used pre-allocated buffers more efficiently.

I measured the performance improvement and noticed a modest difference. All that work helped - but it was still not enough. It was just too slow for my liking.
According to my calculations, the system should be capable of *much* higher
throughput. I decided to keep digging.

Second layer: networking and OS

Underneath the protocol code was a high level interface for networking. Perhaps this wasn't as efficient as it could be? After poring over the code, analysing potential performance issues, I found a few small issues, but no smoking gun.

Now this RTOS has many tasks (threads) running, only one of which is the
networking support. Could this networking task be getting starved for processing time? The interrupt handling latency should be guaranteed and well within the required time. I disabled all non-essential tasks, tried increasing the priority of the networking task, and various other tweaks. None had any impact. The RTOS kernel was perfectly happy and running smoothly.

Keep digging...

Third layer: TCP/IP tuning

The TCP/IP stack we were using has a boatload of parameters you can configure at build time, including its internal buffers. This was a prime candidate for performance issues. I dug up the documentation, went through our configuration, and sure enough - bingo! Several parameters were not at the recommended values for this version of the library. Some buffer sizes needed to be multiples of the packet sizes (eg MSS), and tuned to match other significant parameters. This could have caused fragmented packets or memory buffers, and introduce small but potentially disruptive delays to the flow.

This tuning process took many many hours, and eventually resulted in a decent
improvement in throughput. But was it enough? No - the big dump when the
connection was established wasn't just slow now, it was noticeably jerky and
very *bursty*. I really needed to see exactly what was happening between the two socket endpoints. I needed to understand why it was bursty - and fix it.

Fourth Layer: On the Wire

Having calculated the theoretical peak throughput, I decided there was no good
reason this microprocessor shouldn't be able to maintain a much higher level
of throughput. Time to do some low-level packet analysis.

I set up Wireshark and started capturing packets. At first, everything seemed
ok but looking at the timestamps showed clearly that the transmissions were very bursty. Sometimes there were delays of a few *seconds* between packets! No wonder it was taking so long for a full status dump... but what was causing
this?

Looking at the IP layer, I decoded and inspected the session piece by piece,
from the very first packet. `SYN, SYN-ACK, ACK...` All good so far. But after
transmitting only a few data packets: `NAK`. Retries? Backoff? Delays! What on
earth was going on? The trace showed the micro was resending packets it had
successfully sent. Yet by matching up the sequence numbers, it showed the
packets were being `ACK`ed by the other end. Eventually after receiving a few
out-of-order packets, the receiver tried to back off by increasing timeouts.
This perfectly illustrates the bursty nature of the traffic. But what could
be causing it?

Not leaving anything to chance, I tried changing Ethernet cables to make sure
it wasn't a dodgy connection causing the fault. No dice.

At this point, my best hunch pointed to a bug in the TCP/IP library. Resending
an already acknowledged packet? Madness! Since we had found bugs in this library before, it was quite conceivable. I upgraded the stack to the absolute latest version and reran all the tests. Same problem. Yet according to the forums and bug tracker, nobody else had reported this kind of problem with this stack
before.

I decided some major action was needed. I needed to partition the problem and eliminate large components to isolate the fault.

Isolation

First stop, to write a simple socket server which would accept a client
connection, and then send out packets in a tight loop, as fast as it could. This
would exercise the TCP/IP stack, driver and hardware without any of the protocol or application code. The packets contained a monotonic counter so I could see if any packets were being corrupted or lost.

Running this test and capturing packets on the wire revealed the same problem. A burst of traffic, a flurry of `ACK`s and `NAK`s followed by timeouts and
resends. Curses, foiled again!

Ok, how do I eliminate the TCP/IP stack from the equation? I constructed a UDP ping packet by hand, using parts of the wire capture data to fill in the
relevant fields (such as MAC addresses). I kept a monotonic sequence counter
and copied this into the binary ping blob at the correct offset, which I passed
directly to the driver, with my workstation hardcoded as the destination. I
started with a small delay, in the order of 100ms between ping packets. This
seemed to work ok. But as I decreased the delay, packets were being dropped.
Dropped?!

The only thing between this test and the wire is the device driver and hardware.
Could the driver be corrupting or dropping packets?

Fourth layer: device driver

A code review of the device driver didn't show up anything suspicious. Looking
at the memory management, interrupt handling - it all seemed quite carefully
written. Many hours later, no closer to the problem.

I pulled up the datasheet for the Ethernet controller and started querying the
status registers, halting the microprocessor and printing a diagnostic. There
were no clear errors to be found, so the driver did not appear to be causing the
hardware to fail sending or receiving data.

Fifth layer: hardware

The microprocessor has onboard Ethernet support, which is connected to a
separate MAC (Media Access Control) chip. This MAC performs the actual
electrical interfacing, and is the last piece of silicon before the wire. I
started reading the datasheet and looking at the initialisation sequence in the
driver, which configures the registers in the MAC on powerup. I verified the
correct register flags and values, but while I was reading I noticed there were
some counter registers which collected counts of certain types of media
(physical layer) errors.

I added some code to my minimalist hand-crafted ping test to read these counters from the MAC registers, showing the values before and after the ping burst. Sure enough, the counters were 0 on powerup, and after the ping test one of the error counters had a very large number. Ok, I think we're finally on to something...

Back on the wire

I modified the test program to send out hand-crafted `ARP` packets. The only
other code in play was the driver. I went back to Wireshark and captured
another session. This time, I exported the trace data to a file and analysed
the timing information in the headers.

I then stepped through and counted the number of successful packets sent before a failure. Then the next, and the next. And I started to notice a sort of
pattern. The gaps were something like 9, 17, 33, 51... and eventually it would
come back down and repeat. A regular pattern is very interesting, but what
could be causing this kind of failure?

Stepping back and looking at the regular pattern of successes and failures over
time was like looking at an interference pattern. Like ripples in a pond,
where the waves met, packets were dropped. A colleague observed that this
looked a bit like there were two slightly different frequencies involved...
Wait a minute!

Don't blame the Hardware

It was nearly midnight, and I desperately wanted to talk to the Hardware
Engineer who designed the system. But it would have to wait until the morning.
I fired off an email explaining what we had found, and went home exhausted.

The next day, I walked up to the Hardware Engineer who had a big grin on his
face. "I think I found your problem...", he opened. I was skeptical, but
excited and urged him to explain. "In the last board spin, I rerouted the clock
source of the MAC controller. So the Microprocessor and the MAC were actually running off two different clocks!"

I was elated. This perfectly explained the "interference pattern" we had
observed. The frequencies of the two clocks were supposed to be the same, but were not perfectly aligned. Even a slight difference in frequency would cause a 'beating' effect as they drifted in and out of phase. Much like you can hear when tuning a guitar, and two strings are almost, but not quite, in tune and you hear a lower frequency 'wow'.

So - while the two clocks were aligned, the microprocessor and the MAC
controller chip could reliably communicate, and the packets flowed normally. But as the clocks drifted slightly out of phase, their chip-to-chip communication
was corrupted as the rising and falling signals between them became misaligned. This explained why packets appeared to be sent or received at the higher layers, but were in fact lost in the intermittently garbled transfers. It's almost a marvel TCP/IP worked at all!

The Fix

In the end, it was a simple matter of ensuring both chips used a common clock
source - which required a modificaiton to the PCB routing. But for now, to test
the issue, the Hardware Engineer broke out the soldering iron and fixed the
routing by hand on our development system. (We were fortunate that the clock
signal was on an outer layer of the PCB!) I started the test again, very
nervous to see the results. After days of chasing ghosts, I didn't want to get
my hopes up.

It worked! The hand-crafted arp and ping tests would run for as long as we
liked, and never skipped a beat, running as fast as it could go. Finally, full
throughput was achieved. I queried the registers for protocol and link errors,
and it was all good. I checked the TCP/IP layer diagnostics for errors and
statistics, and there were no red flags. I went back to the original application
firmware and tested out the protocol, monitoring the wire for good measure.

This time, it took less than a second for a full status dump. Finally - success!

Wrapup

So - what began as a seemingly simple software performance problem eventually turned out to be caused by a hardware design fault. And it revealed several other opportunities for improvement along the way. This was a great learning experience, and a very satisfying puzzle to solve!

Random thoughts:

1. TCP/IP is really, really good at surviving unreliable hardware and problems in layers below.
2. Don't mix clock sources between chips!
3. Don't assume that the first problem you find is *causing* the problem.
4. Don't assume the improvement you make is sufficient. Measure!
5. Performance is relative. What can you reasonably expect from the system in front of you?
6. Performance tuning is all about careful measurments and consistent tests. And changing one thing at a time!
7. It's hardly ever the fault of the OS. But it could be. And it's hardly ever the fault of the hardware. But it could be.
8. Don't be satisfied with a fix until you understand *how* it works and *why* it fixes the problem.
9. It is sometimes possible to diagnose hardware problems through analysing software.
10. Persistence usually pays off!

It was about 1975 or so. I was an E-3 level 3 Defensive Fire Control Tech on B-52H models at Grand Forks AFB ND.

The Defensive Fire Control System (DFCS) was adapted from the B-58. The B-58 had one radar. The B-52 had two, one angled left, the other to the right.

One new box was added, the System Control Assembly (SCA) to manage the right side. The left side used the Tracking Control Assembly (TCA), like the B-58. Both worked in conjunction with the Ballistics Computer.

The left side wouldn't track. The TCA was swapped out several times as was the Ballistics Computer. Wiring was checked between the two.

The left side antenna was removed and replaced as was the Controlled Line Platform (CLP), a set of gyros that kept everything aligned, as well as the operator hand control, a joystick with a few switches.

Everything we pulled tested out fine on the test sets. All wiring checks were fine.

I was basically a shop guy, running tests and repairing these larger assemblies. I got out the TOs. I looked at schematics. I found what I thought might be the problem.

The SCA had a wire, inside, that did nothing but pass through.

I suggested we swap out the SCA, a unit with no business in the left side tracking. The E-5 Level 7 scoffed. He had worked B-58s. He knew better.

Swapping out an SCA wasn't hard … two bolts and some connectors but the E-5 wanted this checked before we did the remove and replace, a pain in the ass because it wasn't easy reading the pin numbers on the connectors and making certain you had the right one at each spot, the meter lead all the way in on a female and not touching two male pins at the same time.

After climbing in, checking it several times while holding my flashlight in my mouth and balancing the meter I concluded I was right. It was dead open.

The E-5 scoffed and said I must have had the wrong pins but finally agreed, probably just to shut me up, to swap out the SCA.

Fixed the problem…on a Saturday morning.

When I ran the tests on the SCA it passed. (Tests were automated, run from a punch tape that checked all the functions of the unit.) Because of that he refused to sign off on shipping it to depot for the fix, something beyond base-level authorization

Checking it with the multimeter, with the E-5 and the E-8 shop chief looking over my shoulder, showed what I saw on the flightline. More digging showed the automated tests never checked the pass through.

The unit was shipped to depot but AFAIK the test of that wire was never added to the TO or the tape. It happened once in 15 years on a system that was outmoded.

I managed development of a a real-time stock market analytics workstation back in the Stone Age (the 90s). We took in a feed of all U.S. stock trades at each client site, and combined it with a terminal-based (as in VT-220) trading session at each desktop. The trading sessions would "randomly" freeze. More frustratingly, after minutes and minutes frozen - they would sometimes just unfreeze - and spool out all the data that had been received and keystrokes processed. Went on for months. We captured and replayed every byte and keystroke and couldn't isolate the issue, no matter how hard we tried. One day, the freeze happened in my office. It was a very hot day, and I had turned a fan on at some point. For fun, I tried turning off the fan - unfreeze. On: freeze. Off: unfreeze. After all of the time we spent tracing through lines of our own code, it turned out a small electric charge was causing a bit register in a chip on a board on a LAN card to flip, causing the NETBIOS driver to buffer the input indefinitely. Until another electric zap flipped the bit again and caused all the buffered data to unspool. Intel, who built the PCs we were installing, flew out one of our engineers to show them the problem in person. Whew.

This is an embarrassing story to recall, but here it is.

In my past-life I was a hardware engineer for Unisys working on the design of their A-Series mainframes. These were big iron machines – think very large refrigerator size – that cost and sold for lots of money (often millions per). No, that’s not the embarrassing part ;)

We had a new mainframe that was soon to be released to the market. It was the most powerful (and expensive) one we had ever built, and customers were eager to get it. This was a major product for the company, and a lot hinged on this release. We had a very tight schedule to get our testing complete and stress/tension were running high.

Testing was occurring across all three shifts, but there was one elusive bug that threatened to jeopardize the release. It was so bizarre, and no one could figure it out. Here’s what happened:

At random times, the system would “go down” – which in mainframe terms meant the OS would crash and everything would freeze – a power-cycle being the only way out of the crash. The time between crashes could be as short as a few minutes or as long as many hours – there was no pattern.

We tested every theory imaginable from rouge apps with memory leaks to charting solar flares thinking perhaps some stray gamma rays were thwarting our ECC logic. But no matter where we looked, we couldn't locate the source. This went on for days without a resolution and quickly escalated to senior management given the potential huge financial impact to the company.

Now comes the embarrassing part.

I was working 2nd or 3rd shift – I don’t remember which, but it was late at night. There were three of us on debug at the time. I was walking past the machine when someone shouted out, “It just went down!” We all rushed back to the operator terminal to see what was running right before the failure. As usual, nothing specific.

And that’s when it hit me.

I told the other two to restart the machine … I had an idea.

In those days (early 90s) I had very long hair, carried an over-sized comb in my back pocket, as was the epitome of a dorky nerd trying to be cool. It didn't help that I could do a spot-on renditions of Ice Ice Baby and Funky Cold Medina.

Anyway, after the mainframe was rebooted, I walked back to the machine, combed my hair, and then touched the frame of the computer with my finger. Sure enough, the instant I touched the machine, the other two yelled out, “It crashed!”

Long story short, we had a grounding issue with the computer, and spurious static electricity was randomly causing the machine to crash. Once a grounding plane was installed the problem was solved, machine shipped, customers were happy, and we made a ton of money.

And while I’d love to take credit for amazing powers of deductive reasoning, the truth of the matter is that dumb luck and my ridiculous hair led to the source of one of the company’s most elusive bugs.

if (bitmask & 0xF0 == 0xF0) ...

This cost me a week of chasing on a bare machine (no printf, no debugger, no ICE) in college.

C++ Operator Precedence specifies that the == operator is evaluated before the & operator, so the code is evaluated as:
if (bitmask & (0xF0 == 0xF0)) ...

Due to the lack of strong typing for Booleans in C/C++, that is equivalent to:
if ((bitmask & 1) != 0) ...

The intended meaning was:
if ((bitmask & 0xF0) == 0xF0) ...

This is why I tell my students to always use parentheses rather than relying on operator precedence. Parentheses are free, and (in addition to preventing bugs like this one) communicate your intended meaning to anyone reading the code.

(Edit: BTW this is an example of why strong typing is valuable. C# won't even let that compile.)

My story is about 40 years old, mid 1970s. We had made our first microcomputer product - 8080 based - and it worked, mostly. Except that every few hours it would make an error. Not a large one, just a skip in a count, but since what it was counting was money, it mattered.

There were three of us in the team, and we worked all hours for two weeks trying to track it down. Finally Deadline Monday was approaching and it still wasn't fixed, so Thursday afternoon we set up a program trace and as every instruction was executed, the register set was printed out on a teletype, which was the only printer we had. Every. Single. Step. Clatter-clatter-clatter. This of course reduced the program execution speed to about one instruction every three seconds, but it was our only hope of catching the error.

We took turns watching that thing all weekend, hour after hour, roll after roll of paper. We slept in shifts, lived on pizza and beer. The air was thick with cigarette smoke and the lab smelled like a locker room. Finally, sometime on Sunday, it faulted. There were three lines in an interrupt routine - fetch variable X from RAM, decrement it, write it back. Three instructions. This one time, it didn't decrement.

We looked at the stack, to find out what was happening when the interrupt occurred. We looked at the next instruction after the return - it was, save a register to variable X. The interrupt was working just fine, but its result was immediately overwritten by the routine it interrupted. If the interrupt arrived a microsecond earlier or later it didn't cause an error, which is what made it so hard to catch. It was a valuable lesson for me - I've never made the same mistake since. Variables altered by an interrupt routine must be read-only outside it. (In this case, we put a DI/EI around the external routine to prevent it being interrupted before it finished modifying the variable. You do what must be done when you ship tomorrow.)

Don't judge us too harshly for the obvious error. At that time an 8080 cost a week's wages and none of us had more than a year of programming experience, nor any place to turn for advice except the device data sheets. Everything we knew was self-taught from first principles. Life is too simple for you young whippersnappers of today, with your hard drives and compilers and Java, whatever the hell that is. Assembler and paper tape and debugging with a logic probe, that'll make a man of you. Now you kids, get off my lawn.

Two weeks. It took me two weeks to find this one. A very intermittent bug to find, but crashed the app and usually it seemed to "know" when things were critical and crashed then! So an important one to find.

It was an "executive information system", what today would be called a "management information system" I think. It was a very early example of a client server system. When the system was especially busy (like month or quarter end times) and the user was asking for a particularly large data set (like full sales or production report) the app would – sometimes – crash. But! Not right away. If the crash was going to happen, it would happen 30 seconds to up to ten minutes later. And it only happened for some people, not everyone. Crazy.

This was intermittent enough that for a while we weren't sure if it was actually a real bug. But I was assigned to investigate and fix if possible...

The server would, depending on load, chunk data up into larger messages when the server system was busier. In this system sending data was expensive to the server, so this made perfect sense.

The client would write the data into a large buffer and parse it out. So at these times the buffer would be close to or at max level.

This was some time ago, and low-level debugging was in assembler and hex memory dumps. So the tools were painful and not as slick as they are now.

Eventually I found through a process of careful elimination and systematic experiment that the data buffer had an off-by-one error. We thought we could write up to 4096 bytes, but only 4095 were allocated. (Numbers not accurate, this was a long time ago.)

The value right after that buffer in memory was a boolean used in the UI of the app. (Local var, but statically allocated, so not on the stack.) When the buffer got full and overwrote that boolean byte, then maybe it would change boolean value. This upset the UI with an inconsistent value and ended up (several stack frames later) crashing the app.

But, the UI only noticed this when the user used a certain path (clicks and button presses) through the system. An alternate path would set the bool and correct the value. Some people liked the first approach, others use the alternate. So the problem happened only to some people.

So it only happened when the server sent a full data block, and this tended to happen only during "important" times when a number of users were on the system accessing larger amounts of data.

Intermittent bugs are bad. Memory overwrite bugs are bad. Combining them is worse!

In my college days, I was working on the final year project, we actually wanted to convert the data into excel file dynamically via programming(i.e click the button to convert the data into excel format ). So we started the project to convert data into excel format file. In 2 months of hard work, we made our file conversion project. We started using the project on a daily basis. It works well. Mostly all conversions work well. But some of the conversion files had some issues. Actually, the issue is, data to excel file conversion works perfect, but the file is not opening in Microsoft Excel.

when we try to open the file, we have a popup like this,

Source: Google Images

This Issue is reproduced in the strange converted file. we have no idea for this Issue. Even we don’t know, how to reproduce this issue and fix it. we had to spend at least more than a week to reproduce the issue.

Finally, we reproduce the Issue. Actually, we convert the file into one or more than sheets of single excel file. first, we confirmed that there was no issue with a single sheet excel file. we tested all the cases, it works fine. after, we started with more than two sheets excel file to open in excel. The issue reproduces in a strange case(i.e some files opens, some files not getting open). we started googling about this issue. but still, we were not getting any glue for this issue. after all, we had an idea, actually many developers tried and developed conversion into excel file and many developers may face the issue, so we searched the issue description in GitHub. And finally, we have the case.

Actually in Microsoft excel sheets did not allow the same sheet name to more than one sheets. it will show the error when we save the file in excel. when we generate via the program, we have no restriction to use the same sheet name to multiple sheets. after we found the root cause we solved it.

“Hard” is obviously a moving target. The more experience you have, the easier the fixes get. Of course, more experienced programmers tend to end up working on harder bugs to solve, so it can even out. I think the hardest-for-me bug I had to deal with was one of the first difficult bugs I encountered in my career. Especially because not actually just a code bug. It was a corporate relationship bug.

And it is a long story. Follow me, gentle reader, on this journey back in time.

There was once Company X, that did customized user interfaces for database software sold by Company Y. Company Y also did customized user interfaces for its database software. You see the problem. There were reasons why Company Y needed to support Company X. Nevertheless, the relationship was… tense.

Company X landed a big contract to develop user interfaces for Company Z. There was much rejoicing.

There was a catch. The big contract required Company X to provide a user interface for the Macintosh as well as for PC. And up until that point, Company X had been exclusively a Windows developer. So, Company X was now committed to deliver a Macintosh user interface, based on absolutely no Macintosh experience.

But there was a catch to this catch. The vast majority of Company Z’s employees used Windows. They had gone with Company Y’s database because it would work the best for their Windows users, and with Company X because of its Windows user interface’s capabilities. Macintosh support was a nice-to-have, a checkbox in the corner of the design requirements. So, the contract was very specific about what the Windows user interface should be. But the contract terms covering the Macintosh user interface were almost an afterthought. It had to exist, and it had to meet delivery milestones, and it had to be usable.

And that was where I, the intrepid young programmer, entered the picture. I was hired by Company X as the Macintosh Division. No, that’s not a typo. I would be the Macintosh designer, the Macintosh coder, the Macintosh documentation writer, and the Macintosh field support staff.

Anyway, Company X brought me on, and gave me enough of a budget to purchase one Macintosh and one set of development software. They showed me how their Windows software worked — namely, in ways that were so complicated and so platform-specific that porting them to Macintosh was out of the question. They handed me the floppy disk with Company Y’s new Macintosh library, and a list of the delivery milestones for the Company Z contract. And with that I was launched on my journey.

I knew that I had been hired largely because the stakes were low. Which also meant that I had little to lose. I leaned into it. I didn’t even try to replicate the Windows UX. Instead, I made something very small, but still very customizable. You could reconfigure my little Macintosh UX in five minutes, just by editing resource files. Not just the appearance of the UX, but also the database fields that would be displayed, and the ways that the user could change them. That wasn’t that unusual in Macintosh software at the time, but it was unheard of on the Windows side of things, where they added custom code changes and built separate versions of the applications for each customer. Everything was coming together. I was feeling good. I could actually do this!

Except.

When I tested my code against an actual Company Y database, sometimes it would crash. In all sorts of different places, and in all sorts of entertaining ways. Not always, but often enough to make my code unusable. And the simple test program supplied by Company Y worked fine, so it wasn’t the database or the library. It was something I was doing. Which was odd, because I could see that I was passing the right parameters to things. What was up?

Debugging tools at the time were much cruder than they are now, especially for intermittent bugs. I simplified my code again and again, trying to pin down the problem. Eventually I had a program that wasn’t much longer than Company Y’s test program. It still crashed.

Now I was in a crisis. I couldn’t see what I was doing wrong. It was Mac-specific code, so there was no one at Company X that I could ask for help. This was before the real rise of the Internet, so there were no forums to turn to. I absolutely knew that saying “I think there’s something wrong with this third-party library” was a non-starter for a new programmer. Company Y’s willing to ‘help’ us with the Macintosh was very clear: “Does our test program work? Yes? Then the problem’s in your code. Go away and read the documentation again.”

With no real alternatives left, I started debugging my way into Company Y’s library, in the hope of just learning exactly what part of my code was triggering the crash in theirs. I had one advantage: the folks at Company Y hadn’t built a proper retail build of their library. The library had been built using some settings that would normally be used for debugging. So even though I didn’t have source code for the library, I could still use those debugging symbols to get hints of what was going on. The names of functions, the way that things were arranged in the original source files.

And I finally found the problem. When the library changed the system state in a certain way, it needed to increase the size of a block of memory. And it did that using the C language’s realloc function. And the programmer missed a crucial point: realloc can fail. And when realloc failed, the program would merrily assume that the block was as large as it requested. It would write new data into that ‘blank space’ that in fact was still very much occupied by something else. And then, maybe much later on, some other part of the program would try to access or modify that ‘something else’… and… boom.

The only reason that Company Y’s test program worked is that it was small and did things in precisely the same order every time, and it just so happened that this could never trigger the crash.

I suspect that Company Y’s Macintosh Division was similar to mine.

In any event, I hesitantly approached my Company X managers, and showed my work. And shortly afterwards, there was a meeting between Important People at Company X and Company Y.

And the Important Person from Company X said, we are experiencing crashes with your code, this is unacceptable, aren’t you testing this stuff?

And the Important Person from Company Y said, it’s your code, it’s your problem, don’t come to us and blame us for your issues.

And I tried very hard to sound confident, and I said, it’s your code, and specifically, it’s your call to realloc in your line 234 in your function DoSomethingOrOther in your YourSourceFile.c.

And the Important Person from Company X smiled, and there was a long pause, and I remember ABSOLUTELY NOTHING about the rest of the meeting.

But shortly thereafter, I was handed another floppy disk with another version of Company Y’s Macintosh library. And debugging symbols were still turned on, and there was a new function in YourSourceFile.cpp, and it had some sort of irritated-sounding name, and it fixed the problem. And I shipped my work off to Company Z, and they accepted it, and the “Macintosh support” clause in the contract was dutifully checked off.

The managers at Company X deemed my little Macintosh user interface to be worthy of being offered to other customers. The Macintosh Division remained open for business.

This would eventually lead to me receiving a safety briefing at a federal nuclear facility. “This alarm means you should head to the nearest shelter. This alarm means you should shelter in place. This alarm means you should make the most of your last few minutes.” And this would lead to me reconsidering my life choices.

But that’s another story.

Three. All having to do with rare, nondeterministic events. This means you can't find them easily with a debugger.

1970: I worked on Data General Nova minicomputer serial number #3. [The Nova was essentially the first RISC machine, with a stunningly odd set of arithmetic instructions, including ADCZL# 2,3,SBN. You don't have to know what this means to realize this is an odd machine]. We coded an assembly language device driver... that mostly worked. Occasionally it would fail. (We're doing debugging with front panel switches and a really bad debugger, remember that #3?). It turned that an indexed branch with a negative offset would sometimes go to the wrong place... how? I chased this for days before I got down to an instruction sequence "interruptdisable", "jmp -index[reg]", "interruptenable" where the problem occurred with relatively high frequency (but never when you single stepped it). We decided that disabling interrupts set a flip-flop near the ALU (you know you are desperate when debugging code when you decide to look at the circuit diagram of gates that make up the CPU) and the extra current demand would make the ALU math slightly flaky. We sent the CPU back to Data General, they told us we guessed right, they fixed it and sent it back. Voila, problem solved. Nice to spend weeks to find a design error in somebody else's hardware.

Moral: don't depend on a flaky circuit design.

1974. On a one-of-a-kind 16 bit VM minicomputer I and another fellow (Dennis Brown, hello!) designed had a fancy-shmancy register-chip in it to hold the CPU's registers. I designed an assembler and linker for it; the linker would print a symbol table of names and corresponding addresses on a teletype at the end of the link edit step. Sometimes... the symbol name would print out as complete garbage, but the address was fine and other symbols might or might not be fine. Ultimately we found the culprit: the register-chip bits would turn from zero-to-one sometimes when it got really hot; it would only get really hot when the program was doing heavy duty math in the registers; and the linker was taking a radix-50 encoded symbol and tearing it apart by doing a repeated divide-by-50 (tight loop: compare, shift, subtract, repeat). Problem went away when we blew a lot of air of the register chip. (Freeze spray? in 73?) Cure: complain to the register-chip vendor, get a replacement chip.

Moral: don't depend on a flaky chip.

2012-2014: Using MS Windows, with working thread-switching code on Win32. The thread switching code used MS Win32 API commands to SuspendThread, GetThreadContext, SetThreadContext, ResumeThread. This code was written around 1999 (yes!) and had been stable for 15 years(!). On Windows Vista ... sometimes (once every few million times!) the app doing the thread switching would crash. Try and find such a problem; it occurs abominably rarely and the symptom is "die horribly". This almost made me tear my hair out. I tried huge numbers of experiments, adding consistency checks in an astonishingly large number of places in the code, to little avail. Eventually discovered it was Wow64 emulation of these formerly rock-solid calls.... GetThreadContext *lies* about the thread context, that is, it is supposed to return what is in the registers of a suspended thread, but sometimes returns trash. This is incredibly hard to detect: what do you look at, to see that it is wrong? I didn't debug this so much as recognize, via desperate web searches over 1-2 years, that another person had encountered the same problem; see WOW64 bug: GetThreadContext() may return stale contents. Why didn't I notice on XP-64? Because there it *works*, at least I have run it literally billions of iterations without ever encountering a problem). This is an unforgivable sin for an OS call: a system call for managing thread context that is simply unreliable. To this day, MS has not fixed it; they say Windows 8.1 will tell you when GetThreadContext just lied to you, which is hardly a good cure. [I have no evidence yet that Windows 8.1 tries to tell me this reliably; I have evidence that Windows 10 claims to tell me this]. Cure: my thread switching code now sets a flag when it makes an OS call (in the hundreds of places that it does so, sigh); if the flag is set, I simply don't use SuspendThread/GetThreadContext; at least this solution I trust.

Moral: don't depend on a flaky software vendor. (But hardware you can get fixed).

I wrote a GUI and database that was used by the operations people to make mission critical decisions from. It ran in a failure tolerant environment in a 24*7 operates and control room. This was in the late 80’s or early 90’s and I was using Vax mini computers, mini is a relative term, these were about the size of a large washing machine. There were two processors, labeled a and b

Every so often it crashed. Perhaps once every few weeks. Unacceptable, so I went in to find out what was going on. I wrote some diagnostic software to trap the problems. I isolated it to a specific module, then to a specific routine, then reduced it down to about 5 lines of code. It still ran for weeks without crashing, then it crashed all the time.

eventually I twigged, it only crashed on one machine. In those days, the cpu wasn’t a single chip like it is now. I turned out that one board on one processor was faulty, and it only affected one instruction. The one I used in my algorithm. Since there was an online/standby arrangement, the fault hid behind the online machine and only sometimes failed.

Sometimes your code is perfect, but the hardware is faulty.

A couple of my friends in college, were from the electronics background, and had joined the computer department.

They were new to programming, so they asked me to help with some PL/SQL script that just wouldn't work. The script looked fine. I wrote the same on my computer, and it worked just fine.

They let me bust my ba**s on it for a while, before they told me they had copied it from a book. Dragging my cursor across every character, I stumbled upon the bug, the most epic bug I have ever come across.

They had used two single quotes (apostrophes) in place of an every double quote.

That was a new low in the history of computing.