Common Sense 4 Common People

Error Handling Crimes

Hell, I am furious! I come to work on a bloody Saturday, to review some code so that we can release our new product on time and I stumble twice on one of the most basic coding crimes when it comes to error handling, each done by a different developer.

I know it's natural to make errors, that's why I do the code review in the first place right? Absolutely! The reason that I am furious is that one of the two instances of this crime is long ago commited to our product (an SDK) and thus I cannot correct it... because... I will break backwards compatibility. Damn it!

The other one is not released yet, so it's just a human error; we'll just have to update about 70 calls to some internal function and we are done.

So what exactly is this crime I am talking about?

Simply put, it is a CAPITAL OFFENSE to convert an error code to a boolean Success_or_Failure flag.
A variation of this crime is to convert a rich error code into something like E_FAIL (in the COM world).

In real life, most programs cannot do much in the face of errors, other than (a) not crash (b) let the user know what went wrong.

In real life, when something goes wrong the user will immediately contact technical support. So imagine being the tech personnel, having a frustrated and often angry user telling you "Hey, I tried to do such and such and it fails with the error E_FAIL (Unspecified Error). What the hell is going on?".

Exactly my point! What the hell is going on???? How is our tech support team supposed to provide decent technical support if we don't know exactly what error occurred?

Most developers I have encountered in my career so far don't have the proper understanding and respect for error handling code. They want to write the REAL code not the error handling crap (or heavens forbid... the documentation).
They don't understand or just don't want to realize that in solid systems the error handling code can be up to 50% of the code. Yeap, that's FIFTY percent. And if you don't write it properly then you are NOT doing 50% of your job. Think a little bit about it.

Moreover, in most cases you don't write the error handling code for your end user, you write it FOR YOUR OWN SHAKE, to make YOUR LIFE easier when something goes wrong to the user. Because as sure as death and taxes are, things going wrong to the user is a certainty.

OK, now that I've had my fair share of whining I am not furious any more... Back to business :-)

Have fun!

Demystifying Device Driver Development (DDDD)

It just occurred to me recently that most people don't really have a clue about what it is that Device Driver Developers (DDDs) do for a living. Since I am a DDD myself, among other things, I thought I should make an attempt of describing what I do in a way that Common People can hopefully digest.

Since I enjoy making wannabe hilarious Tom & Jerry dialogs, I will try to put it in a dialog between Jerry the Technical Interviewer and myself. I will intentionally wear the hat of the anti-social, psycho-path, under-the-hood programmer and let Jerry extract all the information from me in a criminal-interrogation like style.

Here we go...

Jerry: So you are a DDD.
Me: Yeap.
Jerry: Can you tell me what it is that you do?
Me: Yeap.
Jerry: I mean you build device drivers right?
Me: Yeap.
Jerry: What exactly is a device driver?
Me: It's just a piece of software that controls a device. Don't you know that? Are you wasting my time?
Jerry: Yes, I guess I do, ehhh I mean I do know, but what exactly is a device?
Me: That's an interesting question so I will dignify it with an answer. A device is a clever piece of hardware. To make things simple, imagine that a device has a little CPU of its own, a CPU that can execute various tasks depending on the hardware at hand.
Jerry: Wow64, there is a CPU on these things? A CPU on a PCI card that I buy off the shelf?
Me: Duh, but of course, how else do you think they would do any useful work?
Jerry: I thought the computer's CPU was the only CPU in the block.
Me: No, devices have CPUs and depending on the hardware they can be pretty sophisticated.
Jerry: That's awesome! What can you tell me about these little CPUs?
Me: For starters, they are not little. Most of them can do several tasks at the same time and they can be pretty darn fast.
Jerry: Hmmm, wherever there is a CPU, there is a program right?
Me: Yeap.
Jerry: Well, where are the programs that these CPUs execute? Firmware stored in flash memory may be?
Me: You'd wish.
Jerry: Please elaborate, I'm getting really curious here.
Me: OK, if you insist... These programs are generated by the device driver.
Jerry: Pardon me?
Me: The programs that the device CPUs execute are generated by the device drivers. Clearer now?
Jerry: Wow64!
Me: Indeed.
Jerry: Generated by the device driver... In what language?
Me: That was a good one! Machine language of course.
Jerry: You mean mov ecx,edx and the like?
Me: Well, in a sense yes. Each device CPU has its own machine language.
Jerry: You are kidding me right?
Me: No way. Don't be an idiot.
Jerry: So the device driver builds machine language programs for the device CPU?
Me: Actually, it's even worse than that. These CPUs usually have many sub-CPUs, execution contexts we call them. The device driver generates programs for several (if not all) execution contexts. Each program is called a context program.
Jerry: Wow64 once again! And where are these context programs located at runtime?
Me: Well, in most cases in main memory, the little thingy you call RAM. Sometimes devices may have onboard memory for storing these programs, but in most cases it's just RAM.
Jerry: In RAM? Wait wait... And how can the device CPU read from RAM? Can it handle virtual addresses and the like?
Me: Man, you are really something! Virtual addresses? What are you talking about? When was the last time you heard the term "Physical Address"? These are the kind of things these CPUs can swallow.
Jerry: Physical Address? What are YOU talking about? Are we talking about DOS here?
Me: LOL. Please realize NOW that devices are what's called "Close To The Metal". They don't and they won't understand virtual crap of any sort. They operate at the PCI level, so they will only understand PCI-level physical addresses.
Jerry: Then how does it all work?
Me: Well the device driver allocates some piece(s) of memory where it intends to store the context programs and then kindly asks the operating system to give it the physical addresses of these pieces of memory. The OS happily obliges and then the device driver prepares the context programs, using physical addresses wherever a jump is required. When the context programs are ready, the device driver sets the Program Counter register on the various sub-CPUs of the device and then starts these CPUs. Ain't it cute?
Jerry: Wowowowowow64!!! That's super cool.
Me: Yeap.
Jerry: If I understand well, the device driver is like a mini operating system for a device, right?
Me: You couldn't phrase it better.
Jerry: Now I realize why being a DDD is so tough! You have to write a mini OS of your own and make the device do your bidding!
Me: Yeah, but the truth is making the context programs and handling their operation is the easy part.
Jerry: Huh? And what is the difficult part?
Me: ERROR HANDLING.
Jerry: Please please, please elaborate!
Me: Well, it's no big deal for an experienced DDD to write a device driver for a well behaved device. The tough thing is to write a driver that does not malfunction or crash the OS in the presence of device errors. These do not happen often so it is really hard to test the error handling code. Actually if you write ANY decent program, user mode or kernel, error handling largely amounts to 50% of the code. If it does not amount to 50% in your code, then you haven't done enough error handling. No developer likes error handling (with the exception of myself of course). It's boring, it's nasty, it's creepy. But unless you do it and you TEST it throughly you can't write decent software. For device drivers this is even more crucial, since in the presence of device errors the best thing that you can hope for is that the device stops functioning (we call it the device is dead). However in the majority of cases (at least in Windows) you get an OS crash.
Jerry: And how do YOU test your error handling code?
Me: That's a trade secret.
Jerry: Com'on now...
Me: I keep an arsenal of malfunctioning devices at my office. Everyone in my team has strict instructions to immediately hand me any device that appears to be malfunctioning. They are my secret treasure.
Jerry: But how can you be sure that your driver handles correctly all possible errors?
Me: I am not.
Jerry: You're NOT???
Me: Well, why do you think it is that people still get BSODs when something unusual happens? Even if you get a driver developed by a team that takes error handling very seriously, they can't test everything, simply because they haven't seen everything yet. So you get BSODs.
Jerry: I am shocked. This is brutal.
Me: Well, as I've said elsewhere BSODs are a blessing. As far as your complaints are concerned, please pass them on to the device manufacturers. If they would put some "test mode" into their devices and make them fail randomly then MY job would be MUCH more easier. But these things would have to go to the chip level and then they would have a significant added cost as you realize. So practically everyone in the industry operates in a "hoping for the best" mode.
Jerry: ...
Me: You can't imagine what I have to go through to make buggy devices operate correctly. In your mind the term "bug" is most probably related to software, but let me break some news to you: There are COUNTLESS bugs in hardware chips and devices, and poor DDDs like me have to struggle hard to ship a decent driver only to have lusers scream their lungs out at ME when they get a BSOD. They scream because probably I didn't handle 100% correctly all the bugs in the hardware chips. But who am I supposed to scream at? Texas Instruments? Intel? You get the picture.
Jerry: Well, I can only say that under this new light I have a whole new appreciation for the work you DDDs are doing.
Me: Darn right you should.
Jerry: My special thanks for this enlightening interview.
Me: May the force be with you.
Jerry: Any last minute advice for Common People who use PCs?
Me: Don't add more RAM to your PC unless you know what you are doing.
Jerry: Excuse me? Don't add more RAM? What harm could additional RAM possibly cause???
Me: I will describe that in a separate post, when I feel like having mercy on the masses. For the time being I know, and you don't. Huh!

Have fun!
Dimitris Staikos

1394 DMA Multiplexing sneak preview

Today I got a new, highly demanding piece of code working in our drivers. What it does is to receive streaming video from multiple 1394 cameras using a single DMA context on the 1394 adapter. Up till now each camera would have its dedicated DMA context. Since there are only four of those on most 1394 adapters, you could only receive 4 camera video streams per adapter. If you needed more you had to install a second 1394 adapter, etc.

DMA Multiplexing was a feature that we have been talking about for years in Unibrain. It would always get pushed back in the schedule since it was considered a "complex" item. Finally we did it. Of course there is more dirty work to get done, but the tricky part is over, and the important result is that the multiplexing DMA context can really handle some serious load without flinching.

I know some people like sneak technology previews so here is a photo of one of my test PCs displaying 6 cameras using a single DMA context:

If you zoom on the actual 5MP picture (1.5MB download), low on the right side you can see the 1394 bus topology, consisting of two PCs (nodes 0 and 9), six cameras, 1 repeater (node 1) and 1 analyzer (node 3). The monitor is on the blue node (node 0).
CPU usage (37%) is pretty decent for a slightly dated single CPU system.

Of course there is no way you can verify, just by looking at the photo, that all six video streams are on a single DMA context. You'll have to trust me on that :-)

Have fun!

Undefined Behavior (NMI quiz - Part 2)

It has been some time since my last Tom & Jerry dialog.

Jerry: Hey Tom, could you help me out here? I am kind of confused...
Tom: Sure thing Jerry! I'd be glad.
Jerry: Do you happen to know what "Undefined Behavior" is?
Tom: Well, isn't it when the results of an operation are unpredictable?
Jerry: I know, but isn't that a definition for this kind of behavior?
Tom: Uhmmm, I guess so...
Jerry: So, you have just defined "undefined behavior".
Tom: Uhmmm, I guess so...
Jerry: So, what's undefined about it, since you just defined it?
Tom: I hate it when you do that! Leave me alone!

Undefined Behavior is a funny term, a misnomer if I am allowed to say.
If you ask me, there is NO such thing as "Undefined Behavior"
Even random behavior is not undefined. It is defined as random.

Instead, there is behavior undefined BY someone.

If you read about Undefined Behavior on WikiPedia you will notice the following in the first paragraph:

... the specification leaves the results of certain operations specifically undefined.

It is the specification that does not define the result of the operations, so implementations are free to do what they feel like, plus tell no one about it (if you are required to document it then it is implementation-defined).

What does all this have to do with NMIs? Well, by now it should be evident.

The NMI I got bitten by last month was caused by "Undefined Behavior", as described someplace inside the OHCI 1.1 specification.

In my case, up to that point in time, the undefined behavior our code was stepping on was not undefined at all. It was perfectly stable and correct behavior. But one day, given the proper hardware combination, it decided to flip and do an NMI instead. Cool, but still perfectly defined :-)

So to answer the quiz, yes, it was once again the software's fault :-)

Have fun!

NMI quiz - Part 1

NMI errors are a form of fatal error that users don't encounter as often as the infamous Blue Screen of Death (BSOD). Here's what an NMI error looks like:

Not as much information as in a BSOD. The user is given no real information about what happened. The most often seen NMI is the Memory Parity Error.

Now, here is a common sense quiz for common computer users:

• You have a piece of driver software that works on all IEEE-1394 OHCI adapters (FireWire), on thousands of different machines, on all versions of Windows (Win2K to Vista).
• HP produces a new server machine. If you use a specific 1394 adapter on this specific machine then you get an NMI error when the drivers load. All other 1394 adapters work without problems on this HP machine.

Who do you think is to blame?

1. The HP hardware. All adapters work on all other machines, so there is something wrong with this machine.
2. The 1394 adapter. There must be a bug in the adapter hardware.
3. The driver software. Obviously there is some bug in the device driver.

Here's a hint to what I think.

Have fun!

Debug Assertions & Breakpoints

If one or more words in the title mean nothing to you don't worry; just ignore it, it's pretty technical stuff for most common people.
I uploaded an article on my current approach to debug assertions and breakpoints both for kernel and user mode programming, together with a piece of source code for you to experiment.