Experimenting With Scrum

I recently tried using some parts of scrum with my team at Microsoft.  We're a development team in test and tend to have a lot of small, independent projects rather than one larger integrated one.  To make matters worse, we work with audio and video and there is a lot of specialized knowledge with each project.  It is non-trivial to move one person to another task.  As such, it is hard to implement scrum as described in the canon.  There is no clear feature backlog and there is no obvious way to define the results of a sprint for the group.  I always wanted to try scrum but couldn't come up with a good way to do it.  Over the past month or two, I tried to approaches.  I'll go into more detail about them in future posts but here are the highlights.

We went through two phases of work recently.  One was fixing bugs and the other was working on new applications.  Each has a different work flow.  Fixing bugs is more fungible, short, and discrete.  Working on new applications is more specialized, the work items longer, and it is less obviously discrete.  For each, I had the team meet once a day for 15 minutes to discuss their status.  It worked better in some situations than in others.  When the work items were longer, the meetings often seemed redundant from day to day.  When the work items were short, it felt more natural to meet daily.

Part 2

Part 3

Part 4

Dual Core Media Center Goodness

I recently put together a new Media Center PC.  The old one was running on a Pentium4 1.7 GHz and was becoming too slow for my tastes.  It ran okay but was becoming slow to bring up the guide, etc.  The day following Thanksgiving, I was able to score an AMD Athlon 64 x2 3800+ for a good price and decided it was time to put together the new machine.  Here is what I ended up with:

AMD Athlon 64 x2 3800+

1 GB RAM

NVidia GeForce 6600GT

NForce4 motherboard (ECS A939) - it was free with the processor.

Two analog tuners (one Emuzed and the other Conexant)

Silverstone LC14 case

This isn't the absolute fastest machine on the block but it has some getup and go.  I'd heard rumors that Media Center glitched on the dual-core processors but following the advice of David Fleishman, I installed new HAL drivers and haven't seen any issues.  The machine is definitely snappy.  I don't have it hooked up to an HD TV yet but it should be capable of doing WMV-HD 1080p decode with hardware acceleration through DXVA-WMV.

The case is quite cool.  It's a little tall but it has a nice black color and fits well with other media components.  Unlike some of its competition, it has space for more than 1 internal hard drive.  In fact, it has space for 3 of them.  It also has space for 2 CD/DVD drives.  The drive covers are better than most for HTPC cases.  They are flip-down doors, not stick-on cover plates.  The only downside is that there is no way to have a card reader in the case.  I suppose that is what the other machines on the network are for.

While I'm on the subject, I should mention that I'm using a Home Theater Master remote control.  This brings up an interesting issue.  Media Center doesn't work well with universal remotes out of the box because it has some technology to stop signal double-taps.  The behavior you'll get is that each button will work only once and then you have to hit another button before it will work again.  The solution is to follow the advice of Michael Swanson who outlines a registry setting that will diable this feature and let your universal remote work.

Language Inefficiencies

   I spent some time yesterday trying to learn Perl.  I'd looked at it some a few years back but never had a use for it.  I now have a need to write a tool for our build environment and so, based on what is available, I am required to use Perl.  The first thing I noticed about Perl is that it is very powerful.  The second thing I noticed is that it is really ugly.  There are too many ways to accomplish the same result.  Many new languages suffer from this same fate, only to a lesser extent.  What I want to talk about is keeping a language small and compact.  If there is a way to accomplish a task, please don't invent another one with slightly different syntax just to save a few keystrokes.  C++ is pretty good about this.  Other than the multiple ways to cast, there is not much redundancy in the language.  The namespace is fairly unencumbered by keywords.  Modern languages like C#, Perl, Ruby, and Python, however, seem to burn namespace like it is going out of style.  They invent new keywords and operators that add nothing to the language.  One of my favorite examples comes from C#.  The operator 'as' strikes me as wholly unnecessary.  The following code accomplishes the same result:

CFoo cf = myObj as CFoo;

- and -

CFoo cf;

if (myObj is CFoo) {

   cf = (CFoo) myObj;

}

In both cases, we are checking if myObj is of type CFoo and if so, setting the variable cf to point to it.  Why the need for as?  What does it add to the language? 

Perl is much, much worse.  A glaring example is the 'unless' operator.  Instead of typing if(!foo), you can type unless(foo).  Much better, right?  No.  There are many other redundancies.  && and 'and' do the same thing.  I can choose whether or not to use parentheses around my subroutine calls.  You can put if before or after the code you want executed.  The list could go on almost indefinitely.  The worst part about Perl is that the advocates are proud of all of this redundancy.  Even newer, more compact languages such as Ruby have redundant operators.

Here is my rule for adding a new operator or keyword to a language:  Does this operator give me the ability to do something I couldn't do before?  If yes, consider adding it.  If no, reject it. 

I'm not here to bash Perl or any of the other languages.  They all have their place and are powerful.  They also all have a large following.  However, it seems like they could be even better if they were a bit more careful.  Making something already possible merely syntactically easier has the effect of making the language more complex.  While making something a bit simpler to express, you have made the language harder to learn and to retain in memory.  If only language authors would think a second time before making an addition to their language.  Having hundreds of keywords and multiple ways to do everything makes the language harder, not easier to use.

Blog Meltdown?

   I've noticed a trend recently.  Or, at least I think I have.  I have certainly noticed a few datapoints that look like a trend.  Blog software is failing to scale.  For all the penetration of blogs and the millions of people hosting them, they are starting to fail.  Here are a few examples:  Scoble's comments started failing.  Wil Wheaton found his blog FUBARdN.  Belmont Club got too big for blogspot.  I'm sure there are lots of other examples but those are the few that I've noticed in the past few months.  In each case, the author of the blog had to move to a new location and start again.  This is a painful process for everyone.  Users need to remember the new url, RSS readers need to be redirected, bookmarks updated, and search engines need to realize that the old site isn't the best answer.  As I write this Scoble's new site is #2 on Google, #8 on MSN Search, and #13 on Yahoo.

   Why are these sites failing?  I think there are two contributing factors.  First, the concept of a blog means that it is often run by a normal person.  That means a person without an IT staff to back them up.  Once a database starts corrupting itself, you have a lot of work if you want to fix things.  This might explain WWdN but not Scoble or Belmont Club which both ran on large blogging sites that should, in theory, have had an IT staff behind it.  Perhaps not an IT staff dedicated to that site, but at least one committed to the code in general.  This comes to my second contention:  much of the blog code is poorly written.  I suspect that much of this code was hacked together quickly and wasn't really built for or tested to high stability.  It takes a different sort of code to merely get something working than it does to get something to scale really big.  I think in many ways the web is just starting to learn this.  Many successful websites are still overgrown proofs of concept.  They haven't had the major shaking-out period that something of this scale normally has.

   Are these three sites just anomalies or are they signs of things to come?  I suspect that there are a lot more failing sites waiting in the wings. 

You Aren't Gonna Need It - Or Are You?

A coworker recently pointed me at this page from the C2 Extreme Programming Wiki.  The basic theory is that, if you find code that isn't being used, delete it.  That is fair enough.  The controversy comes when the code in question is potentially useful in the future.  What then do you do?  The XP advocates on C2 say you should delete that as well.  I don't agree.  If the code in question is potentially useful in the future, I suggest commenting it out (most likely using #if 0).  The arguments against leaving the code in generally revolve around confusing someone.  They might run into the code and try to understand it only to figure out later that it isn't called.  They might be searching the code and  run across it.  There is also an argument that the code will affect compilation times.  If you comment out the code, none of these apply.  The response is often why not just delete the code and if someone really needs it, they can go back in the source control system and retrieve it.  In theory, they can.  In practice, they can't.  I haven't run across a source control system that allows for efficient temporal searching.  Without that, the only way to know that the code exists is to remember that it does.  Not only that, but you also have to remember when it existed.  Somehow, I think that is unlikely to happen very often.  This leads us then to Steve's rules for retaining dead code.  Retain dead code iff:

1)  It does something that will potentially be useful in this area of the code later.  An example might be a slow c-based algorithm for something that is now optimized in assembly.  Someday you may have to port to a new system and you'll want that reference code.

2)  It is not causing you problems.  If this code starts causing problems, get rid of it.

3)  The code works.  If you know it is broken, get rid of it.

This also leads to a feature request for authors of code editors:  Please color code the code between #if 0 and #endif blocks as a comment.

Objective Statements - Do You Need One?

   I've been doing a lot of resume screening lately so I thought I'd write up a few of my thoughts.  Ever since my early days as a hiring manager I've told people not to include an objective statement on their resumes.  More often than not, it is worthless.  Usually it equates to "I want a job."  Well, yes.  I could infer that from the fact that you submitted the resume.  Lately I've had some change in my thinking.  I still find most objective statements useless but a well-crafted one can be useful in the right circumstances.

   What are the right circumstances?  The only time I can envision an objective statement being useful is when you are submitting your resume into a large pool.  If you are submitting directly to the hiring manager, it won't tell him/her anything the presence of your resume does not.  Don't waste the space.  If, on the other hand, you are submitting to a large pool such as Dice or Monster or a large company like Microsoft where many hiring managers will be looking at your resume for disparate positions.

   What should be in the objective statement?  Something like, "To obtain a challenging position in the field of software the utilizes my skills" isn't too helpful.  Don't try to sell yourself via your objective statement.  It isn't a mini cover letter.  It should be a clear statement about what sort of job you are looking for.  "A full time position as a senior software developer" is probably all you need.  If there are more specifics--say you are interested in video--make that clear here too.  "To obtain a position as a program manager in the field of digital video." is probably good.  I think there are two main things you are trying to convey here: 

1)  What class of job are you looking for?  Management?  Development?  Testing?  Program Management?  Architect?

2)  Any specific areas you want to focus on.  Know that if you put down specific areas, it will probably cause you to be overlooked for other jobs.  Saying you want to work on telecommunications means you probably won't be tapped for a job working on video editing.  Make sure you are okay with that.  Generally, I would suggest only doing this if you are really passionate or really experienced in a particular area.

MSN Search Getting Better

   As a good corporate citizen I began trying MSN Search several months ago with the initial release.  At first it was pretty painful.  I would search on MSN, not find what I wanted, then switch over to Google.  Recently, however, I have found that I almost never resort to using Google.  MSN Search is finding what I'm looking for.  In fact, I have noticed several instances of MSN Search being better than Google.  First, MSN seems to have more current results than Google.  Several times lately I have searched for something current and found results on MSN when Google still has none.  A recent example was my last post.  This showed up on MSN just hours after I posted it.  Google took until the next day.  Second, MSN is sometimes giving the more relevant links first.  Let's go back to my last post.  If you search for "Some Perspective On Vista" on Google, my blog will be the 4th link.  The first 3 are pages that link to my blog.  On MSN Search, my blog entry is first, followed by all of the pages that quote it.  In my mind at least, the original source should always be ranked above something quoting it.

Update 10/19/05 - Sigh.  Google is indexing this article now but MSN Search is not.  For what it's worth, Yahoo doesn't have it yet either. 

Some Perspective On Vista

   It's been quite a while since Microsoft released a new operating system.  Windows XP was the last consumer OS to ship and that was 4 years ago.  By the time Vista ships, it will have been 5 years.  There are a lot of consequences of such a long release cycle but the one I want to speak about is perspective.  Rather, the lack of perspective.  Most people today watching Vista come along have no perspective about what a new OS is like before it ships.  When XP was being built, the world was less connected.  There were no blogs.  There were far fewer web sites dedicated to rumors and leaks.  The consequence of this is that most people didn't see what the OS really looked like until it launched.  Those that did tended to read about it in magazines with long lead times and carefully scripted presentations.  Today, the public gets to see a much more raw version of the development process.   Another consequence is that even here inside Microsoft, many fewer people have shipped an OS.  Why does this matter?  I have seen a lot of people reacting to early releases of Vista in a negative light.  It is too slow or it doesn't look different enough or it is buggy.  There are also a lot of positive reactions but those are not the aim of my essay. 

   People who react negatively don't realize that this is always what an OS looks like before it ships.  Think of building a new OS a lot like cleaning a messy room.  The first thing that always happens (at least when I clean a room) is that it gets even messier.  I start organizing things by taking them out of their places and spreading them out.  Before I know it, every surface including the bed and the floors has piles of stuff on it.  If I stopped at this point, the room would be worse than it started.  After sorting everything into piles, I can then put it away neatly and more organized than before.  The result is a much cleaner room.  An OS is similar.  The first thing you do is start removing old, working parts and replacing them with new parts.  These new parts start off less powerful and less stable than the parts they are replacing and only over time become more powerful.  A good example of this is in the world of audio.  The team I work on helped to produce the new audio stack in Windows.  For beta 1 the stack was present but didn't do much the old stack didn't do.  If we did our jobs right, all that work would never be noticed.  If we screwed up, things that worked before wouldn't work now.  For Beta 2 we are able to add all sorts of cool new features on top like per-app audio. 

   The point here is the early releases of the operating system (or any application going through a major overhaul) is going to look worse before it looks better.  Take heart those of you watching the process unfold.  Vista will be an exciting OS with lots of cool new features.  I've seen some of the recent Beta 2 builds and it is looking much more impressive.  I can't say anything about it but if you watch places like Channel 9, you'll probably see some of it soon.  You can also see some of it in the PDC demos which were given about a month ago.

  

Programming for the next guy

   I'm taking a class this week.  The subject is writing WDM drivers.  The class assumes you have some understanding of how operating systems work and teaches you everything you need to know to be prepared to write a WDM driver.  It isn't a cookbook class.  I won't come out and write a driver without needing the DDK but I will have enough understanding to be able to read the DDK and fill in the connections between all of the parts.  I find that when trying to learn something, rather than being told exactly what to do, I learn best by getting a high-level view first and then diving into the details.  This class is taught in that way and I highly recommend it.  The class is offered by OSR.  It is being taught by Peter Viscarola (author of Windows NT Device Driver Development).  If you are new to driver development, take this class.

   The class isn't the main point of this post, however.  Rather, it is some advice that Peter gave a few times during the class.  Someone might ask a question like "Can't I do x in some funky way?" and he would answer, "You could, but no one would expect to see it so don't."  The point he was making is that we, as programmers, should stay away from being clever.  We should, as much as possible, try to do things the same way everyone else does them.  Why?  Because you won't be the only person to work on this code.  Even if you are, the next time you touch it might be a year or two from now.  If you did something clever, the next person to touch it will look at the code and not immidiately understand.  This will have one of two consequences.  Either they will have to spend 10 minutes just trying to understand what it is you did or, worse, they will assume you made a mistake and "fix" it by making it less clever.  Neither of these results is desireable.  Unless you are writing one-off code for yourself* you need to write it in a manner to make it easily understandable so that it can be easily maintained.  This is the intent of the smell test talked about by the authors of Refactoring or of the self-documenting code the XP advocates talk about.

   To be more explicit, whenever possible, follow conventions.  If you aren't the first one to be doing something, don't reinvent the wheel.  If you aren't sufficiently familiar with your domain to know the conventions, it would behoove you to spend some time becoming familiar with how things are done in that space.  Doing so will save lots of time and effort later.

 

* Useful one-off code often ends up not being one-off code.  Always code like you'll be maintaining it.  Someone probably will.

I'm back. Hopefully.

Wow.  It's been a while since I blogged.  I have a lot of ideas but I've been quite busy of late and haven't gotten to them.  I was out of town for 5 weekends in a row from late July through August.  Needless to say, that doesn't leave a lot of time for other stuff.  Hopefully I'll now find, no make, the time to do that. 

OS/2 Finally buried

IBM finally announced the official, final death notice for OS/2.  Back before I joined Microsoft I was one of those who was attempting to jump on the OS/2 Warp bandwagon.  When I was in college I would read about it every week in PCWeek (now eWeek) and was impressed with its capabilities.  I even went so far as to purchase a copy.  $100 was a lot when Win3.1 came free on my Pentium 75.  Even after Win95 came out, I kept trying to run OS/2.  The problem I ran into with OS/2 (which is the same problem I ran into with Linux years later) was that, while the OS had lots of cool features and was, on paper at least, better than Win95, there just wasn't much software for it that could compete with the software for Windows.  Having a shiny new OS with nothing to run on it is like buying a new car but not having a drivers license.  It looks really cool but just isn't that much fun to use.  Now, a decade or so later, it's finally being officially put out to pasture. 

Next Gen Game Console News

Some interesting news (rumors are more like it) about the next generation of game consoles:

The PS3 is rumored to cost $400 street and nearly $500 to make.  Meanwhile, the XBox360 is rumored to cost about $300.  If true, that's great news for Microsoft.  Trying to sell the PS3 at $100 more will be an uphill battle and could take the edge of some of the Sony hype.

Meanwhile, AnandTech has some rumors about the upcoming Cell and XBox360 processors which supposedly come from those who have seen the hardware.  While I'm on the subject, here's an older post from someone on the XBox team talking about the 360 vs the PS3.

For the record, I haven't seen the XBox360 yet and have no inside information.

WMV-HD Is Now Hardware Accelerated

I am please to announce something I find pretty exciting.  Windows Media High Definition Video (WMV HD) is now able to take advantage of hardware acceleration.  Cards from leading vendors such as ATI and NVidia can now assist in the decoding of WMV-HD files.  Depending on the CPU and video card, you can expect to see a 20-40% reduction in overall CPU utilization.  This is often enough to reduce glitching or even allow for playback not otherwise possible.  The acceleration works on WMV-HD streams that are 720p and 1080p.  This includes all of the content (I am aware of ) on the WMV HD DVDs plus most other WMV HD content you are likely to find around.  To take advantage of it, you'll need supporting hardware, updated drivers, the QFE from the link below plus the required DRM QFE (also on the link below).  If you play any WMV-HD and have a sufficiently beefy display card, I suggest you download the update and give it a spin.

Related links:

Microsoft KB Article with the download link.

ATI's Press Release announcing support.

List of supporting NVidia cards.

The Register announcing the release.

AVSForum thread with details and support.

Update:

ATI's Catalyst 5.6 Drivers claim support for WMV9 Acceleration.

 

Broadcast Flag's Terrible, Horrible, No Good, Very Bad Day

Back in February I talked about how the D.C. Circuit Court seemed to question the power of the FCC to implement the broadcast flag.  The court seemed upset with the flag but there was a bit question whether they would find standing (the right to bring a lawsuit) for the those bringing the suit.  The broadcast flag is a piece of administrative legislation which imposes draconian rules on what can and cannot be done with television broadcasts marked with a specific flag.  If upheld, it would have required many digital tuners be removed from the market this July when it went into effect.  Today, the Court ruled, "The FCC has no authority to regulate consumer electronic devices that can be used for receipt of wire or radio communication when those devices are not engaged in the process of radio or wire transmission."   The full ruling can be found here.

In my opinion, this is a great win for consumers.  Forcing extra complication into the video pipeline benefits no one.  Allowing people to back up their television shows or transfer them to another machine is definitely an advantage.  Even better, this potentially allows the use of digital video in devices like portable media center and the like.  If we are to make the transition to digital television in the future, consumers will need to be able to take their shows with them.  I can't imagine consumers easily giving up capabilities for a higher quality signal. 

 

Note:  The opinions expressed in this post are my own.  They do not necessarily reflect the opinion of Microsoft Corporation.

Black Box Testing

   I attended a talk on campus yesterday discussing various aspects of testing.  Part of the talk discussed the need for testers to become better versed in the formalities of testing.  I'll leave that subject for another day.  A portion of the talk, however, discussed an experiment done with some inexperienced testers.  They were asked to create test cases for the Myers Triangle Test.  A lot of the test cases they came up with were not useful.  By that I mean they didn't test the algorithm or they were redundant with other tests.  Some would try inputting something like an 'A' which is invalid and won't pass the string->int conversion function or they would try lots of different numbers that all went down the same code path.  If you look at the underlying code, it is obvious why these tests don't make sense.  Too often though, test plans are full of cases like these.  Why is that?

   I contend that we often test things only at the surface level and don't consider the implementation.  At some point in time I was told that black box testing was a good idea because if you looked at the underlying code, you might make the same flawed assumptions that the developer made.  This is probably also where we got the notion that you shouldn't test your own code.  I never really agreed with the concept of purposeful black box testing but didn't fully challenge the assumption in my mind.  After some reflection though, I am pretty sure that black box testing is almost always less useful than white box testing. 

   Just in case you don't follow, let me define some terms.  Black box testing is testing where you don't understand the implementation details of the item you are testing.  It is a black box.  You put in data, you get out different data, how it tranforms the data is unknown.  White box testing is testing where you have the source code available (and look at it).  You can see that there are 3 distinct comparisons going on in the Meyers Triangle Test. 

   Black box testing can be useful if we don't have the time or the ability to understand what we are testing but if we do, it is always better to take advantage of it.  Without knowing the details, I have to try every potential input to a program to verify that all of the outputs are correct.  If I know the inputs, however, I can just test each code path.  If all triangles are tested for A == B == C, I don't need to Triangle(3,3,3) and Triangle (4,4,4) and Triangle(5,5,5).  After the first one, I'm not trying anything new.  Without looking at the code, however, I don't know that.  Not only does white box testing allow you to see where you don't need to test, it lets you see where you do.  Some years ago I was testing our DirectShow DVD Navigator software.  There is a function for fast forward that takes a floating point number.  From a black box perspective, one would have no idea what numbers to pass.  Just try some and call it good.  In this particular implementation, however, there were different behaviors depending on which number you put in.  For a certain range of numbers, all frames were decoded and just played quickly.  For a higher range, only I-frames were played.  For everything above that range, the navigator started playing only some of the I-frames.  Without looking at the code, I could not have known which test cases were interesting.  I couldn't guarantee that I tried something from every range.

   What about making wrong assumptions if you look at the code?  Won't that cause you to miss things?  Perhaps.  However, test driven development, unit testing, etc. have proven that testing done by the developer is quite effective.  Testers should also have a spec outlining what proper behavior should be.  If the code deviates from that spec, you found a bug (somewhere--it might in be the spec).  If you use common sense, you are unlikely to miss a bug because you make the same assumption as the developer.  If you do, the trade-off for greater test efficiency is probably worth it.  You'll have found many new bugs for each one you miss.

Programming Languages

Interesting article this morning on CNet.  It discusses language usage.  Bjarne Stroustrup (the creator of C++) claims that C++ is not being overtaken by languages like C# and Java.  He claims it is a matter of marketing.  Java has marketing, C# has marketing, C++ does not.  He's probably right about that.  There is no hype machine for C++.  Sometimes though, hype influences reality.  If I'm an aspiring programmer, I'll hear all this cool stuff about .Net, Java, or C#.  I will then be more likely to gravitate toward them.  Surveys show that, as a percentage of all programming, C++ has dropped from 76% in 1998 to 46% in 2004.  That may say more about the size of the programming pool though than about the number of C++ programmers.  Stroustrup maintains that there are 3 million C++ programmers. 

One aspect the article touches on is teaching languages.  When I went to school, pascal was the language of teaching.  In a lot of places, it is currently Java.  Why do universities instist on "teaching" languages?  Why not teach what people really use?  You can't argue that C++ is too complicated.  It is more complicated than C# or Java, sure.  But then again, a lot of your graduates are going to be programming in C++.  By the above survey, about half of them will.  If they can't handle it in school, how are they going to handle it in the real world?  I am all for taking some classes to expose people to different programming paradigms (C++, Lisp, Smalltalk, C#/Java, Python, C, etc.) but the core of the curriculum should be based around one language.  That should be a language that is useful everywhere.  It makes no sense teaching an OS class using Java.  No one but Sun uses it for OS's and we know how successful JavaOS was.

My final thought on the subject is that it shouldn't suprise anyone that Java, VB, and C# and even "languages" like PHP or Perl are gaining so much momentum.  They are more accessible than C/C++.  They are easier to learn and easier to use.  It's harder to shoot yourself in the foot with VB than with C++.  It's also harder to do many kinds of work.  Why not start students with the more complex languages and work down from there.  To learn C# from C++ is easy.  To learn VB after you know C++ is easy.  The other direction, not so much.

So, if you happen to be reading this and just starting out, consider learning C++ early.  You'll be doing yourself a favor.

It's not the idea

I had a chance this afternoon to see one of my favorite writers and thinks, George Gilder.  He came to Microsoft to speak.  Anyway, he said something very interesting.  He stated than patents are not all that valuable because they are open.  Usually having the idea is not worth much until someone can reduce it to practice.  An example he used was that of the microprocessor.  This was something many people had the idea of doing.  It wasn't terribly useful to have that idea until someone figured out to actually manufacture such a beast.  Once that happened, the idea became valuable.  This knowledge about how to do something is what he calls a "latent."  This is an interesting idea and it has, I think, two implications:

1. Companies which are obsessed by patents may be going down the wrong path.  The next Microsoft or Intel won't come from having the right patents but rather having the right latents.
   
2. The USPTO should look very critically at "idea" patents.  If something is an actual mechanism of creating something (a latent which is being made open), a patent may be warranted.  If, on the other hand, this is an idea before its time, it should be rejected.  The only purpose the latter can have is to stop someone who has an idea about *how* to do it from doing it.  Imagine someone patenting the idea of the microprocessor.  It would have made it impossible for companies like Intel to have done what they did.

George Gilder works at the Discovery Institute.  His newest book is called the Silicon Eye.

Unix vs Windows

   Over the past few months I have had the opportunity to take an Operating Systems class at a leading University.  During that time, I have been once again confronted with the whole Unix (*nix, Linux, Mac OSX) versus Windows argument.  It became quite apparent to me during that time that the professor thought that Unix was better than Windows.  It also became apparent to me that he didn't really understand Windows.  Take, for example, our discussion of filesystems.  When referrring to Unix, he talked about inodes.  When referring to Windows, he talked about FAT32.  When talking about memory management, he wasn't sure if Windows still used segments.  When discussing UI programming, he showed some code with the most basic Windows application (create the window, run the message pump, handle windows messages).  The problem was, he saw the message pump and didn't know what the code was doing.  If you don't understand that, you really don't have a right to be criticizing Windows.  The flip side is also true.  If you don't understand Unix, you shouldn't be criticizing it either.

   I have come to the conclusion that most "Windows is better" or "Unix is better" arguments at the wholistic level come down to familiarity.  I have had discussions with a friend who is a BSD guy.  Most of his criticisms are outdated or just plain wrong.  I suspect the same is true to for my criticisms of Unix.  For a given task, one may be better than the other.  Taken as a whole though, both can obviously get the task done if you are familiar with them.  The difficulty is that each is hard to understand so mastering both is really hard.  Also, neither is standing still.  People still argue that Windows crashes a lot (it did in Windows95 but not in WindowsXP) or that Linux can run on a 486 (it can, as long as you don't use a modern UI).  These arguments are outdated.  Often the criticism "Windows can't do X" or "Unix is hard to make do Y" boil down to "I don't know how to do X on Windows" or the corollary on Unix.  It is an argument of familiarity rather than actual ease of use.

   When I started the class, I hadn't done much Unix work for a long time.  It was painful trying to compile, debug, etc.  I still think that things like Windbg are more advanced than DDD/GDB but it is less painful now than it was.  Likewise, I've come to understand some of the Unix file structure and command line tools like vim and grep which makes getting around less difficult.  As I become familiar, it is harder to think "Unix Sucks."  I can still point to many places where it does, or where it feels like it does, but I have to be careful that these aren't stemming from merely my ignorance.  I would challenge any Linux/BSD/Mac OSX/Unix/Solaris afficianados out there to do likewise.

Western Digital Understands Warranties

   I have a hard drive which seems to be going bad on me.  It's a Western Digital SATA drive.  So far, no data loss but the drive has a tendency to disappear after my computer has been on for a long time.  It also tends to start making a clicking noise during the BIOS RAID screen during a warm reboot.  Needless to say, a clicking hard drive that just stops underneath the OS is not something I want vital data on for long.  The drive is still under warranty so I head to the Western Digital web site to figure out how I need to handle the warranty.  Right off the support page there is a link to check on the warranty status of a drive.  Just type in the serial number and it will tell you when the warranty expires.  Slick.  I type that in and find out I have about 9 months left.  Good.  Now, to get an RMA to return this.  Often this is like pulling teeth.  Call someone, send an e-mail to customer service and just hope they pay attention, post to a web forum and wait for a response, these are all options I have had to do in the past.  Not for Western Digital.  Right on the check warranty results page is a box to type in the reason for the return.  They don't seem to really care because they only give you 30 characters to type.  After that, you pick a few options about how you want your return handled and you are done.  Shortly you are notified with an automated e-mail to your account with the RMA number and tracking information.  They'll even ship the new drive before you return the old one.  How cool is that?  I've dealt with my share of warranties over the years but this one is by far the easiest and best to date.  They don't try to hold a profit margin up by making warranty service hard.  The drive only costs $70 these days so their entire profit margin would be eaten up in just one phone call anyway.  Instead, they just let you return it.  No hassle.  Kudos to Western Digital for truly making the customer king.

Interested in being a test developer? We're hiring.

If what I have said about test development excites you and you want to try it, now is your chance.  We are hiring.  We're looking for an experienced developer to work with audio drivers and related technologies.  If you want to get your hands dirty in the lower levels of the OS, come join us.  Details can be found here.

Sony's Cell Microprocessor

Things have been too busy for me to write any more articles lately.  To tide you over, here is something I found interesting:

The Cell Processor is going to be the heart and soul of the next Playstation.  It is a joint effort between Sony, Toshiba, and IBM.  It is supposed to run on everything from the PS3 to workstations.  More interesting than what it will be used in, however, is what it is all about.  What does the Cell processor look like?  What are its advantages and disadvantages compared to traditional CISC or RISC processors?  AnandTech takes a careful look at the Cell.  If you have any interest in processors, give this article a read. 

Update:  Here is another article on the Cell processor, this time by Ars Technica.

Quality Assurance != Testing

Stuart Feldman of IBM research has some thoughts on the issue of testing and quality assurance.  He contends that QA is a process whereas testing tends to be technology-based and an afterthought.  He describes the difference here: 

"What goes into QA? Testing, of course, is a key activity. There is, however, an adage that “you can’t test quality into a product.” A solid test plan should catch errors and give a measure of quality. A good QA plan ensures that the design is appropriate, the implementation is careful, and the product meets all requirements before release. An excellent QA plan in an advanced organization includes analysis of defects and continuous improvement. "

I think what we call what testers do is largely semantic but agree with Stuart on what this activity should be.  If testing is merely about finding bugs, it is insufficient.  It needs to be about assessing the quality of the product as a whole. 

Stuart also spends some time detailing the different levels of testing required for various levels of products.  This ties in well with the software engineering discussion we're having elsewhere on this blog.

hat tip to /.

Why building software isn’t like building bridges

I was having a conversation with a friend the other night and we came across the age-old “software should be like building buildings” argument.  It goes something like this:  Software should be more like other forms of engineering like bridges or buildings.  Those, it is argued, are more mature engineering practices.  If software engineering were more like them, programs would be more stable and projects would come in more on time.  This analogy is flawed.

Before I begin, I must state that I’ve never engineered buildings or bridges before.  I’m sure I’ll make some statements that are incorrect.  Feel free to tell me so in the comments section.

First, making software, at least systems software, is nothing like making buildings.  Engineering a bridge does not involve reinventing the wheel each time.  While there may be some new usage of old principles, there isn’t a lot of research involved.  The problem space is well understood and the solutions are usually already known.  On the other hand, software engineering, by its very nature, is new every time.  If I want two bridges, I need to engineer and build two bridges.  If I want two copies of Windows XP, I only engineer and build it once.  I can then make infinite perfect copies.  Because of this software engineering is more R&D than traditional engineering.  Research is expected to have false starts, to fail and backtrack.  Research cannot be put on a strict time-line.  We cannot know for certain that we’ll find the cure for cancer by March 18, 2005.

Second, the fault tolerances for buildings are higher than for software.  More often than not, placing one rivet or one brick a fraction off won’t cause the building to collapse.  On the other hand, a buffer overflow of even a single byte could allow for a system to be exploited.  Buildings are not built flawlessly.  Not even small ones.  I have a friend who has a large brick fireplace inside their room rather than outside the house because the builders were wrong when they built it.  In large buildings, there are often lots of small things wrong.  Wall panels don’t line up perfectly and are patched over, walls are not square to each other, etc.  These are acceptable problems.  Software is expected to be perfect.  In software, small errors are magnified.  It only takes one null pointer to crash a program or a small memory leak to bring a system to its knees.  In building skyscrapers, small errors are painted over.

Third, software engineering is incredibly complex—even compared to building bridges and skyscrapers.  The Linux kernel alone has 5.7 million lines of code.  Windows 98 had 18 million lines of code.  Windows XP reportedly has 40 million lines of code.  By contrast, the Chrysler building has 391,881 rivets and 3.8 million bricks.

Finally, it is a myth that bridge and building engineering projects come in on time. One has to look no further than Boston's [thanks Mike] Big Dig project to see that.  Software development often takes longer and costs more than expected.  This is not a desirable situation and we, as software engineers, should do what we can to improve our track record.  The point is that we are not unique in this failing.

It is incorrect to compare software development to bridge building.  Bridge building is not as perfect as software engineers like to think it is and software development is not as simple as we might want it to be.  This isn’t to excuse the failings of software projects.  We can and must explore new approaches like unit tests, code reviews, threat models, and scrum (to name a few).  It is to say that we shouldn’t ever expect predictability from what is essentially an R&D process.  Software development is always doing that which has not been done before.  As such, it probably will never reliably be delivered on time, on budget, and defect free.  We must improve where we can but hold the bar at a realistic level so we know when we've succeeded.

Recommended Books On Testing

Another question I was asked via e-mail.  "Do you know of any good books that explain the testing process in detail? I noticed you mentioned Debugging Applications for .Net and Windows, but I am looking for a book that really explains the basics of 'I have this API/application and I want to test it'. "

Let me say up front that I’m not a big fan of testing books.  Most of what they say is obvious and they are often long-winded.  You can learn to test more quickly by doing than by reading.  Unlike programming, the barrier to entry is low and exploration is the best teacher.  That said, Cem Kaner’s Testing Computer Software gets a lot of good marks.  I’m not a huge fan of it (perahps I'll blog on my disagreements later) but if you want to speak the language of the testing world, this is probably the book to read.  Managing the Testing Process by Rex Black gives a good overview of the testing process from a management perspective.  Most testing books are very process-intensive.  They teach process over technique.  How to Break Software by James Whittaker is more practical.  I have read some of the book and have heard James Whittaker speak.  As the title indicates, the intent of the book is to explain where software is most vulnerable and the techniques it takes to break it at those points.

Part of the difficulty with testing books is that there are so many kinds of testing.  Testing a web service is fundamentally different than testing a GUI application like Microsoft Word which is again wholly different than testing a multimedia API like DirectShow.  Approaches to testing differ also.  Some people have a lot of manual tests.  Some automate everything with testing tools like SilkRunner or Visual Test.  Others write code by hand to accomplish their testing.  The latter is what my team does.  Most books on testing will either distill this down to the basics--at which time you have no real meat left--or they will teach you a little about everything and not much about anything.  Read the 3 books I call out above but make sure to adapt everything they say to your specific situation.  Treat them as food for though, not instruction manuals.

Do you have a favorite testing book that I haven't mentioned?  Share your knowledge with others in the comments.

FCC Broadcast Flag has a bad day

In 2003 the FCC ruled that digital television signals could carry a flag which, when set, required specific protections be in place in all hardware/software that handled the signal.  This flag has become known as the broadcast flag.  Recently, several members of the D.C. appeals court heard a lawsuit seeking to block the ruling.  At least two of the judges had unkind words for the FCC.  The flag is currently scheduled to go into effect this July.

News.com:

"You're out there in the whole world, regulating. Are washing machines next?" asked Judge Harry Edwards. Quipped Judge David Sentelle: "You can't regulate washing machines. You can't rule the world."

Reuters:

You crossed the line," Judge Harry Edwards told a lawyer for the Federal Communications Commission during arguments before a three-judge panel of the U.S. Court of Appeals for the D.C. Circuit.

"Selling televisions is not what the FCC is in the business of," Edwards said, siding with critics who charge the rule dictates how computers and other devices should work.

What is a Test Architect?

I was asked a few questions via mail.  Here is the first of some quick answers to these:

   What is the role of a Test Architect?  There is not a single definition of the test architect role.  A test architect is an advanced test developer whose scope is larger and who solves harder problems than the average SDE/T.  The specifics of what they do varies greatly.  They might be solving hard, particular problems.  They might be the one called in to solve the seemingly intractable issue.  They may be designing a new test platform.  Or, they may be determining group test policy.  Any and all of these fall in the scope of the test architect.  The work they do is often similar to that of an SDE/T.  The difference is often one of scope.  An SDE/T will often own a specific technology or be responsible for implementing a part of a system.  A test architect will own the approach to testing or developing entire systems.

   Are you a test architect or do you have a different idea of what one is?  Please leave your opinions in the comments section.  I'd love to see a dialog develop on this subject.  It's something I'm interested in but that isn't all that well understood yet in most quarters.

TDD - First Impressions

      I spent the last two days in a class covering test driven development, unit testing, and refactoring.  I hope to provide a more detailed discussion of what I learned at some later point but for now I thought I'd post my initial impressions.   I've read a lot about TDD, Unit Testing, Refactoring, etc. but I'd never actually *done* test driven development.  The class had several hands-on exercises.  We were using C# and NUnit.  Let me say right here that NUnit is a slick app.  It is unobtrusive which is exactly what you want in a test harness.  This is the report of that experience.

      First off, it felt really strange.  Generally when developing a program, you think about the big picture first and work your way to the details.  With TDD, you end up taking the opposite tack.  Because you have to write a test, fail the test, then write the code, you cannot start with the big parts, instead you start with the small internals.  Of course it is necessary to put some thought into the bigger picture, but very quickly you are driven into the implementation details and that creates a feedback loop for your big picture design.  This feels strange but you become accustomed to it.  The designs that come out seem clean.  Forcing testability causes you to think about cohesion, coupling, redundancy, etc.

      When you are doing test driven development, you are constantly switching between the editor, compiler, and test harness.  You are compiling often.  This is a good thing.  It means that your code is always working.  You don't go on long coding sprees followed by long bug fixing sprees.  Instead, you intermingle the two a lot.  I find it easier to fix an issue as I'm writing it rather than later when I compile some larger chunk of code.

      TDD feels good.  Every time you run the unit tests, you get positive feedback that things are working.  When you make a change, even a radical change, you know that if all the test cases pass that everything is working.  It gives you peace of mind that you cannot get in older coding models.  If you've never done any test driven development, give it a try.  Who knows, you might like it.

      The class was taught by Net Objectives.  This is the 3rd class I've taken from them.  I've also attended a few of their free seminars.  If you have an interest in OO or Agile techniques, check them out.  I highly recommend their work.

Information Technology Leaders

There is a local cable television station run by the University of Washington.  On it there airs a show called Information Technology Leaders hosted by a woman named Laura Shildkraut.  On the show, Laura spends an hour interviewing, you guessed it, leaders in the information technology field.  The guests are CEOs, CIOs, VPs, COOs, CTOs, etc.  Topics covered include not only what they are doing now but what they did in the past and how they got to where they are.  Laura gets some interesting guests.  For instance, she interviews all three Renegades of the Empire (Alex St. John, Eric Engstrom, and Craig Eisler), John Connors, Rick Devenuti, etc.  I find the stories fascinating.  There is also a lot to learn from people who have "made it."  If you have some time, I recommend you check out the streaming video version of the show at http://www.informationtechnologyleaders.com/video.html.

Too Much Test Automation?

            There was a time when testing software was synonymous with manual testing.  Now with the rise of test development and the advent of unit testing, automation is becoming more and more prevalent.  Test automation has many benefits but it is not a silver bullet.  It has its costs and is not always the right answer.  Software development organizations must be careful not to let the pendulum swing too far in the direction of automation.  Automating everything is a way to guarantee you miss many bugs.  It is incumbent upon those in charge of test organizations to find a balance between automated and manual testing.

            It is best to start out with a definition.  Test automation is code or script which executes tests without human intervention.  Simple automation will merely exercise the various features of the product.  More advanced automation will verify that the right actions took place.  Results of these tests are stored in log files or databases where they can be rolled up into reports listing a pass/fail rate.

            Automation has many advantages.  Manual testing is expensive.  It requires people to click buttons and observe results.  This isn’t terribly expensive the first time through but the costs stay fixed as the product progresses.  If you have a daily build (as you should), you pay the cost daily.  This quickly adds up.  Automation is expensive to create but the incremental cost is very cheap.  Automation is also consistent.  No one will ever forget to run a test case with automation.

            Because it is cheap, automation can be run in places where manual testing cannot.  Extensive tests can be run on daily builds.  These can even be run in the wee hours of the morning before everyone shows up for the day.  The inexpensive nature of automated testing is what allows unit tests and test driven development to be possible.

            Some test cases cannot easily be done manually.  When testing an API such as the DirectShow API I often work with, there is no UI to drive it.  At least some minimal coding must be done to expose the API to a user.  In these cases, automation is the obvious choice.

            Despite its advantages, automation is not a panacea.  First, it is not free.  Automation is expensive to create.  If you are able to amortize that cost over a lot of runs, the incidental cost becomes low.  On the other hand, if the test is something that will only be run a few times, automation may be more expensive than manual testing.  Decision makers must consider the high initial cost before committing their organization to automated tests.

            Secondly, automation is limited in scope.  After you have run your automated tests for the first time, you are done finding new bugs.  Never again will you find a new issue.  You might catch a regression, but if you missed the fact that clicking a particular combination of buttons causes the app to crash, you’ll never find that issue.  This is true no matter how many times you run the automation.  On the other hand, high quality manual testers will take the opportunity to explore corner cases.  In doing this, they will find many issues that would otherwise go unnoticed until real users find them after the product is shipped.  This lack of exploratory ability is, in my mind, the Achilles heel of automated testing. 

            The third drawback of automation is that there are things that simply cannot be automated well.  I work in the field of audio-video playback.  Trying to automate video quality tests is hard.  It is very expensive—specific media must be pared with specific test cases.  If you have to account for variable content (say, testing television tuners) or if the algorithm is not fixed (varying video cards or decoders), the task becomes even harder.  Using something like PSNR is possible but is not well tuned to the human visual system and thus is susceptible to false positives and false negatives.  Sometimes there is no replacing a person staring at the screen or listening to the speakers.

            Other problems also exist with automated tests.  Bugs in an automated test may mask a bug in the product and go unnoticed for long periods of time.  Automated testing does not emulate real users using a product nearly as well as a real person.  Manual tests can be brought online more quickly than automated tests allowing bugs to be found sooner.

            What then, is a test manager to do?  Manual testing is too expensive but automated testing is imperfect.  It is imperative that the two sorts of testing are mixed.  Automated testing should be used to verify basic quality but cannot be used exclusively.  There need to be real people simulating the experience of real users.  With the hiring of developers and not scripters as test developers, the quality of automated tests can be made very high.  Unit testing can ensure that automated tests are available early in the cycle.  With the increased sophistication of test developers and automation techniques, automated testing can take a larger role than in the past.  It cannot, however, replace real testers.   

Language Choices

It has been asked several times which language is better for a test developer at Microsoft, C# or C++?  The answer really depends on what group you will be working in.  Different groups require different skills.  It is my perception that C++ is by far the dominant language today but that will not always be the case.  I have friends that are SDE/Ts who work exclusively in C#.  I'm sure there are some on the VB.Net team who do all of their work in VB.  There are probably some on the Office team who use VBA for most of their work.

For now most things are being written in C++ because of the extra runtime speed and the availability of libraries and APIs in the language.  In theory C# could become as fast as or faster than C++.  It has the ability to dynamically compile and thus take advantage of processor and system features not available to the least-common-denominator approach of a traditional compiler.  Most likely, in the real world, it will never quite catch up.  The C# team is working hard to minimize the difference between the two.  See Rico Mariani’s blog for some posts on the subject.

This isn't to say that C# won't become the dominant language though.  Hand-tuned assembly is often faster than C code and C code can be faster than C++.  Those advantages didn't stop C++ from becoming the dominant language, however.  Why is that?  It is because C++ is faster to code in.  It offers native support for object-oriented coding and abstract data types.  These increase the efficiency of the programmer.  This increase in efficiency is accepted at the price of code that runs a bit slower.  C# offers a similar bargain.  Coding efficiency is up but performance is down.  As processors continue to increase in speed, the need for that extra ounce of speed will be less and less advantageous and the increased coding efficiency relatively a greater and greater advantage.  When the APIs for coding Windows applications become managed with Avalon and WinFX, the usefulness of C# will be magnified.

Over the long term, I expect that C# will become a dominant language.  Will it fully displace C++?  That is hard to say.  Certainly not in the near term.

I must end this by saying that I am no expert on C#.  I've done some coding in it but nothing too serious.  There are others on blogs.msdn.com that know a lot more about the subject.

Disclaimer - Please Read

These postings are provided "AS IS" with no warranties, and confers no rights.  This site contains my personal opinions and does not necessarily reflect the viewpoint of my employer, Microsoft Corporation.  The thoughts and opinions expressed herein are solely my own.

Three Reasons To Consider Being a Test Developer

            When it comes to careers in the world of software most people think of programmers or what are more formally known as developers.  Developers are the people who write the software which is consequently sold or utilized by the organization.  I’ll call them dev-developers to distinguish them from test developers.  Sometimes people will also think of testers.  Testers are the people who don’t program but run the software written by developers to find bugs.  What gets lost in the shuffle is a specialized class of developers who are also testers.  We like to call them test-developers.  These are the people that write software to test software.  It is they who will be the subject of this essay. 

            The term “test developer” is sometimes used to refer to a tester who knows a scripting language like Javascript or Perl or maybe even knows VB.  Usually this person has no formal training and takes on only simple tasks.  That is not what I refer to in this essay.  The test developers I am referring to are a specialization of developers.  They write complex code in production languages utilizing computer science techniques.  See my previous essay on why test developers are real developers. 

            So, why should you consider becoming a test developer?  Why not just become a dev-dev instead of a test-dev?  That is the subject of this essay.  There are three primary reasons to become a test-dev rather than a dev-dev.  These are that it makes you a better programmer, the code you write is more broad, and it is at a sweet spot in the development cycle.

            Programmers who are or have been test developers are, on average, better programmers than those who have not.  They have a feel for what is likely to go wrong with software and so code for failure instead of coding for success.  All too often those who have not been testers write code until it works and then stop.  They write so that it can work but not so it will always work.  Test-developers have practice breaking programs and so know where they will encounter problems.  They are thus more successful at anticipating what error conditions may happen and writing resilient code.

            Secondly, test developers tend to write code which exercises the product at a higher level.  Instead of focusing all of your effort on a way to make a widget spin, instead you get to see what happens if multiple widgets spin at the same time or how spinning widgets interact with the IFoo class.  Test developers write code that exercises the product as a whole which is often more interesting and more rewarding than spending time optimizing one little corner case.  You get a more wholistic view of the product which leads to better understanding of how the various pieces interact.  Additionally, you often get to do more fun work like determining ways to break the system, put various pieces together, analyze its performance, etc.

            Finally, test development is at a sweet spot in the product release cycle.  Dev-developers work furiously in the early cycle getting their pieces code complete.  They then work furiously late in the cycle fixing the final few bugs.  The end result is often very long hours.  Test developers, on the other hand, are generally under less pressure.  In the early product cycle, you can only work as fast as code becomes available to you.  In the late cycle, your tests are already in place.  If they aren’t, it is too late to add them.  I don’t mean to imply that test developers don’t work hard.  They do.  They just tend to feel less deadline pressure than dev-devs.

            If you have experiences as a test developer that you feel may benefit someone trying to make a career decision involving test development, please share them in the comments section.

So You Want to Be a Test Developer

            So you have an interest in becoming a test developer, eh?  You’ve come to the right place.  In this paper I will lay out what I think is the best path to take to becoming a test developer or really any sort of developer.  Test developers are really just developers who happen to report in the test org. Becoming a good test developer requires learning not only a programming language but also what to do with it.  You’ll need to learn not only C++ but also about operating systems, algorithms, object-oriented programming, etc.  Becoming a test dev won’t be easy.  It is going to take a lot of reading and a lot of practice.  If you stick with the regimen, you’ll be a real developer.  You’ll have much of what someone would have if they had acquired a CS degree.  If not, well, you’ll be where you are today. 

Starting With a Language

            The place to start is obviously with a language.  If you don’t know how to program, you won’t be much of a programmer.  It is important to know, however, that knowing a language is just the beginning.  Too many people equate knowing C/C++ with knowing how to program.  Sadly, this is not the case.  Think of an author as an example.  Learning the English vocabulary and the rules of grammar will make you a proficient English speaker.  That does not, however, make you a great writer.  Mark Twain wasn’t a great writer because he knew how to diagram sentences.  Knowing what to say is even more important than knowing how to say it.  It is the same way with a programming language.  You can know C/C++ forward and backward and still be a bad programmer.

            The first decision you should make is what language to use to start your journey.  As you progress, you will probably pick up other languages but you have to start somewhere.  You should start with a language that encourages you to do the right things.  C# is an easy language to begin with.  C is simple and easy to learn.  C++ is its more complex big brother.  Most system-level work is done in C++.  That’s probably where you want to start.  If something like VB or Perl strike your fancy, you can use them but know that you’ll be limited in where you can utilize your skills.

            To learn C++, I recommend starting with a book.  The two best ones I know of are C++ Primer by Stanley Lippman and Thinking in C++ by Bruce Eckels.  I prefer these over the “24 Hours” or “21 Days” sort of books.  They may claim to teach you C++ quickly but they won’t.  C++ is huge.  It will take some time to master.  Pick one and read it cover to cover.  Do the exercises if you are that sort of person.  These are both thick books.  Don’t be intimidated.  They’ll both cover everything you need to move onto the next phase of programming.

Don’t stop there though.  You need to put this into action for it to make sense.  Come up with a project and start implementing it.  The project can be anything.  Implement a tool you’ve been itching to have, write a game, whatever.  I started by trying to implement the game of Robo Rally.  I never finished.  You might not either.  The point isn’t to complete the project.  It is to come to an understanding of how the parts work together.  Books teach us to program in discrete little packets.  Each packet makes sense atomically.  The point of writing a real program is to understand how these packets interact.

If you want to become wiser in the ways of C++, there are several other books you might want to read including Effective C++ by Scott Meyers and Inside the C++ Object Model by Stanley Lippman.  Neither book is essential but both will help make you a better programmer.

Going Beyond Language

            As I said, learning a language is just a first step.  Once you are here, you are proficient in some language (C++ most likely).  You are conversant in a language.  You are still a terrible writer.  The next step is to learn what makes a good program. 

This is where things get more complicated.  Learning pointers is hard but understanding algorithms is harder.  If your math is weak, you’ll struggle with some of these concepts.  Fear not.  None of this is rocket science.  It just may take you a bit longer.  If necessary, put down the book you are reading, read some math, then come back.  This should help you get further.  If you run into a roadblock, learn more math and continue until you get through.

There are three major subjects you need to come to understand.  The first is data structures and algorithms.  The second is operating systems and processors.  Finally, there is object-oriented programming.  Once you have these three areas understood, you’ll be well on your way to becoming a good programmer.

Algorithms are what make your programs run fast or slow.  When you are dealing with only a small amount of data, the algorithm you use doesn’t matter too much.  However, when you are dealing with hundreds of thousands or millions of items, the algorithm can matter a lot.  It is important to understand the types of algorithms available and to understand how to measure their efficiency.  Different ways of sorting would be an example of algorithms.  Data structures are ways to organize large amounts of data and maintain accessibility.  Examples are linked lists, trees, hash tables, etc.  Two books which are used a lot for this are Introduction to Algorithms by Thomas Cormen and Algorithms in C++ by Robert Sedgewick.

Understanding how processors and operating systems work will help you understand why the language and the OS do what they do.  Many things that C/C++ do make more sense once you understand how the underlying system works.  Understanding the OS and being able to read assembly language will come in invaluable when it comes time to debug.  The best OS book I know of is Operating System Concepts by Abraham Silberschatz.  It is accessible and covers the critical topics.  For processor/assembly topics, look to Computer Organization and Design by David Patterson. 

Finally, there is object-oriented programming.  Learning about OO techniques will help you understand how to write a program that is maintainable and extensible.  The best place to start is with Design Patterns by Gamma et al.  I suggest using to book to understand how patterns work in general, not to memorize every pattern.  A good book showing the application of patterns and good OO design techniques is Design Patterns Explained by Alan Shalloway.

Where to Next? 

            So you read all 5,000 pages and you still want more?  What can you study next?  The tree branches from here.  There are lots of possibilities.  Some good places to look are for general programming are debugging and good programming practices.  You can also start going down a specific path such as windows programming, graphics, ATL/COM programming, etc.

            Programmers spend an inordinate amount of time debugging the code they just wrote or even code written by others.  Debugging is an art, not a science.  The only way to get good at it is practice.  There are some books you can read which will give you a jump start though.  The best of these is Debugging Applications by John Robbins.

            Learning how to become a better programmer also takes practice but there are a few books written which give you some good pointers.  I won’t list them all here.  Everyone raves about Code Complete by Steve McConnell.  I don’t.  I find it long-winded.  A better book is The Practice of Programming by Brian Kernighan and Rob Pike.  Another book which is really useful is Refactoring by Martin Fowler.  He explains how to take old code and make it better.  Like Design Patterns, I find the first several chapters more useful than the reference that makes up the bulk of the book.  Finally, go find a copy of your groups coding standards document.  Often times these are chock full of good techniques.  They will usually tell you what sort of things to avoid and what good practices to emulate are.

Formal Training

            If you decide that self-driven book learning isn’t for you, are there alternatives?  Yes.  I prefer the classic CS education over professional certificate programs, workshops, or even IT degrees.  To that end, I have found a few programs I believe are worthwhile.  Each of these will allow you to get a full degree or just take some classes.  All are based online.

 

Regis University – Offers a Bachelor of Science in Computer Science.  http://www.regis.edu/regis.asp?sctn=onl

 

Colorado State University – Offers a Master of Computer Science program.  http://www.cs.colostate.edu/cstop/csacademics/csdegrees/onlinemcs.html

 

University of Illinois at Urbana-Champaign – Offers a Master of Computer Science degree.  Students participate in on-campus classes via newsgroups and online videos.  http://online.engr.uiuc.edu/

Bibliography:

Cormen, Thomas.  Introduction to Algorithms.  Thorough introduction to algorithms.  The content is often advanced.

Eckels, Bruce.  Thinking in C++.  Focuses a lot on STL which isn’t used much at Microsoft.  You can download a free copy here:  http://mindview.net/Books/DownloadSites.

Fowler, Martin.  Refactoring.  How to spot good and bad code and what to do about the bad stuff.

Gamma, Erich, Richard Helm, Ralph Johnson, John Vlissides.  Design Patterns.  The first few chapters are the most useful.  They will help you understand how OO designs should work.

Kernighan, Brian, Rob Pike.  The Practice of Programming.  Tips and techniques for being a good programmer from one of the authors of the C language.

Lippman, Stanley.  C++ Primer.  Great introduction to C/C++.

Lippman, Stanley.  Inside the C++ Object Model.  The how and why of C++.

Meyers, Scott.  Effective C++.  50 tips for writing C++.  Each tip is explained.  Along the way, you learn a lot of C++.  Skip More Effective C++ though.

Patterson, David.  Computer Organization and Design.  Covers the MIPS processor not the Intel but the concepts are the same.  Don’t get bogged down in the measurements chapter.

Robbins, John.  Debugging Applications for .Net and Windows.  Great book explaining how to debug programs on windows.  Focuses on the Microsoft programming environment but the general techniques are applicable everywhere.

Sedgewick, Robert.  Algorithms in C++.  Sedgewick has a series of Algorithms books covering everything from C to C++ to Java to Pascal.  The latest versions come in two volumes.

Shalloway, Alan.  Design Patterns Explained.  How to use design patterns in design.

Silberschatz, Abraham.  Operating System Concepts.  Good introduction to how operating systems work.

 

If you have suggestions for additional books, subjects, etc.  Please post them in the comments below.