Functionality Over Elegance

I'm reading Where Wizards Stay Up Late which is a book about the people that invented the internet.  It's the story of ARPA and BBN and the creation of essentially the first computer network.  I ran across an interesting quote from one of the programmers of the first networking gear (the IMP).  Dave Walden said, "There was an infinity of ways we could go wrong, and a substantials number of ways to go right.  Good engineers find one of the ways of going right."  He was pointing out the difference between a computer scientist and a computer engineer.  The scientist looks for the ultimate way of doing things.  The one that trumps all others.  The engineer just looks for a way.*

This isn't to say that all ways are equal.  There are tradeoffs that must be made and their consequences considered.  However, given a time budget, reliability requirements, and performance metrics, a number of solutions often make the grade.  A scientist might continue to investigate which one is best whereas an engineer picks one and moves on. 

The author says it this way:  "At its core, all engineering comes down to making tradeoffs between the perfect and the workable."  He makes the point that when you are trying to ship a product, functionality trumps beauty and elegance. 

As I stated in my essay on the subject of design, sometimes you just need to go with a solution.  It is the job of those of us in the trenches to find a solution and make it work.  It is all too easy to continue to pursue the better in the face of the adequate.  More often than not, doing so leads to overdesign and missed deadlines.  The place where this bites most engineers is in the desire to rewrite.  How often have you heard someone say "This whole module needs to be rewritten."?  Sometimes it really does but more often it comes from a sense of aesthetics.  The code isn't written the way we would have done it or it isn't written as optimally as possible. 

As software engineers, we must remember that the ultimate goal is the shipping of a product.  We must make decisions with that goal in mind.  When faced with the temptation to rewrite a whole section of code, we must resist.  If a simple patch will solve the problem, go with the patch.  The product will thank you.

 

* The use of engineer and scientist here is my choice, not necessarily Walden's.

Merry Christmas

By now you're all probably stuffed full of food and staring at a pile presents not to mention wrapping paper, ribbons, bows, and boxes.  I hope you all had a great day.  Enjoy your time away from the regular routine for a while.  Take the time to do something you don't normally get to--reflect, talk to family, play some games, do some programming for yourself, etc.  Whatever it is, have a great time doing it.  Merry Christmas!

-Steve

Secret Santa is NP-Complete

Every year my group of friends undertakes a Secret Santa gift exchange.  When we started we each drew names from a hat and bought a gift for the names we drew.  Being budding programmers, we soon dispensed with the hat and wrote a program to do the work for us.  In 1999 a friend and I wrote a C++ app to do the work.  Though we've been running it every year, the code hasn't much changed in the intervening seven years.  It is getting dated and needs reworking. 

Toward that end, I've been contemplating the match routine lately.  The current one does something naive like:

Pick a person

Choose a match at random

If there is a conflict with that person, slide to the next one on the list.

Once a match is found, remove those people from the relative lists and pick a new person.

If a match cannot be made, start the process over.

With a small number of people and not a lot of blacklisting (husband shouldn't draw wife's name, etc.), this algorithm will work.  However, for a complicated list of people, this algorithm is nondeterministic and could theoretically run forever.  I was thus searching for a better algorithm.  One which was faster than a complete enumeration of all options and deterministic in nature.

This weekend I took the final for my Formal Models of Computation course.  (Yes, this ties in with the above--be patient) The last thing we covered was complexity classes and the concepts of P and NP.  What follows is a brief description of this concept.  For a more formal handling of the subject, check out the Wikipedia entry. 

In theoretical computer science, they don't use real computers.  Rather, they use formal models called Turing Machines.  These have the same power to solve problems that modern computers do, just a bit less efficiently.  They are a good proxy for real computers.  The speed of these machines is measured roughly in terms of the input.  So given an input of length n, a linear algorithm would take O(cn) time, where c is a constant.  We usually ignore these constants and just call it O(n) time. 

There is a class of problems called P or Polynomial-time which represent those problems that can be solved by a Turing machine in a time which is a polynomial of the input length.  That is, O(n^2), O(n^3), ... , O(n^k).  These are generally thought of as those problems that computers can efficiently solve.

There is another class of problems called NP or Nondeterministic-Polynomial-time.  These represent those problems that can be solved by a nondeterministic Turing machine in polynomial time.  A nondeterministic Turing machine is bascially one that can do more than one thing at once.  When it comes to a fork in the algorithm, it can take both forks simultaneously. 

It is assumed that NP describes a bigger universe of problems than P.  That is, P !=NP.  What takes nondeterministic Turing machines polynomial time takes regular Turing machines exponential time.  That is, they take something like O(2^n) time.

Back to my Secret Santa problem.  It was just after my final that I turned my thoughts back to solving this problem.  It then hit me that what I was trying to do was impossible.  There is a well-known NP-class problem which involves finding a Hamiltonian circuit.  A Hamiltonian circuit is a path that traverses an entire graph by visting each node exactly one time.  It turns out that this is exactly the problem I was trying to solve.  Imagine my Secret Santa problem as a graph where each person is a node and there are edges between all nodes that are not blacklisted.  In this view of the problem, I'm trying to find a path around the graph, visting each node once.  In theoretical computer science this is known as a reduction.

This analysis pretty much dashes my chances of finding an elegant solution to the problem.  There is no true solution other than brute force trying each combination.  With the small number of nodes in my usual matching, this works but I still want something better.  All is not lost, however.  There are some techniques I can use to get close to the solution without necessarily trying all of the combinations which I intend to investigate.  I'll write about them after I understand more.

Wow.  I guess I did learn something practical in that class after all. 

None of Us Is as Dumb as All of Us

That's the subtext on the "Meetings" inspirational poster from Despair.com.  What it means to me is that you'll never get a great design out of a committee.  When designing a piece of software, you want to get input from many people but how do you keep from being paralyzed by input.  Worse yet, how do you keep the design from becoming an incoherent mess with so many people making the decisions?  The simple answer is to elicit feedback from a large group but continue to act autonomously.

It is my contention that things of beauty tend to come from a single mind.  At best, they come from a small group.  The flip side of this coin is that the ugliest things also tend to come from this same group.  A large number of contributors tends toward bland and average.  Look at television or movies.  The best shows are often those where the creators are allowed creative license.  The shows that merely average are many times those where the studio plays a large role.  Let's take one of my all-time favorite shows, Babylon 5.  The initial series was brilliant.  The studio stayed out of the way.  Fans responded.  After it was finished, there was a spinoff called Crusade.  This time the studio came in and meddled.  They demanded more action.  They rearranged the order of the episodes.  They changed the dialog.  The show tanked.

How does this apply to designing software?  Should you just shut yourself in your office (or cube) and design?  Not exactly.  What makes the great come from a small group and the mundane come from lots of input is that a small group has a coherent vision.  Large groups have a hard time operating under a shared vision.  Each memberr has a different idea of what the outcome should be.  This usually leads to a poor design.  The features are often implemented in a nonuniform way or are difficult to work together.  Sometimes, to satisfy everyone, nothing is pruned.  Every option and setting is exposed.

The value of a single decision-maker is coherency.  As I state above, this can lead not only to the best designs but also to the worst.  Doing things completely alone is often a prescription for even greater disaster than doing them by committee.  What then is to be done?  It is a necessity to satisfy customer desires.  A "great" product no one wants isn't really great.  The secret is to listen to idea but not to be beholden to them.

Recently we were trying to make some changes to a shared test shell.  The major changes were fairly noncontroversial.  The small details, however, became a serious sticking point.  People couldn't agree on the exact semantics.  Each person had a preference.  Most were equally valid but no one idea was coming out on top.  Eventually I called for a close to the debate and told the person responsible to just go with whatever convention he deemed best.  This settled the issue and people are (apparently) satisfied with the results.

Too often we get stuck into a binary mindset.  Either we take feedback and treat it as gospel or we ignore feedback altogether--because we know best.  Instead, we need to mix the two. 

My advice when designing something is to have an individual or a small group with a shared idea drive the design.  This ensures a coherent interface and gives a greater chance that the solution will be elegant.  To avoid the trap of irrelevance, this group should solicit feedback from a wide variety of sources.  They must listen to ideas, but be willing to reject them. 

The State of PC Audio

One of the things I own at Microsoft is test development for audio in Windows.  For Vista we did a lot of work to rewrite the audio system and make it more stable and enable more features.  Just recently ExtremeTech published an article on the State of PC Audio.  It talks about the various options for audio including motherboard sound and the various discrete cards available.  If the changes we made for Vista are interesting to you, the article contains a useful drawing of the audio paths in XP and Vista. 

The author complains about the state of motherboard audio.  This is one of the things we're looking to improve for Vista.  In order to get a Windows Vista logo, all machines are required to pass a certain set of qualification tests.  Some of those tests--provided by my team--are there to measure the quality of the audio coming out of a PC.  With these tests, we hope to move the quality of motherboard audio forward.

If PC Audio interests you, be sure to check out these two blogs:

The Audio Fool - One of the people responsible for writing the audio fidelity tests behind the logo.

Larry Osterman - A developer on the audio team.  Also a great source for programming tips and amusing Microsoft history.

New Vista Sound

This is a bit dated but it's worth mentioning anyway now that Vista is officially launched for businesses.  Each major release of Windows comes with a startup sound.  Vista is no different.  It has a sound created by Robert Fripp.  I was lucky enough to be able to attend one of the recording sessions for the new sound.  Scoble captured that session and it is availabe at channel9.

The person responsible for the Vista sound is Steve Ball who happens to be a program manager in my group and sits just down the hall.  He recently gave an interview to NPR where he introduces the new sound.  It is short at just over 2 minutes and goes over the various sounds from past versions as well as the new one.  Here is an article talking about the new sound.

 Update:  In the spirit of accuracy, the session I point to above is not the one that I attended, but is one like it.

What To Unit Test

Several months back I wrote about unit testing.  Following that I received a question from a reader about how to actually carry out writing unit tests.  What should be tested?  How much is enough?  There is no single answer to these questions but I can give guidance.  The actual answer depends on the specifics of what you are trying to test and how much time you have to devote to unit testing.  What follows is a list of the items you should test, in the order you should test them.  If you only have limited time, start at the top of the list and work down until you run out of time.

In each case below, you should begin by writing tests for the positive cases and move on to the negative ones.  By positive cases I mean those things you expect a user to do.  This is the way the feature will be used if it is being used properly.  Call each function with all the equivalence classes of data that they would be expected to see in normal circumstances.  Ensure that the behavior you observe is what you expect.  This means actually verifying that the right behavior took place.  If short on time, you can often use the interface to verify itself.  Calling a Get after a Set to verify that the value was set correctly is a good example of this.  Even better is verifying through some alternative mechanism that the right behavior took place.  This could be looking at internal structures, observing behavior, querying a database, etc.

Negative cases might also be called bad data.  These are the inputs you don't expect.  This includes out of range values, invalid objects, null pointers, improperly initialized structures, etc.  Once upon a time we could ignore these and consider the result of these inputs to be undefined.  In the days of networked computers and the security implications now present in many flaws, they cannot be treated lightly.  It is important to test for these and ensure that the proper behavior is observed.  That behavior could be to ignore the data, return an error, raise and exception, or even crash safely.  The important thing is to understand what the behavior should be and to verify that it indeed takes place.

The first and most important thing to hit is the point where your customers will interact with your code.  This could be an API or a UI* or a web service.  This is the highest level of abstraction and perhaps not the easiest or even the best place for unit tests, but if you only have a little time, this is where you'll get the most bang for your buck. 

The next place to go testing is at the internal interface level.  This means testing the places where the internal objects/functional blocks interact.  If you are programming OO, this means testing all of the public (or friend) interfaces on all of your objects.

Finally, test unit test all of your internal utility functions and private object methods.  Crack the abstraction layer on your objects and probe their internal functionality.

If you do all of this, you'll have a pretty good idea that your feature is functioning as desired and will have warning when something breaks.  Remember to run all of the (relevant) tests before *every* checkin.  Also, be vigilant.  Any failures at all should be fixed before checking in.  It is not okay to ignore a failure.  If it is a false failure, fix the test.  If it is a valid one, fix the code.

 

* You probably cannot unit test the API but if you properly abstract the actual UI, you can write unit tests for the functionality of each UI element.

Bug Taxonomy

When you get toward the end of a product, you have to make the tough calls about what gets fixed and what doesn't.  Most bugs fall into one of two obvious buckets:

• Must Fix -  Something is really bad here.  We cannot ship the product without fixing it.
• Won't Fix - The problem is minor, hard to get to, etc.  It won't be fixed for this release.

However, there are a few bugs which don't clearly fit these buckets.  There is no clear call to make on them.  My group has come up with some fun names for some of these bugs.  Among the taxonomy are:

• Yetis - These are bugs that you'd really like to fix but will be really difficult to find a fix for.  If you find a fix, you'll definitely take it but you won't hold the product until you find one.  Issues in this classification include areas where reproduction is difficult.  Stress crashes, random lockups, and inconsistent behavior all fit into this category.  We thought of using the term Sasquatch but that is too many characters.
• Limpets - These would be more accurately called Remoras, but why we don't is a long story.  These are bugs that we'd really like to fix but it isn't worth the risk to take a change to a particular binary or even the whole product just for that fix.  Taking a change to something already considered done requires another build, test pass, etc. 
• Sharks - Remora are also known as suckerfish.  They are known for attaching themselves to the side of a shark.  The idea here is that, while we wouldn't crack open a binary for a limpet bug, if there is a large fix which must go in, we'd like to take these other bugs with it.  Once you have started changing things and have to redo your test pass, you might as well include those nice-to-have issues too.

What nomenclature do you use to describe bugs?

Vista Goes Gold!

Windows Vista--the product I've been working on for much of the last 5 years--has finally shipped.  Hopefully that will give me some more time to actually blog here.  It has been a long haul and a lot of work--especially over the past few months--but it is now finished.  I'm pretty proud of the final product.  I have been running it on my main machines for a few months now and it has been quite solid. 

A few months back I had a post decribing what the term RC1 meant.  At that point, I also linked to an article showing the times from RC1 to RTM of the various Microsoft products.  Windows 2000 took 166 days.  Windows XP took 53 days.  By my calculations, Vista has taken 68 days.  A little longer than XP but in the same ballpark.

Now there is nothing left to do but wait.  Vista launches to the general public on January 30th.  Office 2007 will launch on that same day. 

Liveblogging Woz

He's talking about his childhood.  How his father managed to get him transisters and diodes because he worked at Lockheed Martin.  He did a lot of electronics.  He didn't know was a computer was for a long time because he was afraid to ask.  Eventually in high school he got a chance to program a computer.  His first program was knight's tour in chess.  It turns out the computer was too slow and would take 10^25 years to complete.  Oops.

In high school he ran across a book called the "small computer handbook" about the PDP-8.  He says he designed (on paper) a PDP-8 based on that manual.  As he learned of new chips, he kept refining the design.

He taught himself coding by sneaking into the Stanford Linear accelerator library and looking at the books and manuals and magazines.  He kept refining his paper designs.  Trying to break his old records.  Going from 65 to 64 chips, etc.

When he went to college, there were no undergrad classes about computers.  He had to take a graduate class on the subject.  In his zeal for programming, he managed to run the computer program 5x over budget for the year.

He's talking about meeting Steve Jobs now.  They went to the same High School but apparently met while Woz was in college at Berkley.  In his 3rd year he was taking all graduate classes.  After his 3rd year, he took a year off to go earn the money for his 4th year.  He ended up working for HP's calculator division.  Despite working on calculators all day, his hobby at night was electronics. 

He's now talking about seeing his first pong game.  At that point, games like pong weren't software, they were hardware.  He had no money and couldn't afford a pong game but he could design one himself. 

Steve Jobs went to Reed college (in Portland, OR) where he skipped most of his classes.  After that, he came back and got a job at Atari.  They offered Woz a job designing games there.  Woz was happy at HP and didn't want to leave but he was willing to take the challenge of designing games for them.  He designed a Breakout game in 4 days and 4 nights. 

Woz observed someone using a teletype machine on the arpanet and wanted one.  Again, he couldn't afford one but he could design one.  He used his TV as the output device.  At Jobs' prompting, he sold his design.

He went to the Homebrew Computer Club and discovered that microprocessors had advanced.  He went on to build his own computer, starting with his video terminal.  After he had one, he needed a language so he wrote a version of basic.  He says writing basic was the hardest part of the Apple I design.  He couldn't afford an assembler so he wrote by hand and then hand-translated it into the 1s and 0s.  Again, Steve Jobs encouraged him to sell his design.  The name wasn't Apple at the beginning.  Woz tried to get HP to produce the computer but they refused.  Because of that, he was able to use the design himself.

The first Apple was priced at $666.66.  They sold about 150 of them. 

The Apple // cost $250 to build and they wanted to sell 1,000 of them.  That's $250,000 in capital.  Apparently Woz tried to sell the design to Atari and Commodore and were turned down by both.  They tried venture capital but couldn't talk business enough to get the money.

They found a source of funding--Mike Markula.  He said he would give them the money but Woz would have to leave HP.  A friend talked him into it because he could stay an engineer.  Sales exploded once Visicalc came out.

That's the end of the speech.  Fun.  If you have a chance to catch him at a book signing during this tour, do it.

Waiting for Woz

Steve Wozniak (co-founder of Apple for those of you in Rio Linda) is on campus to promote his new book iWoz.  The room he is scheduled to speak in hold about 100 people.  The speech is supposed to start in 15 minutes and it is standing room only already.  A little before 1:00 the room was half full.  Crazy.  I guess he really is an icon.  He's an interesting personality.  A total geek.  He spent much of his interview on This Week in Tech talking about a new blue laser he had purchased.  He participates on a Segway Polo team.  He's smart too.  I recall hearing him speak at one point about how he had the whole of the Motoroloa 6502 (the chip powering the Apple //) assembly in his head when he was coding for it.  Should be an interesting speech. I'll try to blog on it later.

Is the Internet Really That Powerful?

About 2 weeks ago Serenity won a Hugo award.  If you haven't seen this movie or watched the TV series that inspired it, Firefly, you should.  It's arguably some of the best TV of recent years.  It's certainly some of the best sci-fi to come down the pipe in a long time.  I was watching the acceptance speech on YouTube and a thought hit me.  Serenity had a *huge* following on the web.  Before the movie came out the hype was really intense.  Then the movie launched and instead of being the next Star Wars it quickly became an also-ran.  The hype didn't translate to lots of people seeing the movie.  It made back the money it cost but didn't make anyone rich.  Recently there was another big internet hyped movie.  Snakes on a Plane received enormous amounts of hype and seemed sure to have at least a huge opening weekend.  It won the weekend but the total box office take was really low.  The hype again didn't translate into people going to see the movie. 

I don't mean to equate these two movies.  Serenity was a great movie.  Snakes is, by all accounts, not very good.  Still, they both had lots of hype.  Lots of blog posts.  Lots of fan websites.  Neither had as many people go see it as was predicted.  Why didn't the hype translate into people going to see the movies?  Is the blogosphere just a echo chamber with a limited audience?  Is it more about the hype itself than the target of the hype?  I don't know the answers but I do find it interesting.  In a day when so much advertising and effort is being made to create a web presence for products, it is important to know whether creating excitement on the web is actually a way to sell products or not. 

What does "RC1" really mean?

With the recent release of Windows Vista RC1, I have seen many comments on places like Slashdot and various blogs which demonstrate there is a lack of understanding of what we mean when we say we have "Release Candidate 1".  I will attempt to explain what we mean when we say RC1.  This is not an exhaustive definition nor is it official but it should give you a good idea what we mean when we say it.

There is an impression that this is the build we intend to release as the final build unless something heinous bug is found in it.  That is often what the term release candidate means in the world of software but not here. 

Back when I started at Microsoft in 1997, we were in the process of releasing IE4 and then Windows 98.  In both of those cases, the release candidate builds were just that, candidates for release.  If we had RC1 on Monday and found a bug in it, on Tuesday we'd have RC2.  If we didn't find a bug on Tuesday but found one on Thursday, then Friday we would have RC3 and so on.  After a half-dozen builds or so, we would have a final release.

When I moved to the NT5 team (what became Windows 2000), it was confusing to me to hear the same terminology mean something very different.  In that world, and in the world of Vista today, what we call a release candidate is more of a waypoint on the way to release than a build we actually think will be the final one.  An RC build is one that we iterate on for a while to drive up the quality (like a beta or one of the CTP releases--only moreso).  A release candidate, in this way of thinking, is a build you release that will differ only in minor ways from the final one.  The bar for bugs goes up substantially after it.  We become really strict about what we will take for the product.  No new features or even major changes are allowed in.  Just bug fixes which don't perturb the system too much.  In this nomenclature, a "release candidate" is not really a candidate for final release.  We know there will be bugs to fix after we release it.  In the WinXP timeframe, there was a month or maybe even two between RC builds.  During that time we still had daily internal builds.  Each release candidate was closer to the final build and the amount of change allowed afterward was smaller than before.

So what do we call the builds we are really considering for release?  We usually call them escrow builds.

 

Update:  I found a nice chart in this Seattle PI article showing the time from RC1 to RTM for Windows 2000 and Windows XP.  For Windows 2000, it was 166 days.  For XP, it was 53.

Management Styles

Joel Spolsky, of Joel On Software, has a series on management styles.  He details three styles of management:  Command and Control, Econ 101, and Identity.  Command and Control is where management dictates everything that happens.  Econ 101 is where management uses economic incentives to get the right behavior.  Identity is getting people to do what you want by making them like you, personally.  These are all straw men and no one really uses only one style but they are useful to understand and help to shape a good mixture of styles.  The command and control model, at its extreme, is micromanagement.  It doesn't allow for any innovation in the leaf nodes.  The polar opposite is Econ 101 where the management identifies some outcomes and pays people to achieve them.  It does not, however, give much instruction on how to achieve that.  Identity is getting people to like you and the organization so they'll do what is best.  This has a similar pitfall as Econ 101 in that it doesn't necessitate helping employees. 

The best solution is a mixture.  Some people are motivated by money and rewarding people economically for contributing to the company can help them.  That cannot be the only motivation, however.  Internal motivation can be an even more powerful motivator.  If people like you, they'll give you their best.  If people are excited about the company's direction, they'll give it their all.  A little command and control can be useful too.  It helps to give the team more than direction.  A very experienced team can get by with only direction but most teams have inexperienced people and they will benefit from being told how to get things done.  The important thing is to make sure the instruction is done with an eye to making an independent worker rather than just getting the immidiate work done.  One of the biggest jobs of a manager is to grow his or her team.  Helping the team mature will help the individuals on the team but also help the output of the whole team. 

Cool Live.com Search Feature

I was watching an internal presentation about Live Search today and ran across this really cool feature.  Live and Google have supported searching within a particular domain for quite some time.  However, it has always required remembering the correct syntax and typing in the full domain name.  Now, Live.com has it integrated into their search results.  It's much more readily available than before.  Try this:

1.  Go to www.live.com

2.  Type in a search.  Try "msdn blogs".

3.  In the upper right hand corner of the results window there will be a slider bar.  Slide this to the right (4 horizontal lines).

4.  Click the words "Search within this site."

5.  Type in a search.  Try "unit tests".

The results are everything within blogs.msdn.com that matches your criteria.

Interview with Clayton Christensen

The Innovator's Dilemma is an eye-opening book that everyone in the technology industry should read and understand.  I recently ran across an audio interview with the author, Clayton Christensen.  In it he gives a brief explanation of the main thesis of his book.  That is that there are certain types of technology that are disruptive.  They are at first underperformant of the market but allow for new uses.  Over time, they become good enough to subsume the previous solutions.  A good example is that of the PC.  When the Apple // and IBM PC first launched, they couldn't do the work of real business.  For that, you needed a minicomputer.  Over time, however, the PC became powerful enough to do everything that a minicomputer could and eventually totally replaced the minicomputer.  In his book, Clay uses examples as diverse as hard drives and earth moving equipment.  In this interview, he also starts to flesh out what he calls the "Law of conservation of modularity" which attempts to explain how the ability to make profits in a market changes over time.  He talks about how this affects Intel and how it will affect the software market.  I don't know that I agree with all of his characterizations but it is definitely thought-provoking.

 

LSharp Discussion Group

I've been playing a little with Lisp lately.  Unlike many other single-sourced languages these days (Python, Ruby, to a large extent C# and Java), Lisp comes in a myriad of flavors and implementations.  Even with the Ansi Common Lisp standard, each one has slightly different abilities.  There are a lot of things, like networking, that are not part of the standard and are done differently in each implementation.  When looking for one to play with, someone on my blog pointed me toward L#.  This looks like an interesting language and is a Lisp based on the Arc principles but built on top of the .Net CLR and its native data types.  This means that you can program lisp but have access to the entire .Net framework.  The L# front had been quiet for a while but now there is a new discussion forum for it which will hopefully spark some interest.  If you have interest in Lisp and want to use it with the .Net libraries, check out L# and the discussion group.

Back From Vacation

I spent the past few weeks on my Grandparents' farm in upstate New York.  When I think of New York I always think of the city and suburbia.  After spending 2 weeks in the upper part of the state, I'm amazed how wrong that view is.  There are hundreds of miles of open territory that has little other than small towns and farms in it.  Amazingly beautiful land.  Too bad I live so far away...

HD-DVD Looks Better Than BluRay

I was in Circuit City the other day and they had a beautiful Sony SXRD television set up playing some BluRay content.  I thought I was going to be blown away.  I wasn't.  The disc playing was a demo disc featuring snippets of lots of movies.  Each one was somewhat washed out.  The edges weren't crisp.  It just wasn't that great a picture.

Now, we have the first head-to-head reviews of HD-DVD and BluRay.  Three titles are now available for both formats.  The results are in.  The winner:  HD-DVD.  HD-DVD had better quality video, better audio, and better interactivity.

Part of this comes from the BluRay camp's choice to use MPEG2 as its format of choice.  Each of the formats has three required video codecs:  VC1 (standardized Windows Media Video 9), H.264 (also known as MPEG4 AVC), and MPEG2.  H.264 appears to be too complex for anything to decode well at this time and neither are using it.  The HD-DVD camp is going with VC1 and the BluRay camp with MPEG2.  VC1 is a much better codec than MPEG2.  It was invented many years after the aging MPEG2 standard.  It can store the same data in roughly half of the space.  This means that HD-DVD can store a better picture in less space.  Also, despite the promises that BluRay would be the higher-capacity format, it presently is not.  BluRay is shipping 25-GB single layer discs whereas HD-DVD movies are shipping on double-layer 30-GB discs.  Between the codec and the capacity, HD-DVD has space for content that is twice as good or twice as much content that is equal.

HD-DVD Podcast

Our Vice President, Amir Majidimehr, gave an interview to XBox Live's Major Nelson.  He and one of the program managers for HD-DVD talk about the market, the discs themselves, scripting, interactivity, codecs, and what makes HD-DVD better than BluRay.  It's a very informative interview.  Load it up on your media player or burn it to CD and listen away.  It's well worth your time. 

Two versions are available:

WMA

MP3

My initial take was that HD-DVD was losing the battle.  The studios all seemed to be siding with BluRay, the PS3 seemed like an unstoppable juggernaut, and HD-DVD didn't seem to have much going for it.  Now all that has changed.  The studios are still largely with BluRay but they'll go wherever the players are.  The PS3 is looking more like a hobbled old horse than the inevitable winner of the race.  HD-DVD has made a decent showing in the market.  It is available and costs 1/2 of the BluRay players.  Things could get interesting.

June Monthly Podcast Update

It's been a busy month (thus the last listing and the sparse postings in June).  No new podcasts were added for the month of June.  The regularly-scheduled programming is the same as it was in May.  I tried a few new podcasts but didn't get into them. 

I listened to The Vintage Gamer but found his style too dry for my liking.  He spends an hour talking about one game--I listened to the Diplomacy episode--but still manages to stay at too high a level for my tastes.  There isn't much meat there and the voice has no energy.

I also tried Linux Reality but found that also not to my liking.  The content is way too high level and is often incorrect.  As someone who has only toyed with Linux, I shouldn't be able to spot glaring errors in the content yet.  The voice here is also lacking energy.

DX10 Interview

The ExtremeTech Podcast for last week was an interview with one of the DirectX 10 architects, David Blythe, and Chris Donahue who is on the Biz Dev side of the house.  The interview covers the new graphics architecture in Vista as well as what has changed for DX10.  There is a lot of useful information in this interview.  You'll get the most out of it if you have some understanding of how graphics pipelines work already but it doesn't get too deeply technical.  There is a lot of exciting stuff made possible by the more generalized shader model and the new geometry shader in DX10. 

We don't do a lot of DX10-related work here in video land but do rely heavily on the new Vista display driver architecture (WDDM).  Here is a presentation David gave at WinHEC a year ago on the subject.  If you are curious, much of what I work on in the video realm falls into that box labeled "DXVA."  You can learn more about it from this WinHEC presentation given by Rodolfo Vargas, a PM that I work closely with.

Let me warn you that the sound quality of the interview is terrible.  If you don't want to suffer your way through it, there is also a transcript of the interview available here. 

Welcome the Audio Fool

   I'd like to welcome a member of my team to the world of blogging.  His blog is called The Audio Fool.  He works on measuring the fidelity of audio devices for Windows.  If you have interest in audio and want to learn about it from a technical standpoint, I suggest you add his blog to your rss reader.  I look forward to reading it.

VOIP - Ready for Prime Time?

   About two months ago my wife and I decided to take the plunge and try out VOIP.  Economically it made a lot of sense.  The pricing at SunRocket is just so much better than the price of our local baby bell.  The question was, would the service be comparable?  The short answer is that while it is not as good as POTS, it is good enough for us.  It may not be for everyone yet though.

   There are three major worries about VOIP phones:

• It may not work with 911


• The voice quality may be low


• It isn't reliable.

Two of the three of these turned out to be non-issues and the third isn't as big a problem as it might have been.

   The 911 problem is largely solved for VOIP.  All of the major providers support Enhanced 911 service.  When you call 911, the call center is provided with information like your address.

   The voice quality is excellent on our line.  At times I think it sounds a bit different but only when I'm focusing on it and that could just be me hearing what I expect to hear.  It is certainly better than most cell connections.  I don't notice any breakup, distortion, or significant latency.

   The reliability isn't all there yet.  POTS never goes out.  Even when your power is down, POTS phones just work.  This isn't the case with VOIP.  If your power it out, it won't work.  If your network is down, it won't work.  When the VOIP company is having issues, it won't work.  When we first had the phone installed, we were experiencing random drops where we could hear what the other person was saying but they couldn't hear us.  After a call (on a cell phone) to Sunrocket, tech support fixed the issue by tweaking something in their device.  Since then, it has been working like a charm. 

   System setup is really easy.  Just plug the device, which Sunrocket calls their "Gizmo," into your network between your cable/dsl modem and your router.  The Gizmo has 2 phone jacks on it into which you plug regular phone line and regular phones.  At my house I have a star network for my phones so I just connected the device to my central phone hub and all the phones in the house worked.

   So, would I recommend it?  Yes.  There is a caveat that you'll have to be a little technically savvy to deal with the glitches that come with networking.  If you can figure out how to reset a cable modem/router every once in a while, you can handle this.  The economics of VOIP are just so much better than land lines.  In our area, a full package of phone plus features like caller ID, etc. costs $35 per month plus long distance and taxes.  Sunrocket is just over $17.  If you have a land line you don't use very much, you can get away with the $9.95/month package.  For $25/month, I can deal with a little inconvenience.

Prescriptive Advice For Successful Unit Testing

   At the beginning of the Vista (then Longhorn) project our team decided that we would implement unit tests.  This was the first attempt in our locale to try to use them.  We had some successes and some failures.  Out of that I have learned several things.  This is an attempt to codify what I have learned and to try to set out a prescription for what I feel it would take to leverage them fully.  What follows are my recommended practices for implementing unit tests:

1. Unit tests must be written by the developers of the code.  Having a separate test team implementing them doesn't work as well.  First, they take longer to write.  The person developing the code knows how it is supposed to operate and can write tests for it quickly.  Anyone else has to spend time learning how it is intended to be used from the minimal (at best) documentation or conversations with the developer.  Having the developer write the unit tests acts as a form of documentation for everyone to follow.  Second, the tests take a lot longer to come online.  They are available days, weeks, or even months after the code is checked into the product. 


2. Unit tests must be written and passing before the code is checked in.  Writing unit tests should be considered part of the development process.  Nothing is code complete unless there are passing unit tests for it.  Checking in code without passing unit tests should be treated like checking in a build break.  That is, it is unacceptable.


3. The unit tests must never be allowed to fail.  Just recently I saw a bug in our database about a unit test that was failing.  The bug was filed several months ago.  This just cannot be allowed to happen if you are going to get the full value out of the unit tests.  Unit tests should act as canaries.  They are the first things to fall over when something goes wrong.  If they ever break, they must be fixed immidiately.  


4. Before any checkin, existing unit tests must be run and made to pass 100%.  This is a corrolary to points 2 and 3 but I want to make it explicit.  Just as you would get a buddy build, a code review, and a smoke test before every checkin, you must also pass the unit tests.


5. Unit tests must be granular and comprehensive.  A test case which plays back some media is not really a unit test for a media pipeline and it certainly isn't sufficient.  At a minimum, each external interface should be tested for all expected values.  An even better system of unit tests would verify internal interfaces directly as well.  A rule of thumb is that unit tests should achieve at least 60% code coverage before the first checkin is made.


6. Standardize a mechanism for storing, building, and running unit tests.  Don't leave it up to each individual or small team.  There should be a standard harness for the tests to be written in.  There should be a convention followed by all for where to check them into the source tree.  Unit tests must be built regularly.  In my opinion, the unit tests should be written in the same harness used by the test team for their tests.  The unit tests should be checked into the build right alongside the code that they are testing and should be built with each build.  A build break in a unit test is just as bad as a build break in the shipping code.


7. The unit test framework must be lightweight.  If the framework is to be the same one the test team uses (highly recommended), it must be one that can be run easily.  If running unit tests requires anything more than copying some files and running an executable, it is too heavy.  Expecting developers to install a whole test framework to run their tests is a prescription for disaster.


8. Run unit tests as part of the daily testing process.  If the tests use the same harness, they can be leveraged as part of the daily tests.  The tests for external interfaces can be especially useful in assessing the quality of the product.

Have any other best practices for unit testing?  Let me know.

Introductory Lisp Books

  I've been doing a little reading about Lisp lately.  In particular, I've read two books on the subject.  ANSI Common Lisp by Paul Graham and Practical Common Lisp by Peter Siebel.  I haven't finished either book but I've read enough to give an assessment of the two.  If you are considering learning Lisp, read on for my recommendation.
   Let me start by saying that both books are good.  Both books assume that you already understand how to program.  Neither book is written for someone who is brand new to the art.  They both cover the subject of ANSI Common Lisp which is probably the predominant dialect of Lisp. 
   Practical Common Lisp starts off with a quick tutorial and moves on to a simple database implementation.  Only after these two examples does Mr. Siebel start describing the syntax and symantics of the language.  After some typical programming book subjects like functions, variable, control structures, and macros, the book goes back to examples.  The examples are easy to follow and give you a good sense of how Lisp is to be used.
   ANSI Common Lisp is written by Paul Graham who also writes on the subject of startups, geek culture, and programming langugaes over at his web site.  Mr. Graham takes a different tack.  After a quick introduction/tutorial, he describes the language via data structures--talking about lists and other data structures.
   My preference is for the ANSI Common Lisp book.  There are two primary reasons for this:  1)  it is more accessible to a C/C++ programmer and 2) it flows better. 
   ANSI Common Lisp approaches the subject from a perspective that is understandable to a C/C++ programmer.  Whereas Practical Common Lisp describes the Let/Setf operators by talking about "bindings", ANSI Common Lisp describes it in terms of variables and pointers.  The first requires becoming fully versed in a new way of thinking.  The latter comes naturally.  PCL also introduces forms and s-expressions but describes them in terms of other Lisp concepts.  This makes it hard to understand unless you already grok Lisp, which, if you are reading the book, probably isn't the case.  ACL avoids these concepts in the abstract and only introduces them with concrete examples.
   PCL also commits what I like to call the carnal sin of programming books.  It uses concepts in its examples before actually introducing them.  For example, in an example on page 94, it uses the first, second, and third functions.  Unfortunately, it doesn't actually explain what these are until page 143.  ACL, on the other hand, seems to build more logically upon the previous pieces it has introduced.
   It should be noted that PCL is available for free as a PDF if that is to your liking.  Personally, I prefer dead trees.

Making Hard Choices - Which Bugs Do We Fix?

We just shipped Vista Beta 2 and we are now on the march toward shipping what we call RCs or Release Candidates.  During this time, we are forced to make many hard choices about what bugs to fix and what bugs we can live with.  There are those who believe that software should ship without bugs but those people have probably never tried to ship a product of any significant size.  There will be bugs in software, just like there are bugs in a new house or a new car.  The important thing is to make sure there are no important bugs. 

Eric Sink is a software developer at SourceGear and I recently ran across an article by him about shipping with bugs.  Eric makes a few points I think worth repeating.  He says there are 4 questions that must be asked:

1) How bad is its impact? (Severity)

2) How often does it happen? (Frequency)

3) How much effort is required to fix it? (Cost)

4) What is the risk of fixing it? (Risk)

The first two determine whether the bug is worth fixing at all.  If the bug doesn't pass muster there, you don't even look at the 3rd and 4th.  A bug being easy to fix is never a reason to take the fix late in a product cycle.  Just because it is a single line doesn't mean we should do it.  There is risk with all fixes.  I've seen the easiest ones go wrong.

The second two can be reasons to reject even righteous bugs.  If the bug is bad and happens often but is very hard or very risky, it still might not be worth fixing.  This is a hard balancing act.  Sometimes the bug is bad enough that you delay the product.  Other times it is possible to live with it.  Making that choice is never a fun one.  As owners of a product, we want all bugs to be easy to fix and have low risk.  Unfortunately, that isn't always the case and when it isn't, hard choices have to be made.

I recommend you read the whole article.  Apparently the one I found is based on a longer article which can be found here.

May Monthly Podcast Update

Another month, another list of podcasts.  A few new ones this month, nothing really fell off of the list.  I think I'm about at my limit.  My commute isn't long enough to support any more than this.

Here is what I listened to regularly this month:

Major Nelson - Covers the world of the XBox360.  News, reviews, interviews with insiders.  I'm not sure there is enough new going on in the world of the 360 to keep my interest forever but for now it stays on the list.

This Week in Tech - Leo Laporte hosts a roundtable discussion about the technology news of the day.  Lots of talk about the Intel Mac this month.

Security Now - Very informative show about the world of computer security.  Cryptography series ended this month and several shows covered more general topics.

The Dice Tower - Best of the board gaming podcasts.  Reviews, news, and top ten lists.  Good place to find new games.  One of the co-hosts, Joe Steadman, left the show this month.  It remains to be seen if the new show will retain the same spark.  I hope that it does.

FLOSS Weekly - Interviews in the world of open source software. 

This Week in Media - Roundtable discussion about digital media creation, distribution, etc.  Part political, part technical.  I'm learning a lot.

HDTV Podcast - Not a lot of depth but good coverage of the news in the world of HDTV.  The whole show is less than 1/2 hour so it's easy to squeeze in.

ExtremeTech Podcast - Yet another round table, this one covering PC technology.  Covers things like vertical coolers, new processors, etc.  A little HDTV is thrown in too.  Twenty minutes long.

Honorable mention:

Engadget - Not something I regularly listen to but their joint coverage of E3 was solid.  Podcasts right after the Sony and Nintendo/Microsoft press conferences plus a more general E3 show.

As always, if you have suggestions of media and/or technology podcasts, send them my way.  I am especially looking for podcasts on project management/programming issues.

Real Estate Finally Goes 2.0

Just a few weeks ago I was searching some real estate listings and it struck me how poor the mapping experience was. It is still very much a "web 1.0" experience. Everything on the page is static. Mapping tools have come a long way in the past few years. It is about time that someone take that technology into the real estate space. It appears that John L. Scott--a real estate company in the Northwestern U.S.--has done just that. It is the first real estate page I've seen utilize "web 2.0" maps.

Several years back online real estate search systems added mapping. At first you only got a high-level view which couldn't be changed. For a while now you've been able to zoom in and scroll around. After using Virtual Earth for a while now, however, the old interface feels very constraining. The zooming and scrolling present in most real estate portals is similar to that provided by MapQuest. You have to click on buttons to zoom and to scroll and these actions are thus much less organic. It is much more natural to use the scroll wheel to zoom and to scroll around by clicking and dragging.

Yesterday morning my wife IM'd me and told me that John L. Scott was using Virtual Earth. Indeed they are and it is such a big improvement, I can't imagine using anything else. Not only can I zoom and scroll without reloading the whole page and through a natural interface, but I can also get an aerial view. Even better, when there is a bird's-eye view available, that too is offered. Imagine being able to see a close-up perspective view of your potential new house. This is so far from the line-drawn map view of old. In my mind at least, this differentiates them from their competition by a substantial margin.

For an example of the old vs. the new approach, check out this randomly selected house on Coldwell Banker and then on John L. Scott. On each page, enter MLS# 26061922 and then select Map Property.

(For the record, I have no affiliation with John L. Scott.)

Testing Systems

This is the third in my series on test harnesses. In this post, I'll talk about systems that do much more than simple test harnesses. Test harnesses provide a framework for writing and executing test cases. Test harnesses focus on the actual execution of test cases. For complex testing tasks, this is insufficient. A test system can enhance a test harness by supplying a back end for results tracking and a mechanism to automatically run the tests across multiple machines.

A test harness provides a lot of functionality to make writing tests easier, but it still requires a lot of manual work to run them. Typically a test application will contains dozens or even hundreds of test cases. When run, it will automatically execute all of them. It still requires a person to set up the system under test, copy the executable and any support files to the system, execute them, record results, and then analyze those results. A test system can be used to automate these tasks.

The most basic service provided by a testing system is that of a database. The test harness will log test results to a database instead of (or in addition to) a file on the drive. The advantages to having a database to track test results are numerous. The results can be compared over time. The results of multiple machines running the same tests can be combined. The aggregate pass/fail rate can be easily determined. An advanced system might even have the ability to send mail or otherwise alert users when a complete set of tests is finished or when certain tests fail.

It is imperative that any database used to track testing data have good reporting capabilities. The first database I was forced to use to track test results was, unfortunately, not strong in the reporting area. It was easy to log results to the database, but trying to mine for information later was very difficult. You basically had to write your own ASP page which made your own SQL calls to the database and did you own analysis. I lovingly called this system the "black hole of data." A good system has a query builder built in (probably on a web page) which lets users get at any data they want without the necessity of knowing the database schema and the subtleties of SQL. The data mining needs to go beyond simple pass/fail results. It is often interesting to see data grouped by a piece of hardware on a machine or a particular OS. The querying mechanism needs to be flexible enough to handle pivoting on many different fields.

Another feature often provided by a test system is the ability to automatically run tests across a pool of machines. For this to work, there is a specified set of machines set aside for use by the testing system. Upon a specified event, the test system invokes the test harness on specific machines where they execute the tests and record the results back to the database. These triggering events might be a specified time, the readiness of a build of the software, or simply a person manually scheduling a test.

Part of this distributed testing feature is preparing the machines for testing. This may involve restoring a drive image, copying down the test binaries and any supporting files they might need, and conducting setup tasks. Setup tasks might be setting registry entries, registering files, mapping drives, and installing drivers. After the tests are run, the testing system will execute tasks to clean up and restore the machine to a state ready to run more tests.

Having a testing system with these capabilities can be invaluable on a large project. It can be used to automate what we at Microsoft call BVTs or Build Verification Tests. These are tests that are run at each build and verify basic functionality before more extensive manual testing is done. Through the automation of distributed testing, substantial time can be saved setting up machines and executing tests. People can spend more time analyzing results and investigating failures instead of executing tests.

It is important that I note here the downside of testing systems. Once you have a full-featured testing system, it is tempting to try to automate everything. Rather than spending money having humans run tests, it is possible to just use the test system to run everything. This is fine to a point but beyond that, it is dangerous. It is very easy to get carried away and automate everything. This has two downsides. First, it means you'll miss bugs. Remember, once you have run your automated tests the first time, you will never, ever find a new bug. You may find a regression, but if you missed a bug, you'll never find it. As I've discussed before, it is imperative to have someone manually exploring a feature. Second, it means that your testers will not develop a solid understanding of the feature and thus will be less able to find bugs and help with investigation. When a system is overly automated, testers tend to spend all of their time working with the test system and not with the product. This is a prescription for disaster.

When used properly, a good test harness coupled with a good test system can save subtantial development time, improve the amount of test coverage you are able to do in a given period of time, and make understanding your results much easier. When used poorly, they can lull you into a false sense of security.

I used a Dremel to upgrade my PC

As you might recall, about 5 months ago I built a pretty cool Media Center PC.  I have been experiencing some times where the lip sync is off.  This seems to be something in the recording because if I play it back, the lip sync is off every time.  Stopping and restarting doesn't fix the problem.  I suspect it was the very old tuner I had in the system.  The solution:  get a new tuner.  As the PCIe bus is supposed to be the wave of the future, I decided to get a PowerColor Theater 550 Pro PCIe.  This tuner is based off of the ATI 550 Pro chipset which gets the best reviews.  The card was even on sale at Newegg for like $65.  It's since dropped $5 and now comes with free shipping--shoulda waited.  The card arrived yesterday and all seemed well.  Then, I went to install it.

When I tried to put the card in my computer, I noticed a problem.  The Zalman heat sink on the north bridge got in the way.  There are 2 PCIe x1 slots and the north bridge was positioned in such a manner as to obstruct *both* of them.  Despite the fan noise, I considered going back to the fan the north bridge came with but that was too tall as well.  What to do?

The solution:  cut off enough heat sink fins to allow the card to fit.  I broke out my Dremel tool, outfitted it with a cutting blade and went to town.  Now, sporting about 1/4 fewer blades than before, the heat sink fits under the PCIe card and all is well.  The card is in the machine and working.  Here's to hoping that my av sink problems go away.

After this fiasco, I started looking around.  Most motherboards out today have similarly situated x1 PCIe slots.  Almost everything I looked at either had the north bridge in the way or, worse, had the slots right next to the x16 display card slot where it would interfere with air flow and oversized fans.  To date not much is shipping in the x1 PCIe form factor.  With the current support, that trend will probably continue for a while.

Advanced Test Harness Features

In a past post I talked about the basic functionality of a test harness.  That is, it should be something that provides a reusable framework for an application, for running and reporting the results of test cases.  There is much more that a test harness can do, however.  It can provide mechanisms for lightweight test development, model based testing, and scripting.

 

Basic test harnesses like the Shell98 I spoke of last time or cppunit are what I would call heavyweight harnesses.  By that I mean that the testing code is statically bound to the harness at compile time.  In the most basic form the harness comes in the form of source code that is compiled along with the test code.  A slightly more advanced model involves a separate library that is statically linked with the test code to form a standalone executable.  This is fine but it means longer compile times, larger binaries, and potentially less flexibility.

 

There is a better way to do this.  The test harness and test cases can be separated.  The harness is compiled into an executable and the test cases are loaded by it dynamically.  I call this a lightweight harness.  As the harness no longer knows what test cases it will be tasked with executing, this requires that the tests are discoverable in some manner by the test harness.  The test cases are usually collected in dlls, jar files, or assemblies which are loaded by the harness.  The harness uses reflection (C# or Java) or a custom interface to discover which tests are in the test file.  This system is much more complex than the static binding but it offers several advantages.  It separates the test cases from the harness, allowing them to be varied independently.  It decreases compile times and reduces binary size.  It can also allow a single instance of a test harness to run many different types of tests.  With a static model, this scenario would require running many different executables.  Nunit is an example of a lightweight test harness.

 

Another feature that advanced test harnesses will have is support for model based testing.  Model based testing is a method of testing where test cases are generated automatically by the test framework based on a well-defined finite state machine.  A good description can be found on Nihit Kaul’s blog.  A test harness which supports model based testing will provide mechanisms for defining states and state transitions (actions) which it will use to generate and execute test cases.  The harness will also need to support a mechanism for verifying that the system is still correct after each transition.  Setting up model based testing usually requires a lot of work up front.  They payoff can be quite high though.

 

In a simple test harness, a test case is not provided any context in which to run.  Imagine the test cases as functions which take no parameters.  They will run exactly the same each time they are executed.  An advanced test harness will provide a mechanism to modify the test cases via parameters or scripting.  The simple method is to allow parameters to be passed to the test cases via a configuration file.  This is useful when you want several tests which vary only on a single parameter.  I wished for this feature when I was testing DVD playback.  In order to test the performance of our DVD navigator on different discs, I needed test cases to play different chapters.  I was forced to create a test case per chapter I wanted to play.  It would have been much simpler to write one test case and then allow a parameter which supplied the chapter to play.  Some harnesses might even provide a full scripting model where you could programmatically call test cases and provide parameters.  It is easy to envision embedding VB, Python, or even Lisp into a test harness and using that to control the test case execution.  The advantages to both of these methods are the ability to easily vary the test coverage without being required to program.  This makes test case creation accessible to non-programmers and saves programmers a lot of time over compilation.

 

Very advanced test harnesses provide mechanisms to execute the tests on a pool of machines.  We might call these test systems rather than test harnesses.  I’ll discuss these in another post.

 

Cryptography Podcast Series

   Security Now just concluded an amazing podcast series on the subject of cryptography.  Steve Gibson does an excellent job of making the world of crypto accessible.  No serious math is needed to understand it.  He covers everything from stream ciphers, to block ciphers, to public key cryptography and certificates.  If you have any interest in the subject, check out this six-part series:

Intro to Crypto

One Time Pads

Block Ciphers

Public Key Crypto

Cryptographic Hashes

Primes and Certificates

If you want to learn more, there are two books I recommend:

Crypto by Steven Levy - A modern history of crypto.  Levy is a great author.

Applied Cryptography by Bruce Schneier - If you want to learn how this stuff really works.  You'll need some math for this book.

Test Harness Basics

   A short while back one of my readers asked what a test harness was.  I will answer that in a pair of posts.  This first post will describe the basics of a test harness.  The second post will talk about more advanced features that a test harness might have.

   When you are writing test cases , whether they are unit tests, tests for test-driven development, regression tests, or any other kind, there are some functions you will find yourself doing each time.  These include the basics of an application, a mechanism for launching tests, and a way to report results.  You could write these functions each time you start testing a new feature or you could write them once and leverage them every time.  A test harness, at its most simple, is just that—a system to handle the elements you’ll repeat each time you write a test.  Think of it as scaffolding upon which you will build your tests.

   A basic test harness will contain at least the following elements:  application basics, test launching, and result reporting.  It may also include a graphical user interface, logging, and test case scripting.  It should be noted that harnesses will generally be written for one language or runtime (C/C++, Java, .Net, etc.).  It is hard to write a good harness which will work across runtimes.

   To run a test, the first thing you need is an application.  Each operating system has a different way to write an application.  For example, on windows if you want any GUI, you need things like a message pump and a message loop.  This handles the interaction with the operating system.  The harness will include code to start the program, open any required files, select which cases are run, etc.

   The next thing you need is a way to actually launch the tests.  Most of the time a test case is just a function or method.  This function does all of the actual work of the test case.  It calls the API in question, verifies the results, and informs the framework whether the case passed or failed.  The test harness will provide a standardized way for test cases to advertise themselves to the system and an interface by which they will be called.  In the most simple system, a C/C++ harness might advertise its cases by adding function pointers to a table.  The harness may provide some extra services to the test cases by allowing them to be called in a specified order or to be called on independent threads.

   The third basic pillar of a test harness is a way to inform the user of which cases pass and which fail.  They provide a way for test cases to output messages and to report pass/fail results.  In the most basic harness, this could be just a console window in which test cases will print their own messages.  Better than that is a system which automatically displays each test case name and its results.  Usually there is a summary at the end of how many cases passed or failed.  This could be textual or even a big red or green bar.  More advanced systems have built-in logging systems.  They provide a standardized way for the test cases to output trace messages informing the user of each important call as it is being made.  The harness may simply log to a text file but it may also provide a parsable format like XML or even interact directly with a database for result retention.

   At Microsoft, many groups write their own test harnesses which are specialized to support the unique needs of the organization.  For example, my team uses a harness called Shell98 which has, among other things, support for device enumeration.  Good examples of a freely available test harnesses are the xUnit series of tests like cppUnit, nUnit, and jUnit.  These are designed for unit testing and are not very feature-rich.  I’ve used cppUnit which is very basic and nUnit which is pretty slick.  The xUnit harnesses do not do any logging and you do not have control over the order in which tests are run.  They are intended for a user to run and visually inspect the results.  The harness I use allows for scripting and outputs its results to a database.

 

(fixed the font issue--silly cut and paste from Word)

Monthly Podcast Update

It's nearing the end of the month so I thought I'd update you on what podcasts are filling my Zen Nano these days.  Here are the ones I'm listening to regularly.

Major Nelson - Covers the world of the XBox360.  News, reviews, interviews with insiders.

This Week in Tech - Leo Laporte hosts a roundtable discussion about the technology news of the day.  Guests this month included Robert Scoble.

Security Now - Very informative show about the world of computer security.  Most of this month covered the basics of cryptography.

The Dice Tower - Best of the board gaming podcasts.  Reviews, news, and top ten lists.  Good place to become introduced to new games.  The hosts are good natured and fun to listen to.

FLOSS Weekly - Interviews in the world of Open Source Software.  Only 3 episodes old but all very interesting.  Guests so far include Cmdr Taco of /. and Ben Goodger of Firefox.

Honorable mentions (things I listened to but don't make my rotation yet):

This Week in Media - Roundtable discussion about digital media creation.  I've only listened once so far but found it intrigueing.  I'll be back for more.

HDTV Podcast - Not a lot of depth but good coverage of the news in the world of HDTV.  The whole show is less than 1/2 hour so it's easy to squeeze in.

You'll notice that Engadget fell off of my list.  They were sporadic at the beginning of the month and there's only so much I can listen to about the Motorola Q.

As always, if you have suggestions, send them my way.  I'm looking for good podcasts on the subject of audio or video production and on programming.  I haven't found any really good development blogs.  Perhaps it is just something you can't do in an audible medium.

Software Engineering - A Valid Aspiration?

I was listening to an interview with Alistair Cockburn tonight on my way home and thought he had some interesting insights into the subject of whether we should aspire to the concept of "Software Engineering."  He had three basic points which I'll relate:

First, engineering as it is conceived today, is a rather new invention.  Before 1950 or so, engineering was less math-intensive and more exploratory.  It was about invention over theory.  As Alistair puts in, after World War 2, engineering got "physics envy" and began to move heavily into theory and math.  This isn't necessarily bad (although there are definite side effects to it), but aspiring to be an engineering trade doesn't necessarily mean the heavy-theory, plan everything in advance model we seem to accept it as meaning.

Second, the term "Software Engineering" doesn't have the origin you probably think it does.  My thought when I hear this is that someone looked at software development and said, "you know, this looks a lot like engineering."  In fact, that's not what happened.  In 1968 there was a NATO conference on software development and, in order to be "deliberately provocative", they decided to use the term "Software Engineering" to describe the industry.  This was done more to force the industry in a particular direction than to reflect where it really was.

His third point is that software development is not really engineering--especially our modern view of that term.  It is rather like a cooperative game.  It is exploratory in nature.  The actions taken are two:  invention and communication.  He expands on this idea in his book, "Agile Software Development."  The advantage of thinking of it as a game is that we can see that there are not right and wrong moves but rather better and poorer moves.  You don't win a game by following a ridgid formula.  Instead, you react to the circumstances.  You need a strategic plan, but you also must remain tactical.  A good place where this becomes useful is in the concept of documentation.  It isn't that documentation is intrinsically good (ISO 9001) or bad (XP), but rather we can decide for each project how much documentation is appropriate.  Weigh the costs and the benefits and decide what to do.

Alistair has some good points here.  Similar to my post about why building software isn't like building bridges, he focuses on the naturally exploratory nature of what we do in software.  It is more about making tradeoffs and reacting to the state of the game than about making a priori decisions which will be applied to the situation in a rigid manner.

 

Part of this interview is typed up on this blog.  The official transcript seems to be here.

 

HD-DVD Launches

The first of the next-generation formats officially launched today.  As of now, you can walk into a store like Best Buy and pick up an HD-DVD player and some discs.  I know at least one person who already has one.  I'm hoping to see what it looks like shortly.

There are 2 DVD players available, the HD-A1 for about $500 and the HD-XA1 for about $800.  That is pretty cheap for a first generation format.  If I recall correctly, that's about where DVD players came out.  It might even be a bit cheaper.  Early adopters always pay a lot. 

There are a whopping 4 discs available right now:  Serenity, The Last Samurai, Phantom of the Opera, and Million Dollar Baby.  Considering a month ago there was talk of launching the player with no discs available, that's not too bad.  It looks like more discs will trickle out each week for a while.  Apparently the first discs and the first players won't have much of the advanced iHD menuing/interactivity yet.  That may have to wait for the next generation of players.

HD-DVD is shipping and BluRay is delayed.  The Samsung player will come out at the end of June now instead of the end of May.  Sony will apparently be launching movies on May 23, but no players will be available to watch them for a month.

I'm not sure quite what this means.  The videophiles are very excited.  I don't get the feeling that the next-gen-DVD war is slowing down their enthusiasm.  What do you think?  Does being out first help HD-DVD?  Will BluRay's larger cache of movies and studios turn the tide?  Does it matter?

I have a hunch that the winner of the next-gen-DVD war will be downloadable content.  More on that later.

 

Dangers of Test Automation Revisited

Here is a comment I received via e-mail in reference to my post on the Dangers of Test Automation.  I found it insightful.  The point is the oversights are easy and automation won't catch them.  Despite a few claims that this is just a deficiency in the test plan, assuming that a test plan is perfect is a dangerous assumption.  There are always going to be holes, just like there will always be bugs in the code we write.  Having manual exploration gives us insurance that we will cover these holes.  Once again, if all of your testing is automated, you are done finding bugs after your first run.

Here is the comment:

I was reading your post:

 

... and I was immediately reminded of a scene from Jurassic Park (the book, not the movie, though that was good too)

 

The park administrator is showing off the high-tech computer systems.  He says "you can search for dinosaurs... I'll just search for all of them... and the big board will show you their current locations."  Sure enough, there they all are.

 

Later it is revealed that, contrary to design, the dinosaurs are breeding.  Someone asks why the extra dinosaurs didn't show up on the computer search.

 

Nedry answers (I'm paraphrasing) "The search allows you to enter an 'expected number' of dinosaurs, which helps it to run faster.  It's a feature."

 

Premature optimization strikes again...

 

HDCP Primer

HDCP stands for High-bandwidth Digital Content Protection.  It is the encryption and handshaking protocol used by DVI and HDMI connections to allow transmission of high definition video content between devices.  In the future, this is the protocol that will be used to connect video sources like HD-DVD and BluRay players and set top boxes to your high definition television.  Ed Felten of Freedom to Tinker gives an excellent overview of the technology.  He describes it in easily-accessible language that doesn't require a Ph.D. in mathematics to understand.

For those who want full details, you can find the full specification and other documents here.

Why I'm Not Buying an HDTV Yet

   Working with video all the time, I should be an obvious owner of an HDTV set.  Alas, I'm not.  I've many times considered purchasing one but I just haven't been able to bring myself to pull the trigger yet.  Why not?  Let me explain why I'm not jumping in yet.

   First, no technology seems quite ready yet.  Each has a pretty substantial downside.  Second, there are still some changes coming that may effect the utility of what we're buying today.

   Let me run through the technologies quickly and explain why I'm not enamored with each one:

• Rear projection - Almost all of the rear projection units on the market today require expensive bulbs that are changed out every few years.  I want something that can match my CRT and not require lots of maintenance.
  
• DLP [Digital Light Projection]- Most DLPs on the market today are of the single-chip variety and use a spinning color wheel to generate the colors you see on the screen.  The problem with this is that, when things move quickly, you can sometimes spot a rainbow effect on the edges.  Like many video artifacts, once you see this, it's hard to stop noticing it.  DLP also has the bulb issue.
  
• LCD projection [Liquid Crystal Display] - There is a fairly pronounced screen door effect unless you are back far enough.  What I mean by this is that you can pick out the individual pixels.  It is like watching TV on the other side of a screen door.  LCD projection has the bulb issue.
  
• LCOS (SXRD/DILA) [Liquid Crystal on Silicon] - My favorite of the projection technologies.  It doesn't have any major shortcomings outside of the bulb issue.  It's still pretty pricey.
  
• CRT [Cathode Ray Tube] - This is your traditional TV set.  Great technology but way too heavy in bigger screen sizes.
  
• LCD - LCDs have very low contrast ratios and thus the dark areas of the screen all tend to blend together.  Trying to watch a night scene can be painful as all of the detail is lost.
  
• Plasma - Plasma has one big drawback:  Burn In.  It is, by all reports, not as bad as it once was but it is still an issue.  Perhaps it isn't when watching TV but if you want to connect a computer or a game console, you have to be really careful.

   So nothing quite does what I want yet.  On top of that, there is talk of redesigning the HDMI connector.  HDCP (the encryption protocol for HDMI and DVI) is still unproven in my mind.  Each time I read an HDTV magazine, I hear about some cable box that won't talk to some TV.  Until this is rolled out on a bigger scale, I still worry that the connections will fail to work.  Finally, 1080p is a potentially interesting format.  Some screens (like SXRD) have 1080p native resolutions but they won't accept a 1080p signal yet.  They take a 1080i signal only.  I want to wait for them to start accepting the big signals.

For a whole lot of detail on the topic of HDTV, check out the AV Science Forum.