Thursday, October 6, 2011

Steve Jobs on the Value of Saying No

I ran across a great segment of Steve Jobs talking at the WWDC in 1997 just after he returned to Apple.  Similar to my post about pruning the decision tree, he speaks about the power of saying no to the bad ideas.  "Focusing is about saying no," he says.  His analysis of what was wrong with Apple at that time was that they had terrible engineering management.  They were doing too many things--interesting things--but had no direction.  When he took over, the decisions that had to be made were not to cut things that were bad, but to cut things that were unfocused.  A lack of focus makes the whole less than the sum of the parts.  Good focus allows the whole to become greater than the sum of the parts.


Two segments are worth watching.  The first is Steve Jobs explaining his philosophy:

The second is him responding to a question about why they cut OpenDoc.  The interesting observation is that OpenDoc was probably better than anything else at some things.  That, by itself, wasn't enough.  It had to be part of a larger vision or it had to go.


 He ends with a great observation about how you have to let the vision dictate the technology and not the other way around.

Wednesday, October 5, 2011

How much did Steve Jobs Mean To the Tech Industry?


This picture says it all.  The front page of Hacker News is completely dominated by the news of Steve's death.  Even as someone who never owned an Apple product, he had a huge influence and raised the bar.  Not just once, but at least 5 major products from him changed the state of the industry.  The Apple // was a watershed for personal computers.  The Mac for UI.  The iPod and especially iTunes for digital music.  The iPhone completely changed the phone business.  The iPad created a whole new category of devices.  His influence will be greatly missed.

Wednesday, September 28, 2011

Pruning the Decision Tree in Test

Yesterday I wrote about the need to reduce the number of things a project attempted to do in order to deliver a great product.  Too many seemingly good ideas can make a product late or fragmented or both.  The same is true of testing a product.  Great testing is more about deciding what not to test than deciding what to test.

There is never enough time to test everything about a product.  This isn’t just the fault of marketing which has a go-to-market date in mind.  It is a physical reality.  To thoroughly test a product requires traversing the entire state tree in each possible combination.  This is analogous to the traveling salesman problem and is thus NP-Complete.  In layman’s terms, this means that there is not enough time to test everything for any non-trivial program.

When someone first starts testing, thinking up test cases is hard.  We often ask potential hires to test something like a telephone or a pop machine.  We are looking for creativity.  Can they think up a lot of interesting test cases?  After some time in the field, however, most people are able to think up a lot more tests than they have time to carry out.  The question then becomes no longer one of inclusion, but one of exclusion.

In Netflix’s case the exclusion was for focus.  This is not the right exclusion criteria for testing.  It is improper to not test the UI so that you can test the backing database.  Instead, the criteria by which tests should be excluded is more complex.  There is no single criteria or set of criteria that work for every project.  Here are some to consider which have wide applicability:

  • Breadth of coverage – Often times it is best to try everything a little rather than some things very deep and others not at all.  Don’t get caught up testing just one part.
  • Scenario coverage – Look for test cases which will intersect the primary use patterns of the users.  If no one is likely to try to put a square inside a circle inside a square, finding a but in it is not highly valuable.
  • Risk analysis – What areas of the product would be most problematic if they went wrong?  Losing user data is almost always really bad.  Drawing a line one pixel off often is not.  If you have to choose, prefer focusing more on the data than the drawing.  Another important area for many projects are legal or regulatory requirements.  If you have these, make sure to test for them.  It doesn’t matter how well your product works if the customer is not allowed to buy it.
  • Cost of servicing – If forced to choose, spend more time on the portions that will be more difficult or costly to service if a bug shows up in the field.  For instance, in a client-server architecture, it is usually easier to service the server because it is in one spot, under your control, rather than trying to go to hundreds or thousands of computers to update the client software.
  • Testing cost – While not a good criteria to use by itself, if a test is too expensive to carry out or to automate, perhaps it should be skipped in favor of writing many more tests that are much cheaper. 
  • Incremental gains – How much does this test case add to existing coverage?  It is better to try something wholly new than another slight variation on an existing case.  Thinking in terms of code coverage may help here.  It is usually better to write a case which tests 10 new blocks than one which tests 15 already covered blocked and 2 new ones.  It is very possible that two test cases are both great, but the combination is not.  Choose one.

There are many more criteria that could be used.  The important point is to have criteria and to make intentional decisions.  A test planning approach that merely says, “What are the ways we can test this product?” is insufficient.  It will generate too many test cases, some of which will never be carried out due to time or cost.  It is important to prune the decision tree up front so that the most important cases are done and the least important ones are left behind.  Do this up front, in the test spec, not on the fly as resources dwindle.

Tuesday, September 27, 2011

Pruning the Decision Tree

A great post by Marc Randolph got me thinking.  He tackles the question of why Netflix made the moves they made recently.  Specifically, why did they spin off their DVD option as a company called Qwixter?  The answer: focus.

What separates successful ventures from failures is choosing to do the right things.  What separates great successes from merely good ones is choosing not to do the wrong things.

When faced with a question about whether to add a feature, the decision often revolves around whether customers will like it.  This is a good first question, but stopping there leads to sub-optimal results.  It is important to ask the next question:  What can we not do by doing this?  Resources are limited.  For each feature that goes into a product, something else(quality, time, other features) comes out.  If that is forgotten, the product will end up with too many features that are good but not great and a product that feels the same.

Shipping a great product then is about the decisions about what features not to implement and what bugs not to fix.  Netflix demonstrates what it looks like to take this very seriously.  They are jettisoning what is a popular part of their company so they can focus on what they think is the future.  Only time will tell if they chose the right strategy, but one has to commend them for making a clear choice.  

Thursday, September 15, 2011

Follow my adventures at //build/

This week I'm attending the //build/ conference where Microsoft is revealing many of the details about Windows 8.  If you want to see what is going on, follow my Twitter feed at,  I'm posting some pictures, linking to summaries, etc.

Sunday, September 11, 2011

Listening to the team

There is an old saying in software that goes something like this, “Scope, Timeframe, and Budget: Pick two.”  Being a tester, I would rephrase this a little as, “Features, Timeframe, Budget, and Quality; Pick three”.  It’s usually possible to hit all three of the first choice as long as you are willing to sacrifice quality.  We’ve all seen products that do this.  They have such potential, but just don’t work.  Over the past year, there were lot of planning efforts going on around me and this was a great chance to observe different behaviors.  One pattern I have seen a few times is worth discussing.  That is managers trying to dictate all 4 aspects of a project.  There are two ways this tends to happen: the dictatorial manager and the persuasive manager.

Before we jump in, it is important to define the four levers.  Features represent the complexity of the product—how many things it can accomplish.  Timeframe is how soon the product needs to ship.  Budget is how much can be spent on the project.  For software projects, this corresponds largely with the number and skill of people that can be put on the task.  No discussion of budget can be complete without at least mentioning Brooks’ lawQuality is the ineffable attribute representing beauty, greatness, or merely suitability for a given task.  In practical terms it means how many bugs are exposed by the product.  Increasing the expectations for any of these increases the pressure on the project.  Within reason, each can be relaxed to achieve the desired levels of the other three.  It is important that managers understand that when they ask to increase features, they are trading off time or that when they ask to reduce time, they are trading off features.

The dictatorial manager is the more obvious of the two.  This is the person that just says, “You will have the following specs done by this date.”  Implied is that this will be done with a fixed number of people (never enough) and of high quality.  This is the pointy haired boss from Dilbert.  When the team raises objections, this manager is unphased.  They stick to their position.  If the team can’t get it done in the timeframe they ask for, they will find someone else who will.  It never occurs to the manager that perhaps what they are asking for is impossible.  The only good news is that this type of manager is often self-correcting.  They are not enjoyable to work for.  The best members of the team have mobility and will leave.  This slowly guts the team of the most capable people and the project slowly fails.  At a quality company this manager will be seen for who he is and removed from such responsibility.

The persuasive manager is less obvious but nearly as bad.  This is the manager who convinces their team that it can do the impossible.  In this case they don’t dictate an impossible situation, they merely get the team to agree to it.  This manager is usually a really nice guy who just can’t say no.  Instead they ask people nicely to sign up for everything marketing and upper management ask for.  Their team likes working for them, it just can’t figure out why it is so over-worked.  The quality of the project suffers as the team goes into an over-worked haze.  This sort of manager is not as quickly self-correcting.  The team does not leave except through burn-out.  They don’t blame the manager because signing up for the work was their idea.  Neither the dictatorial manager nor the persuasive manager will succeed.  Both will usually ship something close to on time because they cannot admit failure.  The quality, however, will be low.  People stop caring once they are made to attempt the impossible.  The team will be much weaker the next go-round because all of the good members leave through disgust or attrition.

A better model is to listen to the team.  Most times a manager does not have control of all of the levers.  Typically timeframe and budget comes down from on high.  A manager cannot magically hire more people.  Except through brinksmanship, they cannot cause the project to ship later.  They do, however, have control over the expect level of quality and the number of features.  A team will usually give signals if they think they are being asked to do too much.  The wise manager listens to this and adjusts plans accordingly.  This is the genius of systems like Scrum and Agile development.  A tenet of these systems is listening to the team.  Using a system of cost estimates and burndown tracking helps bring an objective view of the picture.  Likewise, if the team hesitates or balks at signing up for work, a good manager will re-evaluate.  This doesn’t mean automatically reducing the ask, but it does mean making sure you are convinced it can be done. 

If a manager senses that the workload is too much, it is their responsibility to reduce it.  As much of a manager’s job should be spent deciding what not to do as deciding what to do.  This may generate heat from upper management who wants everything in a limited time, with a fixed budget, and at the highest quality.  Helping upper management understand the situation and even taking the pain if understanding is not conveyed is the responsibility of the manager.  If failure is inevitable, it is better to fail in a place of your choosing (that with the least impact/priority) rather than risk failure on a broader scale at a later date.