Showing posts with label Programming. Show all posts
Showing posts with label Programming. Show all posts

Monday, March 15, 2010

The Complexity Hammer

I’ve been doing a lot of interviewing lately, especially of college students.  There is one tendency I see a that really separates those that are good from those who still have more learning to do.  This is the tendency of the good programmers to see elegant solutions to problems and the corollary that less skilled programmers solve every problem by adding more complexity.  Stated another way, the thing that separates the best programmers from the rest is what happens when they run into a serious issue.  In my observation, the best coders step back and look for a more elegant solution.  The less skilled coders assume that their approach is correct and that another piece of state or another special case is the best choice. 


Here is an example.  The problem has been changed to protect the innocent.  Often times I ask a question similar to the change-making problem.  That is, write a program to enumerate all of the ways to make change for a dollar.  A typical approach might look something like this.  Note, I didn’t actually compile this code so there could be typos.  If there are, I’m sure you’ll let me know.



void MakeChange()


{


  int moneyLeft = 100;


  for (int quarters = 0; quarters <= 4; quarters++)


  {


    if (quarters) moneyLeft –= 25;


    for (int dimes = 0; dimes <= 10; dimes++)


    {


      if (dimes) moneyLeft –=10;


      for (int nickles = 0; nickles <=20; nickles++)


      {


        (if nickles) moneyLeft –=5;


        for (int pennies = 0; pennies <= 100; pennies++)


        {


          if (pennies) moneyLeft—;


          if (0 == moneyLeft) print…;


        }


      }


    }


  }


}


I know what you are thinking, “That’s not the right way to solve this.”  And you would be correct.  However, I have seen a lot of people give basically this solution.  Their failure to solve it correctly the first time isn’t the point of this post.  Rather, it is their response to the problem.  If you haven’t spotted it yet, this will only work correctly for the first time down for loops.  After we get to zero, we never gain the money back from the pennies we spent during the last nickles iteration.  When I point this out, the solution is too often not to step back and re-examine the problem.  “Is there something wrong with this solution?”  Rather the typical reaction is to assume that the solution is mostly right and to tweak it.  One might think of this as a specific case of not asking the 5-Why’s.  The initial reaction is often just to reset moneyLeft at the top of the quarters loop.  When that doesn’t work, more variables are added.  The result solution looks something like this:



void MakeChange()


{


  int moneyLeft = 100;


  int moneyLeftQuarters = 100;


  int moneyLeftDimes = 100;


  int moneyLeftNickles = 100;


  for (int quarters = 0; quarters <= 4; quarters++)


  {


    moneyLeft = moneyLeftQuarters;


    if (quarters) moneyLeft –= 25;


    moneyLeftQuarters = moneyLeft;


    for (int dimes = 0; dimes <= 10; dimes++)


    {


      moneyLeft = moneyLeftDimes


      if(dimes) moneyLeft –=10;


      moneyLeftDimes = moneyLeft;


      for (int nickles = 0; nickles <=20; nickles++)


      {


        moneyLeft = moneyLeftNickles;


        if (nickles) moneyLeft –=5;


        moneyLeftNickles = moneyLeft;


        for (int pennies = 0; pennies <= 100; pennies++)


        {


          moneyLeft—;


          if (0 == moneyLeft) print…;


        }


      }


    }


  }


}


Not exactly an elegant solution and not one that is easy to get right.  There are a lot of subtle cases to think through.  Unfortunately, code like this, or code trying to be like this shows up on my white board too often.  In a simple problem such as this, it is possible to keep all of the cases in your head and get it right.  When the problem becomes larger, however, this is no longer the case.  The programmer with the above solution will fail.  Thus the solution above is not an acceptable answer even though is technically solves the problem.


If one takes a few moments to step back and re-examine the problem, it is easy enough to see that one doesn’t need to track the amount of money left.  It can be trivially calculated when necessary.  This is just a specific case of the principle that one should never keep state that can be calculated.  Tracking such state provides no benefit and offers the possibility that it will differ from the real state.  The solution might look like this:



void BetterMakeChange()


{


  for (int quarters = 0; quarters <= 4; quarters++)


  {


    for (int dimes = 0; dimes <= 10; dimes++)


    {


      for (int nickles = 0; nickles <=20; nickles++)


      {


        for (int pennies = 0; pennies <= 100; pennies++)


        {


          if (100 == (quarters*25 + dimes*10 + nickles*5 + pennies)) print…;


        }


      }


    }


  }


}


Much more elegant.  Fewer state variables and thus less to get wrong.  All of this stems from the idea that one need not track state.  There is no reason to keep a running total of the money.  It’s all readily available at any moment.  It is this key notion that one needs in order to come up with a much improved algorithm.  As long as the programmer doesn’t step back and question the need for tracking how much money has been used/how much is left, they will be stuck adding complexity on top of complexity.  This is a prescription for failure.  This is not an isolated case.  Now that I have noticed this tendency, I can often spot it in interviews or even in code reviews.  The moral of the story:  always look for the elegant solution first.  Can the problem be solved by eliminating something or by looking at the problem differently?  Only once you have eliminated these possibilities should you add more state.  Adding state isn’t always the wrong solution, but it can be a crutch to avoid deeper thinking.


A few notes:


The initial paragraph isn’t quiet accurate.  The best programmers often see the elegant solution up front and get themselves into such trouble much less often.


The final solution is not optimal.  I know this.  Optimizing it would not benefit the example.

Monday, November 23, 2009

A Taste of Stack Overflow DevDays

If you missed Stack Overflow DevDays, there is some audio from it available on Stack Overflow Podcast #71.  I wish there was a longer version of this.  It’s only about 1/2 hour of outtakes from the conference, but it is still interesting to hear.  These snippets are followed by a long discussion with some of the speakers.  The conversation rambles and the audio quality is poor so feel free to stop listening after the conference outtakes.

Thursday, November 19, 2009

Design Patterns Are Not Outdated

A comment left on my answer to a question over on Stack Overflow has me a little worked up.  I've seen this meme come out of programmers more and more and it just doesn't seem accurate.  The statement goes something like this, "Design Patterns were only useful because C++ (or Java) was so broken."  The implication is thus that design patterns belong in the dustbin of history now that we have moved on to more enlightened languages like Python or Ruby.  In this particular instance the commentor was talking about the strategy pattern.  His (?) assertion was that the need for the strategy pattern is not present in a language with first class functions.  My response is three-fold.  First off, this argument is historically ignorant.  The design patterns came as much from Smalltalk as from C++.  Second, it is not strictly true.  First class functions alone don't obviate the need for the strategy pattern.  Finally, providing an alternative implementation does not make the first implementation bad.


The argument that design patterns generally are due to flaws in C++/Java are historically innaccurate.  Design patterns (like much of modern programming) originated in the Smalltalk community.  Smalltalk is dynamic.  It has first-class functions.  Most of the flaws pointed to in C++ which "modern" languages solve did not exist in Smalltalk.  The patterns, then, have value independent of that flaws and the accused language.  For instance, the Strategy pattern is covered not just in the Gang of Four (where it happens to have a C++ example), but also in the Design Patterns Smalltalk Companion which points out that it is used in Smalltalk in the MVC framework at the heart of Smalltalk's GUI.  The controller is a strategy pattern.  It is also used ImageRenderer and a few other examples.


First class functions do not, by themselves, obviate the need for a strategy pattern.  This is more a nit than a real argument, but it remains true nonetheless.  To say a language has first class functions means merely that functions can be passed as data types.  It does not necessitate that the language implements closures.  Closures are a way of capturing the context of the function.  They can be used to retain state between calls to a particular function instance.  Without closures, there is no state.  Without state, many of the strategy pattern uses fail.  Consider for a moment using a strategy pattern to encapsulate encryption algorithms.  Without closures (or objects), you would have to pass the encryption key to the function every time it was used.  This is possible, but not terribly elegant.


The existence of an alternate implementation does not make the original implementation any less useful.  The fact that I can get much of the power of OO out of first class functions and closures does not mean OO is now of no value.  There are advantages to both techniques.  Empirically there does appear to be power in OO that is not captured (easily) by purely functional languages.  Most successful functional (or psuedo-functional) languages have adopted OO features eventually.  See Python, Ruby, Common Lisp, Scala, etc.  All added or have object-oriented features.  Let us return to the example of the strategy pattern.  Is its utility obviated by the use of first class functions plus closures?  In many cases it is.  Certainly it could be in the encryption example.  On the other hand, strategies are often more complex than mere functions can express.  The controller in MVC is a strategy.  In anything but a toy application, the controller will consist of multiple functions.  Sure, one could create these functions with the same closure and thus share state, but is that really a superior model?  I would argue that it is not.  In this case it would seem a less clear mechanism because the fact that the functions are tied together is less discoverable.  OO languages and functional languages each do things differently.  Things that are easy in one are more difficult in the other.  Neither is superior in all respects.


It should be noted that when I say "design patterns" above I am referring to it in the common sense of the object-oriented programming design patterns made popular by the Gang of Four book.  In a more general sense, each language has its own set of patterns and these can also be thought of as design patterns.  Some of the OO patterns are specific to OO languages and the need doesn't translate to functional languages.  Others are more "in the large" and likely translate well to all languages trying to solve big problems.  It is my claim, however, that most of the GoF patterns will be useful in any OO language.  They are not artifacts of particular implementations such as C++ or Java.

Wednesday, October 21, 2009

StackOverflow DevDays

I spent the day at Benaroya Hall for the 1st (annual?) StackOverflow DevDays conference.  Overall eight speakers took the stage on topics from .Net MVC to Python to the Google App Engine.  The room appears to hold just over 500 people and it was filled to capacity with programmers.  There were some vendors in attendance including Amazon.com, Fog Creek Software, and someone showing off HexBugs.


The day started off with a short video titled Scrumms which was a funny spoof on life at Fog Creek and the StackOverflow podcast.  It was quite entertaining.  I hope they release it on the web after the conferences complete in a few weeks.


Joel Spolsky was the first speaker.  I always enjoy reading and listening to him.  This speech was not a disappointment.  He was as entertaining as ever.  The subject matter under discussion was that of design elegance.  He began by pointing out that software often gives the user too much choice.  Often times it interrupts the user’s work flow to ask a question most users are unprepared to answer.  Other times there are options pages with myriad options which no one could know enough to actually use.  What is a “trusted user” on GMail anyway?  He cited a study where one store put out a lot of varieties of jam (24?) for people to try.  Many did, but substantially fewer actually purchased than when the same store put out only a half dozen varieties.  People are intimidated when given too much choice.  Joel recommended the book, The Paradox of Choice.  He then went on to talk about the simplicity offered by companies such as 37 Signals whose products do only one thing, but do it well.  He argued that this isn’t the solution.  Customers will demand more features and to grow, a company must grow its feature set.  More sales leads to more features.  Choice must be offered then, but how to do it in a way that doesn’t alienate the customer?  The solution Joel offered up was to make sure the choices support what the user is doing.  The users should be modeled and choices aligned along the path of their behavior.


Joel was followed by Scott Hanselman who gave an overview of the ASP.Net MVC framework.  This is a web framework built on top of the ASP.Net framework, but which exposes much more direct control to the programmer.  For instance, they now have direct control of their URL’s (yeah!).  Scott was an entertaining speaker although I think he was a bit too self-deprecating about Microsoft.  He spent most of the talk showing the audience various features in existing and upcoming Visual Studio products which make ASP.Net MVC programming easy.


Next up was Rory Blyth talking about iPhone development.  He was an engaging speaker who obviously knows a lot about what he is doing.  I had never looked at iPhone or Objective C development before.  I can’t say I’m terribly impressed.  The tools like adequate, but aren’t as good as Visual Studio or even Eclipse.  Objective C looks like a mishmash of C and Smalltalk.  Rory described learning to develop for the iPhone as the Stockholm Syndrome where you eventually come to love your oppressor.  The iPhone is an attractive target to develop for from a business perspective (maybe), but the SDK doesn’t appear to be the reason people are flocking to it.  One highlight at the end was when Rory showed Novell’s MonoTouch which allows for C# development targeting the iPhone.  This looks like a slick environment even if it is a little pricey.


Following Rory came Joel again with a sales pitch for FogBugs 7.  I have to say I was impressed with the empirical scheduling.  There is a new plugin called Kiln for code reviews which looks alright, but I think I prefer the UI from Malevich better.  For instance, the comments didn’t appear inline with the text when they were being made.  They did later when the submitter saw them though so perhaps I just missed something as Joel blazed through the UI.  If I were a startup, I would definitely consider using FogBugz to handle my planning and bug tracking needs.


Lunch was catered by Wolfgang Puck and was reasonably good consider it was a conference and the whole thing only cost $99.  They divided up the tables into discussion topics (great idea!).  I went to a table about programming languages, but others included agile methods, startups, and a few I forgot.


The first speaker after lunch was Cody Lindley from Ning who was talking about jQuery.  jQuery has been on my list of topics to learn about, but never made it to the top of said list.  It’s fascinating technology, especially considering I just read Crockford’s Javascript:  The Good Parts recently and got a feel for the language as it was meant to be used.  For those that don’t know, jQuery is the most popular Javascript framework right now.  It runs on 35% of all Javascript pages on the web and 21% of all web pages of any kind.  It’s primary use is to make manipulating the DOM (the page structure) much easier.  Boiled down it lets a programmer easily select some portion of the page and apply an effect to it.  In implementation, it appears to be a Javascript function that wraps up a collection of page elements into a class which then provides methods for manipulating the set.  Cody used jsbin.com for his demo which appears to be an online javascript testing and debugging tool.  It looks very nice.


Daniel Racha from Nokia was next up to talk about Qt (pronounced "cute" not "Q-T").  This is a cross-platform UI toolkit and development platform recently purchased by Nokia.  It is also the basis of the K Desktop Environment (KDE) for Linux and is where Webkit originated.  Webkit is the rendering engine that powers Safari and Chrome.  Nokia’s plan is to use Qt as the basis for its phone app development story.  The technology and the tools are both mature and highly capable.  Daniel did a good job selling the merits of Nokia’s tool chain.  Considering the toolkit supports 6 platforms right now (Windows, Mac, Linux, plus several phone OS’s), I can see how this might be the way for cross-phone applications to be written.  Daniel also mentioned the Nokia N900 which apparently is completely open source to the point where end users could upload their own OS.  I can foresee 3rd party variants like those created for the Linksys routers.  This could be an interesting challenge to Apple’s iPhone strategy.


Ted Leung from Sun came to talk about Python.  This is another language I haven’t had time to learn yet.  His slides were terribly hard to read (purple on black—seriously?), but the content was good.  He gave a quick overview of the language basics and then talked about more advanced features like destructuring, generators (simple continuations), decorators, and extensions.  Python has definitely taken a lot from the world of functional programming.


Dan Sanderson was next up and gave an introduction to the Google App Engine.  It looks like an interesting way to build out scalable web sites.  This is the competitor to Amazon’s S3 and Microsoft’s Azure.  It supports the Python and Java virtual machines and so sites can be implemented in Python, Java, or any language that targets the JVM (Clojure, Scala, JRuby, etc.).  With that kind of support it would seem an app programmed to it would be capable of being moved, but that would not be the case.  The app engine is a very different sandbox to play in.  The database is non-relational and doesn’t support SQL (or at least fully SQL), the filesystem is different, there is no direct network access.  In short, once an app is written to the App Engine, it won’t easily run anywhere else.  The environment looks intriguing, but you are putting your company’s fate into Google’s hands.


The day ended with Steve Seitz from the University of Washington talking about some of the advances in 3D image extrapolation.  Steve is behind much of the technology that became PhotoSynth.  Steve’s talk was light on programming content but high on “Wow!” factor.  This was a great way to end the day.  I’m not sure my mind could have taken another heavy talk.  The stuff Steve showed us was mind blowing.  They are able to take regular photographs, process them, and recreate the 3D scene.  Not only that, but newer technology allows for a walkthrough of the site and even full texturing.  You can see it here.  The best is this one:





Overall the day was well spent.  I got to learn about a lot of cool new technologies and renew my excitement for programming.  Now I just have to pick one and find the time to learn it.  If the event takes place again next year, I definitely intend on going.  Thanks to Joel Spolsky and the folks at Carsonified for putting on such a great conference.


For more, find other reviews on Meta.Stackoverflow or watch the Twitter stream.


 

Wednesday, May 27, 2009

Five Books To Read If You Want My Job

This came out of a conversation I had today with a few other test leads.  the question was, “What are the top 5 books you should read if you want my job?”  My job in this case being that of a test development lead.  At Microsoft that means I lead a team (or teams) of people whose job it is to write software which automatically tests the product. 

  • Behind Closed Doors by Johanna Rothman – One of the best books on practical management that I’ve run across.  1:1’s, managing by walking around, etc.
  • The Practice of Programming by Kernighan and Pike– Similar to Code Complete but a lot more succinct.  How to be a good developer.  Even if you don’t develop, you have to help your team do so.
  • Design Patterns by Gamma et al – Understand how to construct well factored software.
  • How to Break Software by James Whittaker – The best practical guide to software testing.  No egg headed notions here.  Only ideas that work.  I’ve heard that How We Test Software at Microsoft is a good alternative but I haven’t read it yet.
  • Smart, and Gets Things Done by Joel Spolsky – How great developers think and how to recruit them.  Get and retain a great team.

 

This is not an exhaustive list.  There is a lot more to learn than what is represented in these books, but these will touch on the essentials.  If you have additional suggestions, please leave them in the comments.

Tuesday, May 12, 2009

Some Programming Languages to Consider Learning

Learning a new programming language can affect the way you think.  While most modern languages are Turing Complete and can theoretically all accomplish the same things, that’s not practically true.  Each language has its own strengths of expressiveness.  For instance, trying to write dynamically typed code in C++ is possible, but a pain in the neck.  You would have to implement your own type system to do so.  Each language makes certain things easy and other things hard.  Learning different languages then exposes you to different approaches.  Each approach provides a different way of thinking and a set of tools supporting that way of thinking.  What follows are some of the languages I’ve learned and what I think they provide.  This list is limited to languages I’ve studied in at least a little depth.  There are many more languages out there that may be useful.  If you have additional suggestions, please make them in the comments.



  • C – This is coding very close to the metal.  Learning it promotes an understanding of memory, pointers, etc.

  • Lisp/Scheme – Once you get past your parenthesis-phobia, it’s a very elegant language.  The big learnings here are treating code as data, metaprogramming (writing programs that themselves write programs), and recursion.  Lisp is also a dynamically-typed language.

  • Clojure – A variant of Lisp that runs on the JVM.  In addition to what you can learn from Lisp, it adds persistent data structures and transactional memory.  Persistent data structures are ones that *never* change.  Once you have a “pointer” to one, it will never change underneath you.  This makes parallel programming much simpler.  Clojure also is more of a functional language than Lisp/Scheme.  It is not purely functional, but allows for the functional style to be followed more easily.

  • Smalltalk – Much of modern programming originated in this language.  Modern GUIs are based on the Xerox Parc Smalltalk systems.  Object Oriented programming was first popularized in Smalltalk.  Extreme Programming, Agile, and Design patterns all found their initial formulations in Smalltalk.  In addition to learning about OO programming, Smalltalk is great to understand message passing.  This gives object-oriented code a different feel than the function call semantics of C++/Java/C#.  Smalltalk is also a dynamic language.

  • Perl – Perl was once known as the duct tape of the internet.  It ran everything.  It has since been surpassed (at least in blogosphere popularity) by other scripting languages like Ruby and Python.  The biggest thing to learn from Perl is regular expressions.  They are built into the core of the language.  Other languages support them but often as a library.  Even those that do support them in the syntax do not usually utilize them so pervasively.

  • C#/Java – These languages both solve the same problems in almost the same ways.  They are a great place to learn object-oriented programming.  It is built in from the ground up.  The OO style is one of function calls and strong interfaces (which distinguishes it from Smalltalk).  These languages also have the largest accompanying libraries.

Saturday, February 28, 2009

10 Papers Every Programmer Should Read

I’m always on the lookout for good reading material.  Michael Feathers over at ObjectMentor has served up a great post entitled 10 Papers Every Programmer Should Read.  I intend to.

Monday, February 16, 2009

Check Out Stack Overflow

I’ve recently become quite addicted to the website stackoverflow.com.  It is a joint venture between Jeff Atwood and Joel Spolsky.  There is an accompanying podcast if you want to hear about the creation process.  The site itself is a question and answer site for programming questions.  Want to know how to do a simple Perl hash equivalency comparison?  Ask.  Want to find the best book on C# for experienced programmers?  Ask.  There is quite an active community and most questions are answered in short order.  You don’t even have to sign up to ask your first question.

If you want to stick around longer than one question, you can answer questions and earn reputation for doing so.  Greater reputation means more abilities on the site.  At one level you can change the tags on questions.  At another level you can vote to close questions.  Still more reputation and you can actually edit the text of questions.  Reputation is granted by users voting for the best answers and questions.  It’s amazing how addicting it can be to try to raise an arbitrary score.

The site has only been open for  a few months and already it is a treasure trove of knowledge.  Joel recently stated that the site gets something like 2 million uniques a month.  As I write this, there are approximately 90,000 questions that have been asked.  Almost all have answers.  This is crowd sourcing at is best.  Once people start linking to it in large numbers, expect to see it shoot up the rankings of programming-related searches.

There are downsides to Stackoverflow’s popularity.  Questions don’t stay on the front page for long.  I suspect they will have to create sub-pages for different topic areas the way Reddit did with its subreddits.

Friday, December 5, 2008

Improving Task Estimation Accuracy

In my opinion estimating how long it will take to write a piece of software is difficult if you haven’t done it before and with software we never have.  The more experience you have, the more you’ll have done similar things and thus the more accurate your estimates will be.  To help build this experience faster, it is important to track how long things actually take. 

Many software developers just start working and don’t really plan.  They have a vague idea what they want to accomplish, but haven’t really thought it through.  This inevitably leads to the task taking longer than expected.  There is a great discussion of this on the Stack Overflow Podcast.  Check out episodes 5 and 16 for the before/after.  There are transcripts available if you don’t have time to listen.  The reason why things take longer is because without careful planning, there are a lot of pieces of work that aren’t taken into consideration.  Basically the programmer makes an unfounded assumption that he doesn’t have to do some work that indeed he does.  In my experience, this is one of the largest contributing factors to things taking too long.

The good news is that most of these unconsidered items can be teased out if sufficient thought is given to the problem.  They are known unknowns.  The questions are usually known, just not the answers.  There are techniques for doing this. 

The simplest is breaking down the problem into small pieces.  On our projects we never let items get bigger than 5 days of work.  This is probably too large.  If you are estimating in chunks larger than a day or two, you are probably glossing over important details. 

Writing a spec in some level of detail is also important.  Be honest with yourself and explain how things will get done, not just that you need to do them.  “Break class Foo into two classes” may not be enough detail.  What are the two other classes going to do?  How will they work together?  My rule of thumb is that a spec isn’t done until someone could pick it up cold and implement a reasonable facsimile of what you intended.  If they can’t, it is too vague.

For estimation of larger projects, there is a process called Wideband Delphi estimation that can help to flush out the underlying assumptions.  It is very lightweight and can be done by a few people in a couple hours for many projects.

Once the task is broken down into small segments and these segments are estimated, it is important to track how long each item really took.  Be honest.  You are only cheating yourself if you report a 3-day work item as having taken 1 day.  This helps to build experience and make better estimation possible in the future.

One more thing to consider is whether you are staying on task.  When it isn’t something we failed to consider, the next highest cause of under-estimation is distraction.  How much time are you actually spending on a project?  If you spend an hour a day working on it and the other 7 hours doing e-mail, meetings, surfing the web, talking to people, etc. you won’t finish on time (unless you account for an 87.5% buffer).

Speaking of buffering, it is my experience that it is easier to estimate uninterrupted time than to try to estimate how much time all of the distractions will take.  On our team we estimate each work item in a vacuum.  This will take 1 day of coding and debugging time.  Meetings, vacations, e-mail and other distractions all fall into another category we call “buffer.”  We usually estimate 30% of an average person’s time will be spent in this buffer state.  This varies with the project of course.  Separating the two allows us to better see where the time is being spent.  It is a much different issue if someone takes 5 days actually working on what was estimated as a 2-day problem than if they take 2 days coding but are distracted with other items for 3 days.

Wednesday, November 26, 2008

Using Perl for Mass In Place Editing

Have you ever wanted to update some text in a bunch of files all at once without a hassle?  I had reason to do this recently and turned to Perl for my solution.  Once I found the implementation quirks, it turned out to be quite easy.  This functionality is pretty well documented, but not for Windows.  There are some gotchas trying to do it there.  What follows is what I found to work.


I had a set of files that were database queries for a particular milestone in our product release.  Let's call it M1.  I needed to update them all for a different milestone that we'll call M2.  Each file was an XML version of a query so text parsing seemed like the logical course of action.  Perl is good at text processing so I went that route.  First off I had to install Perl.


Perl is capable of accepting a single line of code at the command prompt so there is no reason to actually author a perl script.  The command for this is -e (execute?).  To change something inline, you can use the -i (in place) command.  The command looks something like this:



perl -pi.bak -e "s/M1/M2/g" file.txt


The -i.bak means to rename the original files with the .bak extention.  In theory -i alone will delete the originals but ActivePerl wouldn't accept this. 


The -p tells perl to run the command over every line of standard input (i.e. over every line of the file).


The "s/M1/M2/g" command is a regular expression telling it to substitute M2 for M1 globally.  It could be any regular expression.  Note that most examples of this online use only single quotes ( ' ), but this doesn't work on Windows.  One hint:  If the command fails, try adding -w to the command line to generate warnings. 


The above command will change all instances of M1 to M2 in file.txt.  What I wanted to do was to replace it in every file.  Simple, I'll just change file.txt to *.*.  Sorry, no dice.  ActivePerl doesn't accept this nor does it accept *.  Time for some more command-line action.  There is a for command that can be utilized at the cmd prompt which fits the bill.  Use it like so:



for %i in (*.txt) do perl -pi.bak -e "s/M1/M2/g" "%i"


This command will iterate over all the files (*.txt) and execute the command following the do.  You have to quote the trailing %i because filenames might contain spaces.


There you go, quick and dirty text replacement from the command line.  Note that perl regular expressions are capable of much more than simple search and replace.  You can use this technique to accomplish anything they can.


Is there an even simpler way to do this?  There probably is.  If so, please let me know in the comments.


 


12/4/08 - Updated with -p thanks to Maurits.

Monday, June 2, 2008

Refactor To Make Immediate Change Easier

Jeremy Miller gives his reasons for refactoring.  He gives two over-arching reasons.  The first is "To remedy a deficiency in code, design, or architectural quality" and the second, "To make a forthcoming change to the code easier."  I'm very much in agreement with the second, but the first makes me nervous.  Refactoring is not an end to itself.  Changing code to make it of higher quality without an immediate purpose in mind strikes me as wasteful.  Code doesn't need to be pretty until it is being maintained.  The compiler doesn't care how ugly the code is.  The reason to make code of higher quality is because someone will may have to change that code and it will be hard to change in its low-quality state.  However, if change is not imminent, cleaning up the code now is a non-optimal decision.  Suppose the code continues working and no change is necessary in that part of the code base.  Because the compiler doesn't care and no human looks at it, the effort to make it higher quality is wasted.  Instead, it is better to wait until the change is needed to make the updates.  This would seem to be in line with the Lean principle of waiting until the last responsible moment or just in time delivery.  It is better to wait than to refactor pre-emptively.  Refactoring can become a yak-shaving exercise if one is not careful.  To be fair, Jeremy does end with this advice:  "If a structural problem in the code isn't causing any immediate harm or friction, you probably leave it alone for now."

Thursday, May 15, 2008

Design Principle: Don't Repeat Yourself

There's a design principle I neglected to mention in my initial list but which certainly merits attention.  That principle is this:  whenever possible, don't repeat yourself (DRY).  Put another way, do things one time, in one place rather than having the same or similar code scattered throughout your code base.

There are two main reasons to follow the DRY principle.  The first is that it makes change easier.  The second is that it helps substantially when it comes time for maintenance.

I was once told by an instructor in a design patterns class that the Y2K problem wasn't caused by using 2 digits to represent 4.  Instead, it was caused by doing so all over the place.  While the reality of the situation is a little more complicated than that, the principle is true.  If the date handling code had been all in one place, a single change would have fixed the whole codebase rather than having to pull tons of Cobol coders out of retirement to fix all the business applications.

When I first started working on DVD, I inherited several applications that had been written to test display cards.  They had started life as one large application which switched behavior based on a command-line switch to test various aspects of DirectDraw that DVD decoders relied upon.  Some enterprising young coder had decided we should have separate executables for each of these tests so he made 6 copies of the code and then modified the command-line parser to hard-code the test to be run.  The difficulty here is that we now had 6 copies of the code.  Every time we found a bug in one application, we would have to go make the same fix in the other 5.  It wasn't uncommon for a bug to be fixed in only 4 places.

Shalloway's Law:  “When N things need to change and N>1, Shalloway will find at most N-1 of these things.”

This principle applies to everything you write, not just to copying entire applications.  When you find yourself writing the same code (or substantially the same code) in 2 or more places, it is time to refactor.  Extract a method and put the duplicated code in that method.  When the code is used by more than one application, extract the code into a function call that you put into a shared library.  This way, whenever you want to change something, a change in one place will enhance all callers.  Also, when something is broken, the fix will automatically affect all callers.

Note that this is a principle and not a law.  There are times when substantially similar code is just different enough that it needs to be duplicated.  Consider the alternatives first though.  Can templates solve the problem?  Would a Template Method work?  If the answer is no to everything, then duplicate the code.  You'll pay the price in increased maintenance, but at least you'll be aware of what you are getting yourself into.  It might not be a bad idea to put a comment in the code to let future maintainers know that there's similar code elsewhere that they should fix.

Monday, April 28, 2008

Prefer Composition Over Inheritance

It's probably about time to bring my "Design Principles To Live By" series to a close.  This is the last scheduled topic although I have one or two more I may post.

Let's begin with some definitions:

Composition - Functionality of an object is made up of an aggregate of different classes.  In practice, this means holding a pointer to another class to which work is deferred.

Inheritance - Functionality of an object is made up of it's own functionality plus functionality from its parent classes.

For most non-trivial problems, there will be similar code needed by multiple classes.  It is not a wise idea to put the same code in more than one place (a topic for another day).  There are two strategies in object-oriented programming which attempt to solve the problem of duplicate code.  The one most popular in the early days was inheritance.  Shared functionality was implemented in a base class which allowed each child class to inherit that functionality.  A child would just not implement foo() and the parent would do the work.  This works, but it is not very flexible.

Suppose that the shared functionality is some kind of encryption algorithm.  Each child class will only inherit from one base class.  What if there is a for different encryption algorithms?  It would be possible to have multiple base classes, say AESEncryptionBase and DESEncryptionBase, but this necessitates multiple copies of the child classes--one for each base class.  With more than 2 base classes, this become untenable.  It also becomes very difficult to change out the encryption routine at runtime.  Doing so means creating a new object and copying the contents of the old object to it.

Another difficulty is the distortion of otherwise clean class hierarchies.  Each child should have an "is-a" relationship with its parent.  Is a music file and AESEncryptionBase?  No.  Here is a particularly telling examples from Smalltalk.  In Squeak (the dominant open-source Smalltalk implementation), Semaphore inherits from LinkedList.  Is Semaphore a linked list?  No.  A linked list is used in the implementation, but a sempahore is not a specialization of linked lists.

A better approach is to contain the new functionality via composition.  A class should contain instances of objects it needs to utilize functionality from.  In the music file case, it would have a pointer to an EncryptionImpl class which might be AES, DES, or ROT13.  The class hierarchy will stay smaller and the music file implementation does not even need to be aware of which encryption method it is using.  In the Semaphore case, Semaphore would contain a LinkedList object which it would use to do the work.  Clients of Semaphore would not be expecting LinkedList functionality.  Extraneous methods would not need to be disabled.  Composition would also allow for more flexibility later.  If an implementation based on a heap or a prioritized queue were found to be advantageous, they could be without clients of Semaphore knowing. 

Think twice before inheriting functionality.  There are times when it is a good idea such as when there is a logical default behavior and only some child classes need to over-ride it, but if the intent is to utilize the functionality rather than expose it to child class callers, composition is almost always the right decision.

Thursday, March 27, 2008

Don't Blame the Compiler

It's been a busy week without much time to blog.  In the mean time, here's a good reminder.  It's Always Your Fault.  Jeff Atwood explains why it's always the wrong idea to blame your tools.  New programmers especially see something they don't understand and assume that the tools (compiler, OS, libraries, etc.) are at fault.  While this is sometimes indeed the case, it is so rare that it's not worth considering.  Even if you do see an issue in the compiler or OS, you are unlikely to be able to get it fixed so you'll have to understand the issue and work around it anyway.

Wednesday, March 19, 2008

Which Language To Learn Next?

Once you have mastered one programming language, it is a good idea to branch out and learn some others.  Each language is optimized for different things.  Knowing multiple will give you a larger set of tools one of which which may solve the problem at hand better than others.  Additionally, and more importantly, different classes of languages approach problems very differently.  Learning languages from other groups will expand your thinking even if you stick to one language for most of your programming tasks.  C++, Java, and C# are all from a similar language class.  Languages like Lisp or Smalltalk are not.  Learning those requires learning new ways of thinking.  Christopher Diggins from Dobbs Code Talk presents a list of languages he recommends learning and what to get from them.  If you are looking to expand your horizons, check it out.

Monday, February 25, 2008

Classes Should Exhibit High Cohesion

This is part 4 of my ongoing Design Principles series.

When designing the classes in your model, it is important that they each have a specific role to play.  Cohesion means joining together to form a united whole.  Classes and methods should be highly cohesive.  They should have a single function.  Methods on a class should be single function and together the methods on a class should all work toward accomplishing the same purpose.  Adding an add method to a string class is not being cohesive.  Having a "utility" class isn't either.  Classes (or methods) that try to play more than one role are harder to use, harder to test, and harder to change.  The reason is side effects.  If a class does many things, there is a high likelihood that using one part will affect how another part works. 

It is hard to understand the use model of a multifunctional class.  A class with a purpose has an obvious place in a design.  A class without a purpose implies less of its use pattern and is therefore easier to mis-use.  Classes with multiple roles to play also have more complex interfaces.  Each role subtly influences the design of the other roles and may make the use model more complex than necessary.  Simple, self-documenting use models are going to result in the cleanest code with the fewest bugs.

It is important in testing to make sure that one method call on a class doesn't impact other method calls.  If the number of methods on a class gets large, the combinatorial explosion forced on test becomes unmanageable.  Likewise, every time a change is made, all methods on a class must be tested to make sure that they didn't break.  The more methods--and the more disparate the methods--the more burdensome this becomes for test.  That means longer test passes, less new exploratory testing, and a poorer product. 

Most importantly, complex multi-use classes will be difficult to change.  The code underneath will be interconnected and a change that should affect only one role may end up triggering a cascade of changes that impact all of the other roles.  Tradeoffs may be required.  What should be a simple change may mean untangling various pieces of internal state.  It is hard to be comfortable with a change to a complex class.  Unit tests will ease the discomfort, but there is so much that could potentially break that one can never be sure something subtle didn't just change for the worse.

How can a poor design be recognized?  Here are some questions to consider:

  • Is the name generic?  If it doesn't imply a single role, the class may be trying to do too much.
  • Conversely, can this class be given a simple name?  If naming is difficult, the class is too complex.
  • Classes with a large number of methods often are trying to do more than one thing.  Consider breaking them up.
  • Classes should have only one reason to change.
  • Methods that take a lot of options are generally poorly designed.  If a method checks a state and then branches right at the beginning, it should be more than one method.

DoSomething(enum Type, int Data)
{
    switch(Type)
    {
           type1: do something
           type2: do something else
    }
}

Is better written:

DoSomethingType1(int Data)
{
    do something
}

DoSomethingType2(int Data)
{
    do something else
}

Tuesday, February 19, 2008

Is There Value In Code Uniformity?

There seem to be two opposing views of coding standards.  Some think that they should enforce uniformity in code.  Everyone should use K&R braces, leave two lines between functions, and have a space after the if and before the opening parenthesis.  Others (like me) think that there is little value in uniform-looking code.  The value of coding standards is in their ability to stop bugs from getting into the code.  Patterns that are error-prone, like failing to create a new scope { } for the result of an if statement, should be banned.  Others that are odd but not necessarily error-prone should not.  I'm interested in hearing from you, my readers, on your experience here.  Is there more value in having uniform code than I give it credit for?  Does it really make reading code that much easier?  Do bugs really stand out more or is it just format deviation that stands out?


 


2/20 - Ugh, my spam filter stopped every response to this.  Unblocked now.

Monday, February 11, 2008

Arc Is Out

Over 6 years ago Paul Graham told the world that he was working on a new programming language called Arc.  It would be some derivative of Lisp, but otherwise not much was known about it.  Graham is the author of 2 books on Lisp and a popular series of essays on topics ranging from programming to startups.  As he had been mostly quiet about Arc for such a long period of time, many people have forgotten about it or dismissed that it would ever come to light.  Two weeks ago Paul announced the initial version of Arc and made it public.  I haven't had a chance to look at it yet but I intend to.  If you are interested, you can find information at arclanguage.org.  There seems to be some excitement building around it.  Only time will tell if it can build a user base or if it fades like so many other new languages.

Monday, January 21, 2008

Design to Interfaces

This is the 2nd article in the Design Principles to Live By series.


An interface is--for the purposes of this post at least--an abstract definition of the functionality of an object.  It is the signature of a class divorced from its implementation.  Think of it as the list of methods (and properties) that define a class.  Concrete examples are COM interfaces, Interfaces in C# or Java, or even just classes with at least one (but preferably all) pure virtual function in C++.  An interface need not be structural.  In late-binding languages like Smalltalk, it can just be an agreed-upon list of methods that all classes of a certain type will implement.  In the simplest form, it can just be a list of function prototypes in a C .h file.  The important part of an interface is that it abstractly describes a unit of functionality.  It forms the contract between the class implementing it and the class calling that implementation. 


The use of an interface allows client software to be written without a reliance upon or even knowledge of the underlying implementation.  This yields two primary advantages.  First, it allows for parallel development.  Second, it provides for more flexibility in the design.


If a project is being worked on by more than one programmer, interfaces are essential.  The work between multiple coders is usually partitioned into discrete chunks and each person is responsible for one or more of these.  Each of these units will have to interact with the other units in some way.  The wrong way to code is to wait for the other person to finish their work before beginning yours.  If Programmer A is writing some functionality that Programmer B needs, B will have to wait for A to have at least a prototype ready before he can being his work.  However, if programmer A designs an interface, he has created a contract for how his code will be interacted with.  B can then immediately begin work.  The two don't need to coordinate as closely and B is not gates on the work of A.  When they combine their work, it will--in theory-- just work.  Of course, in practice there will be some kins to work out but they should be minimal.


Let's take an example from my RoboRally project of last summer.  There were four of us on the project.  We were scattered across the country and couldn't closely collaborate.  We partitioned the work into four parts and defined the interfaces that formed the seams between our work.  I was coding the game logic.  I was responsible for tracking robot locations, moving pieces around, and calculating the interactions between the robots.  Another person was responsible for the UI that allowed players to choose their moves.  We agreed upon an interface for the players.  They would have a list of cards, and a robot.  We agreed upon the interface for giving a list of robots to my game engine and for the UI to tell me to process a turn.  With that, I was free to write my game engine code without having to worry about how the UI worked.  The UI programmer was also freed up to write the turn-selection logic without needed to know how I would later process the turns.  All he needed to do was to package the data into the agreed-upon format and I would do the rest.


The second advantage of using interfaces is the flexibility they provide.  This advantage is accrued even on a single-coder project.  Because the various functional units in the program are written to interfaces and not to concrete classes or implementations, the details of the implementation can be varied without the client code having to change.  This is the impetus behind the "encapsulate what varies" principle.  It is impossible to predict when the need will arise to change a portion of the code.  The more you use interfaces, the more easily the program will be modified.  This is especially helpful if the variation turns out to be an addition instead of a replacement.  Changing one image encoder for another can be done by changing the implementation of a concrete class, CJpegEncoder.  Clients don't need to know anything changed.  However, if client code is creating the image encoder class directly and statically linking to its functions, adding a second option for image encoding becomes hard.  Each place where the client creates and interacts with the encoder needs to be modified.  If, instead, the client code only uses IImageEncoder, the code doesn't need to care if it is interacting with CJpegEncoder or CPngEncoder.  It makes the same calls.


Another example.  We have tests dating back to the time I began at Microsoft 10 years ago.  A few years back a new testing system was rolled out across the team and our tests needed to conform to it.  The primary thing that needed to change was the logging.  Prior to this testing system, logging was a haphazard affair.  Logs were consumed by humans so as long as the text clearly delineated a pass and a failure, everything was fine.  The new system had a centralized reporting system including an automated log-parsing system.  This required revamping the way each of our tests logged. 


We had 3 main classes of tests.  There was one class which used a common test harness.  This harness could take dlls containing a logger.  The interface to the dll was standardized.  With one new dll we were able to convert a whole class of applications.  No code in the test applications needed to be modified.  The second class did their own logging, but each contained a function, Log(string), that did the logging.  Each of these applications had to be touched, but the modifications were simple.  The Log function was replaced by one that called into the new logging system.  The modifications were quick.  The third class of tests was the minority of our tests, but these were written without any concept of interfaces.  Each did logging in its own way and usually by passing around a file handle which the various test functions used fprintf to log to.  This worked fine, at the time, but the implementation was tightly bound to the client code.  Each line that logged had to be modified.  This took months.  The moral of the story:  Use interfaces.  Even where you don't need them yet.

Friday, January 18, 2008

Prefer Loose Coupling

This is the 3rd post in the Design Principles to Live By series.

Coupling is the degree to which software components depend on each other.  Tight coupling is a high degree of dependence.  In practical terms, it means one component needs intimate knowledge of how another works in order to successfully interact with it.  As much as possible, tight coupling should be avoided.  Tight coupling has many negative consequences.  It can make testing difficult, increase complexity and limit re-use.  Code which is loosely coupled is code which does not need to understand the details of the other components in the system.  Interaction should be through well-defined interfaces and semantics.

When one component is heavily reliant upon another, it cannot be tested except in combination with the other.  Testing become a much more complex task.  Not only does this make life difficult for the test team, but it deters the use of unit tests.  Tight coupling can also reduce the testable surface.  If ComponentA explicitly creates and then utilizes ComponentB and may even rely on side effects of ComponentB, that seam cannot be tested.  ComponentB cannot be replaced which means any interaction with it cannot be tested.  Less testing means less confidence in the product and ultimately a resistance to change later.

Each component that is tightly coupled to another component increases the complexity of the system.  To understand the system requires understanding the details of each coupled component.  Increased complexity in turn increases initial development time, difficulty of change, and thus the resistance to change.  Complex units are harder to understand, design, and build.  Loosely coupled components can be developed and changed independently of each other.  Tightly coupled components musts be created in a coordinated fashion.  This implies a delay to one delays the other and a change to one changes the others.  Change is similarly a cascaded endeavor.  Changing ComponentB requires also changing ComponentA.  Anyone relying on ComponentA may also need to be changed.  What should be a simple fix turns into a long series of changes and trust in the system decays.  Because testing cannot be done in isolation, testing the fix also becomes harder.  Because of this, fixes that might be useful are rejected as too costly.

Finally, tightly coupled components are hard to re-use.  Using one part requires using many others.  The bigger the coupled system, the less likely it is to fit the need.  Large systems have a greater likelihood of requiring unacceptable dependencies or use models.  It is hard to understand the side-effects of utilizing a large, complex code based.  A small, independent component is easier to trust.

Tight coupling is not just a lack of interfaces.  It is requiring an understanding of the semantics of the interface.  It is possible to have a tightly coupled set of components which all have well-defined interfaces.  DirectX Media Objects (DMOs) were envisioned to be small, independent pieces of the DirectShow media pipeline.  Unfortunately, each DMO had its own initialization interfaces.  There was no standard way to set up the objects.  Once set up, the data could flow in an abstracted manner, but setting up a pipeline required knowing what each DMO was so it could be configured properly.  This is a tightly coupled system.  Each application had to know specifically which DMOs it was using and couldn't use others without first being programmed to understand them. 

Another example was a video player application we were working on at one point.  It was supposed to have an abstracted pipeline to playback could be done via MediaFoundation or DirectShow or other pipelines.  One proposed design utilized PropVariants heavily in its interfaces.  Not only are they onerous to use from C++ but they make for terrible interfaces.  They look flexible, but in reality cause tight coupling.  Each user of the interface has to understand what values will be put into those PropVariants. 

Consider this rule of thumb.  If you ever see code like this:

switch(type) {

typeA: do something;

typeB: do somethingElse;

default: error;

}

Stop!  Your code is too tightly coupled.  This isn't the only tight coupling pattern, but it is one of them.  Another is if you are reaching into the guts of an object.  If you are accessing the members of an object and not through a method/property, you are too tightly coupled.