Monday, March 12, 2007

Why Don't PCs Have Hardware Video Decoders?

Robert X. Cringely speculates in his latest column that Apple may soon be adding H.264 hardware decoding and encoding to its Macintosh line.  Cringely is wrong more often than he is right so the truth of this rumor is unknown.  I'm sure someone at Apple is at least studying the idea.  More interesting to me than whether or not Apple is shipping a $50 chip in all Macs to do H.264 is his talk about why no one has done this before.  He hints at a conspiracy of sorts to keep video-accelerating hardware away from the public.  I worked on DVD playback at Microsoft from nearly the inception.  There are good reasons you don't have hardware decoders in your system right now.  Let me explain.

Cringely says, "Maybe the reason is economic (save the $7) or maybe it is political (Microsoft or maybe Apple are for some reason opposed to hardware decoding). But like a lot of real reasons, I think it probably comes down to hubris and the simple fact that by decoding video in software, road warriors have another incentive to buy a more expensive -- and more powerful -- computer."  The truth is, Microsoft did work back in 1997 to enable hardware decoding in Windows.  The support shipped in the form of DirectX Media around Thanksgiving of that year.  I think it was in version 5.2 of the SDK.  This functionality first shipped with Windows in Windows 98.  Several vendors took advantage of this support.  Toshiba had an early system which shipped with DVD decoding in hardware.  I still have one of their full-length PCI boards hanging in my office.  IBM shipped the Thinkpad with hardware decoding for several years.  Dell offered an upgrade to a Luxsonor decoder card.  Creative shipped the DXR2 and DXR3 at retail.

Despite these many offerings, the market for hardware decoders never grew very big.  Why not?  Because software could do the job and it was substantially cheaper.  Whereas a hardware decoder might cost $10-$50 in COGS (just a guess), a software decoder could be had for something like $1 in volume.  Even on early PCs, the CPU was fast enough to decode the MPEG2 streams for DVD.  A Pentium 2 - 266 could almost keep up and a 333 had plenty of horsepower.  Today's operating systems and background processes sap a lot more power than did Windows 98 but there's still plenty of power there.

Rarely, however, does DVD play on a computer though without some sort of hardware assistance.  As early as the ATI Rage II+, display cards included special hardware designed to accelerate MPEG2 decoding.  On Windows 95 and its variants, this hardware was accessed in a propreitary fashion.  With Windows 2000, Microsoft created a standard called DirectX Video Acceleration (DXVA) which is used today by all DVD decoders to utilize this hardware.  Depending on the display chip, you may get more or less acceleration but the software is always offloading at least a portion of the work to the GPU.

In June of 2005 Microsoft shipped DXVA support for WMV-9.  With Windows Vista, we also support accelerating VC-1 (the variant of WMV used in HD-DVD and BluRay) and H.264 via DXVA.

So, is there a conspiracy to get you to buy more expensive PCs just to play video?  No.  It saves money to decode video without a specialized device just for video.  It is for economic reasons that there are not dedicated hardware decoders.  It's cheaper (and just as effective) to decode video in software with a hardware boost than to ship a chip to do just decoding.

So why then is Apple possibly considering a video acceleration chip in the Mac?  There are two possible reasons.  The first is that H.264 is really, really hard to decode.  It is a lot harder to decode than MPEG-2, DIVX, or even WMV9/VC-1.  Support for H.264 acceleration on graphics chips is also behind the support for other codecs.  Probably the more likely reason is for encoding.  Encoding is a lot harder than decoding (because it requires a full decode in addition to the encode portions).  If Apple wants to do something like timeshifting in H.264, it will need hardware to do it.  Media Center (now included in Vista Home Premium) usually uses dedicated MPEG-2 encoding chips for the TV functionality.


  1. > Encoding is a lot harder than decoding
    Could you elaborate on why this is?

  2. Good question.  I may do a full post on this later but here is the short answer.
    Decoding is a pretty linear process.  You do run-length decoding then inverse discrete cosine transforms then copy and paste motion regions.  In advanced codecs you do a little blending afterwards.  These all take a bounded amount of time.
    Encoding, on the other hand, requires more unbounded work.  You need to do a search for motion vectors.  This requires doing sub-pixel comparisons on several frames of video.  After this you do a cosine transform/quantization and finally a variable-length coding.  For the next frame, you need to decode the frame you just encoded so you can do the motion search against what will really be in the bitstream and not the idealized initial picture.  Thus encoding takes a full decode plus some bounded work (DCT and VLE) plus some unbounded motion vector searching.

  3. As I understand it, newer video/image encoding methods specify what the bitstream looks like and how the decoder behaves. It does *not* specify how to encode a stream. Given the constraints of what the decoder can do (operations, time, memory), the encoder has to figure out the most efficient way to represent a movie.
    Encoders have a VBV (Video Bitstream Verifier ??) that is essentially an decoder that verifies the compressed bitstream doesn't exceed certain constratins, such as the amount of memory required, time, etc.
    It's like saying, "here's the postscript spec. It describes how a printer executes postscript instructions. Generate postscript that prints this page." There's a zillion ways to describe a page - the trick is finding the "best" one in the limited amount of time you have.

  4. He does have a point about HD, though. My (admittedly rather old) notebook struggles decoding a full 1080 HD video stream. Seems strange to me that I have to buy a top-of-line HTPC in order to play HD-DVDs (not that I'd ever THINK of buying a standalone player, they're still way too expensive for what you get, in my opinion)...

  5. @DriverDude - You are right.  Most modern codecs specify decoder behavior and let encoders do whatever they want to create a conformant bitstream.
    @Dean - HD streams, especially advanced codec HD streams, are really hard to decode.  There's a lot to be done in the CPU and the raw video stream itself taxes many display cards.  H.264 is the hardest of the lot to decode.  You'll need at least a 3.0 GHz (or equivalent) machine to decode anything in HD without a GPU accelerator.  If you have 60fps content, that's even harder.  HD-DVD and BluRay are designed to push the envelope right now.  Older machines just won't be able to handle them.

  6. I would think we still won't see specific hardware decoders show up in PCs unless the decoder manufacters and content providers are driving it.  I think multi-core advances and GPU advances will make those custom devices DOA.  Although it would be interesting to see a cost comparison between encoding and decoding algorithms and the relative COGS for both approaches.

  7. Good points.  For the most part, economics will make decoding hardware something rare.  There is a chance we'll see dedicated hardware decoders in two instances:
    1)  Low power systems.  Running the GPU takes a lot of electricity.  If you're just playing video, it's possible that a dedicated chip could be less power hungry.
    2)  For a short period before CPUs/GPUs catch up.  Right now H.264 is hard for most systems to decode in HD resolutions.  Your points on multi-core and GPU advances are correct but there may be some time where the average machine doesn't have sufficient power.  Just like with MPEG-2/DVD, this may cause people to ship dedicated hardware for a short while.  I don't see that as a persistent trend though.

  8. Gerard van der LandMarch 16, 2007 at 11:49 AM

    Interesting read. I still have a REALmagic Hollywood Plus MPEG-2 decoder by Sigma Designs somewhere, which came with a Dell Dimension PC. A couple of years later Sigma Designs introduced the "REALmagic Xcard" that did both MPEG-2 and DivX 4.x decoding.
    PC's have been shipping with DVD drives for about ten years now. Since you worked on DVD playback at Microsoft from nearly the inception, a more interesting question you may be able to answer would be: "Why Doesn't Windows Have an MPEG-2 Software Decoder". Is this only about licensing costs? You say those costs are only $1 per copy, probably even less in the volumes in which Windows is shipping, and because Microsoft can ship it's own implementation.
    Windows Vista Home Premium and Ultimate are the first and only versions of Windows that include an MPEG-2 codec. I'm running Windows Vista Enterprise on my notebook and I cannot playback a DVD on it without requiring additional third-party software. In 2007 I find this rather disappointing..

  9. Gerard, I forgot about the Sigma Designs stuff above but we did have a few of those in the labs back in the day.  They never worked too well with our software though, IIRC.
    That's a very good question.  The $1 cost was the cost of the software, not the licensing cost.  As we didn't ship the decoders, I'm just guessing on that cost by the way.  I know it was low but I don't know for sure how low.  My understanding is that the licensing cost for an MPEG-2 decoder is (or at least was) something like $5.  The OEM would have to pay that whether they used hardware or software for their decoding.
    Windows didn't ship with a decoder for most of its history for a lot of reasons but the licensing costs certainly played into the equation.  
    The good news is that Vista Home Premium & Ultimate do come with an MPEG2 decoder built in.

  10. I did a little poking around.  The royalty for MPEG-2 decoding is $2.50 presently.  It was $4.00 before 2002.  You can find the info here:

  11. consumer4beta@hotmail.comMarch 17, 2007 at 9:30 AM

    That's why I think VC-1 is THE BEST, because it's much better than DivX/Xvid/MPEG-4 and DVD, although admitedly inferior to H.264 in compressed size/quality.