Storage Bits

November 23rd, 2009

Light Peak: black hole

Posted by Robin Harris @ 9:14 am

Categories: Infrastructure

Tags: Black Hole, Sony Corp., I/O, Apple Inc., USB, Intel Corp., USB 3.0, Cable, Telecommunications, Personal Technology

Light Peak is a high-speed optical I/O interconnect - starting at 10 Gbit/sec and scaling to 100 - whose parents are 3 of the biggest I/0 screwups in high tech: Sony, Apple and Intel. Can you spell doomed?

And that is too bad for us, because Light Peak - or something like it - is a Very Good Thing.

The need
The Moore’s Law driven merry-go-round of CPU, storage and bandwidth growth needs to get another push - this time with bandwidth. USB 3.0 claims a 5 Gbit/sec bandwidth - like USB 2.0 claims 480 Mbit/sec, HA! - but leading edge consumers are already using that with eSATA.

For these folks USB 3.0 is too little, too late.

But USB 3.0 faces a bigger problem: stricter EMI regulations. The 1 GHz and above band has been lightly regulated but as more services - Wifi and more - move into that range standards groups are clamping down.

Each USB 3.0 cable and port is a gigahertz radio broadcasting in your living room. The cables can be shielded by the emission problems will only get worse with time as devices proliferate.

That’s where Light Peak comes in. An optical interconnect cable doesn’t radiate EMI. It’s immune to problems like ground loops that can afflict copper cables. High-speed optical interconnects are cleaner.

Every interconnect takes up device space and power, requires driver updates and leaves customers with the limitations of older technology. Yet the longer a popular interconnect is shipping the better the drivers can be tuned for both reliability and performance.

The players
Intel, Sony and Apple hope to make Light Peak a universal optical interconnect. Intel has been driving hard to reduce the cost of the optical transceivers and they have made tremendous progress. Apple wants to limit the number of interconnects on its systems. And Sony and the rest of Hollywood dream of a consumer paradise where we spend all day watching carefully DRM’d 3D super high def video on an array of costly displays, speakers and servers.

And everyone wants a physically small and fast I/O port for mobile devices.

Sadly, none of them get I/O.

• Apple. Arguably the worst offender, Apple’s penchant for tiny I/O ports - mini-VGA, mini-DVI, mini-DisplayPort and the ghastly ADC - has inflicted more pain on Mac users than even FireWire’s high license fees.
• Intel. Who remembers that Intel pushed Infiniband to replace PCI? Great I/O tech with no sense of market realities like cost.
• Sony. They’ve pooched Blu-ray. Last consumer hits: PS2, Walkman and Trinitron. And they love DRM.

Intel claims all Light Peak components will ship next year. But they need a launch customer, preferably one with a large following among bandwidth intensive users such as scientists and video pros. Apple fits the bill.

The consumer wild card
Consumer acceptance - even from Apple fans - is not a given. One big issue: the cost of critical bits like hubs, disk interfaces and cables.

When USB was new the extra bits were costly too. The key is how fast Intel can drive optical chip prices down. They tend to be way optimistic - so I’m not.

The Storage Bits take
Light Peak is a great idea and doomed. Between obnoxious DRM, costly optical hubs and switches, Blu-ray style licensing fees, Intel over-engineering and Apple’s penchant for twee little I/O ports, Light Peak is almost certain to come to fail.

Which is too bad. USB2 shows the value and power of a mature general purpose interconnect. Plug in a thumb drive, digital camera, media player, disk drive, scanner, printer, hub or rocket launcher - and they just work.

That takes years, so an interconnect designed to scale to 100 Gbits/sec as Moore’s Law drives down interface costs is a Very Good Thing. Let’s hope our corporate overlords surprise us this once.

Comments welcome, of course. How about giving thanks this week for driver writers whose hard work is among the least appreciated in all of techdom?

November 18th, 2009

Disks: why size means performance

Posted by Robin Harris @ 8:41 am

Categories: Disk drives, Solid State Disk

Tags: Hard Drive, Disk, Performance, Solid State Disk, Robin Harris

Big=fast
Most people keep less than 80 GB of data, including the operating system and their applications, on their hard drive. So why should they buy a 500 GB, 1 or even 2 TB hard drive?

One simple reason: speed. That big hard drive will give you this snappiest performance this side of a solid-state disk.

For many applications, even faster than a costly SSD. For a lot less money.

Why?

Big circles, little circles
Data is laid on disks in blocks called sectors. The sectors are laid down in circular tracks.

Disk engineers saw a long time ago that they could put more sectors on the outside of a disk than on the inside. As the head gets closer to the center of the disk, there are fewer sectors - and your data rate slows down.

How much? The innermost track will commonly have only ~45% of the speed of the outermost track.

But that’s not all
Bigger hard drives have another advantage: higher bit density. So for the same RPM, more bits come out.

The difference is substantial. A 3.5″ 500 GB drive might max out at 80 MB/sec, while a 1 TB drive will reach 100 MB/sec. The new 2 TB drives can reach 140 MB/sec.

1 more thing
Not only do big drives deliver more bandwidth, they also deliver more I/Os per second (IOPS). On a half full drive the head will never have to move across the entire platter to access data, cutting seek times - all else being equal - in half.

Big million dollar enterprise RAID arrays stuffed with 15k drives frequently short stroke their already-fast drives to make them even faster. It works on home systems too.

What about SSDs?
Flash-based solid state disks (SSD) excel at random reads - which is why they boot up a system so much faster than a hard drive. But due to the housekeeping needed to make an SSD look like a hard drive their write performance isn’t nearly as stellar.

A state-of-the-consumer-art SSD, the 80 GB Intel X25-M G2, is spec’d (pdf) at 70 MB/sec for sustained sequential writes.

A 500 GB Seagate Momentus 7200 rpm notebook drive will average almost 80 MB/sec while offering 6x the capacity for a much lower cost.

If you work with large files - photography, music, video - you’ll notice the difference over an SSD.

The Storage Bits take
Installing a big hard drive is, after more RAM, the easiest performance upgrade most of us can make. Even if you don’t need the capacity you’ll appreciate a friskier computer.

Another benefit: after initial break-in, new hard drives tend to be much more reliable than a 3 year old drive. Faster, more reliable and cheaper than an SSD, a hard drive upgrade may be all you need to keep your system happy.

Courteous comments welcome, of course. For more on this topic see Hard disks do get slower with use.

November 5th, 2009

A 4 SSD array: Apricorn pt 2

Posted by Robin Harris @ 9:28 pm

Categories: Disk drives, RAID, Solid State Disk

Tags: Card, Apricorn, Intel Corp., Performance Management, Human Resources, Workforce Management, Robin Harris

In part I of this review I measured the bandwidth of a hard drive version of the Apricorn hard card. That’s a PCI express card with 4 2.5″ hard drives mounted on it.

But a good solid state drive (SSD) is faster than a disk - so how fast is the Apricorn card with 4 Intel X25-M drives? I won’t keep you in suspense: darn fast.

Fast enough to handle full 12-bit RGB 4:4:4 at film’s 24 frames-per-second rate. I striped the 4 SSDs with the Mac’s Disk Utility software RAID 0 with the default 32KB block size, which is about the fastest. I used the Blackmagic Designs Disk Speed Test - which is close to X-bench results.

Fast enough for uncompressed 24fps video.

The X25-M uses the slower multi-level cell (MLC) flash, while the X25-E uses faster - and more expensive - single-level cell NAND flash. While both have impressive 260 MB/sec read speeds, M version writes - 70 MB/sec - are slower than an empty 2 TB Seagate hard drive at 117 MB/sec. Of course, the hard drive will slow down as it fills up and the flash drive shouldn’t.

Apricorn tested the card with 4 X25-E drives and reported over 675 MB/sec writes speeds. I didn’t have that config to test, but their other numbers have been similar to my test results.

For Macs Apricorn offers a new version of the driver that they’ve tested with Snow Leopard. I didn’t try it as I’m in the middle of a couple of jobs but I believe them.

The Storage Bits take
I’m currently testing a couple of external arrays, and while they have their advantages, internal system storage is less cluttered with fewer connectors to cause problems as well as much faster read performance - and writes, with the X25-E.

The nearest competitor is Fusion-io’s ioXtreme card. At 80 GB for $895 it offers comparable read performance and faster write performance - at least on Windows and Linux machines - no Mac driver is available.

The Apricorn supports 4 80 GB X25-M drives to achieve similar performance. At roughly $250 each, a fully configured Apricorn costs $1200 for 320 GB. You do the math. For uncompressed video the extra capacity will be helpful.

Comments welcome, of course. I’ll ship the gear Apricorn loaned me back this week. It was fun while it lasted! I’ve also done work for Fusion-io, but have they sent me review copies of anything? No.

October 30th, 2009

A visit to Microsoft's first store

Posted by Robin Harris @ 5:22 pm

Categories: Marketing

Tags: Apple Inc., Microsoft Corp., Store, Windows Store, Microsoft Windows, Operating Systems, Corporate Communications, Software, Marketing, Robin Harris

Leaving the high desert mountains for a day I went Scottsdale’s Fashion Square mall, where the world’s first Microsoft no, make that Windows store opened. [For more photos, see my Gallery Tour.]

That’s right, the logo on the store isn’t Microsoft. It isn’t even the wavy Windows flag logo. It is a newly styled Windows logo.

Microsoft store logo

That’s how you know this isn’t your father’s Microsoft. It’s way different — sleek, colorful, stylish, modern — just like an Apple store.

The setting
Set in the middle of the upscale mall — Scottsdale is part of the Phoenix metro area — that doesn’t house an Apple store. Good thing, too: shoppers might get confused.

The similarities: white façade surround; large glass windows; spare modernist interior; T-shirt clad employees; stylish hardware. Even the occasional “it just works” tagline. Except for the color logo on the façade and the dark wood table tops and you could be in a Apple store.

Which is not a criticism. Apple stores are nice. Congrats to Microsoft’s team for ditching the brown Zune difference-for-the-sake-of-difference impulse.

The store
The store was busy at one o’clock on a Thursday afternoon. No one bothered me as I came in taking pictures.

The most striking non-Apple feature of the store is the row of thin bezel displays along each side wall. They display graphics, advertising messages and video, including some Xbox output.

Another difference: several of Microsoft’s Surface computers placed around the store. These large screen touch sensitive displays are impressive for their responsiveness and ease of use. Think giant iPhone.

At the back of the store is a small theater area - just like at Apple stores. In front of that is Microsoft’s version of the Genius Bar.

The goods
The merchandise choices are well chosen. The slim, stylish and colorful notebooks are a welcome change from the chunky gray, heavy, notebooks on display at Wal-Mart and Best Buy.

Three rows of tables displayed hardware from Dell, HP, Sony and Lenovo. I also saw a Flip camcorder and colorful pink and red netbooks.

Colorful computer bags and accessories are displayed as well. The overall effect suggests quality, not price, drove product selection.

The help
I was looking at a Sony notebook with a textured surface when a store associate asked me if I had any questions. I asked which version of Windows would he recommend I buy to run a video editing app on my Mac?

He had no idea about Mac and Windows but that he would find someone who did. The first person he asked also had no idea so they led me to a nice Answers lady.

She correctly outlined 3 ways I could run Windows on Mac. OK, and which version of Windows 7 would be appropriate for video editing?

She said Windows home basic. No extra features in the higher-end versions I’d want? No, for only running a program Windows 7 home basic is all you need. She didn’t try to upsell me to a more expensive version.

A 2nd opinion
A noncombatant in the computer wars thought the 2 stores looked alike, but that the Apple store had a cooler vibe - not as hard sell. After 3 can-we-help-you’s at the Windows store she went to a Pottery Barn to relax.

I’d chalk it up to a brand new staff eager to prove themselves. They’ll figure out what works, given time.

The Storage Bits take
The first Windows retail store is an impressive effort. Sure, they stole freely from Apple, but why not? Few civilians will notice or care about the similarities - unless the stores are in close proximity.

That’s when the price differences between Windows and Mac hardware will be most obvious. Stylish Macbook, $999. Stylish Sony, $799. Many a suburban breadwinner will wonder what the difference is. Apple will have to tell them.

The Windows stores are aiming at the mass-market end of the Apple demographic. Soccer moms and small business, not students and designers.

That’s the core of the MS strategy: to fuzz the difference in the consumer’s mind between Microsoft and Apple. Except, of course, any price difference. “We’re as good as Apple, only cheaper!”

Apple is firing back. “#1 in customer satisfaction” the latest ads proclaim. They’re also targeting XP users who face a tough upgrade to W7. With Grand Central Dispatch and OpenCL they’re positioned for the future - as long as they can deliver obvious “wow!” to consumers.

The winners are us, the consumers. Microsoft can’t illegally crush Apple the way they did Netscape 15 years ago. They have to compete on the merits. And Apple will have to work harder to tell its story.

Let the games begin!

Comments welcome, of course.

October 28th, 2009

Long-term Mozy for Mac review

Posted by Robin Harris @ 5:56 pm

Categories: Infrastructure, Software

Tags: Apple Macintosh, Mozy, Redundancy, Backups, Desktops, Hardware, Robin Harris

I work online and live in a small town 30 miles of 2 lane road from Flagstaff, AZ. I don’t have a lot of options when my infrastructure FUBARs.

Redundancy keeps me up and running. A key piece is 3 backup systems:

1. Hourly Time Machine backups of changed files.
2. Nightly system disk backups to a bootable external drive.
3. Online backup to cloud storage system.

Am I paranoid? When a recent OS upgrade failed due to driver conflicts, I needed 2 of the 3 systems to retrieve all my data. If a backup disk had failed I would have used all 3.

Mozy for Mac
For over 2 years, at my own expense, I used Decho’s Mozy for Mac client to back up critical data - some 40+ GB of it. About 3 months ago I switched to another provider.

Why?

It wasn’t reliable. Worse, after 2+ years, it wasn’t getting more reliable. I kept hoping, not wanting to go through another multi-week data backup, but I gave up after the latest version stopped working for several weeks and 2 reinstalls didn’t fix the problem.

Mozy’s tech support people are uniformly polite and responsive. But if the product doesn’t want to work they can’t do much about it.

A sample size of 1
I want to stress that this is only 1 machine’s experience with Mozy - not a statistically valid study. I have almost 6 TB of disk on a 12 GB Intel quad-core Mac Pro. I capture hundreds of GB of video, surf a lot of websites, run several data collection utilities, support a GigE LAN and use data intensive peripherals such as a fast sheet-fed scanner and a firewire HD camcorder.

I started using Mozy for Mac when it was in beta. After a slow start I gave v1.0 an endorsement of sorts:

I started working with Mozy’s Mac client a year ago, blogged about it for a couple of months (see “Mozy Mac client beta watch” one and deux). But after many continuing bugs and failures I got depressed and stopped.

Progress was slow. Painfully slow.

New! Improved! Now It Works!
Now I can to report that their client is out in v1.0 and it works.

At the time Mozy for Mac had no competition in the $50/yr backup space.

That was then.
It did work, but as the OS upgrades continued, so did the Mozy client problems. The tech support response tended towards “uninstall and reinstall” although they would sometimes ask for logs. Mozy preserved the online data, so data didn’t need a complete backup.

In the meantime some credible competition has arrived in the form of Backblaze and Crashplan. I’ve tried both and they both worked. Now I’m doing a long-term eval of one as a paying customer.

More on that in a future post.

The Storage Bits take
Mozy’s parent company, Decho, was created by EMC, a $15 billion company with 40,000 employees, that also owns Iomega, VMware and RSA. Plenty of software talent and plenty of money to hire anything they needed.

But they couldn’t get the product to work to my satisfaction even though I installed the latest versions, trashed plist files and more. The same problems kept coming up.

It was the lack of progress to stability that finally made me dump Mozy for Mac. After more than 2 years it wasn’t getting better.

Which meant it was getting worse. And that isn’t acceptable in a back up product.

Comments welcome, of course.

October 25th, 2009

Mac ZFS is dead: RIP.

Posted by Robin Harris @ 11:34 pm

Categories: Infrastructure, Software

Tags: NetApp Inc., Apple Macintosh, Sun Microsystems Inc., File System, Apple Inc., Mac ZFS, Current File System, Storage, Hardware, Robin Harris

PC file system progress stalled this week with the news on MacOSforge that Apple’s ZFS project is dead.

ZFS Project Shutdown 2009-10-23
The ZFS project has been discontinued. The mailing list and repository will also be removed shortly.

ZFS, developed by Sun engineers, is the first 21st century file system. NTFS and HFS+ are firmly rooted in the 1980s. ZFS has a lot of cool features:

• End-to-end data integrity. Current file systems are prone to many problems - ranging from phantom writes to inconsistent error-handling - that mess up your data. The ZFS architecture eliminated them with parent block checksums.
• Pooled storage. Add a drive and it adds extra capacity, not another volume. Less management.
• No need for journaling. Which is one problem Solid State Drives don’t handle well. Get rid of it and SSDs work better.
• Built-in RAID that is as fast as hardware RAID. Get data protection for a lower cost.
• Low-cost snapshot copy. As a copy-on-write system, ZFS can create new snapshots - once an hour, minute or second - with low CPU and storage overhead. Cruise back in time to just before the virus hit, recover, and life is good.

Apple announced in June ‘08 that Snow Leopard server would support ZFS. But things came apart early this year.

What happened?
Jeff Bonwick, ZFS architect, posted Saturday on an earlier quoted comment:

> Apple can currently just take the ZFS CDDL code and incorporate it
> (like they did with DTrace), but it may be that they wanted a “private
> license” from Sun (with appropriate technical support and
> indemnification), and the two entities couldn’t come to mutually
> agreeable terms.

I cannot disclose details, but that is the essence of it.

Jeff

Indemnification?
Sun is being sued by NetApp, a $3B enterprise storage company, claiming that ZFS infringes on NetApp patents. If NetApp won, Apple would find itself in a tough position unless Sun shouldered the financial damage. That’s indemnification.

Sun has made a (IMHO) strong case that NetApp’s patents should be invalidated by prior art. But with all their other problems and the Oracle purchase it was a headache they and Oracle didn’t need.

Where does Apple go from here?
Apple has hired some smart file system engineers and wants to hire more to work on “state-of-the-art file system technologies for Mac OS X.”

But writing new file systems isn’t easy. It takes 5-7 years for a new file system to achieve the maturity needed to support large-scale deployment.

So if Apple is starting from scratch we have a long wait for real innovation to appear. Like Mac OS XII.

What about Microsoft?
Redmond’s file system gurus are well aware of NTFS issues. And under the covers they are making stepwise enhancements to the architecture and implementation.

But as the NTFS and HFS+ architectures age and the pace of storage innovation increases the gap between what is and what could be grows. It’s like putting a 1001 hp Bugatti engine in a Model T: the power is there but you can’t use it.

The Storage Bits take
This kind of cock-up makes me hate software patents - but that’s another post. As long as law allows companies will try to enforce them.

NetApp missed a golden opportunity to raise their visibility in the consumer market by cutting a deal with Apple directly. “NetApp is powering Apple’s advanced storage technologies” would make the company a lot more visible outside the enterprise market.

NetApp is a good company, but they’ve lost their way lately. Note to new CEO Tom Georgens: with EMC moving aggressively into the consumer space you don’t have forever to reposition NetApp for a consumer-driven world.

Steve Jobs doesn’t get storage. Consumers are generating masses of video and photos at an accelerating pace - and they’ll need reliable, available and dirt-easy storage. Lots of it.

Until the Next New Thing in file systems rolls out of Cupertino, Redmond or, maybe, Redwood City, consumers will stuck with too many BSODs, missing and corrupted files and app crashes. Let’s hope we don’t have to wait too many more years.

Comments welcome, of course. Update:There’s now a Google Code page for MacZFS.

October 22nd, 2009

Net neutrality: a faster Internet for all

Posted by Robin Harris @ 10:07 pm

Categories: Infrastructure, Public policy

Tags: Theory, Content Provider, Comcast Corp., Internet Service Provider, YouTube Inc., Net Neutrality, Hulu, Internet Service Providers (ISPs), Internet, Robin Harris

Want a faster Internet? Then the FCC’s new net neutrality focus is good news for you. Because net neutrality - another name for common carrier - means a faster Internet for all.

This discussion focuses on your local Internet service provider, be it Comcast or AT&T. Content providers already pay to get on the Internet, so the core issue is what your local ISP does.

Gee, that isn’t what ComCast said!
According to Comcast if they can’t charge content providers for priority service they won’t be able to invest to expand capacity. But using game theory we find that offering priority services makes Comcast more profitable while offering you worse service and more expensive for content providers.

Oh, that’s why the telcos doesn’t like net neutrality.

How that works
Let’s say a telco like Comcast strikes a deal with Google to offer 10 Mbit/sec priority service for YouTube. Non-payers, like Hulu, are stuck at 2 Mbit/sec, giving YouTube a 5x advantage.

As YouTube videos are faster, more people watch YouTube, which consumes more network capacity. Hulu slows down along with email, web surfing, video conferencing and all other non-priority apps.

Comcast and YouTube are happy, but all your other services have slowed down. So what does Comcast do when you complain?

Do they invest in more bandwidth so all apps can run at 5 Mbit/sec, reducing YouTube’s advantage to 2x? Or do they simply go to other app providers and sell them “priority” service?

The latter will generate more revenue for Comcast and less performance for the remaining Internet services. Good for Comcast; bad for content providers and bad for you.

How net neutrality works
Under net neutrality your service provider only gets revenue from you, the customer. Your ISP has a clear goal: keep you buying.

Now the ISP is incented to invest in higher quality service or a competitor may come in with a better deal. The free market at work!

In the real world
Game theory is well and good, but does the ISP market really play out this way? In Japan, where net neutrality is the rule, ISP compete fiercely to offer the best service. Japan has had download speeds in excess of 100 Mbit/sec for the last 5 years, with lower prices than we have in the US.

That is only a dream for us in the US, the country that invented the Internet. What’s wrong with this picture?

The Storage Bits take
Aligning private incentives to serve the public good is why we have a government empowered to set rules. When the rules are set wrong or not enforced - as the last 25 years of financial de-regulation has disastrously proved - almost all of us end up worse off.

The few who benefit, be they Wall Street i-bankers, MRI-owning doctors or duopolistic telcos, argue for their “right” to damage the rest of us. But just as our military sacrifices to defend our nation and everyone pays taxes, the privileged can sacrifice some profit potential for the greater good.

As game theory demonstrates, America as a whole will be better off with net neutrality, when ISPs focus on serving customers instead of chiseling money from content providers. Free markets work best when the incentives are aligned to create lasting wealth for us all.

Comments welcome, of course. I’ve drawn heavily from the work of Professors Hsing Kenneth Cheng, Subhajyoti Bandyopadhyay and Hong Guo. Here is a brief, non-technical introduction to their work. Serious econ gearheads will like their recent 55 page, algebra-heavy paper The Debate on Net Neutrality: A Policy Perspective.

See also: FCC unanimously approves next steps toward Net Neutrality

October 20th, 2009

DRAM errors in embedded systems

Posted by Robin Harris @ 9:07 am

Categories: RAM

Tags: DRAM, Network, Memory, Networking, Semiconductors, Hardware, Components, Robin Harris

The fact that DRAM error rates are hundreds and thousands worse than the industry admitted has implications beyond your computer’s main memory. Unprotected DRAM is in many embedded subsystems that your computer relies on - both local and across the Internet.

At any time DRAM errors can affect your data - without any indication that DRAM errors are responsible.

Let me count the ways
DRAM is the fast and handy “scratch pad” memory embedded systems rely on to buffer data in process. Here are some common uses:

• VRAM. Video cards with a gigabyte of memory are common - but who cares about a flipped bit on a frame of video? Bigger worry: the use of GPUs as general purpose multi-processors.
• Network adapters. Network and user data is buffered in DRAM. If a network address is corrupted, data goes astray and retries hit network performance. In large, network-based supercomputers this problem has been already seen.
• Storage controllers. Like network adapters, storage controllers buffer data in DRAM. Another reason for end-to-end checksums like ZFS has - and your file system doesn’t.
• Solid State Disks. NAND flash writes are slow and flash SSDs require frequent data rewrites, so designers use unprotected DRAM to buffer data in transit.

The Storage Bits take
Data corruption is an all-to-common problem. Most of the the time though the corruption isn’t labeled with a big red “Corrupted Data.”

The corruption shows up as unreadable or lost files, lost network packets and connections, unexplained system crashes - anything but the true root cause.

What can users do?

Enterprise users should demand that all embedded systems in the data path use ECC memory. Anything less is NOT enterprise-class.

Personal users - like me - have fewer options. You can insist on systems that have ECC DRAM - good luck finding a notebook that does - but until system designers understand the issues and customers are willing to pay the costs, there is no fix for our rickety PC infrastructure.

Just live with it.

Comments welcome, of course.

October 11th, 2009

Optical storage: RIP

Posted by Robin Harris @ 9:51 pm

Categories: Disk drives, Infrastructure

Tags: Hard Drive, Disk, Capacity, Storage, Blu-ray, BD, TDK, Robin Harris

TDK recently demo’d an impressive technical achievement: a 10 layer 320 GB optical disk - using standard Blu-ray (BD) drive technology.

Too bad it will never be a commercial success. Optical is at the end of the line.

Why do formats die?
When their reliability, capacity, performance, density and cost aren’t competitive. Which is where optical is now - even 320 GB optical.

You probably don’t remember punched paper tape - all the rage in the 60s and early 70s - but it was popular on 16 bit minicomputers back when 4k of RAM was respectable and 64k unaffordable. It was limited to a few dozen KB of capacity and not reliable in long-term use, so when 240KB 8” floppies arrived in 1973 paper tape was toast.

But floppies couldn’t keep up with the growth of applications and data sets. The 100 MB Zip drive was insanely popular when introduced in 1994, but by 1999 the format was on the way out thanks to cheaper and more capacious CD-R drives.

Despite heroic efforts to increase removable magnetic disk capacities - culminating in 2001 with the 5.7 GB Orb drive - today removable magnetic disk media is dead, killed by cheaper optical and more convenient flash media. Just like magnetic killed paper.

Removable: backup and transfer
Removable media is good for 2 things: data backup and data transfer. Tape dominates removable media backup today with capacities rivaling the largest disks.

Thumb drives long ago replaced floppies for smaller file transfers - “sneakernet” - with external hard drives handling large capacities. With 1 TB 2.5” hard drives, even a writeable 50 GB Blu-ray (BD-R) can’t compete with a small hard drive in transfer speed or capacity.

TDK’s problem
Which gets us to the 10x Blu-ray problem: even if they started selling it there’s no market. Why?

• Capacity. Successful optical media capacities have been competitive with current disks - CD-ROM in the early 90s; DVD-R in the early 2000s. Multi-layer Blu-ray will never be more than a small fraction of hard drive capacities.
• Performance. 24x Blu-ray transfer rates are half that of today’s disks. And as capacities increase, disks get faster. Not so with Blu-ray: 48x, if it happens, will be the outer limit.
• Reliability. Early adopters report that BD burner disks often don’t play on commercial players. That will get fixed, but multi-layer DB-R will have to solve it again.
• Density. Managing a single piece of media is much simpler than managing 6 or 10. Hard drive density makes them much more convenient.
• Cost. BD-playing DVD drives haven’t been popular on PCs, and BD burners are way more expensive, as is the media. A FireWire or USB hard drive can be had for less than $100, has much faster access times, higher capacity and faster data transfer. With volume BD-R prices will come down - but where will the volume come from?

Multi-layer BD-R has advantages, especially if current BD players can be updated to use it. But there is no commercial justification for distributing content on 320 GB optical disks and there isn’t likely to be one.

Hollywood has a real chance to make 3D work, but 3D HD movies will fit fine on BD. Put a 3D “Band of Brothers” on a single disk? OK, but really, getting up every 50 minutes to change disks isn’t so hard, is it?

The Storage Bits take
New optical formats will get introduced - like 750 MB Zip drives and 5.7 GB Orb drives did - but they’ll stumble around the fringes of consumer acceptance before a quiet death. Many of the same forces that are killing BD - downloading, upconverting, cost - are closing in on optical media in general.

DVDs will be around for years - even as CD-Rs still are - but the focus is shifting to online storage and local disks. The industry has yet to crack the code on massive home disk storage, but that day is coming.

You’ll buy HD 3D content online, download it, store it in your digital library, and watch it when and where you want. If your house floods your content suppliers will let you download again. Who needs the hassle to burn disks?

The one remaining piece is for hard drive vendors to get serious about building archive-quality hard disks. I love their technology, but they aren’t the most forward looking group.

Courteous comments welcome, of course. Anyone want to buy a vintage USB Zip drive?

October 4th, 2009

DRAM error rates: Nightmare on DIMM street

Posted by Robin Harris @ 10:04 pm

Categories: Uncategorized

Tags: DRAM, Error, DIMM, Hardware Failure, Memory, Semiconductors, Hardware, Components, Robin Harris

A two-and-a-half year study of DRAM on 10s of thousands Google servers found DIMM error rates are hundreds to thousands of times higher than thought — a mean of 3,751 correctable errors per DIMM per year.

This is the world’s first large-scale study of RAM errors in the field. It looked at multiple vendors, DRAM densities and DRAM types including DDR1, DDR2 and FB-DIMM.

Every system architect and motherboard designer should read it carefully.

If you can’t trust DRAM . . .
Here are some hard numbers from DRAM Errors in the Wild: A Large-Scale Field Study by Bianca Schroeder, U of Toronto, and Eduardo Pinheiro and Wolf-Dietrich Weber of Google.

The Google servers use ECC DRAM that typically corrects single bit errors and reports double bit errors. It is a rare notebook or consumer desktop that supports ECC.

You could be having DRAM problems and not know it because even the system doesn’t know.

Non-ECC DRAM is more common
Most DIMMs don’t include ECC because it costs more. Without ECC the system doesn’t know a memory error has occurred.

Everything is fine until the data corruption means a missed memory reference or an incorrect value or a flipped bit in a file writing to disk. What you see is a “file not found” or a “file not readable” message or, worse yet, silent data corruption - or even a system crash. And nothing that says “memory error.”

Conventional Wisdom
The industry take on DRAM is summed in a quote from an old AnandTech FAQ that took the industry at its word:

Everyone can agree that hard errors are fairly rare. . . . For the frequency of soft errors. . . . IBM stated . . . that at sea level, a soft error event occurs once per month of constant use in a 128MB PC100 SDRAM module. Micron has stated that it is closer to once per six months . . . .

An even bigger surprise: it appears that hard errors, not soft errors, are the dominant error mode - the reverse of the conventional wisdom.

Good news
The study had several findings that are good news for consumers:

• Temperature plays little role in errors - just as Google found with disk drives - so heroic cooling isn’t necessary.
• The problem isn’t getting worse. The latest, most dense generations of DRAM perform as well, error wise, as previous generations.
• Heavily used systems have more errors - meaning casual users have less to worry about.
• No significant differences between vendors or DIMM types (DDR1, DDR2 or FB-DIMM). You can buy on price - at least for the ECC-type DIMMS they investigated.
• Only 8% of DIMMs had errors per year on average. Fewer DIMMs = fewer error problems - good news for users of smaller systems.

But something to think about for large-memory servers running, say, in-memory databases.

Bad news
Besides error rates much higher than expected - which is plenty bad - the study found that error rates were motherboard, not DIMM type or vendor, dependent. This means that some popular mobos have poor EMI hygiene. Route a memory trace too close to noisy component or shirk on grounding layers and instant error problems.

Hardware failures are much more common as well and may be the most common type of memory failure. Google replaces all DIMMs with hard errors - as do most data centers - as a matter of policy.

Other interesting findings
For all platforms they found that 20% of the machines with errors make up more than 90% of all observed errors on that platform. There be lemons out there!

In more than 93% of the cases a machine that sees a correctable error experiences at least one more in the same year. They don’t get better by themselves.

High quality error correction codes are effective in reducing uncorrectable errors. There are “chip-kill” DIMM/mobo combinations that can detect and correct 4 bit errors, but few vendors make those.

Besides costing more, ECC DIMMs are about 3-5% slower than unprotected DIMMs. Few of us would ever notice that small a performance hit, but gamers might care.

The Storage Bits take
You’d think that given the several decades of semiconductor DRAM usage that this study would be old news. I did.

Like most folks I accepted industry assurances that DRAM is reliable. My main machine today uses power-hungry fully-buffered ECC DIMMs.

But I was surprised when I checked out my memory section of “About this Mac” and discovered that 1 of my 6 2GB DIMMs was reporting correctable memory errors. Time to see if the “lifetime” warranty means anything.

I suspect this is another example of the industry’s code of omerta. Big system vendors have scads of data on disk drives, DRAM, network adapters, OS and filesystem based on mortality and tech support calls, but do they share this with the consuming public? Nothing to see here folks, just move along.

Kudos to Google for doing the long-term research required for substantive results and then sharing those results with the rest of us. Data is what makes your computer YOUR computer, and it is worth protecting. Forking over a bit more for ECC mobos and DIMMs may be worth it for serious users.

I expect ECC systems will become a lot more popular in the years ahead.

Comments welcome, of course. Can someone please document how to access ECC error reporting on Windows and Linux machines too? Thanks.