Storage Bits

November 5th, 2009

A 4 SSD array: Apricorn pt 2

Posted by Robin Harris @ 9:28 pm

Categories: Disk drives, RAID, Solid State Disk

Tags: Card, Apricorn, Intel Corp., Performance Management, Human Resources, Workforce Management, Robin Harris

In part I of this review I measured the bandwidth of a hard drive version of the Apricorn hard card. That’s a PCI express card with 4 2.5″ hard drives mounted on it.

But a good solid state drive (SSD) is faster than a disk - so how fast is the Apricorn card with 4 Intel X25-M drives? I won’t keep you in suspense: darn fast.

Fast enough to handle full 12-bit RGB 4:4:4 at film’s 24 frames-per-second rate. I striped the 4 SSDs with the Mac’s Disk Utility software RAID 0 with the default 32KB block size, which is about the fastest. I used the Blackmagic Designs Disk Speed Test - which is close to X-bench results.

Fast enough for uncompressed 24fps video.

The X25-M uses the slower multi-level cell (MLC) flash, while the X25-E uses faster - and more expensive - single-level cell NAND flash. While both have impressive 260 MB/sec read speeds, M version writes - 70 MB/sec - are slower than an empty 2 TB Seagate hard drive at 117 MB/sec. Of course, the hard drive will slow down as it fills up and the flash drive shouldn’t.

Apricorn tested the card with 4 X25-E drives and reported over 675 MB/sec writes speeds. I didn’t have that config to test, but their other numbers have been similar to my test results.

For Macs Apricorn offers a new version of the driver that they’ve tested with Snow Leopard. I didn’t try it as I’m in the middle of a couple of jobs but I believe them.

The Storage Bits take
I’m currently testing a couple of external arrays, and while they have their advantages, internal system storage is less cluttered with fewer connectors to cause problems as well as much faster read performance - and writes, with the X25-E.

The nearest competitor is Fusion-io’s ioXtreme card. At 80 GB for $895 it offers comparable read performance and faster write performance - at least on Windows and Linux machines - no Mac driver is available.

The Apricorn supports 4 80 GB X25-M drives to achieve similar performance. At roughly $250 each, a fully configured Apricorn costs $1200 for 320 GB. You do the math. For uncompressed video the extra capacity will be helpful.

Comments welcome, of course. I’ll ship the gear Apricorn loaned me back this week. It was fun while it lasted! I’ve also done work for Fusion-io, but have they sent me review copies of anything? No.

October 30th, 2009

A visit to Microsoft's first store

Posted by Robin Harris @ 5:22 pm

Categories: Marketing

Tags: Apple Inc., Microsoft Corp., Store, Windows Store, Microsoft Windows, Operating Systems, Corporate Communications, Software, Marketing, Robin Harris

Leaving the high desert mountains for a day I went Scottsdale’s Fashion Square mall, where the world’s first Microsoft no, make that Windows store opened. [For more photos, see my Gallery Tour.]

That’s right, the logo on the store isn’t Microsoft. It isn’t even the wavy Windows flag logo. It is a newly styled Windows logo.

Microsoft store logo

That’s how you know this isn’t your father’s Microsoft. It’s way different — sleek, colorful, stylish, modern — just like an Apple store.

The setting
Set in the middle of the upscale mall — Scottsdale is part of the Phoenix metro area — that doesn’t house an Apple store. Good thing, too: shoppers might get confused.

The similarities: white façade surround; large glass windows; spare modernist interior; T-shirt clad employees; stylish hardware. Even the occasional “it just works” tagline. Except for the color logo on the façade and the dark wood table tops and you could be in a Apple store.

Which is not a criticism. Apple stores are nice. Congrats to Microsoft’s team for ditching the brown Zune difference-for-the-sake-of-difference impulse.

The store
The store was busy at one o’clock on a Thursday afternoon. No one bothered me as I came in taking pictures.

The most striking non-Apple feature of the store is the row of thin bezel displays along each side wall. They display graphics, advertising messages and video, including some Xbox output.

Another difference: several of Microsoft’s Surface computers placed around the store. These large screen touch sensitive displays are impressive for their responsiveness and ease of use. Think giant iPhone.

At the back of the store is a small theater area - just like at Apple stores. In front of that is Microsoft’s version of the Genius Bar.

The goods
The merchandise choices are well chosen. The slim, stylish and colorful notebooks are a welcome change from the chunky gray, heavy, notebooks on display at Wal-Mart and Best Buy.

Three rows of tables displayed hardware from Dell, HP, Sony and Lenovo. I also saw a Flip camcorder and colorful pink and red netbooks.

Colorful computer bags and accessories are displayed as well. The overall effect suggests quality, not price, drove product selection.

The help
I was looking at a Sony notebook with a textured surface when a store associate asked me if I had any questions. I asked which version of Windows would he recommend I buy to run a video editing app on my Mac?

He had no idea about Mac and Windows but that he would find someone who did. The first person he asked also had no idea so they led me to a nice Answers lady.

She correctly outlined 3 ways I could run Windows on Mac. OK, and which version of Windows 7 would be appropriate for video editing?

She said Windows home basic. No extra features in the higher-end versions I’d want? No, for only running a program Windows 7 home basic is all you need. She didn’t try to upsell me to a more expensive version.

A 2nd opinion
A noncombatant in the computer wars thought the 2 stores looked alike, but that the Apple store had a cooler vibe - not as hard sell. After 3 can-we-help-you’s at the Windows store she went to a Pottery Barn to relax.

I’d chalk it up to a brand new staff eager to prove themselves. They’ll figure out what works, given time.

The Storage Bits take
The first Windows retail store is an impressive effort. Sure, they stole freely from Apple, but why not? Few civilians will notice or care about the similarities - unless the stores are in close proximity.

That’s when the price differences between Windows and Mac hardware will be most obvious. Stylish Macbook, $999. Stylish Sony, $799. Many a suburban breadwinner will wonder what the difference is. Apple will have to tell them.

The Windows stores are aiming at the mass-market end of the Apple demographic. Soccer moms and small business, not students and designers.

That’s the core of the MS strategy: to fuzz the difference in the consumer’s mind between Microsoft and Apple. Except, of course, any price difference. “We’re as good as Apple, only cheaper!”

Apple is firing back. “#1 in customer satisfaction” the latest ads proclaim. They’re also targeting XP users who face a tough upgrade to W7. With Grand Central Dispatch and OpenCL they’re positioned for the future - as long as they can deliver obvious “wow!” to consumers.

The winners are us, the consumers. Microsoft can’t illegally crush Apple the way they did Netscape 15 years ago. They have to compete on the merits. And Apple will have to work harder to tell its story.

Let the games begin!

Comments welcome, of course.

October 28th, 2009

Long-term Mozy for Mac review

Posted by Robin Harris @ 5:56 pm

Categories: Infrastructure, Software

Tags: Apple Macintosh, Mozy, Redundancy, Backups, Desktops, Hardware, Robin Harris

I work online and live in a small town 30 miles of 2 lane road from Flagstaff, AZ. I don’t have a lot of options when my infrastructure FUBARs.

Redundancy keeps me up and running. A key piece is 3 backup systems:

1. Hourly Time Machine backups of changed files.
2. Nightly system disk backups to a bootable external drive.
3. Online backup to cloud storage system.

Am I paranoid? When a recent OS upgrade failed due to driver conflicts, I needed 2 of the 3 systems to retrieve all my data. If a backup disk had failed I would have used all 3.

Mozy for Mac
For over 2 years, at my own expense, I used Decho’s Mozy for Mac client to back up critical data - some 40+ GB of it. About 3 months ago I switched to another provider.

Why?

It wasn’t reliable. Worse, after 2+ years, it wasn’t getting more reliable. I kept hoping, not wanting to go through another multi-week data backup, but I gave up after the latest version stopped working for several weeks and 2 reinstalls didn’t fix the problem.

Mozy’s tech support people are uniformly polite and responsive. But if the product doesn’t want to work they can’t do much about it.

A sample size of 1
I want to stress that this is only 1 machine’s experience with Mozy - not a statistically valid study. I have almost 6 TB of disk on a 12 GB Intel quad-core Mac Pro. I capture hundreds of GB of video, surf a lot of websites, run several data collection utilities, support a GigE LAN and use data intensive peripherals such as a fast sheet-fed scanner and a firewire HD camcorder.

I started using Mozy for Mac when it was in beta. After a slow start I gave v1.0 an endorsement of sorts:

I started working with Mozy’s Mac client a year ago, blogged about it for a couple of months (see “Mozy Mac client beta watch” one and deux). But after many continuing bugs and failures I got depressed and stopped.

Progress was slow. Painfully slow.

New! Improved! Now It Works!
Now I can to report that their client is out in v1.0 and it works.

At the time Mozy for Mac had no competition in the $50/yr backup space.

That was then.
It did work, but as the OS upgrades continued, so did the Mozy client problems. The tech support response tended towards “uninstall and reinstall” although they would sometimes ask for logs. Mozy preserved the online data, so data didn’t need a complete backup.

In the meantime some credible competition has arrived in the form of Backblaze and Crashplan. I’ve tried both and they both worked. Now I’m doing a long-term eval of one as a paying customer.

More on that in a future post.

The Storage Bits take
Mozy’s parent company, Decho, was created by EMC, a $15 billion company with 40,000 employees, that also owns Iomega, VMware and RSA. Plenty of software talent and plenty of money to hire anything they needed.

But they couldn’t get the product to work to my satisfaction even though I installed the latest versions, trashed plist files and more. The same problems kept coming up.

It was the lack of progress to stability that finally made me dump Mozy for Mac. After more than 2 years it wasn’t getting better.

Which meant it was getting worse. And that isn’t acceptable in a back up product.

Comments welcome, of course.

October 25th, 2009

Mac ZFS is dead: RIP.

Posted by Robin Harris @ 11:34 pm

Categories: Infrastructure, Software

Tags: NetApp Inc., Apple Macintosh, Sun Microsystems Inc., File System, Apple Inc., Mac ZFS, Current File System, Storage, Hardware, Robin Harris

PC file system progress stalled this week with the news on MacOSforge that Apple’s ZFS project is dead.

ZFS Project Shutdown 2009-10-23
The ZFS project has been discontinued. The mailing list and repository will also be removed shortly.

ZFS, developed by Sun engineers, is the first 21st century file system. NTFS and HFS+ are firmly rooted in the 1980s. ZFS has a lot of cool features:

• End-to-end data integrity. Current file systems are prone to many problems - ranging from phantom writes to inconsistent error-handling - that mess up your data. The ZFS architecture eliminated them with parent block checksums.
• Pooled storage. Add a drive and it adds extra capacity, not another volume. Less management.
• No need for journaling. Which is one problem Solid State Drives don’t handle well. Get rid of it and SSDs work better.
• Built-in RAID that is as fast as hardware RAID. Get data protection for a lower cost.
• Low-cost snapshot copy. As a copy-on-write system, ZFS can create new snapshots - once an hour, minute or second - with low CPU and storage overhead. Cruise back in time to just before the virus hit, recover, and life is good.

Apple announced in June ‘08 that Snow Leopard server would support ZFS. But things came apart early this year.

What happened?
Jeff Bonwick, ZFS architect, posted Saturday on an earlier quoted comment:

> Apple can currently just take the ZFS CDDL code and incorporate it
> (like they did with DTrace), but it may be that they wanted a “private
> license” from Sun (with appropriate technical support and
> indemnification), and the two entities couldn’t come to mutually
> agreeable terms.

I cannot disclose details, but that is the essence of it.

Jeff

Indemnification?
Sun is being sued by NetApp, a $3B enterprise storage company, claiming that ZFS infringes on NetApp patents. If NetApp won, Apple would find itself in a tough position unless Sun shouldered the financial damage. That’s indemnification.

Sun has made a (IMHO) strong case that NetApp’s patents should be invalidated by prior art. But with all their other problems and the Oracle purchase it was a headache they and Oracle didn’t need.

Where does Apple go from here?
Apple has hired some smart file system engineers and wants to hire more to work on “state-of-the-art file system technologies for Mac OS X.”

But writing new file systems isn’t easy. It takes 5-7 years for a new file system to achieve the maturity needed to support large-scale deployment.

So if Apple is starting from scratch we have a long wait for real innovation to appear. Like Mac OS XII.

What about Microsoft?
Redmond’s file system gurus are well aware of NTFS issues. And under the covers they are making stepwise enhancements to the architecture and implementation.

But as the NTFS and HFS+ architectures age and the pace of storage innovation increases the gap between what is and what could be grows. It’s like putting a 1001 hp Bugatti engine in a Model T: the power is there but you can’t use it.

The Storage Bits take
This kind of cock-up makes me hate software patents - but that’s another post. As long as law allows companies will try to enforce them.

NetApp missed a golden opportunity to raise their visibility in the consumer market by cutting a deal with Apple directly. “NetApp is powering Apple’s advanced storage technologies” would make the company a lot more visible outside the enterprise market.

NetApp is a good company, but they’ve lost their way lately. Note to new CEO Tom Georgens: with EMC moving aggressively into the consumer space you don’t have forever to reposition NetApp for a consumer-driven world.

Steve Jobs doesn’t get storage. Consumers are generating masses of video and photos at an accelerating pace - and they’ll need reliable, available and dirt-easy storage. Lots of it.

Until the Next New Thing in file systems rolls out of Cupertino, Redmond or, maybe, Redwood City, consumers will stuck with too many BSODs, missing and corrupted files and app crashes. Let’s hope we don’t have to wait too many more years.

Comments welcome, of course. Update:There’s now a Google Code page for MacZFS.

October 22nd, 2009

Net neutrality: a faster Internet for all

Posted by Robin Harris @ 10:07 pm

Categories: Infrastructure, Public policy

Tags: Theory, Content Provider, Comcast Corp., Internet Service Provider, YouTube Inc., Net Neutrality, Hulu, Internet Service Providers (ISPs), Internet, Robin Harris

Want a faster Internet? Then the FCC’s new net neutrality focus is good news for you. Because net neutrality - another name for common carrier - means a faster Internet for all.

This discussion focuses on your local Internet service provider, be it Comcast or AT&T. Content providers already pay to get on the Internet, so the core issue is what your local ISP does.

Gee, that isn’t what ComCast said!
According to Comcast if they can’t charge content providers for priority service they won’t be able to invest to expand capacity. But using game theory we find that offering priority services makes Comcast more profitable while offering you worse service and more expensive for content providers.

Oh, that’s why the telcos doesn’t like net neutrality.

How that works
Let’s say a telco like Comcast strikes a deal with Google to offer 10 Mbit/sec priority service for YouTube. Non-payers, like Hulu, are stuck at 2 Mbit/sec, giving YouTube a 5x advantage.

As YouTube videos are faster, more people watch YouTube, which consumes more network capacity. Hulu slows down along with email, web surfing, video conferencing and all other non-priority apps.

Comcast and YouTube are happy, but all your other services have slowed down. So what does Comcast do when you complain?

Do they invest in more bandwidth so all apps can run at 5 Mbit/sec, reducing YouTube’s advantage to 2x? Or do they simply go to other app providers and sell them “priority” service?

The latter will generate more revenue for Comcast and less performance for the remaining Internet services. Good for Comcast; bad for content providers and bad for you.

How net neutrality works
Under net neutrality your service provider only gets revenue from you, the customer. Your ISP has a clear goal: keep you buying.

Now the ISP is incented to invest in higher quality service or a competitor may come in with a better deal. The free market at work!

In the real world
Game theory is well and good, but does the ISP market really play out this way? In Japan, where net neutrality is the rule, ISP compete fiercely to offer the best service. Japan has had download speeds in excess of 100 Mbit/sec for the last 5 years, with lower prices than we have in the US.

That is only a dream for us in the US, the country that invented the Internet. What’s wrong with this picture?

The Storage Bits take
Aligning private incentives to serve the public good is why we have a government empowered to set rules. When the rules are set wrong or not enforced - as the last 25 years of financial de-regulation has disastrously proved - almost all of us end up worse off.

The few who benefit, be they Wall Street i-bankers, MRI-owning doctors or duopolistic telcos, argue for their “right” to damage the rest of us. But just as our military sacrifices to defend our nation and everyone pays taxes, the privileged can sacrifice some profit potential for the greater good.

As game theory demonstrates, America as a whole will be better off with net neutrality, when ISPs focus on serving customers instead of chiseling money from content providers. Free markets work best when the incentives are aligned to create lasting wealth for us all.

Comments welcome, of course. I’ve drawn heavily from the work of Professors Hsing Kenneth Cheng, Subhajyoti Bandyopadhyay and Hong Guo. Here is a brief, non-technical introduction to their work. Serious econ gearheads will like their recent 55 page, algebra-heavy paper The Debate on Net Neutrality: A Policy Perspective.

See also: FCC unanimously approves next steps toward Net Neutrality

October 20th, 2009

DRAM errors in embedded systems

Posted by Robin Harris @ 9:07 am

Categories: RAM

Tags: DRAM, Network, Memory, Networking, Semiconductors, Hardware, Components, Robin Harris

The fact that DRAM error rates are hundreds and thousands worse than the industry admitted has implications beyond your computer’s main memory. Unprotected DRAM is in many embedded subsystems that your computer relies on - both local and across the Internet.

At any time DRAM errors can affect your data - without any indication that DRAM errors are responsible.

Let me count the ways
DRAM is the fast and handy “scratch pad” memory embedded systems rely on to buffer data in process. Here are some common uses:

• VRAM. Video cards with a gigabyte of memory are common - but who cares about a flipped bit on a frame of video? Bigger worry: the use of GPUs as general purpose multi-processors.
• Network adapters. Network and user data is buffered in DRAM. If a network address is corrupted, data goes astray and retries hit network performance. In large, network-based supercomputers this problem has been already seen.
• Storage controllers. Like network adapters, storage controllers buffer data in DRAM. Another reason for end-to-end checksums like ZFS has - and your file system doesn’t.
• Solid State Disks. NAND flash writes are slow and flash SSDs require frequent data rewrites, so designers use unprotected DRAM to buffer data in transit.

The Storage Bits take
Data corruption is an all-to-common problem. Most of the the time though the corruption isn’t labeled with a big red “Corrupted Data.”

The corruption shows up as unreadable or lost files, lost network packets and connections, unexplained system crashes - anything but the true root cause.

What can users do?

Enterprise users should demand that all embedded systems in the data path use ECC memory. Anything less is NOT enterprise-class.

Personal users - like me - have fewer options. You can insist on systems that have ECC DRAM - good luck finding a notebook that does - but until system designers understand the issues and customers are willing to pay the costs, there is no fix for our rickety PC infrastructure.

Just live with it.

Comments welcome, of course.

October 11th, 2009

Optical storage: RIP

Posted by Robin Harris @ 9:51 pm

Categories: Disk drives, Infrastructure

Tags: Hard Drive, Disk, Capacity, Storage, Blu-ray, BD, TDK, Robin Harris

TDK recently demo’d an impressive technical achievement: a 10 layer 320 GB optical disk - using standard Blu-ray (BD) drive technology.

Too bad it will never be a commercial success. Optical is at the end of the line.

Why do formats die?
When their reliability, capacity, performance, density and cost aren’t competitive. Which is where optical is now - even 320 GB optical.

You probably don’t remember punched paper tape - all the rage in the 60s and early 70s - but it was popular on 16 bit minicomputers back when 4k of RAM was respectable and 64k unaffordable. It was limited to a few dozen KB of capacity and not reliable in long-term use, so when 240KB 8” floppies arrived in 1973 paper tape was toast.

But floppies couldn’t keep up with the growth of applications and data sets. The 100 MB Zip drive was insanely popular when introduced in 1994, but by 1999 the format was on the way out thanks to cheaper and more capacious CD-R drives.

Despite heroic efforts to increase removable magnetic disk capacities - culminating in 2001 with the 5.7 GB Orb drive - today removable magnetic disk media is dead, killed by cheaper optical and more convenient flash media. Just like magnetic killed paper.

Removable: backup and transfer
Removable media is good for 2 things: data backup and data transfer. Tape dominates removable media backup today with capacities rivaling the largest disks.

Thumb drives long ago replaced floppies for smaller file transfers - “sneakernet” - with external hard drives handling large capacities. With 1 TB 2.5” hard drives, even a writeable 50 GB Blu-ray (BD-R) can’t compete with a small hard drive in transfer speed or capacity.

TDK’s problem
Which gets us to the 10x Blu-ray problem: even if they started selling it there’s no market. Why?

• Capacity. Successful optical media capacities have been competitive with current disks - CD-ROM in the early 90s; DVD-R in the early 2000s. Multi-layer Blu-ray will never be more than a small fraction of hard drive capacities.
• Performance. 24x Blu-ray transfer rates are half that of today’s disks. And as capacities increase, disks get faster. Not so with Blu-ray: 48x, if it happens, will be the outer limit.
• Reliability. Early adopters report that BD burner disks often don’t play on commercial players. That will get fixed, but multi-layer DB-R will have to solve it again.
• Density. Managing a single piece of media is much simpler than managing 6 or 10. Hard drive density makes them much more convenient.
• Cost. BD-playing DVD drives haven’t been popular on PCs, and BD burners are way more expensive, as is the media. A FireWire or USB hard drive can be had for less than $100, has much faster access times, higher capacity and faster data transfer. With volume BD-R prices will come down - but where will the volume come from?

Multi-layer BD-R has advantages, especially if current BD players can be updated to use it. But there is no commercial justification for distributing content on 320 GB optical disks and there isn’t likely to be one.

Hollywood has a real chance to make 3D work, but 3D HD movies will fit fine on BD. Put a 3D “Band of Brothers” on a single disk? OK, but really, getting up every 50 minutes to change disks isn’t so hard, is it?

The Storage Bits take
New optical formats will get introduced - like 750 MB Zip drives and 5.7 GB Orb drives did - but they’ll stumble around the fringes of consumer acceptance before a quiet death. Many of the same forces that are killing BD - downloading, upconverting, cost - are closing in on optical media in general.

DVDs will be around for years - even as CD-Rs still are - but the focus is shifting to online storage and local disks. The industry has yet to crack the code on massive home disk storage, but that day is coming.

You’ll buy HD 3D content online, download it, store it in your digital library, and watch it when and where you want. If your house floods your content suppliers will let you download again. Who needs the hassle to burn disks?

The one remaining piece is for hard drive vendors to get serious about building archive-quality hard disks. I love their technology, but they aren’t the most forward looking group.

Courteous comments welcome, of course. Anyone want to buy a vintage USB Zip drive?

October 4th, 2009

DRAM error rates: Nightmare on DIMM street

Posted by Robin Harris @ 10:04 pm

Categories: Uncategorized

Tags: DRAM, Error, DIMM, Hardware Failure, Memory, Semiconductors, Hardware, Components, Robin Harris

A two-and-a-half year study of DRAM on 10s of thousands Google servers found DIMM error rates are hundreds to thousands of times higher than thought — a mean of 3,751 correctable errors per DIMM per year.

This is the world’s first large-scale study of RAM errors in the field. It looked at multiple vendors, DRAM densities and DRAM types including DDR1, DDR2 and FB-DIMM.

Every system architect and motherboard designer should read it carefully.

If you can’t trust DRAM . . .
Here are some hard numbers from DRAM Errors in the Wild: A Large-Scale Field Study by Bianca Schroeder, U of Toronto, and Eduardo Pinheiro and Wolf-Dietrich Weber of Google.

The Google servers use ECC DRAM that typically corrects single bit errors and reports double bit errors. It is a rare notebook or consumer desktop that supports ECC.

You could be having DRAM problems and not know it because even the system doesn’t know.

Non-ECC DRAM is more common
Most DIMMs don’t include ECC because it costs more. Without ECC the system doesn’t know a memory error has occurred.

Everything is fine until the data corruption means a missed memory reference or an incorrect value or a flipped bit in a file writing to disk. What you see is a “file not found” or a “file not readable” message or, worse yet, silent data corruption - or even a system crash. And nothing that says “memory error.”

Conventional Wisdom
The industry take on DRAM is summed in a quote from an old AnandTech FAQ that took the industry at its word:

Everyone can agree that hard errors are fairly rare. . . . For the frequency of soft errors. . . . IBM stated . . . that at sea level, a soft error event occurs once per month of constant use in a 128MB PC100 SDRAM module. Micron has stated that it is closer to once per six months . . . .

An even bigger surprise: it appears that hard errors, not soft errors, are the dominant error mode - the reverse of the conventional wisdom.

Good news
The study had several findings that are good news for consumers:

• Temperature plays little role in errors - just as Google found with disk drives - so heroic cooling isn’t necessary.
• The problem isn’t getting worse. The latest, most dense generations of DRAM perform as well, error wise, as previous generations.
• Heavily used systems have more errors - meaning casual users have less to worry about.
• No significant differences between vendors or DIMM types (DDR1, DDR2 or FB-DIMM). You can buy on price - at least for the ECC-type DIMMS they investigated.
• Only 8% of DIMMs had errors per year on average. Fewer DIMMs = fewer error problems - good news for users of smaller systems.

But something to think about for large-memory servers running, say, in-memory databases.

Bad news
Besides error rates much higher than expected - which is plenty bad - the study found that error rates were motherboard, not DIMM type or vendor, dependent. This means that some popular mobos have poor EMI hygiene. Route a memory trace too close to noisy component or shirk on grounding layers and instant error problems.

Hardware failures are much more common as well and may be the most common type of memory failure. Google replaces all DIMMs with hard errors - as do most data centers - as a matter of policy.

Other interesting findings
For all platforms they found that 20% of the machines with errors make up more than 90% of all observed errors on that platform. There be lemons out there!

In more than 93% of the cases a machine that sees a correctable error experiences at least one more in the same year. They don’t get better by themselves.

High quality error correction codes are effective in reducing uncorrectable errors. There are “chip-kill” DIMM/mobo combinations that can detect and correct 4 bit errors, but few vendors make those.

Besides costing more, ECC DIMMs are about 3-5% slower than unprotected DIMMs. Few of us would ever notice that small a performance hit, but gamers might care.

The Storage Bits take
You’d think that given the several decades of semiconductor DRAM usage that this study would be old news. I did.

Like most folks I accepted industry assurances that DRAM is reliable. My main machine today uses power-hungry fully-buffered ECC DIMMs.

But I was surprised when I checked out my memory section of “About this Mac” and discovered that 1 of my 6 2GB DIMMs was reporting correctable memory errors. Time to see if the “lifetime” warranty means anything.

I suspect this is another example of the industry’s code of omerta. Big system vendors have scads of data on disk drives, DRAM, network adapters, OS and filesystem based on mortality and tech support calls, but do they share this with the consuming public? Nothing to see here folks, just move along.

Kudos to Google for doing the long-term research required for substantive results and then sharing those results with the rest of us. Data is what makes your computer YOUR computer, and it is worth protecting. Forking over a bit more for ECC mobos and DIMMs may be worth it for serious users.

I expect ECC systems will become a lot more popular in the years ahead.

Comments welcome, of course. Can someone please document how to access ECC error reporting on Windows and Linux machines too? Thanks.

September 21st, 2009

What does 6 Gbit SATA mean to you?

Posted by Robin Harris @ 10:45 am

Categories: Disk drives

Tags: Disk, Performance, I/O, Serial ATA, Performance Management, Storage, Hardware, Human Resources, Workforce Management, Robin Harris

Seagate announced hot new disk today: a 6 Gbit SATA interface; 64 MB of cache; and 2 TB of capacity. Time to replace your old disk drives?

No rush
Each of these features is a good thing. But only the capacity is usable today.

The faster SATA interface is available on just a couple of high-end PC motherboards. The good news is that the interface is backwards compatible with the 1.5 and 3 Gbit versions. While you don’t gain any performance from the faster interface today, you don’t lose anything either.

History suggests that the new interface will be fairly common in 12 to 18 months. Since disks have a 3 year useful life you could get 2 years of higher performance if you bought the new drive today — and buy a new PC in 12-18 months.

Where’s the performance?
Where and when you actually see improved performance from the higher speed interface is the real question. A sequential read from a 7200 RPM drive can’t saturate a 3 Gbit link let alone 6 Gbit.

That is where the 64 MB cache comes in. If the data the operating system is requesting is in the cache, the cache can saturate a 6 Gbit link.

Let’s run some numbers.
Any 6 Gbit interface is capable of roughly 600 MB/sec of user data after accounting for protocol and encoding overhead (I’m assuming the drivers are well tuned - which may not be true for some time). Delivering 64 MB from cache will take 100 ms, while a sequential read from the desk could take 600 ms.

6X speed up sounds good. But how likely are you to see it?

If you’re doing small block or random I/O the answer is “not very.” Disk firmware predicts future data requests using the concept of “locality of reference.” The idea is that a request for a block of data will be followed by another request near that block. The disk reads ahead and loads the cache with data it expects the operating system to request.

It is a great concept, but with small requests the data transfer time is dwarfed by the I/O system overhead. And if your I/O is really random, locality of reference isn’t very helpful either.

What about sequential I/O? Even a 100 MB video file will overwhelm a 64 MB cache. Many disks don’t use the cache in sequential I/O because of cache latency. It’s faster to skip the cache and go direct from the read head to the SATA interface.

OK, where DOES it help?
Seagate tells me there are 2 cases where the larger cache offers noticeably higher performance. The first is in non linear editing (NLE), where multimegabyte video clips are flipping around.

The second is a Media Center PC. There large sequential I/O’s are coupled with relatively low audio or video data output rates. The disk can get ahead of the system demand and fill the cache with data that’s ready to go.

Luckily for 6gig SATA chip vendors these are high-growth apps.

The Storage Bits take
What will really drive demand for the new 6 Gbit interface is the new super speed USB 3 due next year. Capable of 300-500 MB/sec USB 3 will enable a new generation of high bandwidth peripherals.

Seagate’s new 6 Gbit interface and larger caches will also become more important as disk drive areal density grows, increasing R/W speeds. Today the disk’s higher performance is only important as part of an end-to-end system design capable of processing and delivering higher bandwidth on a sustained basis.

Comments welcome, of course. Update: Commenters have questioned the 3 year useful life, even suggesting I’m a shill for drive vendors. Hardly. I’m a data driven guy. You can review the data from Google and Carnegie-Mellon’s Parallel Data Lab in posts I wrote 2.5 years ago: Everything you know about disks is wrong and Google’s disk failure experience. Note 1: Annual Failure Rates spike in year 3. Note 2: massive skepticism of drive vendor claims. Note 3: great comments on both posts.

But the term “useful life” means more than “works.” It is a combination of reliability, capacity, expense and performance. A Model T Ford might “work” but it is past its useful life, at least in the US. Likewise, a 10 year old 4 GB drive may still work, but the 60kWh you spend each year to run it would buy you a new USB thumb drive that is more reliable with almost no operating expense.

But the biggest issue is that disk drives are mechanical devices and they wear out. Sure, I back up my data 3 ways, but I also replace my disks every 3 years. What is your data worth to you? End update.

September 16th, 2009

Terabit 3-D chips will keep the flash party rolling

Posted by Robin Harris @ 10:52 pm

Categories: Solid State Disk

Tags: 3D, Samsung Electronics Co. Ltd., Cell, Chip, Semiconductors, Storage, Network Technology, Hardware, Networking, Robin Harris

Higher flash bit density and lower cost is the major driver of flash’s success in the last five years. But many in the industry, including SanDisk CEO Eli Harari, have predicted that flash will hit a wall: 3 or 4 generations of process shrink before flash became slower - and less reliable - than disk.

But at the 2009 Symposia on VLSI Technology and Circuits in Kyoto, Toshiba and Samsung showed technology that could take NAND flash to 1 Tbit per chip. In five years you could be carrying a 4 TB drive on your keychain.

Let’s get vertical
Ever smaller feature sizes have made NAND flash cheaper and denser. Multi-level (MLC) flash - where a cell stores two or more bits - has further increased density.

But shrinking feature sizes also shrinks the number of electrons that represent a bit of data. Vendors already use sophisticated signal processing to determine if a bit in a cell is a one or zero, but as the number of electrons in a cell shrinks it takes more time to determine a cell’s state. And the number of writes declines as well.

If you can’t get smaller get higher. 3-D cell technology stacks cells one on top of the other.

Cost is the problem
The hard part is adding layers without adding cost. Otherwise flash loses the cost advantage that has made it the industry darling and consumer favorite.

Toshiba, the inventor of flash, demonstrated a 32 Gbit prototype using 3-D cell technology. But Samsung, the world’s largest manufacturer of flash today, seems to have an early lead in 3-D technology.

Samsung announced something called VG-NAND (Vertical Gate). In this design the electrode and dielectric layers are laid down and then tiny vertical grooves are batch processed into the chip so control gates can be implanted. This gives high density without a lot of extra processing.

Here’s what it looks like:

Vertical Gate NAND cross-section. Image courtesy Samsung.

The Storage Bits take
Flash emerged as an economic competitor to DRAM only three years ago. In that short time it has had a dramatic effect on storage system architecture and cost.

Shrinking feature sizes have driven that success, but semiconductor physics imposes hard limits. 3-D chips and more investment in process technology promise to keep flash marching forward for at least the next five years.

Comments welcome, of course. Back in the early 80s many warned that alpha radiation would flip bits in 1 Mbit DRAM chips and render them useless. The engineers figured it out. More detail on 3-D flash in this report from Nikkei Electronics Asia.