July 24th, 2008
DTrace and the Linux bunker mentality
Since I don’t usually get many chances to work with DTrace I can’t claim to have any serious hands on experience with it - but I do know that it’s insanely great because it works well, it’s easy to use, and it doesn’t impose significant loads or risks of its own.
DTrace is an open source Sun product that made its debut with Solaris 10 but has since been ported to MacOS X and various other BSD derivatives. IBM is reported busily copying it into AIX (along with containers, zones, and whole bunch of other things they ridiculed when Sun introduced them) and only Linux is left without significant corporate or community support for a DTrace port.
Instead the Linux kernel group has been working on a thing called SystemTap. Here’s a bit of what DTrace developer Brian Cantrel had to say about that in his blog:
—
The interest in DTrace on Linux is heating up again — this time in an inferno on the Linux 2008 Kernel Summit discussion list. Under discussion is SystemTap, the Linux-born DTrace-knockoff, with people like Ted Ts’o explaining why they find SystemTap generally unusable (“Do you really expect system administrators to use this tool?”) and in stark contrast to DTrace (“it just works”).
While the comparison is clearly flattering, I find it a bit disappointing that no one in the discussion seems to realize that DTrace “just works” not merely by implementation, but also by design. Over and over again, we made architectural and technical design decisions that would yield an instrumentation framework that would be not just safe, powerful and flexible, but also usable. The subtle bit here is that many of those decisions were not at the surface of the system (where the discussion on the Linux list seems to be currently mired), but in its guts. To phrase it more concretely, innovations like CTF, DOF and provider-specified stability may seem like mind-numbing, arcane implementation detail (and okay, they probably are that too), but they are the foundation upon which the usability of DTrace is built. If you don’t solve the problems that they solve, you won’t have a system anywhere near as usable as DTrace.
So does SystemTap appreciate either the importance of these problems or the scope of their solutions? Almost certainly not — for if they did, they would come to the same conclusion that technologists at Apple, QNX, and the FreeBSD project have come to: the only way to have a system at parity with DTrace is to port DTrace.
—-
So why isn’t the Linux kernel group doing what the BSD people did: seeing a better idea and grabbing it with both hands?
DTrace is open source, the ideas behind it are public, the MacOS X port is a success - and, minor limitations aside, exceedingly useful. So why are the Linux people fighting to protect an obviously bad idea? Because many of them see Sun as the enemy? because they’re still reacting to SCO by pretending that Linux isn’t Unix and they’re a copy-nothing shop? because they don’t have IBM’s cheerful certainty that the customer won’t know where the ideas came from? because they didn’t invent it? or, as I think, all of the above?
There’s no catch here: if SCO eventually wins its case (as I’m sure it will) the losers will be IBM and patsies like Novel, not Linux -and taking a few good ideas from the Solaris side of the Unix house to incorporate in the Linux side is well within open source norms, not seriously impeded by licensing, and good for both.
So why not? my guess (and that’s all it is) is that some of the power players are suffering severe not invented here syndrome, leveraging the anti-Sun feeling in the x86 Unix community to inflict their opinions on others, and generally displaying a lack of technical integrity in trying to re-invent a square wheel alternative to an industry standard for no good reason other than personal antipathies.
July 23rd, 2008
How to lose customers by not even trying
The other day my saintly mother in law asked me to help her find a Volvo to rent for a week in Victoria. Hertz has them, so I said I’d check the web and get her an estimate and a phone number to call to make her reservation.
I use Firefox on Solaris - a rare combination but one that works reasonably well for most sites. Not, however, for Hertz.com: their javascript pagecode, checks for a wide variety of browsers but essentially groups them as “ie4up” or “not ie4up” and never really uses the detail in the not IE category to accommodate users.
In my case, for example, I get the default front page reservation form - with no warnings that after I fill the thing there’s no place to go because the “Submit” button is hidden under some ads on the right and can be neither found nor clicked.
So do the obvious, right? Use my HertzID to log in and see if that produces a working page - it doesn’t; instead the login process informs me that “A valid member number must be entered to log in or to create/find a password. [NZX005].”
Helpful huh?
So I did the obvious: checked the Avis site (which worked properly) and helped her reserve a Ford Taurus instead of an S60.
Then I looked at the code and the servers.
The Hertz site runs on Windows Server 2003 and seems to have been ported from Windows 2000. The Avis site runs on Linux and seems to have come from Solaris 8.
The Hertz site downloads this warning first:
Copyright (c) 2005 The Hertz Corporation All Rights Reserved. (Unpublished.)
The information contained herein is confidential and proprietary to The Hertz Corporation and may not be duplicated, disclosed to third parties, or used for any purpose not expressly authorized by it. Any unauthorized use, duplication, or disclosure is prohibited by law.
The Avis site does not have this - although both use Omniture tracking with appropriate copyright acknowledgments.
Comments like: “// 032504 - dmr”; embedded in the Hertz code suggest that it was written by someone whose first commitment wasn’t to Hertz - and comments like this one:
//alert(’Condition = ‘+(!((isIE4up)||(isNN7up)||((isMozilla)&&(mozillaVersion>=1))||(isFirefox)||(isFirebird)))+’\nNetscape 7+: ‘+isNN7up+’\nisMozzila and up: ‘+((isMozilla)&&(mozillaVersion>=1))+’\nisIE4up: ‘+isIE4up);
add emotional depth to that assumption.
In contrast the comments embedded in the Avis code look like this:
/*
* JavaScript method responsible for setting tracking variables and
* submitting the form when the “Express” button is clicked.
*/
In total the impression I get from the Hertz site is that the company doesn’t care about it - that’s it’s there because everyone’s got a website and having one ticks a box.
In contrast my impression is that somebody at Avis cares about the customer experience - and that’s really my bottom line: one of these two sites works, and made a sale for that company. The other site doesn’t, and lost Hertz its customer.
July 22nd, 2008
Apache’s open source governance model
In our industry the market dominant players often have the worst products, but that’s pretty clearly not true for the Apache Foundation.
The core Apache servers power the web: combining dominant market share with dominant performance and stunning software reliability - and because that combination is unusual, we have to ask why and how?
A big part of the why is historical: Apache got its start with the first NCSA servers and so inherited an installed base, a solid code base, and, most importantly, the loyalty of many of those involved in building out the original server ideas.
Part of this is a matter of perception: Apache dominates only the middle market, not its fringes. If you want the easiest possible route to customer sign-off and don’t care about the costs of ownership - then Apache is not the answer. Similarly if you really only care about a specific subset of secure transaction management, can dictate all other software used in the system, and face hardware limitations requiring that you focus on server cycles - then Apache/Tomcat is not the answer.
But a majority share of the long term how has to be put down, I think, to an effective governance model for the foundation and its community.
Here’s how the Apache foundation describes the key to their organizational success:
Meritocracy
Unlike other software development efforts done under an open source license, the Apache Web Server was not initiated by a single developer (for example, like the Linux Kernel, or the Perl/ Python languages), but started as a diverse group of people that shared common interests and got to know each other by exchanging information, fixes and suggestions.
As the group started to develop their own version of the software, moving away from the NCSA version, more people were attracted and started to help out, first by sending little patches, or suggestions, or replying to email on the mail list, later by more important contributions.
When the group felt that the person had “earned” the merit to be part of the development community, they granted direct access to the code repository, thus increasing the group and increasing the ability of the group to develop the program, and to maintain and develop it more effectively.
We call this basic principle “meritocracy”: literally, govern of merit.
What is interesting to note is that the process scaled very well without creating friction, because unlike in other situations where power is a scarce and conservative resource, in the apache group newcomers were seen as volunteers that wanted to help, rather than people that wanted to steal a position.
Being no conservative resource at stake (money, energy, time), the group was happy to have new people coming in and help, they were only filtering the people that they believed committed enough for the task and matched the human attitudes required to work well with others, especially in disagreement.
After explaining the structure of the ASF, we will see how the meritocracy relates to the various roles.
Notice that what distinguishes the Apache foundation from thousands of other organizations isn’t the idea that people should become what they can do, but their consistent execution on this idea - basically most organizations talk about devolving authority to the leading edge, but then weight the actual execution the other way to gradually centralize control. Apache, like the U.S. Military (and any well run Unix shop) weights its execution the other way: to actually put people on the cutting edge in charge.
And that, I think, is the bottom line message from Apache’s success: you may need a centralized organization for supply, co-ordination, and legal cover but in general organizations are more successful if tactical decision making is entrusted to those doing the actual work -leaving those nominally in charge at the organization’s co-ordination and resource center to facilitate rather than get in the way.
July 21st, 2008
If sameness is a recipe for losing, how do you win?
Kind of obvious, isn’t it? but lets do an imaginary walk on the wild side: lets imagine that you just got a letter from Walmart notifying you that your manufacturing company is one of about thirty being asked to bid on supply of some retail product line they’ve been sourcing in China and now want to get locally.
It’s a big opportunity - even being one of the five or so they’ll pick for the initial retail trial could add significantly to your volume for the year and beyond that; well, margins might be thin but Walmart shelf space really is a lot like money in the bank.
You make a great line in widgets, but so do most of those other guys - and, like everybody else, you know you’ll face design and process compromises somewhere between the stuff coming into your inventory and final packaging to meet Walmart’s rather predatory pricing expectations - but it’s all do-able, and the negative impact selling at Walmart will have on your premium lines will be manageable. So it’s good business, and you want it: but how do you win?
Somebody’s brother in law probably does work near the top ranks in Walmart purchasing, but not yours -and you’re not a haloed national brand either, so as far as they’re concerned you’re just another over-priced supplier.
So if the real meaning of the statement that you’re as good as the other guys is that they’re as good as you, where do you get a real competitive advantage? Not from check-marks on the bid summary: yes you use qualifying ERP/SCM software; yes, you can serialize by promotion; yes, you can load RFID tags and customize packaging; yes, yes, and yes - and it all looks great until you realize the other guys got on the list because they can too.
And it’s not going to be in product costing either - Walmart does not buy retail: every supplier comes under pressure to cut Walmart’s total costs including those of products, inventory, delivery, marketing, returns, and warranties. Worse, while a tiny percentage difference can cost you the business, you have to remember that price cutting is ultimately a self-limiting process - start down this road and eventually every product going out the door will carry away a tiny bit of your company’s value to its owners, its employees, its lenders, and its community.
So where do you look for that unique edge: the thing Walmart’s buyers are going to carry back to their bosses because you’re the guy that’s getting them promoted?
I think the generic answer is unique innovation: in product engineering, in packaging, in manufacturing and in anything you can do (and the other guys can’t) to reduce Walmart’s cost without reducing your own margins beyond the point of no return.
To some extent your ability to innovate is constrained by your financial strength but that will be true for most of the other guys too - and Walmart’s people know perfectly well that even divisions of larger companies don’t have an easy time getting at the money. The bottom line is that banks lend money on business track records and opportunities: meaning that as long your ability to spool up deliveries isn’t at obvious risk, the financial strength issue will be more of a check-off box for their auditors than a real decision factor.
To some extent your ability to innovate is driven by the people you have - but that’s true of the other guy too. You all hire from pretty much the same pool and it’s usually true that a majority of the more senior people in each company worked for a competitor before coming on board. As a result most companies will have individual stars: you’ve got a packaging genius, he’s got some great manufacturing people, and the next guy has someone with an intuitive grasp of distribution logistics that nobody else can match - but no single player is likely to have a long term human resource advantage.
So what that leaves is organizational structure: not so much the people and tools you have, but how you deploy them.
Look at your individual cost centers: sales, engineering, manufacturing, purchasing, distribution, finance, executive; and within each group you’ll find that the tools they use dictate the local structure - so both what you do and how you do it in each of these organizational units are pretty much standardized across your industry.
So what does that leave? Probably the most important organizational bit of all: the communications and control structure under which all of these groups interact for the common good. And not only is this critical to overall success, but it’s almost invisible - meaning that your competitors will be slow to understand what happened and adapt if you can find a way to gain a competitive edge by making advances in this area.
Look closely at it, and the chances are good that effective change is possible and will meet few, if any, entrenched organizational barriers - meaning that for most companies in your position this is a good place to put some serious management attention.
Look at the processes by which a customer opportunity eventually becomes a contributor to net earnings and what you see is essentially an interlocked set of communications and memory processes - things IT can be really good at. Imagine your company’s information processing and communications infrastructure as its nervous system and you see the point: make it faster, make it more accurate, give it finer point control in the way that some people have finer muscle control than others, and you have the basis for a real competitive advantage - one you can hope to implement without upsetting operations inside each of the groups you’re tying together.
The opportunity here is simple and critical: if you can use IT to distribute responsibility - to free your guy to take action while his counterparts in competitor companies are drafting memos or scheduling meetings, then you’ve got an organizational competitive advantage.
In theory you can do this with any IT toolset, but in practice the management baggage that goes along with each IT toolset means that your technology choices can work for you, or against you. Most of your competitors, for example, will be using either Windows or OS/400 based systems - and both of those have strong centralizing tendencies that will work against what you’re trying to do, and like tide waters hammering on rocks, eventually wash your efforts away.
Now while this doesn’t mean that you can’t use these toolsets within a larger organizational effort to distribute responsibility and control to the people who do the actual work; it does mean first that you’re fighting your own tools and, more importantly, that you can’t beat the other guy doing it - because something like nine out of ten of your competitors use the same ideas, the same software, the same hardware, and the same inter-changeable people and once you succeed they’ll be doing exactly the same things in same ways.
Bottom line: the real meaning of market monopolies in IT is averaging - you can keep pace by copying your competition: but you can’t beat them doing it.
And that’s the problem: IT innovation is spelt R-I-S-K, but without it you’re going nowhere - and there an IT architecture that’s internally consistent with the desire to push authority out the edges as a way of giving your organization a long term competitive advantage: Unix, with smart displays like Sun Rays.
The Unix/Smart display architecture gives you the cost and risk minimizations that go with highly centralized processing - and enables the organizational edge that goes with distributing control to the people who do the work. When it works, it gives you the best of both - and you get a real, long term, competitive edge by enabling your people to react faster; to focus more on the customer; to just use, rather than work against or around, their IT tools; and, to avoid much of the management overhead incident on centrally controlled organizations.
You can also, of course, lose big trying this - building momentum for positive cultural change is enabled by technical choices, but made to happen by your people. Get the right people, and you’ll get good results - get the wrong people and you’ll get set back after set back. But that’s always the bottom line in business: if you can win, you can lose. But if you win, you win big: get off the level playing field, break out of the technology ruts the other guys are stuck in, and you’ll be golden all the way to the top.
July 19th, 2008
The iPhone meets Journalism
Sometimes The Onion just doesn’t take itself seriously. Yesterday, for example, it published a review of Time Magazines’ Obama coverage that includes gems like these:
According to the Time reporter, work on the profile was often harder than he had anticipated, with Obama at times dodging questions about whether or not he played a musical instrument, and about what Monopoly piece he thought best represented his candidacy and why.
“Situations like these are when you have to get on the phone and talk, not only to his mother, but to his aunt, his uncle, a Boy Scout leader, or maybe even one of his camp counselors growing up,” Sherwood said. “And if they don’t return your call, you turn to Sunday school teachers and former babysitters?anyone who is willing to go on record and say that Barack Obama was a really good kid who was destined for great things.”
Added Sherwood, “It’s all about getting the factoids out in the open.”
and:
Time managing editor Rich Stengel said he was proud of the Obama puff piece, and that he hoped it would help to redefine the boundaries of journalistic drivel.
“When the American people cast their vote this November, this is the piece of fluff they’re going to remember,” Stengel said. “Not the ones by Newsweek, Harper’s, The New Yorker, The Atlantic, The New Republic, The Economist, Nightline, The Wall Street Journal, or even that story about lessons Obama learned from his first-grade teacher we ran a month ago.”
Now that’s pure politics; couldn’t possibly happen in technology reporting, right?
Wrong.
About a week ago I saw a series of email requests by someone who presents as a journalist asking for dirt on the iPhone 3G - and when I asked google news just now for hits on iPhone (fails OR failed OR failure) it provided 971 of them - with summaries like these:
Apple fails BlackBerry test
Financial Times, UK - 21 hours ago
iPhone’s failure to address other issues make it less than an ideal device for business. The two most glaring problems from a businessHas Apple Lost Its Mojo?
InternetNews.com - 20 hours ago
Many stores sold out of iPhone 3Gs on the morning of the first day. That means both Apple’s physical and digital supply chains failed to meet demand.Apple: Failure to Launch
Techtree.com, India - Jul 15, 2008
The reasons for this sorry state of affairs - Apple servers couldn’t survive the extreme traffic caused by over-enthusiastic iPhone users wanting toWhere the iPhone Failed
BusinessWeek - Jul 3, 2008
In one sense, next week’s launch of the 3G iPhone marks a significant defeat for Apple. Of course, the original iPhone was a huge success in the US and the11 Great Apple Technologies That Failed
PC Magazine - 12 hours ago
But the Newton can be seen as the forerunner of the iPhone, a much more successful product. And the Newton had several innovations that still look …Steve Jobs Decision Behind iPhone Apps’ Achilles’ Heel
InformationWeek, NY - Jul 17, 2008
[Update 2, July 17, 8:20 pm: With due respect to the commenters below, who’re complaining that I failed to note that Apple indicated a fix at WWDC: The
All in the interest of getting the factoids out, right?
And on the Macworld et al editorial side of things? They make Time Magazine look good - but then, like Obama, these guys change their deepest moral and intellectual convictions at the drop of an opportunity.
July 18th, 2008
From Chapter one: Data Processing and the IBM Mainframe
From Chapter one: Data Processing and the IBM Mainframe
This is the 9th excerpt from the second book in the Defen series: BIT: Business Information Technology: Foundations, Infrastructure, and Culture
Note that the section this is taken from, on the evolution of the data processing culture, includes numerous illustrations and note tables omitted here.
Roots (Part Three: The System 360 and Controls)
As discussed in Chapter Two, the System 360 wasn’t the only computer product available in the sixties or seventies and many users were influenced by ideas generated outside the COBOL/360 world’s digital continuation of 1920s card processing methods. That, of course, led to increasing user unhappiness as the gulf between what they read about computers and what their own systems people delivered widened.
The resulting stand-off between Finance, through which Systems normally reported and not traditionally a bastion of user empathy, and the user community led to the evolution of a number of governance best practices. Those, in turn, rapidly co-evolved a system of formal controls - a set of minimum standards which, if adhered to by both users and systems, will nominally mark a secure and successful operation.
These controls achieved their clearest formulation to date in the CoBiT Framework for Governance, Control, and Audit for Information and Related Technology developed by the Information Systems Audit and Control foundation (ISACF) and available through isaca.org.
CoBiT sets the gold standard for audit and control work in mainframe data centers.
The CoBiT Framework establishes four governance domains:
- Planning and Organization;
- Acquisition and Implementation;
- Delivery and support; and,
- Monitoring
These four domains relate to a total of 34 high level processes giving rise to 302 specific control objectives like:
3.6 Workload forecasting
Controls are to be in place to ensure that workload forecasts are prepared to identify trends and to provide information needed for the capacity plan
The key control documents are:
- The service level agreement (SLA). This document sets out the user’s service expectations, when the system will be available, what processes are in place to react to problems, and who is responsible for what delivery step. All other controls are, from the user perspective, either subservient to, or triggers for, the SLA.
- Systems documentation is seen as key to ensuring continuation of systems services. Are procedures documented? are the SDLC stepwise documents available for every significant system?
- Role documentation and the separation of functions. Is production wholly independent of testing? and so on.
- Proper process documentation and the separation of responsibilities are seen as integral to the security of information processing. Do the people who fill key roles have the experience and certifications needed for those roles? Are change steps demarcated by organizational walls that require management knowledge and approval to cross?
- The disaster plan describes how the data center will react in the event of a predictable disaster such as equipment failure, fire, or flood.
- The capacity plan, which may have human resource and succession attributes, sets out the data center’s short and long term plans to meet growth in user processing needs.
The importance of the service level agreement is hard to over state. The SLA is the “peace treaty” between users and the data center against which subsequent actions or demands from either side can be judged. All other documents are sub-servient to it - the disaster recover plan is, for example, simply evidence that the SLA can be honored during the aftermath of a systems failure. Furthermore, annual data center budgets and new project authorizations are often set by committees with user input specifically required under the SLA. Similarly, service appeals processes are generally spelled out along with the rights and obligations of both sides.
Nevertheless it is common in organizations to find that the SLA has fallen into disrepute or is simply no longer maintained or enforced. In those situations users will often be found, on investigation, to have bypassed the data center for services they consider critical and to have adopted an attitude of resigned contempt for the data center in areas where use of its services is unavoidable.
July 17th, 2008
Dreams come true: network guy holds city hostage
A few well chosen words from the technically knowledgeable Jaxon Van Derbeken, a staff writer for the San Francisco Chronicle as published on July 15/08:
(07-14) 19:23 PDT SAN FRANCISCO — A disgruntled city computer engineer has virtually commandeered San Francisco’s new multimillion-dollar computer network, altering it to deny access to top administrators even as he sits in jail on $5 million bail, authorities said Monday.
Terry Childs, a 43-year-old computer network administrator who lives in Pittsburg, has been charged with four counts of computer tampering and is scheduled to be arraigned today.
Prosecutors say Childs, who works in the Department of Technology at a base salary of just over $126,000, tampered with the city’s new FiberWAN (Wide Area Network), where records such as officials’ e-mails, city payroll files, confidential law enforcement documents and jail inmates’ bookings are stored.
Childs created a password that granted him exclusive access to the system, authorities said. He initially gave pass codes to police, but they didn’t work. When pressed, Childs refused to divulge the real code even when threatened with arrest, they said.
…
Authorities say Childs began tampering with the computer system June 20. The damage is still being assessed, but authorities say undoing his denial of access to other system administrators could cost millions of dollars.
Officials also said they feared that although Childs is in jail, he may have enabled a third party to access the system by telephone or other electronic device and order the destruction of hundreds of thousands of sensitive documents.
Authorities have searched Childs’ home and car for a device that could be used in such an attack, but so far no such evidence has been found.
…
Childs, according to payroll records, earned $126,735 in base pay in 2007 and additional premium pay of $22,534, for a total of $149,269. Vinson said the extra money was apparently compensation for being on-call as a trouble-shooter.
Besides the shiny new definition of optical storage, I think the reporter here missed Wikipedia’s photo of the “device” the police were looking for in the back seat of the guy’s car.
All joking aside, lots of bosses worry about situations like this - and I’ve known rather a lot more than one (comes from being a consultant) IT guy who dreamt of doing something like this, but I’ve never seen anyone actually pull it off.
So I want to know two things: $149,269 for a network guy?! ; and, how?
I mean, really, how? the system is said to be running normally so he didn’t mass encrypt the databases - and what that leaves is BIOS changes if the servers are Windows and password control changes if the lockouts are based on router/switch controls. Either way, however, it’s a scriptable fix requiring, at worst, local connection to each device and a reboot/interrupt with a fully scriptable delete and reconfig - figure ten to fifteen minutes per device for a pretty finite number of devices.
So - somebody tell me: how could this kind of thing be done?
July 15th, 2008
Sun’s ZFS/Flash initiative
Up to now, the coolest thing about ZFS, besides really making RAID cheap and easy to implement, has been its ability to silently correct the bit errors that creep in as data is stored, read, and written - a facility that’s been particularly important to the raidz implementation.
In the near future, however, that’s going to change and the coolest thing about ZFS is going to be its ability to make intelligent use of large amounts of flash memory to simultaneously speed disk I/O while letting you lower platter rotational speeds (and thus both wear and power use) to something on the order of 5400 RPM.
One of the core developers, Adam Leventhal, has an interesting article in the July ACM on how this going to work - the technology is non obvious, but the bottom line is simple: much faster, cheaper, and more reliable storage for big installations.
Properly configured systems using ZFS with flash in the storage hierarchy ahead of traditional disk should offer dramatic (order of magnitude) throughput gains on things like database transactions - and virtually eliminate some processing crisises I’d guess virtually all serious sysadmins have had to face.
Disk reconstruction delays and risks will, for example, essentially disappear - and if you mirror on two of the new JBOD arrays, layer in flash, and run something like Oracle or PostGresSQL, almost all of your backup and recover delays will disappear too.
More interestingly, there are oddball RDBMS admin problems that will get easier to resolve: a lot of production systems, for example, get constrained when databases grow beyond the point that backup and PC style table inversions (aka “cube” computation) can be done in the time available. In the past the right answer (switch to something more modern that queries the production system directly) has usually been administratively impossible, the fast answer (dump to text and use Perl) usually produces howls from outraged PC people, and the wrong answer (recreate the database schema you need on /tmp and run the inversion in memory mapped space) often becomes the only one that both works and doesn’t create extensive conflicts with the MCSE crowd.
No more - with the ZFS/Flash layer in your storage hierarchy you can flash freeze the database without stopping production, and then do both your off-line backups and inversions at production I/O rates using whatever cycles are available -whether the users are on-line or not.
Most sysadmins will have run into this problem at some point - but there’s a special variation too: one that’s rare in general business but important to Sun’s military and telco markets. What happens in these cases is that business needs make any delay on some transactions data unacceptable; so you set things up to cache the critical stuff in memory while keeping only a few indexes to the collateral data there - and then the database grows faster than you can get more memory. As a result you find yourself trying to continually re-optimize your index and caching to keep up - and that activity itself then causes more problems (especially when your boss brings in consultants “to help”). With ZFS managing the storage hierarchy and flash at the front, however, the distinctions disappear from view - and your system self-adapts as volumes change.
You can’t buy ZFS/flash yet - and I’m guessing that when you do (early next year? - but note that it took Sun three years from the introduction of ZFS to its first new JBOD products) you’ll pretty much need to run a workload measurement utility whose results determine the custom configuration you’ll be ordering from Sun. Timing aside, however, I think that the bottom line on this for anyone now using Solaris for larger applications is that this stuff is going to be important - and that getting a running start by reading what you can and experimenting with the key ideas now (using ramdiskadm with ZFS) will pay off for you.
July 15th, 2008
Do problems with Wikipedia presage social networking’s end?
Wikipedia is supposed to be the on-line encyclopedia for everyone, but what it has become is something entirely different: an early and illustrative warning of the collapse from informed social networking to propaganda.
Wikipedia’s central premise is that inviting anyone and everyone to scrutinize and correct encyclopedia information is the best guarantee of honesty and completeness - it’s a very democratic idea and broadly similar to Raymond’s comment about no bug being invisible to a million eyeballs.
Like democracy itself, this ideal assumes that the participants are both educated and honest: educated enough to understand the issues, and honest enough to fairly report facts and consequences separately from speculation and opinion.
Look at Wikipedia as a focal point for niche groups - those interested in each major subject grouping - and it’s a pretty clear application of the same ideas underlying something like sourceforge or any other open source community center. In that context, therefore, my concern about Wikipedia’s apparent failure to meet its mission mandate is that this illustrates something fundamental about social networking: specifically, that it inevitably degrades to the lowest level acceptable to the most committed players in each niche.
Consider two illustrative niches: a controversial one focused on climate change, and the apolitical discussions centered around processor architectures.
On climate change consider this extended quotation from a National Review article by Lawrence Solomon:
—
In theory Wikipedia is a “people’s encyclopedia” written and edited by the people who read it - anyone with an Internet connection. So on controversial topics, one might expect to see a broad range of opinion.
Not on global warming. On global warming we get consensus, Gore-style: a consensus forged by censorship, intimidation, and deceit.
I first noticed this when I entered a correction to a Wikipedia page on the work of Naomi Oreskes, author of the now-infamous paper, published in the prestigious journal Science, claiming to have exhaustively reviewed the scientific literature and found not one single article dissenting from the alarmist version of global warming.
Of course Oreskes’s conclusions were absurd, and have been widely ridiculed. I myself have profiled dozens of truly world-eminent scientists whose work casts doubt on the Gore-U.N. version of global warming. Following the references in my book The Deniers, one can find hundreds of refereed papers that cast doubt on some aspect of the Gore/U.N. case, and that only scratches the surface.
Naturally I was surprised to read on Wikipedia that Oreskes’s work had been vindicated and that, for instance, one of her most thorough critics, British scientist and publisher Bennie Peiser, not only had been discredited but had grudgingly conceded Oreskes was right.
I checked with Peiser, who said he had done no such thing. I then corrected the Wikipedia entry, and advised Peiser that I had done so.
Peiser wrote back saying he couldn’t see my corrections on the Wikipedia page. I made the changes again, and this time confirmed that the changes had been saved. But then, in a twinkle, they were gone again. I made other changes. And others. They all disappeared shortly after they were made.
Turns out that on Wikipedia some folks are more equal than others. Kim Dabelstein Petersen is a Wikipedia editor who seems to devote a large part of his life to editing reams and reams of Wikipedia pages to pump the assertions of global-warming alarmists and deprecate or make disappear the arguments of skeptics.
I soon found others who had the same experience: They would try to squeeze in any dissent, or even correct an obvious slander against a dissenter, and Petersen or some other censor would immediately snuff them out.
Now Petersen is merely a Wikipedia editor. Holding the far more prestigious and powerful position of administrator is William Connolley. Connolley is a software engineer and sometime climatologist (he used to hold a job in the British Antarctic Survey), as well as a serial (but so far unsuccessful) office seeker for England’s Green party.
And yet by virtue of his power at Wikipedia, Connolley, a ruthless enforcer of the doomsday consensus, may be the world’s most influential person in the global warming debate after Al Gore. Connolley routinely uses his editorial clout to tear down scientists of great accomplishment such as Fred Singer, the first director of the U.S. National Weather Satellite Service and a scientist with dazzling achievements. Under Connolley’s supervision, Wikipedia relentlessly smears Singer as a kook who believes in Martians and a hack in the pay of the oil industry.
—
If you just enter “global warming” in google the first result you get points to the Wikipedia entry Connolley controls - and if you just wanted a two minute briefing on the subject you’d never know that the article is utterly and relentlessly dishonest.
It’s possible to argue, however, that the global warming hypothesis was once grounded in science; has now become so politicized because this is the only way to get action; and therefore that the expression of activism through both censorship and exaggeration is justified. Moral issues aside, this is a means justifying the ends argument dependent on the provability of the underlying assumptions - a defense for Wikipedia which, whatever its merits with respect to climate change, would clearly not apply to something as apolitical as computer processor architectures.
And yet, the same kinds of distortions and omissions are found there.
Thus what’s been allowed into the Wikipedia’s primary article on CPUs focuses on Intel x86, includes numerous statements offered as fact that are simply not true (everything from the assertion that Intel was the first to create the microprocessor to the claimed design equivalence of Intel’s multi-core packaging to Sun’s CMT/SMP architecture), and essentially denies AMD’s role in repeatedly forcing innovation within the x86 framework.
Exclusions work the same way too: just as the climate change article allows no hint of dissent, the CPU article gives the reader no hints that PPC derivatives dramatically out sell and out perform x86 while using less power and costing significantly less per operation per second.
What’s going on in both cases is that sub-groups of the general community have captured these niches and are now using Wikipedia as a marketing tool for their viewpoints - and while that’s expected and reasonable for agenda sites like groklaw or dailykos, it’s fundamentally inappropriate in a site nominally dedicated to the provision of objective information.
But, given the structure, how could it be otherwise? Wikipedia invites community members to contribute and edit information - and given that people who advocate a position are more likely to be deeply committed to a particular viewpoint than their audience, how could site editors who’re neither omniscient nor omnipresent hope to prevent the slide from objectivity to advocacy?
I don’t believe they can, and therefore see Wikipedia’s inability to maintain credibility in the face of its usefulness to propagandists as illustrative of a hypothesis I want to put forward for discussion: that no widely accessible social networking site can simultaneously give its users write access to the source material and protect itself from takeover by people with axes to grind.
July 14th, 2008
In the information age, IT innovation means everything
Here’s a bit from Sun president Jonathan Schwartz’s blog:
I remember a dinner I had a while back with the CEO of a global financial services firm. As one of his first acts as CEO, he’d canceled an enormous outsourcing contract, and I’d asked him why - his response has stuck with me. “Banking is a technology business. Pure and simple. I can’t win if I don’t have my own team.”
Independent of his views on outsourcing, I’ve heard the same point made by many (but not all) financial services executives - banking (like big swaths of telecommunications, media and retailing) has become a technology business, where every ounce of performance and differentiation matters. Even, and especially, in the midst of market turmoil.
I think this is true - but I don’t think it’s limited to the financial industries. Everywhere you look the opportunities are in things that get away from the bland one size fits all strategy of using the same software, on the same hardware, and with the same interchangeable people, as everyone else.
In the crying until you laugh category, for example, it turned out recently that both sides and the Judges involved in a recent U.S. Supreme Court judgment missed the applicability of an explicitly on-topic Act of Congress: the military justice provisions in the the National Defense Authorization Act (2006) - apparently because the legal research tool used by all three groups turns a blind eye to military and related law.
Less spectacularly, but more generally, the overall Canadian and American economic advantage in manufacturing stems almost entirely from the combination of advanced engineering with a committed, knowledgeable, labor force - both of which can be imagined as peaks on pyramids whose lower layers are based on information and information exchange.
At the political level this is what the English Firsters are about, but at the IT level this is about the impact of computing, the value of the internet as a competitive tool, and the role innovative IT has played, and continues to play, in turning high cost, and high quality, resources into competitive advantages in world markets dominated by subsidized, lower cost, lower quality, producers.
In some sports, like skiing, swimming, and running, the outcomes of enormous efforts in training and preparation depend on edges whose effects are measured in fractional seconds - and in today’s hyper competitive volume manufacturing businesses the compromises made when you apply general tools to specific problems guarantee failure precisely because each generalization adds the equivalent of fractional seconds to each step in the process.
You can see this writ large, for example, in the way Boeing’s fortunes in the global market for commercial aircraft since about 1980 have consistently moved in a lagged lockstep with its IT strategies: pulling strongly ahead of the competition during its 7×7 development era adoption of the Vax with Unix and NCD displays, falling dramatically backwards during the jihadist Windows period - and, more recently, showing an emerging resurgence with improving computational diversity, an end to some crippling out-sourcing contracts, and the wider adoption of key applications and ideas developed at Lockheed and other defense contractors.
You can’t really do a survey of reality in a blog, but look around you: everywhere you see a significant economic (or business) success these days you see innovative IT - and none of that comes from adopting the other guy’s ideas, the other guy’s software, or the other guy’s IT people.
Wintel’s market dominance has, I think, had the effect of raising barriers to innovation in “mainstream” computing - with the effect that almost all business application and telecommunication innovation going on now takes place in the L’Unix universe: Solaris, MacOS X/BSD, and Lintel, simply because that’s where the barriers are lowest.
All of which leads to a pair of simple, bottom line, prescriptions for larger businesses caught in competitive environments: kill the out-sourcers, and start a Unix skunkworks - because using the same people, the same ideas, the same software, and the same hardware as the other guy is a recipe for stagnation; not innovation, and not success.
Paul Murphy (a pseudonym) is an IT consultant specializing in Unix and related technologies. See his full profile and disclosure of his industry affiliations.
Essential Topics 
Essential for School
- Help get your kids get A+ ready with Microsoft® Office 2007 Home and Student Edition
- Buying a PC? Don't forget Microsoft Office 2007!
- Easier formatting. Better Results. Visit Office Online now to learn more.
- Attention back-to-school shoppers: Find your laptop reviews here
Recent Entries
- DTrace and the Linux bunker mentality
- How to lose customers by not even trying
- Apache’s open source governance model
- If sameness is a recipe for losing, how do you win?
- The iPhone meets Journalism
Most Popular Posts
- Dreams come true: network guy holds city hostage
- Reasons to buy a Mac
- Sun's ZFS/Flash initiative
- Do problems with Wikipedia presage social networking's end?
- From stunned outrage to stunned laughter in two seconds
- Harry Potter and the da Vinci book of Sudoku
Top Rated
- Achieving magical results in IT+9 votes
- Sun's ZFS/Flash initiative+6 votes
- How to lose customers by not even trying+5 votes
- The top Sun Ray myths+4 votes
- DTrace and the Linux bunker mentality+3 votes
- Dreams come true: network guy holds city hostage+3 votes
- From Chapter one: Data Processing and the IBM Mainframe+2 votes
- Reasons to buy a Mac+2 votes
Premier Vendor Content Whitepapers, webcasts & resources from our Power Center Sponsors
- Hardware Assisted Virtualization to Mitigate Security Risks
-
In this podcast Intel's Malcolm Harkins and HP's Manny Novoa chat about the latest issues in security technologies, notably the emergence of hardware assisted virtualization.
- Watch the Podcast on the Intel Open Port General Blog >>
