Category: News
March 24th, 2008
Fixing the unfairness of TCP congestion control
Bob Briscoe (Chief researcher at the BT Network Research Centre) is on a mission to tackle one of the biggest problems facing the Internet. He wants the world to know that TCP (Transmission Control Protocol) congestion control is fundamentally broken and he has a proposal for the IETF to fix the root cause of the problem.
The Internet faced its first congestion crisis in 1986 when too much network traffic caused a series of Internet meltdowns when everything slowed to a crawl. Today’s problem is more subtle and lesser known since the network still appears to be working correctly and fairly. But underneath that facade and illusion of fairness, a very small percentage of users hog most of the Internet’s capacity suffocating all other users and applications.
Solving the first Internet meltdown crisis
In October of 1986, the Internet began to experience a serious of “congestion collapses”. So many computers were piling their traffic on to the network at the same time that the network came to a grinding halt and no one got any meaningful throughput. By mid 1987, computer scientist Van Jacobson who is one of the prime contributors to the TCP/IP stack created a client-side patch for TCP that saved the day. Every computer on the Internet - roughly 30,000 in those days - was quickly patched by their system administrators.
Jacobson’s TCP stack patch worked by causing a computer to cut the flow rate of its TCP stream in half as soon as it detects any packet loss. Packets are lost whenever the routers relaying them receive more packets than they can forward and the router begins to randomly drop packets across the board. But whenever a computer sees an acknowledgement that its packet arrived successfully, it quickly and continually increases its flow rate with every acknowledgement until it experiences another packet drop at which time it cuts its throughput in half again. This became known as the AIMD (Additive Increase Multiplicative Decrease) algorithm where the sending computer is constantly probing for the maximum allowable bandwidth by repeatedly increasing throughput until it crosses a line and gets knocked down.
Jacobson’s AIMD algorithm also allowed a new TCP stream to open up and quickly rise to equilibrium where it attains the same flow rate as all other TCP streams. Conversely when a TCP stream ended transmission, the extra bandwidth freed up would be evenly distributed amongst the remaining streams. Van Jacobson’s patch was so successful that it became a part of the TCP standards and it hasn’t fundamentally changed for over 20 years and according to Bob Briscoe, Jacobson’s algorithm is the “fifth most cited academic paper in all of computer science”.
Under Jacobson’s algorithm which sought out to balance the flow rate (throughput) of each TCP stream, the system was more or less fair to everyone who wanted to use the network so long as everyone used an equal number of TCP streams. Since people typically used one TCP stream at a time and people had limited usage on those time-sharing computers in the 1980s, Jacobson’s algorithm was adequate for the problems of that era. While it was possible for someone to open two FTP downloads or uploads at a time and get double the total throughput than anyone else, this wasn’t a big problem when applications and operating systems were mostly limited to text and computers were limited to academic and large corporate institutions. But as time went on and as the number of applications and users grew, it was only a matter of time before the fairness of the system would be exploited.
March 16th, 2008
Japan's ISPs agree to ban P2P pirates
Four of Japan’s largest Internet provider organizations have come to an agreement with copyright holders on how to tackle the illegal file trading on P2P (Peer to Peer) networks. Comprised of about 1000 major and smaller Japanese Internet providers, the four organizations agreed to target flagrant copyright violators by first warning them and then banning them if their behavior doesn’t change.
According to the Daily Yomiuri Online, the Internet providers two years ago attempted to disconnect users anytime they detected the use of Winny (a popular Japanese P2P application) or any other file-sharing software. But that ran afoul of the Japanese Ministry of Internal Affairs and Communications because of concerns of privacy and the providers abandoned that practice. This time the Internet providers seem to have learned from the past and they’re going to be much more targeted by going after the most obvious transgressors of illegal file trading.
When the copyright owners see a list of IP (Internet Protocol) addresses downloading their copyrighted content, they’ll send that list of violators to the ISP (Internet Service Provider) and the ISP will warn and then ban the copyright infringers if necessary. This method doesn’t involve any of that politically dreaded DPI (Deep Packet Inspection) since the copyright owner merely needs to look for their own content on the popular file trading sites and ask for a list of peers by merely participating in the file trade. Not only does this method avoid privacy concerns, it also happens to be the most practical if not the only way of attacking the problem since many file trading applications are already completely encrypted against packet snooping.
Update 5:40AM - Just to make myself extra clear since many people refuse to believe that we are not talking about deep packet inspection here. P2P in Japan like the latest “Perfect Dark” application (successor to Winny and Share) is already fully encrypted at both the protocol and data level. That’s encryption is completely bypassed since the content owners merely need to download the Winny, Share, and Perfect Dark and look for their own content that’s being pirated. Then all they need to do is connect to it as if they were a user and then download the content to see if it is indeed their content. Then they already have a list of IP addresses that participated in that file exchange. There’s no decryption, key cracking, or deep packet inspection going on here.
Japan is considered one of the most connected broadband nations on the planet with widespread 100 Mbps broadband service. Many people in this country believe that by simply offering more capacity, there would be no need to manage the network since congestion problems would be gone. But Japan teaches us that no matter how much capacity you throw at the problem, congestion will always be a problem and the vast majority of it will be caused by P2P traffic.
At the iGrowthGlobal Panel on Network Management on Capitol Hill (my recap here), I met Haruka Saito who is Counselor for Telecom Policy from the Embassy of Japan. Mr. Saito was my fellow panelist and he shared the following data with the congressional and FCC staffers in the audience. He presented the following data from the Japanese Ministry of Internal Affairs and Communications which had been studying the issue of Net Neutrality in Japan for more than a year.
[Updated 3:15PM - I had incorrectly stated that 1% consumes 63% of all traffic because I read the charts wrong. The corrected text is in bold below.] As you can see, the utilization levels especially for uploads are dangerously high and that P2P traffic absolutely dominates both upload and downloads by a very large margin. Winny, WinMX, and Share (a successor of Winny) dominates the P2P usage. From this data, the P2P users that make up 10% of all Internet users in Japan hog ~75% of bandwidth resources and 1% of all Internet users in Japan consume 63% of that 75% share. That means just 1% of users consume 47% of all the Internet traffic in Japan. It’s no wonder the ISPs in Japan want a solution that cuts off the most egregious illegal file traders who also happen to be the biggest bandwidth hogs.
March 7th, 2008
Asus' 8.9" Eee draws crowds at CeBIT
Here in CeBIT 2008, crowds descended on Hannover Germany to see the latest technologies. Germany is certainly a lovely country but there’s nothing lovable about the 5.60 Euro per gallon gas prices.
CeBIT is certainly one of the more unique conventions I’ve been to since everything is spread out over a square kilometer and it’s like going to 10 mini conventions. While you get some outdoor air between the halls, don’t expect any fresh air with all the smokers there. The temperature delta certainly makes proper attire a challenge because it’s too warm inside and freezing outside.
Asus had a massive presence in building 26 which is one of the more popular spots at CeBIT and they managed to draw crowds wanting to get a closer look at the new and improved 8.9″ Asus Eee PC. The new 8.9″ Asus Eee comes with more SSD flash storage, a bigger LCD screen with 1024×600 resolution, a better quality webcam. The same Pentium M 900 MHz CPU is the same as the original Eee. [See gallery for a close-up view.]
 |
 |
The Windows XP model comes with 8 GBs of SSD flash memory when the Linux model comes with 12 GB of SSD flash memory. So far we only know that the price will be 399 Euros (which typically means it will be fewer in dollars for the US market), but we don’t know if there will be a price difference between the Linux and Windows XP model. It is possible that the price of the flash memory offsets the licensing costs of Windows XP.While holding the lightweight Eee with one hand, I tested the quality of the Mic and the Webcam and confirmed that the quality if fairly good. The Webcam is definitely much better quality than the old Eee. The Eee also comes with a wired 10/100 Ethernet port as well as 802.11g. The one down side to the Eee is that it doesn’t have a DVI output and instead has a DB-15 VGA port.
Here’s a comparison of the older 7″ Asus Eee versus the 8.9″ Eee. As you can see, the screen is much bigger and the color and contrast appears to be much better. The speakers had to be moved to the bottom of the laptop because the bigger screen pushed them off the lid. You can also see that the track pad is also larger.
I wouldn’t doubt if people buy the 12 GB Linux version and use NLite to install a trimmed down version of XP though having Linux on this device is extremely useful if you’re going to use it as a security auditing tool. The 8 GBs of SSD is more than enough to hold the OS and key applications and a $60 16 GB SDHC card is more than sufficient to hold plenty of movies and data. With the larger screen and nicer webcam and adequate microphone, it becomes a great Skype video conferencing solution. The bottom line is that the Asus Eee is very pleasing in the hands and it runs Windows XP very quickly if you keep bloatware/crapware off of it.
March 5th, 2008
AT&T's degrading service and my landlord's ban on Comcast
With all the negative attention headed towards Comcast lately, AT&T’s problems seem to be slipping below the radar. Unfortunately for me, those problems are first hand for me as I’m personally suffering degradations in speed. As if getting 1200 Kbps downstream on a so-called 1500 Kbps service and all those outage problems (example here and here) weren’t bad enough, my AT&T DSL service has declined. I suppose I could count myself lucky compared to my Mom’s neighbor who only got 320 Kbps service after AT&T unilaterally and without permission “upgraded” his bill to the 1500 Mbps service without upgrading his performance. Seem my DSLReports.com speed tests below.
The results above were performed at the nearest locations to my home and they were performed on idle servers with barely anyone using them. That pretty much confirms the problem is on AT&T’s end and possibly on the last mile. My Mother’s so-called 768 Kbps service only delivered about 330 Kbps but after the AT&T fixed some wiring problems outside the house, the performance went up to about 600 Kbps. I’ll have to call AT&T and see if they can do anything about my problems when I get back home.
It’s gotten so bad with my service that I’m actually starting to yearn for some of those “evil” TCP resets from Comcast to grace my router. Even more frustrating is that Comcast might actually be offering DOCSIS 3.0 with 15 Mbps downstream and 2 Mbps upstream in my area, but I live in one of these draconian housing complexes that force us to pay for bundled inferior analog cable service even though I don’t use it. The FCC has ruled against these types of exclusive contracts but I don’t think that can overturn my current situation. I think I’m finally motivated enough that I want to start a petition with the neighbors to demand the right to use Comcast. I’ll definitely have to bring this up the next time I go to Washington DC before Congress and the FCC.
Some people have told me that I should have looked at the contract before I moved in but it really isn’t that simple. There are about a thousand homes in the same multi dwelling unit and we don’t have much of a choice on where we live when an old and small home in Silicon Valley is $650,000. I do find it ironic that I’m now begging and fighting for the right to get Comcast service while others are fighting to kill Comcast.
This isn’t to say that Comcast is always good and AT&T is always bad although I’ve always gotten much better service from Comcast when I actually had a choice between the two. But it is so critical that we have competition between the two so that they have to fight for my business. The way it stands now, AT&T pretty much knows that I have no other game in my area and they have zero incentive to deploy U-Verse in my neighborhood let alone fiber-to-the-node like Verizon’s FiOS service.
What’s even more frustrating is that this isn’t a rural area problem since I’m in the heart of Silicon Valley with about 4000 homes jammed tight in a two block by two block neighborhood. It would be a Verizon FiOS installer’s dream deployment with homes packed so tightly together. With my landlord out of the way, I’d have DOCSIS 3.0 15 Mbps service to choose from and AT&T would prioritize jumping in here with U-Verse service. These are the real problems facing consumers today and not whether a few bandwidth hogs get throttled or not and I hope others will join me in a worthwhile cause.
March 3rd, 2008
A geek's trip to Capitol Hill on Network Management
I appeared before congressional and government staffers on Capitol Hill for a panel on Network Management sponsored by iGrowthGlobal. This was my first time in Washington DC and while it was a little cold for my Californian bones, it was a beautiful city and seeing the capitol of the nation was certainly a worthwhile experience. One thing that struck me was how large and spread-out the Capitol was with so many Government buildings several miles apart.
The panel was moderated by Scott Wallsten, VP for Research and Senior Fellow of iGrowthGlobal. I met Mr. Wallsten at the Net Neutrality summit held at University of San Francisco last month where the two of us presented on separate panels. The rest of the panelists for this event were:
- Melvin Ammori, General Counsel, Free Press
- David Burstein, Editor, DSLPrime
- George Ou, Editor at Large, ZDNet
- Haruka Saito, Counselor for Telecom, Embassy of Japan
- Christopher S. Yoo, Professor of Law and Communications, University of Pennsylvania
Christopher Yoo -
After a brief introduction by Scott Wallsten who explained that the order of the presentations will be reverse alphabetical order, Christopher S. Yoo kicked off his presentation. Professor Yoo explained that networks, like roads, aren’t built for everyone to use them at the same time. Yoo gave the example that if a person wants to know how fast he can travel on a freeway, he wouldn’t know until he got there because we can’t predict exactly how many other people will be on the road at the same time. Yoo explained the difficulty in projecting network capacity and that we can’t always be right when determining whether more capacity or network management was the answer. Sometimes more capacity is the answer, sometimes network management is the answer and we shouldn’t lock ourselves in to one solution or the other.
Haruka Saito -
Next up was Mr. Haruka Saito from the Embassy of Japan. Mr. Saito explained that Japan had been studying and debating the issue of Network Neutrality in Japan for about a year and a half and he offered a lot of hard data gathered in Japan. Japan is one of if not the most connected nation in the world when it comes to broadband deployment with 100 Mbps fiber deployments and despite this abundance of capacity, even I was shocked that they were running in to congestion problems.
When the traffic chart was broken down in to color-coded regions showing application usage, P2P easily ate the lion’s share of resources and dwarfed everything else on the chart. Mr. Saito went on to explain that 1% of the users primarily through P2P consumed around 50% of the total capacity and this pretty much mirrors every other study I’ve seen elsewhere in the world regardless of capacity. The debate in Japan was who was going to pay for this excessive usage and whether the heaviest users should start paying more money.
George Ou -
Next up was me and I gave a presentation based on my Comcast versus Vuze and Comcast before the FCC article. After Mr. Saito’s presentation, it certainly made my job a lot easier showing my charts on how BitTorrent and P2P were effectively the primary bandwidth hogs. I explained that the vast majority of all web applications like Web surfing, YouTube, Apple iTunes video downloads, Xbox Live Marketplace video downloads, and other applications like email almost never use any upstream capacity. Even though there are applications like Skype High Quality Video Conferencing which can fully saturate the upstream pipe, its duration is relatively short which significantly lowers its average load on the network.
I then explained that Vuze using the P2P model shifts nearly all of its server, storage, and bandwidth costs to its customer’s computer and the broadband providers while all other video distribution services pay for their own distribution costs. Then I explained that Cable networks and Wireless networks are shared-medium networks that are constrained in capacity and that they weren’t built nor sold to be content servers for the rest of the Internet. Wireless networks are even more scarce in terms of capacity because of the scarcity of spectrum and many of the smaller ISPs would be put out of business if the Government made rules banning P2P throttling or P2P blocking. Without those smaller wireless ISPs that cover the rural areas that the larger companies don’t want to cover, those Americans living in rural America would be cut off from the Internet and possibly even their phone service. We have plenty of choices on getting content but few choices on broadband carriers and the Government must keep this in mind when making network management policies.
David Burstein -
David Burstein went up next to give his presentation though he didn’t actually have a presentation ready so he improvised the presentation. After indirectly but clearly referring to Professor Yoo as an “idiot”, Burstein told the audience that if only Comcast would upgrade to DOCSIS 3.0, then there wouldn’t be any need to manage the network. That seemed to fly in the face of the hard network traffic data that Mr. Saito presented indicating that even a 100 Mbps per home dedicated fiber network would have congestion problems due primarily to P2P traffic. Burstein continued to insist that a measly DOCSIS 3.0 network (which is 120 Mbps shared between a few hundred users) would somehow be immune to congestion problems.
Even stranger was Burstein’s testimony that it would only cost Comcast 10 cents per user per month to upgrade everyone to DOCSIS 3.0. When pressed where he got such a number, Burstein Then he admitted it was only a guess but insisted that until someone proves him wrong, then everyone should laugh in the faces of his doubters. I didn’t bother challenging Burstein on the spot since there were so many other things I wanted to say, but I will respond to him here.
If we take Burstein’s estimate at face value, then we would have to believe that a DOCSIS 3.0 CMTS (Cable Modem Termination System) along with a ~250 DOCSIS 3.0 cable modems could be had for a cheap total of $50 for the entire neighborhood per month. Now bear in mind that the typical DOCSIS 2.0 modem costs about $60 and a CMTS is about the size of a 40U rack and falls under the category of very specialized networking gear. A more common Cisco switch half the size would easily cost a quarter million dollars so it wouldn’t be surprising if a CMTS costs upwards of half a million dollars. With 500 users on a CMTS loop (Cable TV with typically half of them subscribing to cable broadband), the costs will at least be $1000 per user for just the CMTS and we haven’t even begun to look at the costs of upgrading the surrounding infrastructure to support the higher capacities and the cable modems.
[Update 3/4/2008 - Dave Burstein has asked me to issue a correction that he stated it was 10 cents per user PER MONTH. I do apologize for my error, but it doesn't really change the fact that the correct number from Burstein has little to do with reality. At 10 cents per user per month, it would take 10,000 months or 833 years to break even on a minimal $1000/user investment.]
Marvin Ammori -
Marvin Ammori from the Free Press went up and also improvised a presentation. He kicked it off with a cheap shot saying how he was glad that Professor Yoo and I didn’t bring a busload of chair warmers and attempted to paint the two of us as industry shills. Ammori then went on to build a straw man argument that he thought my position was that YouTube didn’t pay their fair share of the Internet. Ammori obviously never saw my article from last year where I ripped Ed Whitacre’s statements that Google didn’t pay their fair share on Internet connectivity. After Ammori finished his presentation, I let my displeasure be known that I spoke as a proud American citizen who was in Washington DC for the first time with no one paying me to speak.
One other interesting tidbit was the fact that Mr. Ammori who admittedly never heard of the word “BitTorrent” up until a few months ago claimed that BitTorrent will only do 4 upstream sessions. Since Ammori told us that he heard it from Professor Edward Felton [waiting for Ammori's clarification on who he heard it from], somehow that overturns my testimony that BitTorrent was a bandwidth hog that opened 10s of upstream sessions. The reality was that certain BitTorrent clients will default to 4 upstream sessions for each torrent, but multiple torrents meant multiples of 4. The other interesting claim that Ammori made was that BitTorrent was intelligent and kind enough to back off when your neighbor was trying to use something like a web or email application. Where exactly Ammori got this information wasn’t clear, but I’d like the Free Press to show me some documentation for a protocol that no one has ever heard before.
[UPDATE 3/4/2008 - Ammori emailed me that he didn't say it was from Ed Felton despite the fact that he mentioned Ed Felton's name in the closest proximity to as far as my memory is concerned. Ammori writes in his email that he had named David Reed, David Clark, and Ed Felton as the three expert witnesses he cited, but has so far refused to clarify which one told him that BitTorrent maxes out at 4 upstream sessions. Strangely, Ammori seemed a lot more confident of his source when testifying before the government to bolster his claims and discredit mine but now he refuses to clarify his source when he is shown to be wrong. At this point I don't know if Ammori was given the wrong information or didn't understand what he was told, but either way he gave bad testimony.
Instead of offering clarification, he took a few more shots at me the same way that he attacked Richard Bennett implying that we're somehow not qualified and that we're "brought in" by Comcast which has no truth. Then just as he did at the panel last Friday, he insists that his sources are better even though none of his sources have disputed anything I or Richard Bennett has said. Richard Bennett is one of the pioneers of the Internet and he's written some very informative and articulate articles on this matter and he's also faced off with Ed Felton in podcasts. You can hear the podcast for yourself but I think you'll find that Richard Bennett held his own against Ed Felton and Richard has far more expertise on this particular subject matter.
During his presentation, Ammori also tried to discredit the data I showed where P2P seeding was pretty much the only application that hogged the upstream. In the context of the hard data presented by Mr. Saito from the Japanese Ministry of Internal Affairs and Communications showing that P2P was undoubtedly the upstream and downstream bandwidth hog, it was shocking that Ammori would try to continue disputing that fact. Ammori basically argued that we can't really know if the charts I used (copy here) are legitimate or not and he made a habit of trying to discredit me with no factual data to counter. It will be interesting to see if he's willing to explain exactly which expert he was citing.]
During the informal panel debate after everyone had spoke, I brought up the fact that Comcast gives you web space to post content which operates 10 times faster than any BitTorrent seed. This apparently wasn’t good enough for Mr. Ammori and he felt that this was somehow impinging on his right to free speech since he couldn’t serve out high-definition video content from his own home. Never mind the fact that we’re in a unique time in history where for the first time user generated content on YouTube can have a huge impact on the election. Anyone can put up a political ad on YouTube and get millions of people to watch it if the video was clever enough, but the fact that Ammori couldn’t serve it in High Definition from his own home was somehow a violation of his first amendment. But the fact of the matter is that you can serve HD video from your own home if you pay for a commercial-grade Internet connection that allows you to host servers. What you don’t have the right to do is buy a cheaper residential-grade Internet connection, hog the scarce resources by serving content to the whole world and violate the terms of service.
So to sum it up, it was knee deep in politics experience but it was all worthwhile. I felt honored that I had contributed something to my Government and my Nation.
[Update 3/4/2008 - Since this post is obviously being told from my viewpoint, I will be happy to link to any of the other speaker's blogs rehashing their experience if they write anything regardless of whether I agree with them or not.]
March 2nd, 2008
Intel christens Silverthorne as "Atom"
Intel has officially announced its new branding for the “Silverthorne” processor and the “Menlow” platform. The Silverthorne processor will be called the “Intel Atom”. The Menlow platform will be called “Intel Centrino Atom”. The Intel Atom processor will be used in the Intel Centrino Atom platform. The new Atom logos are shown below.
Intel released technical details of the new Silverthorne processor last month at ISSCC 2008. This latest announcement gives Silverthorne and Menlow their official branding and their official logos. Intel also released high resolution die shots at the right hand side of their press release. A cut down rotated version of the die shot is shown below.
Here’s a summary of the new “Atom” processor:
- Equivalent on single-threaded performance to original Pentium M “Banias” processor. Faster if SSE3 instructions are used in the application or if multiple threads are involved.
- 0.6W TDP (Thermal Design Power) to 2.5W TDP
- Up to 1.8 GHz and DailyTech says sources inside Intel are saying that the 500 MHz version goes down to 0.6W TDP.
- Idle power consumption can drop as low as 0.01W to 0.1W
- Deep power down C6 state
- Optimized register-file and cache 6T bits cells
- CMOS mode on quad-pumped FSB IO
- Split IO power supply
- Single CPU core 2-issue in-order pipeline
- SMT (Symmetric Multithread) architecture
- 25mm^2 die size (2500 CPUs per 300mm diameter wafer)
- Can achieve 2GHz core frequencies at 1.0V
- Intel VT (Virtualization Technology)
- Intel 64 architecture (formerly EM64T and compatible with AMD64)
Intel’s press release also mentions the processor codenamed “Diamondville”. DailyTech reported some leaked information that Diamondville would be released in a single and dual-core version at 4W and 8W TDP. Diamondville will be soldered on to an Intel 945GSE chipset motherboard and judging from the photo, it looks to be a replacement for the D201GLY and D201GLY2 developing market platforms. The Intel D201GLY2 uses a lower power Celeron 220 (Core Solo architecture) with a TDP of 17W so Diamondville is a huge boost in energy efficiency. The current D201GLY and D201GLY2 also utilizes a third party SIS chipset which doesn’t support S3 sleep/suspend states while the Diamondville 945GSE platform will.
Given the fact that it’s highly unlikely (too expensive) that Intel would design a whole separate CPU for this type of a solution, it is very possible that Diamondville is simply a soldered-on-motherboard derivative of Silverthorne and the dual-core version is simply an MCM (Multi Chip Module) version of Silverthorne. AnandTech’s Anand Lal Shimpi seems to agree with this theory and goes on to explain that the slightly higher TDP with slightly lower 1.6 GHz clock is simply due to a higher voltage allowing for much higher yields. Since this is for the low-cost value market segment, that theory makes a lot of sense.
At present time Intel seems to be hinting that Diamondville will also carry the “Atom” branding but they’re vague on the specifics. What is certain is that the emerging market will enter in to a whole new level of energy efficiency and the appliance/embedded do-it-yourselfers like me are drooling over Diamondville’s power specifications.
February 28th, 2008
Microsoft's free enterprise search is a must try
At the Heroes Happens {here} event in LA yesterday which saw the launch* of Windows Server 2008, one of the relatively hidden gems of the event in my opinion was Microsoft’s free** Search Server 2008 Express. It’s is a streamline install of Office SharePoint Server 2007 with almost all the enterprise search features that most users would want and is a must download for any Windows Server shop.
Even if you didn’t own Windows Server 2003, 2003 R2, or 2008, it would seem like a great way to build a very cheap enterprise search engine appliance with a minimal Windows Server 2003 or above license and a simple 1U server for less than $2000 which is a LOT less than a $30K starting price Google Search Engine appliance with a 500K document cap. Update 7:28PM - Wiredguy in the talkback pointed out that Google’s Mini search appliance starts at $3K, but that only indexes 50K documents and it doesn’t tie in to Active Directory as seamlessly and lacks Exchange support. If you’re a Windows shop with an IIS server sitting around with low CPU utilization which is quote common, adding Microsoft’s Search Server 2008 Express costs nothing.
So why would you want an enterprise search engine for your company or organization? Windows Vista (and XP users who add Windows Desktop Search or Google Desktop Search) know how useful it is to have relatively instant indexed search results for any document or email in their computer. But those benefits stop at the local computer because you don’t want every user crawling the network data resources redundantly since it would bring the whole network and server infrastructure to a halt.
An enterprise search engine gives you a centralized intranet website where users could go to a URL like search.mycompany.com and find any document in their entire corporate LAN (and to a lesser extent the WAN and some Internet sites due to bandwidth considerations). Google’s online search engine is great but it’s stopped dead in its tracks at the corporate firewall and there’s no way it can search your Exchange or Lotus Notes mail server or your file server documents. The enterprise search engine bridges an essential gap between desktop search and google.com. Documents or emails that would have been glossed over and forgotten about instantly pop up on the enterprise search server.
The search results are security-trimmed and active directory integrated so that the user will only see the documents that they have permissions to access. With an Intranet IIS web server set for seamless Active Directory authentication, the user merely goes to the search portal and they’re logged in automatically. The server can also be tuned to crawl the network at off-peak hours with full or incremental searches.
Microsoft’s Search Server Express comes preloaded with the following search connectors.
- File servers
- Web sites
- SharePoint websites
- Exchange Server public folders
- Lotus Notes
To make Search Server 2008 Express work, you’ll either need a free SQL Server 2005 Express database backend or Microsoft SQL Server 2005 and above. Using the free SQL Express will limit the server to 1 GB and 4 GB database size. Under most document sizes, a 4 GB index should allow you to index more files than the 500K document cap imposed by the $30K edition of the Google Search Engine appliance. Buying a SQL server license will still end up being far cheaper than buying the Google appliance. No matter what your opinion of Microsoft, I think this is one of those things that’s definitely worth a try. Enterprise search is finally affordable and it should become a fixture in any company’s server room or datacenter.
* This was also a 3-month post launch party for Visual Studio 2008 and 6-month pre-launch party for SQL Server 2008.
** People who already own a copy of Windows Server 2003, 2003 R2, or 2008.
February 26th, 2008
FCC hearings: Comcast versus Vuze
The FCC held its hearing on Comcast’s Network Management practices at Harvard University yesterday. Vuze executive Gilles BianRosa whose company filed one of the two FCC complaints against Comcast reportedly told the FCC yesterday that BitTorrent does not hog bandwidth. Since most Internet experts would dispute that claim, I generated the following hard data on the bandwidth consumption of various applications that run on the Internet.
Note: Richard Bennett who was an expert panelist at yesterday’s hearings informed me that BianRosa claimed that BitTorrent didn’t exceed the contracted limit. That however ignores the explicit “no server” clause in the terms of service and no broadband service was built to be fully saturated 24×7. This is why commercial grade T1 lines that offer less than half the speed of broadband connections costing 8 times less are $400 per month.
Bear in mind that the data below is in reference to upstream (upload) bandwidth consumption in kilobits per second since that is the focus of these FCC hearings. Also note that applications like web surfing hardly use the upstream at all since it’s primarily your clicks and URLs that are being transmitted to tell the web server where you want to go.
The following is a graph of the above chart
* Corporate VPN telecommuter worker using G.722 codec @ 64 Kbps payload and 33.8 Kbps packetization overhead
** Vonage or Lingo SIP-based VoIP service with G.726 codec @ 32 Kbps payload and 18.8 Kbps packetization overhead
*** I calculated that I Sent 29976 kilobytes of mail over the last 56 days averaging 0.04956 Kbps
It is interesting to note that before the advent of P2P applications, Broadband users were primarily downloaders and rarely did they ever upload. It is for this reason that Broadband networks were built asymmetrically and heavily favored the downstream. Servers in data centers with commercial-grade Internet connections served and transmitted content and consumers consumed that content by downloading them.
If you’re downloading video from a service like Apple iTunes, Microsoft Xbox Live Marketplace, Netflix, or YouTube, you’re only downloading and not uploading anything. Those services pay a lot of money for their own datacenters filled with servers, their own bandwidth, and/or they pay services like Akamai to cache and distribute their content over the entire Internet.
Vuze on the other hand uses a different business model where they don’t pay for their own bandwidth and they expect their users to contribute their upload bandwidth to make the service work using the BitTorrent protocol. Vuze basically gets free distribution because they enlist their own customers to be their servers and bandwidth providers using their own computers and broadband connections. So instead of paying for commercial distribution, Vuze offloads their bandwidth on to the broadband providers.
<Next page - Exacerbating the Cable and Wireless spectrum scarcity>
Disclosure: Many people have asked me for the source of the data so I will put out the following disclaimer. As I already indicated in the first paragraph of this article, I am the original source of those charts and graphs. I’ve written extensively on VoIP bandwidth consumption as the former Technical Director of TechRepublic. Before TechRepublic, I built and designed networks for a living. I worked on the routing, the switching, and the traffic engineering of Intranet and Internet based networks. The in-use bitrates I cited are detailed and include packetization overhead and they can be independently verified.
February 24th, 2008
Leaked Intel Nehalem performance projections over AMD Shanghai
It appears that the rumors about Intel’s next major microprocessor “Nehalem” being a huge juggernaut may be true according to leaked documents from Sun Microsystems (removed Sunday night). The slides appear to be inadvertently placed on Sun’s publicly accessible website and “jokerman” posted the link on Aceshardware (thanks to tip from ZDNet reader JumpingJack). The slides looks like the real thing meant for Intel’s partners and they’re probably well known in the server industry.
Reliable sources have reported in the past that Intel’s Nehalem processor will have three channels of DDR3 memory per CPU versus two channels of DDR2 memory per AMD Barcelona or upcoming Shanghai processor. That would mean that AMD’s massive memory bandwidth advantage will turn in to a large memory bandwidth. So what does this mean for Intel Nehalem’s performance? Take a look at the following charts I generated after carefully measuring the length of the performance bars on a pixel level.
Since Intel’s charts were normalized to an Intel E5160 dual-core processor on SPECint_rate_base2006 and SPECfp_rate_base2006, I had to start somewhere and make some guesses on the base performance. I used Intel’s highest published SPEC CPU integer and floating point score of 60.8 and 45.1 for the E5160 processor as of 2/23/2007. This is probably not the exact reference point that Intel used so the numbers might be off a little.
When I compared my extrapolated numbers to the published SPECint scores for all of the shipping products other than the E5160, I found that Integer performance was 2% to 7% too low and the average was 4%. When I compared with published SPECfp scores other than the E5160, I found that my extrapolated numbers were all 4% too high for all models except the Opteron 2220 extrapolation which was 12% too high. To adjust for this, I raised the SPECint estimates 4% and dropped the SPECfp estimates 5% and generated the following chart which is a closer match to the published scores.
I tend to believe that the second adjusted chart is more accurate. We’ll most likely know by the end of this year what the actual scores are, but I doubt they will be more than 5% to 10% off from these estimated projections.
So how can Intel pull off such a massive performance boost over their current reigning champion “Harpertown” X5482 processor? Consider the fact that Intel’s current generation 45nm Harpertown processors lead the benchmarks despite the memory bandwidth disadvantage because of a much faster execution engine and larger cache. Then we factor in the fact that Intel will implement SMT (dual threads per CPU core), improve the already-fast execution engine of Harpertown, and feed it with three channels of DDR3 memory per CPU instead of the old shared front side bus. AMD’s Shanghai on the other hand is essentially a die shrink, a cache size boost, and a clock speed boost. Taking all these things in to consideration would easily explain how Intel could widen the lead so far.
I would also note that Intel’s leaked slides compare these processors in pairs where the Opteron 2220 DC (Dual Core) faces off with the Intel E5160 DC processor and the 2222 faces the X5365. These two pairs represent a snapshot in time to when the products competed against each other. The last two pairings on top may be generous to AMD since Barcelona processors aren’t shipping yet because of the TLB bug whereas Intel launched the X5482 in November 2007. AMD’s Shanghai processor didn’t have first silicon until four months after Intel showed off their first silicon at spring IDF 2007 in September, but the difference is that Intel has showed the Nehalem running a real Operating System while AMD has not done the same for Shanghai.
Since it usually takes one year from first silicon to production parts, it’s a bit hard to believe that Shanghai will ship at the same time as Nehalem. But even if it does ship at the same time as Nehalem, the competition from Intel looks very daunting if these estimates are anywhere close to being accurate.
February 19th, 2008
One year till death of analog TV in USA
Analog TV will no longer be broadcast one year from today on February 19th 2009, are you ready for the conversion to digital TV? This may or may not affect you so here’s what you need to know to avoid losing TV reception!
The first question to answer is whether this government mandated change to over-the-air digital TV will affect you. If you have any older TV that depends on rabbit ears or external UHF/VHF antennas for analog TV reception, you will no longer get any TV reception a year from today. If you’re already using cable or satellite for TV reception, you have nothing to worry about. If you have a newer TV that supports the ATSC standard for either standard definition and/or high definition reception, then skip the coupon section below to the antenna section.
If you have an older TV that relies on analog TV signals, there is good news for you but you better act fast. The US government has a $40 coupon program that can either cover some or all of the cost of a converter box. You need to apply for it here and each household is entitled to two coupons but there are a finite number of coupons so you need to act before they run out. Once you’ve obtained the coupons in the mail, you can buy one of these certified converter boxes priced between $40 and $70. Update 7:15AM - There are a total of 33.5 million coupons available and as of right now for this update, roughly 3.3 million coupons have already been ordered. Last week the count was 2.9 million ordered.
You’re not home free yet since your existing rabbit ears on your TV set won’t work anymore. At the very least you’ll need an indoor passive antenna or a powered indoor antenna like this Philips amplified UHF/VHF antenna which sells for as little as $20 online or $48 at Circuit City. If you already have one of these antennas you can hook it up and give it a try as soon as you get the coupon and converter box and see if you can get all the digital TV channels you want.
Ideally, you need an outdoor UHF antenna which typically sell for $40 to $80 but the hard part is running the coax cable from the roof to the TV. You’ll typically need around 100 feet or more of RG-6 cabling to attach the outdoor antenna to your TV which costs around $20 online. If you already have a roof antenna and the cabling in place, you’re really in luck. AntennaWeb.org has some great information on choosing antennas and others pointed out that http://tvfool.com is great too. With a good antenna in place, it will allow you to receive free standard definition digital television over the air. You can even get high definition digital reception with any of the HDTVs sold in the last year for as little as $400.
February 13th, 2008
Comcast traffic management issue before FCC
Today is the deadline for the FCC call for comments on the Comcast traffic management case brought about by a formal complaint from the Free Press and Public Knowledge. As a former network engineer who designed networks and servers and as someone who has written extensively on these matters, I thought I would summarize the issues in a clear and concise manner.
Background
Independent groups last year found that Comcast was sending TCP RESET packets to BitTorrent seeders at various times of the day to cut back the number of upload sessions they could have. A BitTorrent seeder is someone who is not downloading but acting as a dedicated and peer-to-peer file server. BitTorrent downloads or uploads while downloading were not affected. Various groups complained that this was possibly illegal protocol discrimination using forged TCP RESET packets while Comcast maintained that this was reasonable network management to assure fair distribution of bandwidth to all their users.
The upstream contention problem
A typical Cable broadband network such as Comcast operates under the DOCSIS 1.1 standard which offers 10 mbps of upstream bandwidth and 40 Mbps of downstream bandwidth bandwidth shared amongst the neighborhood. Since the typical user has a static upstream cap of 384 kbps, it would be possible for 26 BitTorrent seeders and/or BitTorrent uploaders to completely jam the upstream pipe rendering the entire network unbearable. Since a typical Cable broadband company provisions between 50 and 400 users (typically somewhere in the middle) per cable loop, it is possible for ~10% of the users can jam the entire upstream network which ultimately affects downloads as well since services can’t be asked for. This is further complicated by the fact that DOCSIS networks use a reservation system for upstream traffic on a collision network. Too many requests for upload slots and the requests collide and no one gets to transmit anything.
Accusations of discrimination
Some have complained that this was content discrimination. But Comcast does not discriminate based on content; Comcast discriminates against excessive upstream usage that chokes up their entire broadband network. The EFF complains that this was “protocol discrimination” against BitTorrent and other P2P (peer-to-peer) applications, but it is a fact that BitTorrent and P2P are the biggest upstream bandwidth users. Since BitTorrent seeders who only continuously upload throughout the day can be reasonably classified as dedicated servers, they actually fall under prohibited services under Comcast’s TOS (Terms Of Service).
Blocking versus delaying
Comcast says they’re merely delaying BitTorrent seeders from uploading to their peers while their critics say they are blocking. It is true that Comcast blocks BitTorrent seeds when the broadband network is very busy, but they do allow BitTorrent seeding at most other times of the day. Network Engineer and Internet pioneer Richard Bennett explained this best in his comment to the FCC that since BitTorrent and other peer-to-peer applications all have the ability to resume transmission at where they left off, temporary blocking of seeders effectively acts as a delaying mechanism. The file eventually gets served to the remote party outside of Comcast’s network intact.
Consumer versus commercial Internet connection
The reality is that Comcast customers were never blocked, throttled, or delayed from receiving any services; they were delayed from offering hosting services (BitTorrent seeding) that were technically prohibited to begin with under the terms of service. Comcast’s consumer broadband service technically doesn’t have to act as a commercial hosting service to other customers in and outside of Comcast’s network so the fact that they permit seeding most of the day seems like a reasonable compromise. Furthermore, BitTorrent users who are downloading are continuously uploading during the download without any delaying action so it isn’t as if Comcast refuses to participate in P2P uploads.
Blocking of Lotus Notes
Comcast’s network management mechanisms did have a bug in them that accidentally blocked Lotus Notes traffic, but this issue was fixed months ago when the issue was first brought to the attention of Comcast. All software and hardware implementations have bugs and we expect the service provider to act in good faith and repair the problems as soon as possible. In this particular case, Comcast appears to have acted quickly and properly by fixing the problems that blocked Lotus Notes.
The complaint to the FCC
The Free Press and Public Knowledge filed a formal complaint to the FCC to immediately enjoin Comcast from these network management practices before the merits are decided and the facts weighed. This is an unreasonable request since Comcast customers would be harmed by network traffic jams due to the lack of any traffic management. The Free Press and Public Knowledge also demanded fines of $195,000 per infraction which would amount to over $2 trillion dollars if we counted every Comcast customer. This is obviously impossible since it exceeds the gross revenue of any corporation in the USA.
February 7th, 2008
RBAC problems wipe out AT&T DSL in California
If things weren’t bad enough last night with the computer problems I had, things got worse when AT&T decided to do an unannounced maintenance. This was sort of similar to the massive network outage last year where the network goes down but they don’t even bother to tell their own first level support. You call in to tech support and they tell you jump through a bunch of hoops and crawl under the table to find your cable modem model numbers and detach your router and all the usual nonsense. Then they tell you that they might have to send a tech over the next day and how they won’t charge you if the problem isn’t on your end. Well I knew the problem wasn’t on my side so I demanded to be escalated to level 2 support where they confirmed my suspicions.
It turned out that AT&T was doing a 6-hour (12AM to 6AM) “maintenance” on a dozen of their California RBAC (Role Based Access Control) systems this morning which is their PPPoE authentication servers. This is exactly what I suspected since my DSL light was still on indicating that the link to the DSLAM was operational. The last place I lived two years ago my AT&T (SBC back then) DSLAM would die once a week so I know a DSLAM outage when I see one. This kind of service is ludicrous to me because if you’re doing this kind of authentication system maintenance, there should be redundant systems in place or they should simply let everyone on the network even if they can’t authenticate. It’s not like anyone can steal DSL access that easily anyways and we’re talking a short period of time. I’ve run my authentication servers for many years without ever having an outage and it’s ludicrous that AT&T would put their users through this nonsense.
In light of past cases where AT&T doesn’t tell level 1 support about these maintenance and outage issues and putting their customers through tech support hell, this seems to be a systematic breakdown in AT&T’s support infrastructure. I don’t know what it’s going to take to knock some sense in to AT&T’s customer support, but this just isn’t acceptable. It not only frustrates the level 1 support team and makes for unnecessary work on to the maintenance department; it’s just plain bad customer service.
February 7th, 2008
First experiences with Vista SP1 RTM
[UPDATE 1/12/2008 2:55PM - Looks like my fellow blogger Ed Bott may have pulled through and found the answer in the quotation below. The lesson in this is to always update the motherboard BIOS when upgrading to a new OS. This shouldn't be too much of a surprise since this same rule applies to upgrading memory and CPUs as well.
Ed Bott: 2.07 (BIOS for IBM Thinkpad T60) is ancient, and according to the changelog Vista support was added in 2.09, so your BIOS is not Vista-compatible. The most up-to-date BIOS is 2.20:]
[UPDATE 1/12/2008 2:55PM - It looks like the IBM ThinkPad T60 lockups may have something to do with Vista SP1 after all. It locks up within 30 seconds when I boot the Vista SP1 fresh install DVD or when I boot Windows Vista that was upgraded to SP1 from a different DVD. My IT person loaded Windows XP on the laptop and it runs smoothly. He will load Windows Vista without SP1 and see if it is stable as well. More updates to come. Update 3:15PM - Looks like Vista without SP1 crashes too. This reminds me of the lone desktop machine I had last year that ran fine for a year on XP but was never able to load a fresh install of Vista without it crashing. This could be one of those hardware problems that only manifest itself when being taxed more by something like Windows Vista. Either way, we're trying to get to the bottom of this and this laptop is going back to where we bought it from.]
[Update 3:55PM - Seems like a hardware issue with this specific IBM ThinkPad T60 since Microsoft tells me they have plenty of T60s that are running fine with SP1. I was trying to recover some log files for Microsoft by booting the Windows Vista SP1 fresh-install DVD and it hung there and locked up the mouse too. That would seem to at least rule out DivX and it was a mere coincidence on the exact timing of the lockup. Heck the battery on it is dead too and the screen came with some scratches so it's time this dog of a laptop goes back to the IT department.]
I completed the first two installations of Vista SP1 RTM upgrade last night on to my primary desktop computer and my first Vista laptop meant to be my new work computer. The result is a near death experience with my desktop computer, and then a real death experience with the laptop (caused by hardware and not Vista SP1). I guess I should count myself lucky that it wasn’t my main computer that died since I haven’t migrated to the laptop for work yet. I will try to get some help from Microsoft to see if we can resolve this issue.
Near death with desktop computer:
The desktop computer almost didn’t make the upgrade but finally managed to pull itself out of the gates of hell. The SP1 upgrade on both computers took more than an hour to install along with multiple reboots. When the desktop system finally allowed me to log in, it went in to non-aero mode and it refused to let me flip in to aero. The sound was temporarily messed up but I managed to get it working after I enabled the sound. 5 minutes after I logged in the Windows SP1 upgrade finally told me it was finished which seems strange since you would think the user should be locked out until everything was done. Since I couldn’t get aero running I figured I’d try rebooting but the next reboot just seemed to hang on a black screen with a working mouse pointer for 5 minutes so I tried rebooting again. On that last reboot everything finally came up and I breathed a sigh of relief. [UPDATE 3:55PM - Microsoft says they are working with the driver developers on this to smooth out the install process]
Death of a laptop:
The laptop computer on the other hand went a little smoother on the SP1 upgrade and worked fine for about two hours until I installed the latest DivX codec and the whole machine just locked up after Vista popped up the Windows experience feedback prompt. Now this laptop locks up the entire computer within 15 seconds of logging in and there’s no way I even have time to run system restore to see if I can get it to the state right after I installed SP1. All I see is a locked up Vista screen and the laptop is as useful as a bookend. It is possible that this could be a hardware issue but the laptop was working fine up until this point. [UPDATE 3:55PM - It appears to be most likely a hardware issue with this specific IBM ThinkPad T60 laptop]
I don’t know if Vista SP1 just doesn’t like DivX or if it was just a coincidence and something else is causing this problem. I have the same DivX codec installed on my desktop computer this week but it was installed before last night when I installed Vista SP1. It’s quite possible that installing this version of DivX after SP1 will kill the computer but if this is the true, Microsoft needs to issue a warning and block this codec from installing after SP1 has already installed. If you’re planning on installing SP1 on your computer, DO NOT install DivX codec after you’ve installed SP1 until after I verify what’s going on and update this blog. If you have DivX codec installed already, then it doesn’t seem to be a problem.
Minute long login times for domain connected computers
The other problem I was told that Windows Vista SP1 would fix was the minute long login times for a Vista computer joined to an Active Directory. This turned out to be false at least in my case since it still takes 55 seconds of looking at the “Welcome” message after I type in my password. This doesn’t seem to be a problem coming out of suspend mode if you’re already logged in so it would only affect you if you reboot or log off the computer, but it’s annoying as hell and it really makes me think twice before using Vista in a business environment until these issues are solved.
[UPDATE 4:05PM - Microsoft explained to me that until a laptop at least logs in once on the corporate LAN and cache the domain controllers correctly, it will exhibit a 20 second delay per each domain controller the laptop knows of. That's a neat solution and all, but I know quite a few mobile workers who never go in to the corporate LAN and they need a solution where they can simply VPN in and get all this nonsense sorted out automatically and painlessly without flying in to an office with a permanent LAN or WAN connection to the Domain Controller]
Some improvements after SP1
My desktop computer seems to be a lot healthier now after I installed Vista SP1. The Vista install seemed to have gotten corrupted to the point that IE7 was locking up left and right while I kept getting these error messages from Windows Media Center Store Upgrade Manager shown in the figure below. [UPDATE 4:15AM - Looks like I may have spoken a bit too soon and the talkback tool here still locks up IE7 pretty hard on this computer. I'm not sure if it's related to the talkback or something else on that page that's causing it. All I know is that IE7 has been locking up hard on my Windows XP laptop and Vista machine for the last month or more.]
So far that error message hasn’t popped up yet [UPDATE 3:55PM - The message popped up again and it appears I need to rebuild the database score in Windows Media Center] and IE7 seems to have stabilized now and it’s no longer locking up the CPU to 100% utilization on a single CPU core[UPDATE 3:55PM - I'm working with Microsoft to figure this issue out and will update since it's still locking up in the talkback section]
The Windows networking indicator icon shown in the screenshot below seems to be a lot more responsive in discovering your network location. It completes in a few seconds after you log in rather than sometimes wait up to a minute pre-SP1. The laptop (before it died) also exhibited the same responsive network indicator icon. [UPDATE 3:55PM - Microsoft says quite a bit of work went in to the TCP/IP stack. This seems to be a good thing.]
Windows Movie Maker for Vista still a stinker:
If you’re wondering why I even bother with the DivX codec and Dr. DivX video encoder, it’s because Microsoft’s Windows Media Encoder 9.0 is old and doesn’t support high definition HDV formats yet and Windows Movie Maker for Vista is still garbage. If you attempt to use Windows Movie Maker to encode HDV videos, it will only encode one corner of the video at standard resolution and leave out most of the rest of the video frame. [UPDATE 3:55PM - Microsoft says they can encode HDV footage without problems so the problem may be caused by some codec conflicts. They also say that Microsoft Expression Encoder ($300) will work much better, though that's quite a bit more money than I want to spend since Dr. Divx is free.] The latest version of Dr. DivX (which requires the DivX codec) will handle High Definition .dvr-ms files and let you encode in to the desired video format for DVD set-top box playback or for YouTube optimized format. I’d love to be able to encode in to the 1080p Windows Media Advanced Profile format that Windows Media Encoder 9 promises but the software simply doesn’t work.
Making things worse, I was hoping Windows Movie Maker which only comes with the Premium or Ultimate Edition would actually be a complete application by now but I would be disappointed again. The old Windows Movie Maker in Windows XP was a free download and it allowed you to select the part of the tape you want to record off your DV camcorder but HDV format wasn’t supported at all. The new Windows Movie Maker for Vista does support HDV format but it teases you by asking you if you want to “Import entire videotape or just parts?”. Then it only gives you the option to “Import the entire videotape to my computer” as shown in the screenshot below. If anyone knows of a cheap or free non-bloated HDV capture program for Windows, please tell me in the talkback.
Fortunately DivX codec and Dr. DivX works on my desktop computer since they were installed before SP1 so I will need to get verification of the problems on my new work laptop.
Now at this point it’s still to early to draw any conclusions about Vista SP1 and the problems I faced may be unique to my particular setup or hardware. It certainly doesn’t make a good first impression for me and you need to be careful anytime you install a major upgrade like this. Please check back here for updates and status reports on these problems.
February 4th, 2008
Windows Server 2008 and Vista SP1 RTM today
Microsoft has reached a major milestone today for its Windows Server and Client products. Windows Server 2008 and Windows Vista Service pack 1 have been released to manufacturing today which means they will soon be available to IT customers and consumers. Windows Server 2008 replaces the venerable Windows Server 2003 while Vista SP1 upgrades the somewhat controversial Windows Vista. If this looks like a coincidence that Vista SP1 and Server 2008 launched at the same time, it’s not. These two products share the same kernel and they were finished together and launched together by design.
Windows Server 2008 will have key enhancements in Virtualization both on the OS kernel side and the hosting side, but the hosting side of the equation won’t appear for another six months in the form of Windows Hypervisor. The OS kernel side optimizations come in the form of “enlightened” (AKA paravirtualized) IO optimizations for video, storage, networking, and memory. The Hypervisor will take advantage of these kernel enhancements to reduce the overhead associated with virtualization. Other virtualization vendors will most likely license or negotiate rights to these kernel enhancements in virtualization if they wish to host Windows Server 2008 efficiently. Older server operating systems like Windows 2000 and 2003 server will later be retrofitted with just the I/O optimizations but not the full kernel modifications that optimize Memory and CPU operations.
Windows Server 2008 will also have a stripped down headless operation mode called “Core installation” that increases reliability and security because it reduces the code foot print. This in turn also reduces the need for reboots because components that would normally need to be updated simply won’t be installed in the first place. Server 2008 will also have a fast kernel mode IIS web server as well as enhancements to Routing and Remote Access such as SSTP (Secure Socket Tunneling Protocol). SSTP puts a NAT- and proxy-friendly wrapper around the PPTP and L2TP protocol for trouble free VPN access.
Windows Vista will get some sorely needed enhancements on stability. The size and scope of enhancements and changes to Windows Vista over previous generation Windows XP has resulted in some major growing pains both in OS and driver stability. While many of these issues have already been hammered out, annoying problems like a minute long wait to login a Vista machine in to an Active Directory domain and slow network file copies are now fixed in Vista SP1. Other controversial features like a Windows Vista kill switch have been removed. On the usability front, the aforementioned SSTP feature in Windows Server 2008 can now be leveraged using the new SSTP client in Windows Vista SP1. In the coming weeks, I will be eager to test both of these products.
February 4th, 2008
ISSCC 2008: Details on Intel Silverthorne
At this year’s ISSCC 2008 (International Solid State Circuits Conference), details of Intel’s new 45nm Silverthorne will emerge. Intel CTO Justin Rattner held a press briefing last Wednesday to preview some of the highlights of this week’s highly technical ISSCC conference in San Francisco.
Credit: Intel Corporation (from ISSCC preview presentation)
Intel Silverthorne is a brand new Intel x86 processor for the Menlow platform developed from the ground up for low-cost and ultra-low power applications. This includes UMPC (Ultra Mobile PCs), MID (Mobile Internet Devices), set-top applications, some embedded applications, and eventually for smart phone applications though this initial generation may not be suitable yet. Its small 25mm^2 die size on a 45nm process allows 2500 chips to fit on a single 300mm diameter wafer allows for extremely economic production.
From Rattner’s press conference last week, we know that Silverthorne will launches in the first half of 2008 but Rattner will not give a yes on a Q1 launch in response to one of the questions. The first Silverthorne dies were publicly shown in April of 2007 in IDF China so it’s quite possible that we’re looking at a second quarter launch. Rattner also explained that Silverthorne was a dual-issue in-order pipeline architecture with HT (Hyper-threading) and that this was better than hyper-threading in out-of-order architecture. I later got verification via email that the HT type was SMT (Simultaneous multithreading) and not SoEMT (Switch-On-Event Multithreading).
The slides shown by Rattner indicated that Silverthorne had a power consumption below 1W and up to 2W and that it was “10x lower power than ULV Dothan”. The Dothan was the second generation Pentium M product and ULV parts had a TDP (Thermal Design Power) of 5W. I later got clarification via email that Silverthorne processors can have TDPs as low as 0.6W with lower clock speeds and higher clocked parts will have a 2 watt TDP. I spoke with analyst David Kanter of Real World Technologies and he explained that 0.6W which doesn’t factor in chipset power consumption might be too high for smart phone applications. However, its immediate successor in the Moorestown platform which may launch late 2008 may solve that problem with its SoC (System on Chip) design.
Update 3:10PM - There are quite a few inaccurate reports out there on Silverthorne’s power consumption. They have reported the power consumption of Silverthorne as 0.6W to 2W which is not correct. 0.6W is actually a TDP rating which describes PEAK power consumption. Actual idle power consumption can dip down to 0.01W for some models and 0.1W for other models. Intel is not saying too much more right now but it is reasonable to assume that this extremely low power state is designed to maximize battery life in Smart Phones. Keeping a continuous Skype or SIP application presence in a UMPC or MID device to receive calls is now a possibility.
The 2 GHz variant of the Silverthorne processor will operate at 1 volt and it will have performance equivalent to a first generation “Banias” Pentium M notebook processors circa 2003. Rattner confirmed this was for single-threaded performance on a broad range of applications. This would seem to imply that with multithreaded applications, the performance would be even higher than Banias which lacks Hyper-Threading.
Here are some additional quotes pulled from Rattner’s slides:
- Deep power down C6
- Optimized register-file and cache 6T bits cells
- CMOS mode on quad-pumped FSB IO
- Split IO power supply
Here are some additional email responses:
- 0.6W to 2W measured TDP power on real world applications – over the lifetime of the processor/architecture
- Can achieve 2GHz core frequencies at 1.0V
- Will support features such as Digital Media Boost (SSE3), Intel Virtualization technology, Intel 64 Architecture support, HT
February 1st, 2008
San Clemente chipset gives HP lead on energy efficiency
The January 30th 2008 batch of test results are in for SPECpower_ssj2008 energy efficiency benchmark and it looks like Hewlett Packard has claimed the energy efficiency lead with their newest low-cost 2U HP Proliant DL180 G5 server. The secret to their success appears to lie in the selection of the Intel 5100 series “San Clemente” chipset. While the detailed SPECpower disclosure doesn’t actually mention the chipset anywhere, the power characteristics, the six memory DIMMs, and the ICH-9 storage is a dead giveaway.
To see where the modern servers stand on power consumption, I’ve plotted out some ESTIMATED charts to compare the results. Since the AMD system from Colfax International has 8 registered DDR2-667 DIMMs and the HP San Clemente system has 6 registered DDR2-667 DIMMs, I’ve had to adjust them both down to 4 DIMMs to do a fair comparison with the other Intel systems which used 4 DIMMs. To do this I had to use an approximation based on known measurements for memory power consumption and I subtracted 1.875 watts to 3.75 watts for each registered DDR2-667 DIMM on a linear sliding scale based on load percentage. That means I subtracted 7.5 watts for the AMD system at idle and 15 watts for the AMD system at peak power. For the HP San Clemente system I subtracted 3.75 watts at idle and 7.5 watts at peak loads.
Since it was shocking that a dual-processor eight-core 3 GHz Intel system was drawing lower power than a dual-processor four-core 2.4 GHz AMD system, I thought something might be a little off. I realized that Colfax had used a pair of redundant 700 watt power supplies whereas the HP San Clemente system uses a single 750 watt power supply which means the power supply for the AMD system is relatively inefficient. At this point I had to make a reasonable guess at PSU (Power Supply Unit) efficiency and I guessed that the HP single power supply had to be around 80% efficient whereas the Colfax dual-PSU would be around 70%. Therefore I estimated the power consumption of the AMD system had it used an 80% efficiency power supply instead of a 70% efficient power supply.
Unfortunately this is a rough educated guess so the accuracy is dropping quickly but I wanted to take a reasonable shot at it to level the playing field on PSU efficiency. Companies in the future when making SPECpower submissions should avoid using dual power supplies and stick with 4-DIMM configurations so that we can get apple to apple comparisons and measurements. For now the following estimated power consumption graph is what I came up with.
The thing that really sticks out is the fact that the Intel 3 GHz 45nm E5450 processor system uses less power most of the time than the special low-voltage variant of the Intel 2 GHz 65nm L5335 processor. This shows how drastic an improvement Intel made using HKMG (High-K Metal Gate) materials and a shrink to the 45nm process.
The DIMM and PSU adjusted power consumption for the AMD Opteron 2216HE 4-core 2.4 GHz system has dropped significantly by more than 32 watts at the peak but it’s still more power hungry than the Intel 8-core 3 GHz E5450 at less than 80% load. Despite the fact that AMD takes a deeper clock speed dive down to 1.0 GHz at idle while Intel only dives down to 2 GHz, Intel’s C1E state seems to dominate the power savings.
This can also tell us something about the “Barcelona” quad-core “HE” (High Efficiency) 1.9 GHz system because it has a TDP of 79 watts which is 11 watts higher than the 2216HE under maximum load per CPU. Realistically the difference will be smaller than 11 watts per CPU and probably more like an 8 or 9 watts difference so an AMD 2347HE 1.9 GHz dual-processor 8-core version would probably consume 16 more watts. That would likely put the AMD 8-core 2347HE 1.9 GHz server at higher power consumption level than the 8-core 3 GHz E5450 Intel server running on a San Clemente chipset. That seems counter intuitive since Intel’s TDP rating for its 45nm 3 GHz processor is 80 watt TDP and that doesn’t even count the memory controller on the motherboard.
When looking at the difference between the HP San Clemente chipset based server and the HP 5000 series chipset based server, there is roughly a 32 to 40 watt difference even though the two CPUs are identical. Most of that difference is due to an extra 6 to 7 watts per FBDIMM and the remaining power delta is mostly due to the newer chipset on the motherboard. Had both of these servers had 8 DIMMs, the power gap would have been approximately 26 watts wider because of the extra power consumed by the FBDIMMs on the Intel 5000 series chipset.
Next I plot out the power-adjusted ESTIMATED energy efficiency numbers. I adjusted all the systems to four DIMMs and gave the AMD Opteron system a boost in power supply efficiency from an assumed 70% efficiency to 80% efficiency. Again this is a rough guess but it’s reasonable considering the fact that Colfax used a dual 700W power supply instead of a single 750W power supply. If Colfax International is reading this blog then I would suggest to them not to shortchange their own results in the future and use 4 DIMMs and a single PSU like everyone else.
Hopefully the next batch of results will give us some performance numbers on faster single-socket systems using the Bigby chipset and a 45nm processor so we can see how high on the efficiency scale those servers will go.
January 30th, 2008
Painful lesson in OLPC mesh networking for Mongolians
The Mongolians have had a painful lesson on mesh networking according to the OLPC current events webpage. Broadcast storms in the overly dense mesh environment along with excessive mDNS broadcast traffic seem to have crippled the Gobi desert experiment. Here’s an excerpt:
We have painfully discovered the limitations of the mesh and current collaborative software in Mongolia, where the convolution of the number of laptops with bugs #5335 (more mDNS traffic than expected) and #5007 (mesh repeats multicast too much) make the perfect storm, which prevents anybody from using the network. We will continue to improve the mesh performance, but clear guidelines are needed as to what network infrastructure to deploy under what conditions. Once a certain density of students is exceeded, a wired backbone and conventional access points will be required.
The limitations of mesh topology are well known in the wireless engineering community and I’ve raised the issue and pointed out the limitations last September. Each mesh hop you add increases the propagation delay as well as multiply the radio traffic and congestion. Performance on a mesh network is fundamentally many times slower than a non-mesh network and when the density gets high enough, the system simply breaks down.
When on a tight budget, I had always recommended the usage of a cheap $60 router running open source DD-WRT would have sufficed and you get a free router with it which you need for IP sharing anyways. The addition of a high-powered antenna would allow the access point to hear distant signals from faint clients and it will amplify the broadcast signal. A simple in-door $26 9 dBi antenna placed up high can easily cover a small school. A $60 12 dBi outdoor antenna positioned on the roof would easily cover an entire campus. If you put two centralized Access Points and large antennas on channel 1 and 11 (avoid adjacent channels because of channel bleeding) in the 2.4 GHz spectrum, you can load balance and have redundancy if one set of AP/antenna fails.
My fellow blogger and teacher Chris Dawson feels that the ability to do peer-to-peer collaboration with or without an Access Point has great potential. But peer-to-peer wireless collaboration could have been done with regular ad hoc networking technology without the expense or problems of a full 802.11s mesh implementation.
The inclusion of full 802.11s stack has been challenging. The need for a radio system that stays on and continues to forward packets even while the laptop is off added unnecessary expenditure to the OLPC XO and it unnecessarily drains the laptop batteries. When you multiply this expense and complexity across all the clients and realize that the wireless access point comes free with the router, it becomes clear that this may not have been the best design decision.
January 26th, 2008
Network Neutrality Summit at University of San Francisco
I will be speaking at the Network Neutrality Summit this morning at the University of San Francisco. They will be streaming this event LIVE at ustream.tv. Fireworks start at 9:00AM which is the panel I will be on with:
- Richard Clarke - AT&T
- Lawrence Spiwak, Phoenix Center for Advanced Legal & Economic Public Policy Studies
- George Ou - ZDNet
- Marham Erickson - Open Internet Coalition
- Timothy Wu - Columbia Law School
Hope to see you there or catch it live on the Internet.
January 18th, 2008
Don't believe the low bit-rate 'HD' lie
Update 6:00PM - Here’s what fake HD video looks like.
Last week at CES, Comcast announced their “HD” video on demand download service over its future DOCSIS 3.0 that allows 4 minute downloads of entire HD movies. Attendees at MacWorld this week were told that disk-based HD formats like HD DVD and Blu-ray are essentially obsolete because you can simply download “HD” movies from your Apple TV 2.0 box on demand. Microsoft started offering HD downloads for the XBOX360 starting in late 2006. You can even watch “HD” videos from ABC right from the web. There are even YouTube competitors that offer user uploaded “HD” content. There’s just one minor little problem, it’s not HD.
As I’ve tried to educate my readers last year with my blog “Why HD movie downloads are a big lie“, these so-called HD movies use very low bit-rates compared to even standard definition DVDs let alone something like HD DVD or Blu-ray DVD. Raw uncompressed 1080p video at 60 frames per second is about 3000 mbps so even HD DVD’s 28 mbps needs to be compressed about 107 to 1 with the H.264 or VC-1 codec. By all reasonable standards this needs to be the minimum bit-rate for acceptable loss in quality on 1080p video.
Updated 4:30PM - Standard definition 480i DVD movies are typically 5 to 8 mbps (megabits per second) MPEG-2 whereas these so-called HD wannabes weigh in at a pathetic 1.5 to 4 mbps of 720p H.264. Apple’s new HD service is capable of 4 mbps which simply isn’t enough to be considered HD. XBOX360 downloads are 6.8 mbps 720p VC-1 so they’re semi-decent borderline HD. Marketing will push the nicer sounding “720p” aspect of the video but they don’t tell you it’s way too compressed to offer good video fidelity. Blu-ray has a maximum bit-rate of 40 mbps while HD DVD offers a maximum of 28 mbps. Over the air broadcasts can be up to 24 19.38 mbps.
Modern video compression codecs like H.264 or VC-1 can hide these compression artifact problems much better than MPEG-2 video compression but there’s only so much it can do. At best you might get away 50% more compression over older compression technology but 1.5 to 4 mbps H.264 will not be better than 8 mbps MPEG-2 under most video complexity requirements. The only time 4 mbps 720p will look better than 8 mbps 480i is when the video on the screen is almost entirely stationary or it’s a low-complexity video such as animation movies. Under most normal circumstances, the low bit-rate 720p so-called “HD” video will be inferior though many companies are betting that consumers won’t know any better.
So the bottom line is that so-called “HD” video from Microsoft’s XBOX360 HD download service and Apple’s new Apple TV service or any other web download service is simply not HD by any respectable definition. These companies cannot and should not use the “HD” name with video that is lower fidelity than standard DVD. As for Comcast, there’s not much detail on it but I highly doubt it’s more than 4 to 8 mbps even on DOCSIS 3.0 because its 160 mbps total capacity is divided between 50 to 400 customers. Only FiOS technology with its massive 620 mbps per 32-user capacity and possibly U-Verse (but slower than real time) has sufficient last-mile capacity to deliver true HD movie downloads at the quality of HD DVD and Blu-ray technology.
I’m not saying that you shouldn’t buy these services from Apple, or other services that offer low bit-rate 720p video downloads, but consumers must be aware of the fact that they’re slightly worse than a 1080p up-converted DVD. Microsoft’s XBOX service is border-line HD that is slightly better than DVD but nowhere near 1080i over-the-air HD broadcast quality.
January 16th, 2008
Why DIDN'T the MacBook Air get the new 45nm CPU?
Intel launched their brand new 45nm mobile dual-core processors last week with 60% smaller packaging size. Yesterday Apple announced their Über-sleek MacBook Air ultra-slim notebook which also uses a specially designed Intel dual-core CPU with 60% smaller packaging. Naturally I assumed the new MacBook Air uses Intel’s latest Penryn-class 45nm technology with low leakage hafnium metal gates and I called Intel for confirmation of this “special” processor. I thought to myself: What’s so special about it if every PC vendor can use the same shrunken CPU?
To my surprise, Apple didn’t use the newest 45nm mobile processor with 107mm^2 die size; they really did use a “one-off” “Merom” 65nm 143mm^2 die designed-just-for-Apple CPU from Intel. Intel specially designed a larger 65nm core with a specially designed package that’s 60% smaller. This means instead of using the latest 45nm processors that are faster and more energy efficient and are already that small without any special packaging, Apple got a “special” 65nm chip.
This begs the question why Intel doesn’t make its new 45nm packaging even smaller than the current 60% reduction in size if it can reduce its packaging by 60% on 65nm technology. It also begs the question why Apple had to go to the trouble of a tailor made 65nm part when the 45nm part launched 3 weeks before the launch of the MacBook Air. Several other PC makers were already showing off their 45nm based notebooks last week at CES.
I spoke to a few people about this and asked for some theories and we came to a somewhat reasonable guess so I’ll offer these up as some possible reasons. For a product as specialized at exotic as the MacBook Air, the design would have needed to start some time ago. When that design started, it may not have been a certainty if 45nm Mobile Penryn would be ready to ship with MacWorld and there may not have been working samples to start the design process.
Despite the fact that other PC makers have 45nm based notebooks ready to launch, none of them are this sleek. So ultimately it doesn’t really change the appeal of the MacBook Air and it will be the thinnest notebook on the market. In 20/20 hindsight perhaps it would have been better if the MacBook Air had shipped with a 45nm CPU and maybe we’ll see a quick refresh from Apple to the new processor since the size is obviously not a problem. It’s just that “special” in this case isn’t a flattering thing when referencing the older CPU used in the MacBook Air, but the MacBook Air is still every bit special in a flattering way.
George Ou is Technical Director of ZDNet. See his full profile and disclosure of his industry affiliations.
SponsoredWhite Papers, Webcasts, and Downloads
- Email Security and Archiving - Clearer in the Cloud Google The time is NOW for businesses and organizations of all sizes to implement ... Download Now
- Three Steps You Need to Know to Stop Data Loss Varonis Sensitive data exposed to misuse or loss... it is the stuff of nightmares ... Download Now
- The True Costs of Virtual Server Solutions VMware In an economic environment that is repeatedly heralding the message "do ... Download Now
- See why AND is the new OR. Watch the video.
- The Creeps Are Coming: Are You Ready?
- See how much space you can save with our calculator.
- Stay current on the latest trends in our blogs.
Recent Entries
- Saying goodbye to ZDNet
- 55W PC power supply powering the dual-core computer
- Fixing the unfairness of TCP congestion control
- HDMI survival guide for home theater
- The cheapest way to do VoIP is still analog
Blogs From Our Sponsors
Top Rated
Premier Vendor Content Whitepapers, webcasts & resources from our Power Center Sponsors
- Keep Up With The Latest In Document Management with The DocuMentor.
-
Doc delivers the scoop on today's enterprise content management, printer maintenance, and all other issues related to document management. It's the DocuMentor Blog.
- Learn more >>
- New Online Dashboard for IT Leaders
-
Read about top issues IT decision-makers face every day, plus get cost-effective solutions to real-life IT problems.
- Learn more >>
- Save time with automated shipping solutions
-
The Business Essentials Guide provides you useful tools and templates to help grow your business and save you time with automated shipping solutions.
- Visit the UPS Business Essentials Guide
- The more you simplify, the more you save
-
When you transition from your existing Red Hat environment to SUSE Linux Enterprise from Novell, you can recognize dramatic cost savings, perhaps as much 50%
- Learn more >>
- Microsoft Dynamics CRM Online - Free Six-Month Trial for Eligible Organizations
-
Microsoft Dynamics CRM Online provides fast online access, simple contact management and better sales performance for a low monthly cost - the best value on the market today.
- Learn more about the free, six-month trial offer>>
Archives
ZDNet Blogs
- All About Microsoft
- The Apple Core
- Between the Lines
- BriefingsDirect
- Collaboration 2.0
- Dev Connection
- Digital Cameras & Camcorders
- Ed Bott's Microsoft Report
- Emerging Tech
- Enterprise Web 2.0
- Forrester Research
- Googling Google
- GreenTech Pastures
- Hardware 2.0
- Home Theater
- iGeneration
- Irregular Enterprise
- IT Project Failures
- Laptops & Desktops
- Lawgarithms
- Linux and Open Source
- Managing L'unix
- The Mobile Gadgeteer
- On Sustainability
- Rational Rants
- The Semantic Web
- Service Oriented
- Smartphones and Cell Phones
- Social Business
- Social CRM: The Conversation
- Software & Services Safari
- Software as Services
- Storage Bits
- Team Think
- Tech Broiler
- Technology and the Global Supply Chain
- Tom Foremski: IMHO
- The ToyBox
- Virtually Speaking
- The Web Life
- ZDNet Education
- ZDNet Government
- ZDNet Healthcare
- Zero Day
White Papers, Webcasts, and Downloads
- Why Isn't Server Virtualization Saving Us More? A Few Small Changes May Dramatically Increase Your Efficiency VMware Companies have rapidly adopted server virtualization over the past few ... Download Now
- The Impact of Virtualization Software on Operating Environments VMware Today's use of virtualization technology allows IT professionals to ... Download Now
- Reducing Server Total Cost of Ownership with VMware Virtualization Software VMware VMware virtualization enables customers to reduce their server TCO and ... Download Now
SmartPlanet
- Thought-provoking progressive ideas on diverse topics that intersect with technology, business, and life, and matter to the world at large. Visit SmartPlanet
- More from IBM
-
- Innovate your business' process model, play against the market, compete against others on our scoreboards and WIN! Try INNOV8 2.0: A BPM Simulator
- Enabling Real-World Business Transformation through IBM Service Management Read the EMA Analyst Report
