future of the data center

This article from Ars Technica discusses a talk over the summer by Merrill Lynch’s chief technology architect, Jeffrey Birnbaum on “stateless cloud computing” – most concretely on distributed file systems.

Birnbaum believes that one of the key foundational elements of a stateless computing environment is a networked storage system that enables ubiquitous availability of software. The file paths of the individual applications should be based on clearly defined nomenclature, much like the domain of a web site. All application dependencies should be accessible through the network filesystem, and version numbers should be expressed with the path nomenclature.

Big distributed file system – sure. Why should version numbers be expressed with the path nomenclature (a Plan9 idea, btw)?  Now we go on to the ancient problem of caching distributed data.

The obvious challenge posed by rolling out worldwide network storage infrastructure is scalability. If everyone in a global organization is depending on a network storage solution, then it needs to be fast and consistently reliable. The solution that Birnbaum proposes is regional mirroring and caching. The storage system would be universally synchronized between mirrors that have all the data. Caching can also be used at individual facilities to further improve performance. To achieve this kind of global scalability, he says, the best approach is similar to that of Akamai.

So even with a non-globally distributed file system, the problem of shared access is non-trivial. A global file system makes things quite challenging. Suppose we have a file recording trades and the Singapore, London, NY, and Espanola main offices all are reading and writing at the same time. Caching and cache coherency is an utter nightmare.  Akamai, like Google, solves the problem of massive amounts of distributed data by focusing on “delivery” – otherwise known as “read only content” or “many readers one writer” and with no requirement for true synchronization.  But the ML problem is more difficult even if we ignore multiple writers because, presumably, you want Singapore to actually see every trade made in Espanola even though for Akamai, it’s ok if the cache is not fresh. How to solve multiple readers and writers is something else as well.

These concepts don’t cover a whole lot of new ground yet. Much of this was already possible with conventional thin-client systems. The point at which it becomes immensely valuable, according to Birnbaum, is when all of these technologies are used together with virtualization to abstract the processes away from the hardware. Once this is done, individual operations can seamlessly float around data centers and balance out in a manner that offers a more optimal level of resource utilization.

And this seems to me to gloss over the even harder problem. Imagine a serious Oracle application “seamlessly floating” from some set of machines in one data-center to another set.  I can’t imagine how that works. Imagining little jobs floating is easier, but is that really an interesting problem? And this brings us to the most interesting claim:

He claims that 61 percent of a company’s enterprise server capacity goes completely unused and proposes an automated load balancing solution—

SIXTY ONE PERCENT!  Think of the power use.

Advertisements

Operating system research – 16 years perspective

It’s somewhat funny and somewhat sad to read this thread on the old USENET. Starting out with Andy Tanenbaum’s proposed list of accepted truths (most of which I thought wrong at the time)

GENERALLY ACCEPTED AS TRUE BY RESEARCHERS IN DISTRIBUTED SYSTEMS

  • – The client-server paradigm is a good one
  • – Microkernels are the way to go
  • – UNIX can be successfully run as an application program
  • – RPC is a good idea to base your system on
  • – Atomic group communication (broadcast) is highly useful
  • – Caching at the file server is definitely worth doing
  • – File server replication is an idea whose time has come
  • – Message passing is too primitive for application programmers to use
  • – Synchronous (blocking) communication is easier to use than asynchronous
  • – New languages are needed for writing distributed/parallel applications
  • – Distributed shared memory in one form or another is a convenient model

and then Rob Pike’s refutation entitled “Andy Tanenbaum hasn’t learned anything”. A key point that comes up later in the discussion is Tanenbaum’s (incorrect) assertion that a “factor of two” performance loss is nothing to worry about. The date of the discussion is interesting, because in a few years the Linux Tsunami washed away most of the landscape of this discussion. As for my contribution it is very disturbing to see that I have not learned much about those topics in the last 14 16 years.

It is clear that Rob Pike was right on many more issues than AST, but that seemed clear at the time too.

Subject: Andy Tanenbaum hasn’t learned anything

From: rob@alice.att.com (research!rob)
Date: 6 Apr 92 20:06:28 GMT
Approved: comp-os-research@ftp.cse.ucsc.edu
Newsgroups: comp.os.research
Organization: AT&T, Bell Labs
Sender: usenet@darkstar.ucsc.edu

The implementers of Plan 9 are baffled by Andy Tanenbaum’s recent posting.
We suspect we are not representative of the mainline view, but we disagree
at some level with most of the “GENERALLY ACCEPTED” truths Andy claims.
Point by point:

– The client-server paradigm is a good one
Too vague to be a statement. “Good” is undefined.
– Microkernels are the way to go
False unless your only goal is to get papers published.
Plan 9’s kernel is a fraction of the size of any microkernel we know and offers more functionality and comparable or often better performance.
– UNIX can be successfully run as an application program
`Run’ perhaps, `successfully’ no. Name a product that succeeds by running UNIX as an application.
– RPC is a good idea to base your system on
Depends on what you mean by RPC. If you predefine the complete set of RPC’s, then yes. If you make RPC a paradigm and expect every application to build its own (c.f. stub compilers), you lose all the discipline you need to make the system comprehensive.
– Atomic group communication (broadcast) is highly useful
Perhaps. We’ve never used it or felt the need for it.
– Caching at the file server is definitely worth doing
True, but caching anywhere is worthwhile. This statement is like saying ‘good algorithms are worth using.’
– File server replication is an idea whose time has come
Perhaps. Simple hardware solutions like disk mirroring solve a lot of the reliability problems much more easily. Also, at least in a stable world, keeping your file server up is a better way to solve the problem.
– Message passing is too primitive for application programmers to use
False.
– Synchronous (blocking) communication is easier to use than asynchronous
They solve different problems. It’s pointless to make the distinction based on ease of use. Make the distinction based on which you need.
– New languages are needed for writing distributed/parallel applications
`Needed’, no. `Helpful’, perhaps. The jury’s still out.
– Distributed shared memory in one form or another is a convenient model
Convenient for whom? This one baffles us: distributed shared memory is a lousy model for building systems, yet everyone seems to be doing it. (Try to find a PhD this year on a different topic.)

How about the “CONTROVERSIAL” points? We should weigh in there, too:

– Client caching is a good idea in a system where there are many more nodes than users, and users do not have a “home” machine (e.g., hypercubes)

What?

– Atomic transactions are worth the overhead

Worth the overhead to whom?

– Causal ordering for group communication is good enough

We don’t use group communication, so we don’t know.

– Threads should be managed by the kernel, not in user space

Better: have a decent process model and avoid this process/thread dichotomy.

Rob Pike
Dave Presotto
Ken Thompson
Phil Winterbottom

Distributed shared memory – an idea that never made any sense to me.

[edited to reflect the passage of time][ and again to add the Pike/Presott/Thompson/Winterbottom text]

Avaya's bizzare business model

Years ago, we got some equipment from Avaya – not terrible, but not wonderful. A junior tech with no signature authority signed a contract with them that says in big letters on the top “until 2005”. In 2007 we got rid of the equipment and now have spent over a year attempting to cancel the contract – getting one absurd response after another from Avaya’s abusive collections department.  I’d never do business with these people ever again and I advise anyone else considering using their equipment to get all contracts carefully reviewed. What an attitude!

Avaya's bizzare business model

Years ago, we got some equipment from Avaya – not terrible, but not wonderful. A junior tech with no signature authority signed a contract with them that says in big letters on the top “until 2005”. In 2007 we got rid of the equipment and now have spent over a year attempting to cancel the contract – getting one absurd response after another from Avaya’s abusive collections department.  I’d never do business with these people ever again and I advise anyone else considering using their equipment to get all contracts carefully reviewed. What an attitude!