Single core cell phone solution and marketing vs engineering

FSMLabs is finally able to discuss the single core cell phone handset solution we developed with Infineon Technologies last year. This system is cool technology – literally – since it allows parts count reduction on the handset and that reduces power consumption and that reduces heat production. More to the point, in a volume driven business, removing some hardware components phoneis big news. I hope that some of the “no software royalties” hard liners will now pause and consider why paying royalties (or a production license) in order to not have to ship chips and batteries for those chips may be, you know, profitable.

The technology is very interesting. Essentially, we squeeze a lot of performance out of an ARM9 – making it jump back and forth between user interface, radio control, and wireless stacks. Turns out, the ARM9 is up to the job – it was being held back by poorly designed software.

There have been many product announcements of the imminent release of Linux based single core cell phone solutions. I think that in this industry people sometimes forget that there are hard technical problems. Just having a visionary CEO say “make it so” and let loose the loudly singing marketing flock doesn’t actually get the bits to move right. Our software worked because of the smart engineering effort of Cort Dougan, Michael Barabanov, and Zwane Mwaikambo and the engineers they supervised and the work of the engineers at Infineon and also the excellent evaluation work done by Infineon engineering management that allowed them to avoid the mistake that everyone else made in searching for paths to this product. (of course there is a lot more to the phone than FSMLabs work, but RTLinux is agile enough to move the processor to the right place at the right time.)

Microkernels and why academic OS research is boring

Andy Tanenbaum writes a defense of microkernels that (1) misses the content of Linus Torvald’s critique, (2) ignores the most relevant paper on software development, David Parnas’ Software Jewels paper, and (3) pretends RTLinux does not exist. The problem with microkernels is that they are not modular and the problem with all the “nearly ready for prime time” microkernels is that they are not real products and the problem with many new academic OS projects is that they don’t have much new in them.

Parnas asks why we don’t see more elegant, simple, kludge-free “jewels” in systems software and he, rather gently, explains the compatibility, time pressures, and other constraints that make “clean and lean” so elusive. Parnas is just too nice to hammer the point home. Anyone can write small fast clean operating system that doesn’t do anything useful. Over and over, researchers stumble on the discovery that as you add minor inessential feature after minor inessential feature (all in response to some importuning victim who actually needs to do something with your creation), you recreate the “bloat” that wasn’t needed when your software didn’t do anything. In fact, the natural result of making a nice clean simple lean fast OS into something useful is the creation of another implementation of POSIX. VMS and then Windows NT are very POSIX like, despite the intentions of the designers and so is Mach in OS-X and so are many traditional RTOSs. After all this time, this should not be so surprising anymore.

Linus Torvald’s response to Tanenbaum is pretty clear, but Tanenbaum misses the point:

Linus also made the point that shared data structures are a good idea. Here we disagree. If you ever took a course on operating systems, you no doubt remember how much time in the course and space in the textbook was devoted to mutual exclusion and synchronization of cooperating processes. When two or more processes can access the same data structures, you have to be very, very careful not to hang yourself. It is exceedingly hard to get this right, even with semaphores, monitors, mutexes, and all that good stuff.

My view is that you want to avoid shared data structures as much as possible. Systems should be composed of smallish modules that completely hide their internal data structures from everyone else. They should have well-defined ‘thin’ interfaces that other modules can call to get work done. That’s what object-oriented programming is all about–hiding information–not sharing it. I think that hiding information (a la Dave Parnas) is a good idea.

And this gets both Torvalds and Parnas wrong. Information hiding is only good design when the hidden information is not needed by the software it is hidden from! If you hide information that you need to share you’re just wasting time. A great example of real modularity is the splitting off of the command interpreter (first in CTSS (corrected)) from the kernel. This split is possible because the designers recognized that there is very little information exchange between kernel and interpreter. An example of a fake module is the attempt of most microkernels to split virtual memory paging and storage caching. The most elegant and efficient message passing interface that can be imagined cannot fix the problem that too much information must be shared to make the resulting monstrosity run properly. Here’s Torvalds

The fundamental result of access space separation is that you can’t share data structures. That means that you can’t share locking, it means that you must copy any shared data, and that in turn means that you have a much harder time handling coherency. All your algorithms basically end up being distributed algorithms.

And Tanenbaum’s response is that sharing data structures is hard! Well, yeah – many engineering systems are complicated and hard to do right. During the 30 years of development of CTSS, Multics, UNIX, Plan9, and Linux, many components were split off into modules. What remains cannot be broken into decoupled parts in any obvious way. Let no man or woman split what Dennis and Ken have wrought without a damn good reason. Until the hardware changes or someone discovers something new about software structure in operating systems, complaining that this kernel model is complicated seems as useful as complaining that petroleum refineries are too big and filled with chemicals. The POSIX model is a high performing engineering design – it works well and it is just perverse to assume its success is due only to the ineptness of everyone else.

It would be very interesting to study a production operating system like Linux or Windows XP and try to discover hidden modularity – functions that appear to not need to be bound to each other. Do such functions exist? Perhaps – one was discovered only a decade ago. The basis of RTLinux is the recognition that “real-time” can be separated from a time-sharing kernel and put into a module that can operate in a decoupled manner. The result was immediately useful but RTCore, the RTLinux real-time kernel, does not correspond to any of the traditional “modules” advocated by microkernelistas. There is no reason to suppose more such unconventional modules cannot be found. If we keep going back to the same boring list of “modules” that are familiar to generations of students forced to look at that ridiculous layer cake picture of an operating system, however, we’ll keep getting the same lack of results and the same “almost as fast as, almost complete” projects.

GreenHills, foreigners, and the gummi bear threat

When GreenHills launched their anti-Linux offensive, they crossed a line – and I doubt they even knew it. To cast doubt on the value of Linux in critical applications, GreenHills tried to create a scare about the nationality and “loyalty” of engineers working on Linux. This is stupid on three levels. First, software security is an engineering property, not something passed on like a, a, well, a virus. There are established methods for vetting security, by test, by inspection of the design, and by inspection of the design process – none of these involves the visa status of the developers. Second, software is the flagship international product and has been since that Hungarian fellow von Neumann was working in Princeton. Maybe things look different in Santa Barbara, but in the rest of the world, software development is something that discriminates on the basis of talent and work and not anything else. Aldrich AmesFinally, it’s absurd to think that software companies can vett their employees better than government security agencies and we know that those agencies can employ people who work for the other side for years. Does Dan O’Dowd stalk around the GreenHills office looking for subtle signs of disloyalty and contraband? I tried to do that here, but other than Matt’s bag of gummi-bears, I saw nothing life threatening. Xenophobia is a poor substitute for solid engineering and it is a more real danger than hypothetical undercover programmers putting trojan horses into unsuspecting software.

Pitiful formal methods in computer science

FSMLabs is making its way through the DO178 requirements and we’ve given a lot of thought to Common Criteria over the last couple years. One of the advantages of DO178 is that it does not accept the totally untested and frankly unlikely premises of “formal methods”.

The reason mathematics has advanced so much was not because of the Euclidean axioms-lemma-theorem straitjacket, but in spite of it. Luckily, when we actually discover mathematics, we do it the Babylonian way, empirically and algorithmically. It is only when it is time to present it, that we put on the stifling Greek formal attire.

so says Doron Zeilberger

And the mystery quote. Who said this?

“That mathematics reduces in principle to formal proofs is a shaky idea” peculiar to this [20th] century. In practice, mathematicians prove theorems in a social context. […] It is a socially conditioned body of knowledge and techniques.”