Program structure and threads

Threads were initially used for telephony systems where each request to connect could be given to a new thread to manage. The simplification was, in theory, dramatic. The programmer could just write the code for handling a single line, and the OS and network code would multiplex operation over 100,000 calls. The programmer does not have to worry about what operations block  and delays. Imagine writing this same code using some sort of request queue. The only way to do it would be to make extensive use of asynchronous I/O, something that was uncommon when threads became popular.  In fact, any multithreading design seems convertible into a single threaded design via use of asynchronous events and asynchronous I/O. But these methods are not noticeably easier than threads.

rcvm(fd, &buf, n, func); // call “func” when the data arrives – pass it the return val and arguments to read as well

then

while(1){

rcvm(fd,&ptr++,n,got);

if(noresources)wait();

}

got(fd,&p,n,code){

look at message

queue_response

return;

}

Worse or better?

why databases are more interesting than operating systems

A discovery Cort Dougan and I made 10 years ago when we first got a glimpse at the mighty Oracle building on the Bay and realized that, commercially, operating systems are an afterthought.

This article shows that things are just getting better in a $500 million project!!

When tested, the new system failed to find matches for terrorist-suspect names that were spelled slightly different from the name entered into the system, a common challenge when translating names from Arabic to English. It also could not perform basic searches of multiple words connected with terms such as “and” and “or.”

Because the format of the data in the current database is “complex, undocumented, and brittle,” some significant data will be lost when the system is replaced by Railhead, according to the congressional report. For example, scraps of information such as phone and credit-card numbers found when law-enforcement and intelligence officials empty a suspect’s pocket, often called “pocket litter,” will not be moved to the new system.

Railhead was supposed to be completed by year’s end but has been delayed. Nearly half of the 72 so-called “action items” for the program were delayed as of June, congressional investigators found.

Common Weakness Enumeration

The common weakness enumeration is an amazing document. Imagine if there was such a document for architecture/construction. That document would contain admonitions like – “remember to put in structural supports for upper floors” and   “don’t lay floors on dirt” or “make sure there are no free unterminated wires left hanging out of walls.”  Here is the first entry in the CWE

The software, when constructing file or directory names from input, does not properly sanitize absolute path sequences such as “/path/here.”

Here’s another one

Without proper access control, executing a SQL statement that contains a user-controlled primary key can allow an attacker to view unauthorized records.

Ok. And you thought I was exaggerating. Just a few down we see:

The accidental addition of a data-structure sentinel can cause serious programming logic problems.

This means something like “don’t put an end of record marker in the middle of a record and don’t put a 0 in the middle of a C string.”

The next one comes with a great example.

An algorithm in a product has an inefficient worst-case computational complexity that may be detrimental to system performance and can be triggered by an attacker, typically using crafted manipulations that ensure that the worst case is being reached.

The example is here. It’s an example of a security feature that can be misused.

And here is a weakness that appears in many guises

The software fails to adequately filter user-controlled input for alternate script syntax.

The example is here.

Here we are in 2008, and, in desperation, organizations are being asked to certify that their software is developed in such a way that user commands are not passed directly to data base engines that do not check permissions!

Google's engineering culture

Article in the New Yorker on Google contains a fascinating description of a product design meeting:

Page and Brin had wanted an upgrade of an existing product, and they were unhappy with what they were hearing from the engineers. At first, they were stonily silent, slid down in their chairs, and occasionally leaned over to whisper to each other. Schmidt began with technical questions, but then he switched roles and tried to draw out Page and Brin, saying, “Larry, say what’s really bugging you.”

Page said that the engineers were not ambitious enough. Brin agreed, and said that the proposals were “muddled” and too cautious.

“We wanted something big” Page added. “Instead, you proposed something small. Why are you so resistant?”

The head of the engineering team said that the founders’ own proposed changes would be too costly in money, time, and engineering talent.

Schmidt, the only person at the meeting wearing a tie,  tried to summarize their differences. He noted that Brin and Page wanted to start by deciding the outcome, while the product team focused first on the process, and concluded that the engineering would prove too “disruptive” to achieve the goal.

In many organizations, “process” issues either internal to the engineering team or to the management teams are allowed to
gerbil dominate product development. Decisions on whether to provide some capability may be made, effectively, by an engineering team leader who is intent on protecting his/her authority over engineering resources, or a sales manager who has some dimwitted idea about market segmentation or internal power struggles or …..

The huge advantage companies like Google, or the Microsoft of 20 years ago, have is that the top management have technical and market understanding sufficient to overrule such concerns. It’s grimly hilarious to hear the standard stories about “customer centric” and “shareholder value” objectives from people working inside companies that are really dominated by internal politics and the career ladder plotting of management. Such companies running into a well financed monster like Google, have about as much chance as a gerbil in the piranha tank.