Why Google's security arm doesn't have a war room

The computer security industry historically borrows military defence concepts to combat digital threats, literally creating war rooms where experts follow attacks in progress on huge screens with phones ringing off the hook.

Not so at Google's Postini email security service provider unit. Instead, computerised systems monitor three billion messages per day that flow in and out of customer systems and pass through Postini's thousands of machines in datacentres around the US and in Europe before hitting the internet. The Postini system is highly automated, distributed and scalable, characteristic of all of Google's operations.

Google's Gmail anti-spam efforts are separate from those of Postini, which Google acquired two years ago, although it follows similar computerised operations and the teams have started to integrate the processes.

Postini represents Google's commercial push into email security, offering a subscription-based service to more than 50,000 customer companies and organisations and more than 15 million business users. In addition to protecting email from spam and viruses, Postini offers compliance and archiving services.

Sentinels and canaries

About 35 members of the Postini Site Reliability Engineering team have access on their machines to a dashboard that shows the number of transactions that the Postini service is handling per second, as well as the message per minute rate and graphs of the error percentage rate obtained from a test system known internally as Sentinel, according to Craig Croteau, who leads the group.

The Sentinel system has devices located on dedicated pipes into the internet with daemon software, automated programs that run in the background that routinely send out test messages to gauge the performance of the flow through the Postini infrastructure.

If there is a problem with a round-trip test message, indicating possible congestion, it will show up on the dashboard.

"It's a canary in the system," a tiny data stream that serves as an early warning system so potential issues can be stopped before they become major problems, Croteau said.

The Sentinal system posts the information to a database that feeds into the dashboard; one of several different data collection engines that are superimposed on the dashboard. Traffic monitors generate message rate graphs while the system extrapolates rates from live log scraping. Telemetry, remote measurement and reporting, is served up in multiple views.

Postini uses multiple fail-over sites, and if a potential problem is detected, the customer message flow is moved to a backup system. Unlike typical cloud hosting providers, Postini's subscription service does not store the customer's data on its servers but provides the protection services as the data passes through the Postini gateway.

In traditional network operation centres someone sitting in front of a screen notices a rise in error rates or some other problem, then conducts triage and follows a set workflow procedure for dealing with events, according to Croteau.

"There's a built-in lag," he said. "It can take minutes, 15 minutes, to do something," especially if the worker is...

Click here for the next page