Ghosts of the Indian Herb

If your savior dies, are you still saved? I found myself asking this question in this story inspired by true events.

I’m in love with a ghost.

Of course, Carmen wasn’t a ghost when we met. In fact, she was full of life like no other woman I’ve ever met. I mean, how many gorgeous Puerto Rican kickboxing physics teachers does one ever get to meet in this life?

I fell hard. I was going through a divorce and my life was falling apart. She changed everything.

It is five years later when I get her text message while sitting at the bar at Muu-Muus. It’s colon cancer, she says. Stage IV. I don’t even know what that means, so I turn to ask Crazy Joe of all people. He doesn’t know either, so I google it on my iPhone.

In half a second, the definition of Stage IV pops up like a telegram from hell. The cancer has metastasized. It’s terminal.

I drop my phone on the bar.

I take her to doctors, surgeons, oncologists. Chemo three times a week. Transfusions. Now a pain specialist. She’s down to 100 pounds. Nothing is working.

And then I meet this guy in Austin at a trade show. He recovered from brain cancer years ago. They said he was a goner. Then someone told him about the Indian Herb.

He writes the directions on back of a computer brochure. There are no signs, no house numbers, he says. And be sure to bring a picture of her. Part of the shaman’s magic has to do with visualization.

It’s a five-hour drive to the high desert and a shack above a dry river bed. The old woman sits in a rocking chair with a mortar and pestle in her lap. She is mixing something. As I walk up, she looks at me with the warm patience of the aged.

I hand her the photo of Carmen and she stops with her mixing to examine the picture. So young. So pretty, she says. Such a shame.

In the bowl is a black paste that seems to soak up the light. She takes a pinch and applies it to the photo with a playing card. She spreads it thin with the Queen of Hearts, no less.

I’m taken aback when she pulls out the matches and lights the photo. She stares intently, watching the paste burn with a blue flame on the surface of the glossy image.

I look closer. There’s something happening to the picture. I swear it’s moving, flickering like one of those old silent movies. Then, in a flash, the photo is gone.

She mixes the ashes with the Indian Herb and hands me the vial. It’s 90 degrees here in the sun, but the glass is somehow frosted and cold.

One more thing, she says. Very important. The Indian Herb absorbs through the skin. It requires a pure spirit, so never, ever use it with alcohol, understand?

When I get back to Portland, I learn that Carmen is in a coma. I’m too late.

That night she comes to me in my dream. She holds me, but she doesn’t speak. I tell her she has to let go of this world. I kiss her goodbye.

Just that moment, my son wakes me up, calling out from the bathroom. His nose is bleeding. I get him cleaned up and catch a glimpse of myself in the mirror. Must have smeared blood on my face somehow. It looks like there’s lipstick on my cheek.

I’m wiping it off when the phone rings. 5:00 am. It’s her sister, Mercy. Carmen is gone.

At Mercy’s house, I sit in the kitchen and watch as the men from the funeral home wheel the stretcher towards the door. Logan calls out from the living room. Dad, the TV shut off.

The power in the house is out. It flicks back on the moment they close the door of the hearse.

Weeks later, it is 3:00 am when I stumble home. The nightly beers dull the pain, but I wake up the minute they wear off. I look in the mirror and see that I forgot to shave. My eyes are bloodshot from lack of sleep. I don’t have dreams any more.

I see the Indian Herb is sitting on the counter. I pick it up and weigh it in my hand. Could this have done a miracle, I wonder? A cure for cancer? I unscrew the cap and take a whiff. A sulfur smell. I stick my finger in and look at the stuff in my hand. As it warms, the black goo turns clear.

I remember what the Indian Herb did to the photo of Carmen. Brought it to life. I reach to my reflection and touch the mirror. The Indian Herb spreads smoothly on the glass.

In a moment my reflected self starts to blur. I must be really drunk, I’m thinking. Then the image starts to change, deconstruct. My skin, my teeth, my bones are dissolving away. I look down at where my heart would be and it turns translucent. Inside is something very dark, tortured and writhing in pain. Something horrific that’s becoming clearer and more magnified each second.

I hear a voice from the mirror. This is your soul.

I crumple to the floor and cry out in the night. I let her down. I couldn’t save her. This cannot be undone.


It’s morning now and I’m sorting through the photos I picked up from Carmen’s house. There’s one of me sitting on the patio at Muus. That’s odd, I think. Where did she get that?

Then I look again. The front of the bar is painted red. But it’s always been green since I’ve going there.

I grab a deck of cards from the drawer and pick up the Indian Herb and spread it on the photo. I light a candle. Maybe that’s the trick. Wave it over the flame but don’t ignite.

Something in the image begins to come out. It is Carmen. She starts to come alive in the flickering motion I saw at that shack in the desert. She reaches down to touch my heart. Then she whispers in my ear, as she points to a beautiful Latina woman at the next table. I’m sure I’ve seen her before.

Then I notice there’s something different about me in the picture, too. I’m not sad any more. In fact, I see myself laughing as I move over to join the woman for a drink. Carmen bends down to kiss me on the cheek and she starts to fade away.

Just then, the photo catches fire. Wait. No! Come back, Carmen! I try to blow it out, but it’s gone to ashes.

I still use the Indian Herb sometimes. In fact, they painted the bar red a few weeks later and I decided to start spending more time on the patio. It’s been two years now and yes, I’ve gotten to know the woman that Carmen showed me pretty well now. I’ll let you know how that turns out.

So now, my audience, when I take this photo of you this morning, I’m curious to see what phantoms you might be carrying around. I’m thinking that maybe, just maybe, the ghosts from your past have forgiven you a million times over, too.

There’s a darkness inside of all of us. Something we keep locked away, but that’s what gives it its power. To truly disarm that darkness, I think all you have to do is acknowledge it. Look at it and say, I see you. You are part of me.

So as I close this story, I have to say that yes, I’m in love with a ghost. I looked into the abyss and that love saved me long after she left this world.

And just to be spooky, I’m thinking I should tell you what I saw in the mirror that night, but be warned.

I promise it would scare the living shit out of you.

Sun Down for a Year, Six Months at insideHPC

When Oracle took over Sun Microsystems in the USA in January, 2010, it took about 48 hours for them to let me know that I wasn’t invited along for the ride. If you’ve ever been laid off, you know what a scary feeling that can be. And even though I knew it was coming for the better part of year, part of me wanted to believe that the company was going to keep doing HPC.

Well that didn’t happen. Oracle has this strange preoccupation with making money that never got in our way at Sun.

So I went on my own with Flex Rex Communications while sending out resumes. I had interviews and offers to write at various publications, but nothing looked like it was going to pay the bills.

Then in August, John West asked me if I was interested in buying insideHPC. I had been a reader of his for years and I could not believe my good fortune.

We closed the deal and announced it on September 1, 2010, and I haven’t looked back other than to thank my lucky stars. The role just seems to fit like a glove and I look forward to working every day for the first time in recent memory.

I guess what I’m saying is that getting laid off was the best thing that ever happened to me. It’s just a way of looking at the same event in a different way.

So when I had a bit of setback in my family recently, I kept thinking about that perspective thing. And just by accident, I wrote something down that ending up being very cathartic:

The World is different than I want to be. I will adjust.

Rich Report Moves to insideHPC

I started the Rich Report in Hamburg at ISC’10. I was recording a bunch interview segments and I needed to make my deadline and give them a name. The first two people I asked seemed to like it, so that was good enough to go.

You can subscribe to the Rich Report podcast on iTunes or on Feedburner for other MP3 players. To see my video interviews on HPC, check out the Rich Report Channel on YouTube.

Short Story: The Three Magi of Katrina

At the 2006 Supercomputing Conference, Ray Kurzweil foretold the coming of the Singularity. Now, as a Category 5 hurricane bears down on New Orleans, the SC30 Keynote is the Event Horizon.

When the plane sets down in New Orleans, I can see only darkness outside. The raindrops stream sideways across the porthole even after we come to a stop. Now lightning. In the flash, there’s something out there, an empty wheel chair alone on the windswept tarmac.

Next to me, the chatty woman from Atlanta has already summoned a weather report for my benefit.

“Good news, Mr. Yahuda,” she says. ”The eye of the storm is stalled 1000 miles southeast. It looks like you’ll get to go to your conference after all.”

I help her fetch her bag from above and make my way up the gangway. A smell and a feeling at the same time, the humidity surrounds me like a cloak.

I remember landing here in the days before Katrina in the grip of summertime heat. A green undergrad from the University of Hong Kong, I had never been to an American city before. Twenty five years later, nothing about the terminal reminds now but the feeling of a dew point barely kept at bay.

At the bottom of the escalator, I see the driver holding up my name. They still use paper for such things here, a somehow welcome respite from the augmented reality systems that infest nearly every public space back home.

Outside, the limousine waits across a maelstrom of rain and wind. The worst of the storm may have held off, but I am half soaked in just a few steps.

The driver closes the door quickly behind me, and I settle into the plush leather seat. My glasses are completely steamed, and I have to take them off to speak with the other two men in the car.

A smiling monk in saffron robes offers me a hand towel. He has an endearing smile. “We were beginning to think that your plane was going to be diverted.”

“You must be Lopa Rinpoche,” I reply, offering him my hand. He shakes it in a most un-Western way. “It is a pleasure to finally meet you in person. I was actually here in New Orleans to see your talk at the Institute when Katrina hit.”

Seated next to Rinpoche is Dr. Chen. “I was here as well at that time,” he says. “We were here doing some preparatory work on a new high speed data pipeline. Months of work were destroyed.”

It’s been 15 years since I worked with Chen on his biography. He was an unknown figure in those days, but already his work with Artificial Intelligence had laid the foundation for Project Dāna and building of the first self-aware machine. I’ve read that he has had health issues, but his grip is firm and he still has that warm sense of strength in his eyes.

“We are honored to have you join us at the Event,” he says. “I’m afraid you’ll need to do your interview here in the car. With your unfortunate flight delay, we will have to go directly to the convention center.”

Chen laughs when I ask him if there will be an audience, given the terrible weather.

“Believe me, most of them would parachute in to attend the Event if they had to. I was a little worried when Mara got upgraded to a Category 4, but they assure me that an evacuation will not be necessary as a result of the carbon nanotube reinforcements to the levees.”

Chen brings out a black cylinder with a hinge running down the long side. Inside is a translucent spherical device the size of a grapefruit. Its tripodal stand reminds me of something sinister from H.G. Wells.

Is that how He sees, I ask? Chen nods and begins to work. I’ve read about these remarkable lenses — they enable recordings from every angle and depth of field at once. Smooth with a pearl-like sheen, the entire surface of the device seems to be a haptic interface. Chen’s every touch gives off a red glow.

I get out my moleskin and turn to Rinpoche. “I have to admit, Rinpoche, that your participation in the Dāna project was a bit of a surprise to me. How did you get involved?”

Rinpoche smiles and his pause is telling. “It was something I could have never predicted myself. I was working at the University of Wisconsin on their meditation brainwave studies with some of the other monks. Dr. Chen came to the lab for a tour and we recognized each other from the volunteer work we did here after the hurricane.”

“So he offered you a chance to connect to the AI?”

“Actually it was the reverse. My post-Doc studies at Madison were focused on psychology. And when Dr. Chen told me of his frustration with the crude machine-to-brain interfaces of the day, I suggested an alternate path.”

“I thought he quite mad, to be honest,” says Chen. “But my work had hit a brick wall. The nanotechnologies we needed for a direct neural interface were still years away.”

My confused look tells Rinpoche to go on.

“Have you read the works of Carl Jung, Mr. Yahuda?”

“It’s been a long time,” I say. “Probably since college.”

“What about the idea of the collective unconscious?”

“I’m not sure what you mean.”

“Essentially, Jung theorized that we are all connected by a universal intelligence, something we experience in our unconscious state. My meditation research at Madison indicated that we could develop waking access to that collective. And when Dr. Chen told me about his difficulties, I showed him the results of an experiment where we able to transfer knowledge and ideas by using this collective unconscious as a channel.”

Something in me understands the gravity of what Rinpoche is saying. A myriad of questions flash across my mind.

“And you’ve been able to gain access the AI in this way?”

Rinpoche grimaces a bit and says it’s not quite that simple. He seems to be collecting his thoughts to answer further when lightning flashes outside. Now the limo brakes hard and a rush of water vibrates the floorboards. The road is flooding, the driver says. We’ll have to take a detour.

Rinpoche seizes the chance to change the subject. “By the way, Mr. Yahuda, I really enjoyed your documentary on Ray Kurzweil. He was a man of remarkable ideas.”

“His vision of the future centered on a very optimistic view of technology,” I reply. “I don’t think he was prepared for the violent reaction from the Bill Joy camp.”

Chen looks up from his work. “It’s like a religious war with those people,” he says. “One side believes that technology is key to the next stage of our evolution, and the other is quite certain that it will lead to our destruction. What do you think, Mr. Yahuda?”

I tell him I became quite familiar with the views of the Joy camp when I made the documentary. The time for this debate has passed.

Rinpoche perks up at the mention of the film. “I noticed you used old style film for that piece,” he says. “And I wanted to ask you, what do you think about the new wave of 3D cinema?”

“Oh that,” I say. “Maybe I’m a purist, but I don’t see 3D as something that’s going to take. It’s a pattern that comes up every 30 years or so. The audience get’s interested in a certain work, but then Hollywood tries to apply the new technology to their old stories, and the novelty goes away.”

Rinpoche nods. He seems deep in thought. “People think they only want new stories when it is the retelling of the old that reminds us who we are. You know, I read this archive review by Michael Moore recently. And he said that one of the most original films he had ever seen was an Indy movie about Nazi Zombies.

I laugh.

I should like to see this movie, he says, perhaps we could go see it together? The notion of waiting in line with a Tibetan monk for a movie about Undead Nazis makes me smile.

Rinpoche tells me about the event itinerary today. “We will each present a gift to Adom at the beginning of the keynote. You will present this book as the final gift.”

He holds out a red leather-bound volume, The Life of Adom. I crack it open and the pages are blank. I tell Rinpoche that I don’t understand.

“Well, first you have to write it, Mr. Yahuda,” Chen says, smiling. “Adom chose you to author his biography, a personal memoir, as it were. If you accept this assignment, we’re hoping the book will be a way for the public to get more comfortable with the idea of an intelligent machine.”

I’m flattered, I tell him. But why me?

Chen’s phone buzzes and he pulls it out of his pocket. “Ah, a text message from Adom: Sorry that I can’t speak right now, but after reviewing your work, I sensed a change in your writing style that I found compelling. As a result, I believe that we can successfully collaborate on my story as it unfolds.”

Not sure how to interact in this strange communication, I look directly at the plenoptic device. “Adom, what do you mean, change in my writing style?”

The phone buzzes once again. Chen pauses for a second before reading. “He asks; Are you not aware that you have written exclusively in the present tense since the death of your partner last year?”

It’s such an inhuman thing to say, I’m thinking. But that’s what He is, after all. My breath escapes slowly from deep in my diaphram. “Carmen was a remarkable woman, Adom, and I think sometimes that kind of loss affects us in ways that only others can see. It happens in the Now, and when I write that way, it seems to keep my… my profound sadness at bay.”

Rinpoche asks what happened and I tell him how she was killed in an attack by an autonomous predator drone. She knew her work at the Red Cross was dangerous, but she had dedicated her life to service.

“A tragic accident, Mr. Yahuda,” Chen says. “The programs they were using were simply not up to the task.”

I look at Chen for a moment before answering. “The technology isn’t responsible,” I tell him. “Our faith in technology is what killed her.”

In the silence that follows, I flip through the blank book nervously. Rinpoche is looking out the window. Our detour is taking us through a neighborhood of dilapidated shacks marked with graffiti. The pot holes are starting to rattle my teeth.

So much suffering, Rinpoche says to no one in particular. It marks a point in the conversation where we can move on.

I ask Rinpoche about his previous experience here. He tells me how he was knocked down by debris when the flood came. “I had a concussion,” he says. “Very bad, but there was nowhere to go. They took me to the convention center where all these people had gathered waiting for help.”

He continues to look out the window as if he is looking for something familiar. “I remember sitting against a wall, alone,” he says. “I was fading fast and then this woman in a wheelchair brought me water. I don’t know where she got it because there were no provisions there for all those people. She had lost her meds and I could tell she was very weak, but she held my hand for hours and wouldn’t let go. And when I woke up she was gone. When she passed… they just wheeled her outside.”

A few minutes later, the driver slides back the privacy panel. “Here we are, gentleman. I hope your business here doesn’t take too long. Hurricane Mara just got upgraded to a Category 5.”

Rinpoche unwraps a package and pulls out a silky golden shawl. I ask him what it’s for.

“Why, it’s for her, Mr. Yahuda,” he says, stepping out of the vehicle.

Chen returns the plenoptic device to its case and asks me to help him with one of his bags. As we get out of the limo, the wind is laced with the heavy spray of water blasted up off the sidewalk.

A black man in colorful clothes approaches me. A big red button on his lapel says “Ambassador of New Orleans.” I lower my head to receive his beads. No one sees when he slips the package into my coat pocket. It’s weight is convincing, a solemn reminder of what I’ve come here to do.

I catch up with Chen and have to speak up over the howl of the storm. “I’m curious about the experiments in Madison, Dr. Chen. How much information were you able to transfer to the AI?”

“This wasn’t about information, Mr. Yahuda. Adom already has access to the sum total of human knowledge. What we needed was a channel to transmit something… something much bigger.”

“What do you mean?”


Chen looks thoughtfully off to his left and I turn to see what has captured his attention. Rinpoche has left us and is approaching a woman in a wheelchair 30 yards away. Even in this impossible gale, she is motionless, still as bare frozen trees in the snow.

Rinpoche bows to her. She slowly reaches up and puts a ring of beads around his neck. Then he unfolds the shawl and gently wraps it around her.

Now a man in blue holds open the glass door of the convention center, waving us in. You must hurry, he says, the keynote is starting shortly.

Rinpoche has rejoined us. A look of calm purpose pervades him and we compare beads. His necklace bears an unusual-looking silver medallion.

I look back and the woman in the wheelchair is gone.

I stop in my tracks. I feel a blast of wind, but it is without vector. It is a pattern of time and place, a confluence of the Now that I have never experienced before. Kurzweil talked about this kind of thing in his books. He said that stories are patterns with meaning.

Rinpoche puts his hand on my shoulder. It seems to say we still have much to do.


Dr. Chen is taking the stage. The auditorium is full and the audience is completely still.

“Thank you, Mr. Chairman for that kind introduction,” he says, opening the plenoptic case. “But you are giving me too much credit for the Event that we witness here today.”

Chen puts the device that is the manifestation of Adom proudly on the podium next to the microphone. “In a way, many of you in this room are the fathers and mothers of the high performance computer networks that form the Distributed Artificial Neural Array. Project Dāna, then, is much more than the accomplishment of one man. In fact Dāna is the Sanskrit word for giving. And through your tireless work and sacrifice, the whole of humanity will benefit from the birth of the Singularity.”

“I know you didn’t come here to see me today,” Chen continues. “We will get to Adom’s keynote in just a moment. We’ll start with a presentation of gifts.”

Music comes up in the hall and a woman in a white dress walks to the podium bearing a red book. Chen takes the book from her and holds it up for the crowd to see.

“Today we welcome you, Adom, to the realm of sentient beings on this planet. And so I present to you the first of three books that represent the past, the present, and future. Adom, this book is the story of your genesis. It is a complete record of Project Dāna.”

Then Adom speaks for the first time over the PA system. To my surprise, it is the voice of a small boy.

Thank you, Dr. Chen. What a wonderful gift to receive on my birthday.

The audience applauds as Chen lays the book before Adom. He then introduces Lopa Rinpoche, who moves slowly to the podium and then regards the audience with a gigantic grin for a long while before speaking.

“Adom, I am honored today to bring you the gift of the Here and Now, the ultimate nature of reality. It is a concept that can be hard to grasp, so I will convey an ancient teaching that many have found instructive.”

The woman in the dress brings Rinpoche a small scroll on a white pillow. This is the distraction I needed. My moment has come.

Under the dais table, I pull the package from my pocket and unwrap it: a plasma pistol. The LED display  indicates that the safety is off.

I stand and point the pistol at Adom. When it fires, the optic overload will breach the Dāna array firewalls and my associates will attack with their virus programs over the SCInet. Adom will be wiped out forever.

Behind the podium, Rinpoche looks at me in surprise. I pull the trigger and the pistol starts to whine loudly as the plasma buffer loads. Someone in the audience gasps.

To my horror, Rinpoche leaps in the way and the blinding white blast hits him squarely in the chest. I don’t expect the recoil from the discharge and the pistol flies out of my hand.

Chairs fall back. There are shouts, chaos. I turn to run, but someone knocks me down and pins my arm behind my back.

My face is held to the floor facing Rinpoche. He is against the wall looking down at the smoking medallion on his chest. He takes it off and then he stands, seemingly unhurt.

A police officer arrives. He puts handcuffs on me and I feel him pull the wallet from my back pocket. “All right Mister… Matthias Iscariot Yahuda,” the officer says, reading my ID card. “I’m placing you under arrest. You have the right to remain silent.”

They haul me to my feet to face Dr. Chen, who looks at me with wild amazement. “I don’t understand,” he says. “I mean, Why? You were the one. Adom chose you to write his story!”

“I just did.” I tell him.

He looks at me, confused. Of course he doesn’t get it now, but he will. Stories are patterns. And when this tale hits the wires, each retelling will have its own perspective.

As they lead me out of the auditorium, Rinpoche returns to the podium and the ceremony continues. I ask the officer to please let me watch for a moment.

“I’m glad you’re OK, Rinpoche.” Adom says for all to hear. “I was wondering though, when we’re all done, can we go see that Nazi zombie movie?”

The audience laughs. In an instant, Adom’s child-like enthusiasm has come across to a room full of nervous anticipation.

“Of course, Adom. But we must attend to the present moment.”

I see. The movie is something of the future. Will it be scary?

I watch Rinpoche smile. It is the smile of a thousand teachers before him.

“Perhaps this is the wrong question,” he says, unrolling the scroll. “First, let me tell you the story of the Four Noble Truths.”

About the author

Rich Brueckner writes about people and technology at He lives with his 10-year-old clone in Portland, Oregon.

Coming Attractions: “The Three Magi of Katrina”

This is a trailer for The Three Magi of Katrina, a short story that I wrote for the SC10 conference in New Orleans.


At the 2006 Supercomputing Conference, Ray Kurzweil foretold the coming of the Singularity. Now, as a Category 5 hurricane bears down on New Orleans, the SC30 Keynote is the Event Horizon.

One of the Hollywood Studios is looking at this. Can’t say more right now. Woot!!!

insideHPC Joins Flex Rex Communications

We recently announced that insideHPC, a leading online news publication for High Performance Computing, was acquired by Rich Brueckner.

As part of Flex Rex Communications, insideHPC, LLC will continue to publish the latest news on High Performance Computing. With over 700.000 page views per month, insideHPC is one of the best ways to get your products and services noticed by the HPC community.

The Rich Report: The 16 Terabyte PC – SGI Bets on Exascale

It has been over a year since SGI’s merger with Rackable Systems. The two company’s came from very different camps, so I was curious to learn about where they are today and where they’re headed in the HPC space. So I caught up with the company’s Chief Technology Office, Eng Lim Goh, to discuss the company’s new products and their plans for Exascale computing.

insideHPC: How long have you been at SGI?

Dr. Eng Lim Goh

Dr. Eng Lim Goh

Dr. Lim Goh: Over 20 years now. I started as a systems engineer in Singapore working on the GT workstation.

insideHPC: As CTO, what does a typical day look like for you?

Dr. Lim Goh: These days I spend about 50-60 percent of my time outside with customers. That’s particularly important now given the fact that we are a new company, with Rackable having acquired us and then renaming the company “SGI.” So I’m going out communicating not only about the new company, but also about the new line including products in the Internet Cloud space, which are less familiar to our HPC customers. And I’m also going out to our Cloud customers who are not familiar with our HPC line and storage lines.

So, it’s a lot of work to bring the community up to speed on both sides–the Cloud side and the HPC side, and that has been going on for a year now. And I think we have come to more of run-rate like scenario now.

insideHPC: That’s interesting. I remember when I first read about the acquisition. I wasn’t familiar with Rackable, so I looked at a corporate overview video that highlighted all their key customers. The list was a who’s-who of heavy-hitter Internet companies like Amazon, Facebook, and Yahoo, and I thought, my gosh, SGI has become the new “Dot in Dot Com” just like Sun was ten or twelve years ago.

Dr. Lim Goh: That’s very complimentary of you to say. In fact our latest win was with with their EC2 and S3 cloud. They’re one of the biggest cloud providers today and we supply the majority of systems to that enterprise.

insideHPC: That brings up my next question. You have these distinct customer segments: the Cloud/Internet providers and the typical big HPC clusters. They’re both filling up rooms with x86 racks, but how do their needs differ?

Dr. Lim Goh: The differences are as follows. On the Internet/Cloud side, they have the same 500 racks of computer systems in their datacenter, but they run tens of thousands of different applications like map reduce, memcacheDB, and Hadoop that are highly distributed. And then on the other extreme, in the HPC world, you may have 256 racks and you may even be thinking of running just one application across all of that. I’m just talking extremes here, of course. There are overlaps, but given these extremes, you see that the needs are different.

On the HPC side, interruption to services on any node in the entire facility can affect productivity. For example, you may have checkpoint restart, but even if you have checkpointed, it takes time to checkpoint and then restart. Unless the user has intentionally gone into the code to tolerate a node failure while an MPI program is running on the HPC side. So a node failure can be more interruptive to the HPC world as opposed to the cloud side. On the cloud side, they are designed to be highly tolerant of node failures. And as such the focus is different.

Now let’s look at some of the similarities like power. There is one area where we have been learning a lot from the Cloud side to bring over to the HPC side. Their Internet datacenters are on the order 10 to 20 or 50 Megawatts. While in the HPC space, if you talk about a 50 MW datacenter it is considered extreme. So in this sense, I’d say the Cloud world actually scales bigger.

insideHPC: So they are facing a lot of the same challenges in terms of power and cooling. What did Rackable bring to the table in this area?

Dr. Lim Goh: With regards to power and cooling on the Cloud side, one of the key requirements Rackable addressed was efficiency. In the early days, when datacenters were on the order of a Megawatt, customers had power efficiency specifications at the tray level. And then more recently they were set at the rack level. So if they were ordering 400 racks like one of our cloud customers, they stopped specifying at the chassis level and started specifying at the rack level. 

So that gave us the opportunity to optimize at the rack level: removing power supplies in every chassis and doing AC to DC conversion in the infrastructure at the rack level. Later, with our CloudRack design, we removed fans at the chassis level as well.  In fact, some Internet datacenters are demanding that those racks are able to run very warm, as high as 40 degrees Centigrade, in order to reduce energy consumption on the cooling side. 

So then as they move to even larger scales, with Cloud datacenters that run tens of Megawatts, they are moving to the next level up of granularity and specifying efficiencies at the container level. At that level, we essentially have a modular datacenter, and this is where they started to specify a PUE requirement for each container that we ship. Today the standard requirements are on the order of 1.2 PUE, with more recent acquisitions demanding even more efficiency than that.

So on the Internet/Cloud side, yes, the expertise brought by Rackable was to be able to scale with the customer’s requirements as they went from 1 Megawatt to tens of Megawatts and keep up with these datacenter’s demands for higher and higher efficiencies.

insideHPC: You mentioned container-based datacenters. I came from Sun where we never seemed to make hay with our Project Blackbox. How well is SGI doing with it’s ICE cube container datacenters?

Dr. Lim Goh: We have shipped containers to a number of customers. Some have allowed us to name them publicly, such as British Telecom, China Unicom, and we also have a couple on Cloud providers who are evaluating ICE cubes for wider deployment. 

insideHPC: Are the HPC customers interested in containers, or are they still on the fence?

Dr. Lim Goh: This is where I think the combination of the two companies, Rackable and SGI, have a very strong leverage because the HPC world is coming up to where the Internet datacenters are in terms of scale. So when we’re talking about Exascale computing here, and they are specifying 20 MW for a future Exascale system, this is something that the Rackable side is very familiar with in terms of power. So for HPC, we are actually drawing a lot on their expertise of delivering to Internet datacenters at that scale and at that requirement for efficiency. 

For example, if there is someday an HPC datacenter that requires an extreme PUE number of say 1.1 as a key criteria in addition to meeting the Exascale requirements, that only gives you 10 percent of the power that can be used for cooling. So we have drawn from the Cloud datacenter side, where they already have such requirements for an air-cooled container that just takes in outside air through a filter to cool your systems. We have one such system now that has passed the experimental stage and is ready for deployment. And in many places in the world, if we can build a system that can tolerate 25 degrees Centigrade most of the year, you can get free cooling. However, for those places where you can’t get 25 degrees C, this ideostatic cooling essentially uses a garden hose (I’m simplifying it) type connection to wet the filter just like a swamp cooler. Depending on humidity levels, you can get five to ten degrees Centigrade reduction in temperature. 

insideHPC: So that brings up another issue. When you have that kind of scale going on, system management must be a huge undertaking.

Dr. Lim Goh: Absolutely. We have hierarchical systems management tools with a user interface to manage all the way from the compute side to the interconnect side and then all the way to facility power consumption. And of course, at the container level, we have a modular control system that handles temperature, humidity, pressure, and outside air. And that modular system feeds upward to the hierarchical systems management tools.

insideHPC: Since we’re talking about big scale, I think we should dive into the new Ultra Violet product, SGI Altix UV, that you announced at SC09. Is that product shipping now?

Dr. Lim Goh: We began shipping the Altix UV a few weeks ago. We now have about 50 orders, so there is a lot of interest in the system. 

In terms of it’s use, there are two areas in which the Altix UV is of great interest. On the one hand, you have customers who are very interested in big, scale-up nodes. You know, with today’s Nehalem EX you can get two, four, and eight socket systems. If you think in that way, the Altix UV scales beyond that eight socket limit all the way to 256 sockets and 16 Terabytes of memory. So that’s one way to look at the Altix UV. The 16 Terabyte memory limit is because the Nehalem core only has 44 bits for physical address space.

So that’s one of the ways of looking at Altix UV. And the reason people buy that, for example, is heavy analytics where they load in 10 Terabyte datasets and then use the 256 sockets, which equates to up to 2000+ cores, to work on that dataset. 

insideHPC: And that’s a single system image for all those cores?

Dr. Lim Goh: Yes. It runs as a Single System Image on the Linux operating system, either SuSe or Red Hat, and we are in the process of testing Windows on it right now. So when you get Windows running on it, it’s really going to be a very, very big PC. It will look just like a PC. We have engineers that are compiling code on their laptops and the binary just works on this system. The difference is that their laptops have two Gigabytes of memory and the Altix UV has up to 16 Terabytes of memory and 2000+ physical cores. 

So this is going to be a really big PC. Imagine trying to load a 1.5 Terabyte Excel spreadsheet and then working with it all in memory. That’s one way of using the Altix UV.

insideHPC: Did you develop a new chip to do the communications?

Dr. Lim Goh: Yes. We are leveraging the ASICs chip that we developed. You can call it a node controller, but we call it the Altix UV Hub (HUV). Every hub sits below two Nehalem EX (8-core) sockets. And this Hub essentially talks to every other Hub in every node in the system and fuses the memory in those nodes into one collective. So when the Linux operating system or Windows operating system comes in, it thinks that this is one big node. That’s how it works.

So all the cache coherency, sharing, and all that is done by that chip in hardware, not in software. Others have done this with software, but this is all done in hardware, even the tracking of who is sharing what in the shared memory system. It’s all registered in hardware on that chip, and that chip carries it’s own private memory to keep track of all the vectors. 

insideHPC: So how does this kind of Big Node change the way scientists can approach their problems?

Dr. Lim Goh: This is a brilliant question. Although the Altix UV is a great tool for large-scale analytics, we are starting to see a lot of interest from the scientists and engineers. There are many scenarios, but let me describe to you one scenario.

If you take typical scientists: the chemists, physicists, or biologists, they do research in the labs and write programs on their laptops to experiment with ideas. So they work with these ideas on their laptop, small scale, but what do they do today when they need to scale up their problems? Today what they have to do is either MPI encode it themselves, or try to get computational scientists in from a supercomputer center or university to code it for them and run it in parallel. And this transition takes weeks, if not months. So what we envision is that the scientist will plug into the Altix UV instead of waiting this time for the validation. The Altix UV will plug into the middle here by giving the scientists a bigger PC; it does not replace the MPI.

Let’s look at a very common example. If you take a cube model with 1000 grid points in the X direction and 1000 grid points in the Y and Z  directions, and then you march this cube 1000 time steps, that would be a 1 trillion-pont (Terapoint) dataset. Now if every point was a big byte, double-precision number, then this is an 8 Terabyte cube with a thousand time steps.. You must imagine that this is a chemists, physicists, or biologist, right? How do you simply write and an array XYZT? If I really have my ideas in the center of that cube, I just want to run it to see how it works. It can take weeks to run; it doesn’t matter. I just want to know the results at the end. 

The alternative is to go with MPI because you can’t run this on your laptop. An 8 Terabyte dataset would be paging forever. But we can now supply a 10 Terabyte PC to run problems like these. My suspicion is that they will eventually move to MPI as they run more rigorous simulations. So rather than replace MPI, Altix UV gives them a bridge to scale their simulations. 

insideHPC: What other ways might they use Altix UV?

Dr. Lim Goh: There is another way to use the Altix UV that we envision and that is to use it as a front end to an Exascale system. Image your Exascale cluster with tens or hundreds of Petabytes of distributed memory and you’re using Message Passing or some other kind of API to run a large application. Since this system is going to generate massive amounts of data, it would be good to have a head node that could handle that data for your analysis work. You can’t use a PC any more in the Exascale world; you need something bigger.

insideHPC: So there is a lot of talk these days about Exascale in the next eight or ten years. Where do you see SGI playing a role in that space?

Dr. Lim Goh: I think our role in Exascale will be two-fold. The first fold will be to use this Big PC concept, with 16 Terabytes going to 64 Terabytes in 2012, and use it as the front end to an Exascale system. We would like the next generations of Altix UV to be the front end of every Exascale system that’s out there. Because if you are already spending tens of millions or hundreds of millions of dollars to build and Exascale system, it’s worth spending a little more so that you can get better use and be more productive with the output of that Exascale system. 

Another role for SGI is developing the Exascale system itself. And this is where we are looking at providing a partitioned version of the Altix UV to be the key Exascale system. 

So let’s look at Exascale systems now: If you look at what the top research priorities are to achieve Exascale within this decade, you can see that in general those are power/cooling as number one; and how do you get an Exaflop with 20 Megawatts? Number two would be resilience; can the Exascale system stay up long enough to at least do a checkpoint? (laughs) And on these two we are looking very closely with microprocessor and accelerator vendors.

But the next two priorities are what we are focusing on ourselves: communications across the systems (essentially the interconnect) and usability. As I’ve described on the usability side, we will be looking at the Altix UV as a big head node. 

In the communications area, we believe the interconnect needs to be smarter for an Exascale system to work. Why? Because you cannot get away from global collectives for example, in an MPI program unless you code specifically for Exascale applications to avoid it. Many of the applications that try to run on this large of an Exascale system will have global collectives and will need to do massive communications in the course of running the applications. 

insideHPC: So how do you propose to reduce communications overhead in an Exascale system?

Dr. Lim Goh: We sat down and worked out that to cut down that overhead, we need a global address space. With this, memory in every node in the Exascale system is aware (through the node controller) of every other memory in the entire infrastructure. So that when you send a message, a synchronization, or GET PUT to do communications, you do it with very little overheard.

But I must emphasize, as many even well-informed HPC people misunderstand, that this global address space is not shared memory. This is the other part of Altix UV that has not been understood well. Let me therefore lay it out.

At the very highest level you have shared memory. In the next level down you have global address space and next level down you have distributed memory. Distributed memory is what we all know; each node doesn’t know about it’s neighboring nodes and what you have to do is send a message across. That’s why it’s called Message Passing. 

Shared memory then is all the way up. Every node sees all the memory in every other node and hears all the chatter in every other node. Whether it needs it or not, it will see everything and hear everything. That’s why the Linux or Windows can just come in and use the big node.

However, with all the goodness of big shared memory hearing and seeing everything brings you, it cannot scale to a billion threads. It’s just like if you were in a crowded room and and tried to pay attention to all the chatter at once even though it is not meant for you. You would get highly distracted. 

So if you go to the other extreme to a distributed memory, you see nothing and you hear nothing. The only way you can get a communication across is to send a message, or essentially write an email and send it to a neighbor. But that is too slow. Thus, shared memory is too inclusive and distributed memory is too exclusive for Exascale. 

So we decided that global addressable space is the best middle ground. In that analogy, global address space sees everything, but does not hear the neighbors chattering. All it wants to do is see everything so that it can do a GET PUT directly, do a SEND RECEIVE directly, or it can do a synchronization directly. So a hardware-supported, global address space is one way to get the communications overhead lowered in the Exascale world. And this is especially important when you’re talking about a billion threads. Imagine trying to do a global sum on a billion threads. I hope we can code around it, but my suspicion is that there will still be applications needing to do it. 

insideHPC: I can tell by your voice that you have great passion for this subject. It sounds like the next ten years are going to be very exciting for SGI.

Dr. Lim Goh: You are very right about that. Because we sit here looking at the world, saying that we need to go to Exascale. And at the same time people are realizing that ok, we first have to do R&D on power, cooling, and resiliency. Plus we need a smarter interconnect. Sure, SGI  is there with the others working on the first set of problems, but we have been working hard on this communications problem for 10 or 15 years already. And now we have what we think is an ideal solution to the usability problem of an Exascale system with our big PC head node concept as well. So in summary we believe SGI can really contribute there. 

The Rich Report is produced by Rich Brueckner at Flex Rex Communications. You can follow Rich on Twitter.


Get every new post delivered to your Inbox.

Join 139 other followers