A year memoir of Fukushima 3-11 eathquake | A geek's diary

It has been almost one year from 3-11 Quake and Tsunami disaster in Fukushima. I was there, I saw and remembered it well. Yesterday I was attending Defcon Japan meeting and talk about this with the guy sat next to me. He urged me to write it, and this is it, I dedicate this writing to all 3-11 quake victims and to fellow IT/security engineers everywhere to let you know the actual feeling of what had happened on-the-site.

I was working in the Fukushima xxxx network for trouble shooting a a network packet filtration, when earthquake started on March 11th last year. When it was started 6(six) of us was inside of the secure data center. I remember that I was analyzing a weird SYN/ACK packets sent and received between proxy layer and core switch in within, so I was standing in the front of one 7feet high server's rack analyzing and a monitor console for the captured network packets. The server's racks in the room itself contains of 3 blocks of 20U (has big gaps within U's)racks of 10feet long. Shortly, when it hit us there, it was happening so fast but every glimpse of what I saw was perfectly recorded in my brain like sequences of a slow motion "flashback" movie, that often plays over and over in some of my nightmare until now, so yes, I think I can pretty much tell you the exact details. Now, a year had passed after the disaster, there were nights to woke up with the nightmare with seeing these visions and this is the first time ever I finally dare to write it.

When it was started, it was just like the previous smaller quakes previously happened during the period, it was started with the horizontal movement and slowly followed by the strange deep grunt voice of "Boom, boom, boom..!" coming from the center of the earth, then it went faster.. The vertical rocks then started to merge with the horizontal movement. I stopped my work, exit the shell right away, slam down the console and looked up. We were all stunned from our posts since we had 2 more groups separated in the network consoles and tables. We looked at each others for at least in the first 10secs, afterward one of us said "This one is getting harder and harder isn't it?", and that was the word that triggering our first action. About 20Us size server's racks (around 7ft height) was started to rock so hard, and every edge of racks bumped to its space limit safety block (actually it was dig and made with slack of 1 meter size for the quake precaution) with the loud voice, when the bumping on each hit got so hard, the balls inside of the rack's conjunctions was popped out and fly like a bullets. Some smaller server's, routers, switch's part was starting to fall (or FLY will be more like it..). I assure you there is no loose parts in that data center but the hit was so hard so it can throw stuff that was not tightly screwed. The situation is getting into a chaotic scene, still, at that time most of us were trying to save the service first. I saw four of the engineers run and hold the rack by their both hands while others were doing some shutdown procedure.

In a short moment after, I didn't recall how much seconds or minutes, the quake was started to rock us with a hoopla-like movement that so strong so I felt that I was going to die right there. I made decision, to yell out loud to others to leave/forget the racks & to just run out of the data center room and out of the building a.s.a.p; can't remember much; but I was saying those with grabbing my stuff and two engineers and pull them all to the exit door.

At that time the call coming to my mobile from Tokyo and I was reported the same condition (around 3secs established call then it disconnected), well, later on I knew that when the caller heard my yell to other fellow engineers in the data center by phone, my yelling made the callers running out of the building too, good for them. Meanwhile inside the data center the electricity was starting to go off. This causing us cannot get out of the room for the security authentication card cannot be read due to power off (surprisingly the backup generator was NOT running), Jammed at the door, I took a glimpse and heard sounds from some server's consoles that were still running by the UPS to hold the services for the shutdown procedure of important network devices. And that time was when the mobile service of the earthquake alert sounds started to "buzz" (those alert were a way too slow, meant nothing to us at that moment...), I was starting to wonder what would happen next if we don't hurry to get out that room..That was the point where we started to be panic, and when you panic on such situation, trust me, it will rush into your head and override your common-sense..

Finally using the manual procedure the door was managed to be opened up and we started to just run out of the IDC door to the hall, we ran like hell to the building's exit door, at that time the building was rocked so hard so we kept on running and managed not to fall at the same time. Myself and 5 of us was successfully went out of the exit door, when one engineer went back to the IDC to get a battery, and I went back in to chase him back there and held the IDC door for him (I don't know why I chased him back in.. I guessed maybe it was better to have 2 persons back in a room than one..)

He took what he needed and we managed to fly off the exit door (again!) and at the time the concrete blocks of the building was starting to fall to the parking lot and exit door's nearby.. Those were a big blocks made of concrete, if one of it hits my head I will surely K.O., I guess. So we were very very lucky.

The first sight I saw after stepping out of the building was a rather big concrete blocks falls all over near by the exit door, I saw some cars parked near to building was already wrecked by those fallen concrete bricks in the parking area and right in front of us there was a slim 5th floor building was rocked side-to-side until (more or less) almost 25degree each.. seriously, I was amazed watching that building was not falling at that point.. that building was throwing its antennas and etc parts from its roof like rockets in its some swings. That time, it was the first time that I realized that I carried my bag, I put it somewhere in the parking lot space and a car was moved on its own by the quake and one of the tire was (literally) on my bag..but I didn't even remember when was the time I grabbed it. We were gathered there until 15:40 at that time, before I decided to go to Fukushima station, to find a way back home. The worst part of all is the news, the tsunami news can be viewed by the TV inside of the cars.., we were thinking of the end of the world that time..

One of the members was driving me half way to the station, which was a hard stuff since no traffic light works and many panic drivers thinking only their own priority, We only could drive halfway, I continued to go to biggest train station by foot and reach there to find about almost 200peoples standing in the front of the station building which cannot be entered since the quakes-echo still came like waves with the quite big scale (within 4 to 5 each in Japanese scale called "shindo").

To make our disaster perfect, the snow started to fall. I saw some office workers without the coat out there too. Can't use any mobile devices, no phone calls..I presumed all of the antennas went down. I dropped idea of phone access and tried my luck to check every internet connection I had...or I knew, after some rapid dial here and there found that one service's antenna connection can be used, yet there was a problem in resolving DNS (since the infra based connectivity was ruined), hurried putting some known public worldwide DNS IP in every resolve.conf of my devices, afterward I tried to check my house & office IP but got timed-out. So does the Tokyo IDC. At that time VoIP call came into my mind. After the 10th effort connecting to a server in United States, I finally got it. I was thinking to make connection alive as long as possible. I tried to make phone call to the hard lines, avoiding mobile ones, and successfully made some. Many people in there seeing me with amazed since they can not even call, so I was explaining put loud to use the Wifi to connect a specific service's antennas and VoIP service so they might made a call, and the ones with PCs started to tried it.

Meanwhile, my PC was connected to the internet and I can get the internet radio on, people started to gather. At that time the first nuclear plant explosion news came into our ears I guess. I was a kind of news center, and keep on doing that until my PC went out of battery (no power plug since all buildings blacked out). We were starting to freeze, it was snowing, afterward, police station nearby managed to get taxi drivers to pick them as passengers (5person in each taxi) and evacuate them from station to the nearest residence area (usually there were plenty of taxicabs in front of station but the quake made them runaway), while I was staying there & start arranging my way back home, which will be another long story that took me 3days to get back there..

The moral of this true story is; Number One, in disaster like this: human is first & machine is next! As an one of unix sysadmin we are pledged to make a service lasts as long as possible, maybe longer than human's lifetime, that was why in BSD we said Daemon to services. BUT, in a practical way, no human means no service. We should say over and over that saving a service is important yet is up the some certain condition of a disaster, when the condition goes near to live or death, just drop everything and run off. Damage will occur and so be it, because without your life the damage cannot be fixed. P.S.: If the service is involving lifeline level of a lot of people, maybe it will worth to die for, and that's is a different one. This priority will be hardly remembered during the disaster, so next tip is important..

Number two is, to train yourself to handle this kind of situation! We should pay more attention to the disaster simulation handling. No person who born with the gift for handling these pressure, to train yourself to act automatically in the disaster is a must! Which is not just a merely training but think & imagine it as real as possible. I am telling you, no brain can overcame fears caused by these panic & stress, and the cure of it is your rehearsal effort.

Number three, be resourceful in media connectivity, as much as possible in your daily life (internet, radio, TV, anything!). As engineer you SHOULD at least know that you can be connected to network as many way as possible. If you have a hand-phone make sure it has FM/Radio on it, if you have smart-phone be sure to have WiFi function on it, If you use Wifi, make sure you have some services to connect to & do not depend into one connectivity. If you have latest smart phone, make sure it can do tether on it. This diversification of connectivity know-how will be your weapon to survive in disaster. And don't forget your bag. You have many surviving tools in it, more than you know it, you might need it later.

---
Based on personal experience,
Twitter: @unixfreaxjp
*) all of the picture attached is to be viewed publicly in some media & all rights are copyrighted & reserved.