土曜日, 2月 25, 2012

#InfoSec - A year memoir of Fukushima 3-11 | A geek's diaster diary

It is almost a year from 3-11 Quake and Tsunami disaster in Fukushima. I was there, I saw and remembered it well. Yesterday I was attending Defcon Japan meeting and discussing this with the guy sat next to me. He urged me to write it, and this is it, I dedicate this writing to all 3-11 quake victims and to fellow IT/security engineers everywhere to let you know the actual feeling of what happened on-the-site.

I was working in the Fukushima xxxx network for trouble shooting some security filtration scheme of our solution, when earthquake started on March 11th last year. When it was started 6(six) of us was inside of the secure data center. I remember that I was analyzing the weird SYN/ACK packets sent and received between proxy layer and core switch within so I was standing in front of one rack watching the monitor console for the captured packets. The server's racks itself contains of 3 blocks of 20U racks of 10feet long. Shortly, when it hit is there, it happened so fast yet every glimpse that I saw was recorded & formatted in my brain in a sequence of slow motion flashback that I often have it as nightmare until now, so I can tell you the exact detail of it. A year had passed after that disaster, there were nights to woke up with the nightmare with seeing these visions and this is the first time ever I finally dare to write it.

When it was started, it was just like the previous smaller quakes, the horizontal movement began slowly followed by the strange deep voice of "Boom, boom, boom..!" coming from the center of the earth, which went faster, then the vertical rocks starting to merge with it. We were stunned for at least at the first 10secs, afterward one of us said "This one is getting harder isn't it?", and that was the word that triggering our first action. About 20Us size server's rack (around 2meter height) was rocked so hard, and every block of rack bumped to the space limit safety bare-bone (actually it was made with slack of 1 meter size) with the loud voice, the balls inside of the rack's conjunctions was popped out like a bullet, and some server's, routers, switch's part was starting to fall (or FLY will be more like it..). Still, at that time we aimed to save the service first, four of the engineers run and hold the rack by their both hands. the rest of us starting the shutdown procedure.

At that time the quake started to rock us with a hoopla-like movement, I made decision, to yell out loud to others to leave/forget the racks & to just run out of the data center room and out of the building a.s.a.p; can't remember much; but I was saying those with grabbing 2 engineers and pull them all to the exit door.

At that time the call coming from Tokyo and I was (likely) reported the same condition (3seconds established call), well, later on I knew that when they heard my yell to other fellow engineers in the data center by phone, my yelling made the callers running out of the building too, good for them. Meanwhile inside the IDC the electricity starting to black out. This causing us cannot get out of the room for the authentication card cannot be read due to power off (surprisingly the backup generator NOT running). Jammed at the door, I took a glimpse to the servers and made sure the UPS running and holds the services for the shutdown procedure of every clustered servers. At that time the mobile service of the earthquake alert sounds started to "buzz" (those alert were a way too slow, meant nothing to us at that moment...), I was starting to wonder what would happen next if we don't get out that room..that was the point where the panic started to rush overriding the common-sense..

Finally using the manual step the door was opened up and we started to just run for the building door, on the way there through the exit door, at that time the building was rocked so hard so we run and managed not to fall at the same time. I saw 5of us went out to the exit door, yet one person went back to the IDC to get the battery so I chased him back there (I don't know why I chased him back in.. I guessed maybe it was better to have 2 persons back in a room than one..)

He took what he needed and we managed to fly off the exit door and at the time the concrete block of the building was starting to fall to the parking lot and exit door's nearby.. We were very lucky I guess. The first sight I saw after stepping out of the building was a rather big concrete block falls all over near by the door, I saw some car parked near to building was already wrecked by those bricks in the parking area and right in front of us there was a slim 5th floor building was rocked side-to-side until more or less almost 25degree each.. seriously, I was amazed watching that building was not falling at that point.. that building was throwing antennas or etc parts on top like rockets in some swings too. That time it was the first time I realized that I carried my bag, a car was moved by the quake and one of the tire was (literally) on my bag..but I didn't even remember when was the time I grabbed it. We were gathered there until 15:40 at that time, before I decided to go to Fukushima station, to find a way back home. The worst part of all is the news, the tsunami news can be viewed by the TV inside of the cars.., we were thinking of the end of the world that time..

One of the members was driving me half way to the station, which was a hard stuff since no traffic light works and many panic drivers thinking only their own priority, We only could drive halfway, I continued to go to biggest train station by foot and reach there to find about almost 200peoples standing in the front of the station building which cannot be entered since the quakes-echo still came like waves with the quite big scale (within 4 to 5 each in Japanese scale called "shindo").

To make our disaster perfect, the snow started to fall. I saw some office workers without the coat out there too. Can't use any mobile devices, no phone calls..I presumed all of the antennas went down. I dropped idea of phone access and tried my luck to check every internet connection I had...or I knew, after some rapid dial here and there found that one service's antenna connection can be used, yet there was a problem in resolving DNS (since the infra based connectivity was ruined), hurried putting some known public worldwide DNS IP in every resolve.conf of my devices, afterward I tried to check my house & office IP but got timed-out. So does the Tokyo IDC. At that time VoIP call came into my mind. After the 10th effort connecting to a server in United States, I finally got it. I was thinking to make connection alive as long as possible. I tried to make phone call to the hard lines, avoiding mobile ones, and successfully made some. Many people in there seeing me with amazed since they can not even call, so I was explaining put loud to use the Wifi to connect a specific service's antennas and VoIP service so they might made a call, and the ones with PCs started to tried it.

Meanwhile, my PC was connected to the internet and I can get the internet radio on, people started to gather. At that time the first nuclear plant explosion news came into our ears I guess. I was a kind of news center, and keep on doing that until my PC went out of battery (no power plug since all buildings blacked out). We were starting to freeze, it was snowing, afterward, police station nearby managed to get taxi drivers to pick them as passengers (5person in each taxi) and evacuate them from station to the nearest residence area (usually there were plenty of taxicabs in front of station but the quake made them runaway), while I was staying there & start arranging my way back home, which will be another long story that took me 3days to get back there..

The moral of this true story is; Number One, in disaster like this: human is first & machine is next! As an one of unix sysadmin we are pledged to make a service lasts as long as possible, maybe longer than human's lifetime, that was why in BSD we said Daemon to services. BUT, in a practical way, no human means no service. We should say over and over that saving a service is important yet is up the some certain condition of a disaster, when the condition goes near to live or death, just drop everything and run off. Damage will occur and so be it, because without your life the damage cannot be fixed. P.S.: If the service is involving lifeline level of a lot of people, maybe it will worth to die for, and that's is a different one. This priority will be hardly remembered during the disaster, so next tip is important..

Number two is, to train yourself to handle this kind of situation! We should pay more attention to the disaster simulation handling. No person who born with the gift for handling these pressure, to train yourself to act automatically in the disaster is a must! Which is not just a merely training but think & imagine it as real as possible. I am telling you, no brain can overcame fears caused by these panic & stress, and the cure of it is your rehearsal effort.

Number three, be resourceful in media connectivity, as much as possible in your daily life (internet, radio, TV, anything!). As engineer you SHOULD at least know that you can be connected to network as many way as possible. If you have a hand-phone make sure it has FM/Radio on it, if you have smart-phone be sure to have WiFi function on it, If you use Wifi, make sure you have some services to connect to & do not depend into one connectivity. If you have latest smart phone, make sure it can do tether on it. This diversification of connectivity know-how will be your weapon to survive in disaster. And don't forget your bag. You have many surviving tools in it, more than you know it, you might need it later.

Based on personal experience,
Twitter: @unixfreaxjp
*) all of the picture attached is to be viewed publicly in some media & all rights are copyrighted & reserved.

0 件のコメント: