Three months IT recovery project – from crash to glory

[Bakal Shick]

Minutes of three-months recovery project

Recovery: how technology downfall brings up new ideas and possibilities

Many factors contributed to this horrifying chain of events when normal IT people thought that everything lost and then everything gained in September, and then go through the same experience once again in October.  How could that happened to reasonable people? And yet, it could. And it did.

 

First crash

Aug 22, Tue – Synology scheduler ran its last backup job and stopped working after that date
Aug 29, Tue – users begin to report seeing “Out of space” message on Synology despite hundreds gigabytes of free space on all RAID disks.
Aug 31, Thu near midnight – people can’t access Synology – “Not enough disk space?!”
Sep 01, Fri – Synology failed due to overwhelmed very small internal disk. No one can connect.
Evaluating the loss: all contacts, entire staff calendar, all web sites are off.
Sep 01-04 – Labor weekend hiking trip to NH with extreme concern how to get our staff’s data back
Sep 05, Tue – Recovery day
07:49 connected to Synology via puTTY console!
08:28 directory /etc contains something hidden, something big
12:18 found 2 gigantic files in /etc/PM2 directory
12:42 IT guys with trembling fingers, issue a command to remove offending files
13:19 Everything works! Unit that carried all data and all sites recovered
Sep 09, Sat
Total WP site upgrade announced by Administration
Obtain SSL for DWC.com and all other corporate sites
Copy DWC directory to initiate WP upgrade
Sep 10, Sun – connect to Synology WebDAV server – remote network drive
Sep 12, Tue – start comparing WP themes to buy: MH Magazine
Sep 16, Sat – found WP theme Hueman and Hueman Pro
Sep 17, Sun – almost bought Hueman Pro for $39, but sales link disappeared
Sep 22, Fri – revive Photoshop CS5 – it woks in Windows 10!
Sep 23, Sat – start file DWC Requirements.xlsx to convert needs into features
MH Magazine ($49) selected as a new theme for future upgraded WordPress
Start learning features and tricks to customize new professional WP theme
Sep 26, Tue 15:53 – find a place where to insert code to display unique SWC header image

Second crash

Sep 30, Sat – 05:29 Synology continuously restarts and eventually dies
Evaluating the damage: this time at least we have the most important passwords written down and we have a week old backup datof all   D/B dated 09-23. Last data and code backup discovered date August 2016.
Jump start Amazon Linux instance to host all site externally
13:27 – launch a new Amazon Linux instance
21:20 – Synology will not turn on at all – another loss?
Oct 01, Sun – All we need is a new power supply. That is easy!
Order replacement power supply for Synology all the way from Washington state ($110)
15:24:39 Transfer Editor Calendar site from Synology to Linux server with new PHP7
Oct 02, Mon – Synology: your unit is still on warranty! Amazing!
Oct 05, Thu – magically Synology is back on without human intervention.
Time to run all required backups and contingencies
Oct 06, Fr 23:37 Editor Contacts site on Amazon server works on internal Node.js port
Oct 08 , Sun 13:44 – SSL certificate work with Node.js
Stuff Contacts site works on port 80!
18:16:11 – implement reverse proxy and SSL on Amazon Apache server – success!
Oct 12, Thu – Should we let one of the editor go to Moscow for an anniversary?
Synology dies second time, and take one of the RAID disks with it.

 

Road to recovery

Oct 13, Fri – replacement unit arrived from Redmond, WA
We are back online this time with 4TB WD Red drive ($150)
All this time – implement the Requirements without core modifications
Oct 16, Mon – MH Magazine child theme works without loosing all formatting
Now we can modify child theme without touching any core or parent files
Oct 19-23 – trip to Moscow
Nov 03-14 – resolve broken Scheduler with Synology support – everything works!
Now 15, Wed 21:47 fix a show-stopping issue with Post name permalinks not working on Amazon Linux
Four problems solved at once: old and new links are working, foreign links working, SEO is happy
Now 16, Thu 05:34 – issue an editorial “Who is my President?”
Nov 17, Fri – copy DCW.com to on Amazon as prod.WDC.com as a future site
Nov 18-25 – trip to Dominican Republic
Now 23, Thu – seriously consider Mergebot service ($26/mo) to sync DEV and PROD sites
Nov 29, Wed – upgrade office VW cell plan from 2 to 5GB and save money
Dec 02, Sat – Migrate and upgrade WordPress site to Amazon server.
Dec 04-05 – write this log of recovery and revival

[Author]

Posts