Home Forums Inside Inner Circle Three months IT recovery project – from crash to glory

Viewing 1 post (of 1 total)
  • Author
  • #5378
    Bakal Shick

    Minutes of three-months recovery project

    Recovery: how technology downfall brings up new ideas and possibilities

    Many factors contributed to this horrifying chain of events when normal IT people thought that everything lost and then everything gained in September, and then go through the same experience once again in October.  How could that happened to reasonable people? And yet, it could. And it did.


    Mysterious Synology

    First crash

    Aug 22, Tue – Synology scheduler ran its last backup job and stopped working after that date
    Aug 29, Tue – users begin to report seeing “Out of space” message on Synology despite hundreds gigabytes of free space on all RAID disks.
    Aug 31, Thu near midnight – people can’t access Synology – “Not enough disk space?!”
    Sep 01, Fri – Synology failed due to overwhelmed very small internal disk. No one can connect.
    Evaluating the loss: all contacts, entire staff calendar, all web sites are off.
    Sep 01-04 – Labor weekend hiking trip to NH with extreme concern how to get our staff’s data back
    Sep 05, Tue – Recovery day
    07:49 connected to Synology via puTTY console!
    08:28 directory /etc contains something hidden, something big
    12:18 found 2 gigantic files in /etc/PM2 directory
    12:42 IT guys with trembling fingers, issue a command to remove offending files
    13:19 Everything works! Unit that carried all data and all sites recovered
    Sep 09, Sat
    Total WP site upgrade announced by Administration
    Obtain SSL for DWC.com and all other corporate sites
    Copy DWC directory to initiate WP upgrade
    Sep 10, Sun – connect to Synology WebDAV server – remote network drive
    Sep 12, Tue – start comparing WP themes to buy: MH Magazine
    Sep 16, Sat – found WP theme Hueman and Hueman Pro
    Sep 17, Sun – almost bought Hueman Pro for $39, but sales link disappeared
    Sep 22, Fri – revive Photoshop CS5 – it woks in Windows 10!
    Sep 23, Sat – start file DWC Requirements.xlsx to convert needs into features
    MH Magazine ($49) selected as a new theme for future upgraded WordPress
    Start learning features and tricks to customize new professional WP theme
    Sep 26, Tue 15:53 – find a place where to insert code to display unique SWC header image

    Second crash

    Sep 30, Sat – 05:29 Synology continuously restarts and eventually dies
    Evaluating the damage: this time at least we have the most important passwords written down and we have a week old backup datof all   D/B dated 09-23. Last data and code backup discovered date August 2016.
    Jump start Amazon Linux instance to host all site externally
    13:27 – launch a new Amazon Linux instance
    21:20 – Synology will not turn on at all – another loss?
    Oct 01, Sun – All we need is a new power supply. That is easy!
    Order replacement power supply for Synology all the way from Washington state ($110)
    15:24:39 Transfer Editor Calendar site from Synology to Linux server with new PHP7
    Oct 02, Mon – Synology: your unit is still on warranty! Amazing!
    Oct 05, Thu – magically Synology is back on without human intervention.
    Time to run all required backups and contingencies
    Oct 06, Fr 23:37 Editor Contacts site on Amazon server works on internal Node.js port
    Oct 08 , Sun 13:44 – SSL certificate work with Node.js
    Stuff Contacts site works on port 80!
    18:16:11 – implement reverse proxy and SSL on Amazon Apache server – success!
    Oct 12, Thu – Should we let one of the editor go to Moscow for an anniversary?
    Synology dies second time, and take one of the RAID disks with it.


    Road to recovery

    Oct 13, Fri – replacement unit arrived from Redmond, WA
    We are back online this time with 4TB WD Red drive ($150)
    All this time – implement the Requirements without core modifications
    Oct 16, Mon – MH Magazine child theme works without loosing all formatting
    Now we can modify child theme without touching any core or parent files
    Oct 19-23 – trip to Moscow
    Nov 03-14 – resolve broken Scheduler with Synology support – everything works!
    Now 15, Wed 21:47 fix a show-stopping issue with Post name permalinks not working on Amazon Linux
    Four problems solved at once: old and new links are working, foreign links working, SEO is happy
    Now 16, Thu 05:34 – issue an editorial “Who is my President?”
    Nov 17, Fri – copy DCW.com to on Amazon as prod.WDC.com as a future site
    Nov 18-25 – trip to Dominican Republic
    Now 23, Thu – seriously consider Mergebot service ($26/mo) to sync DEV and PROD sites
    Nov 29, Wed – upgrade office VW cell plan from 2 to 5GB and save money
    Dec 02, Sat – Migrate and upgrade WordPress site to Amazon server.
    Dec 04-05 – write this log of recovery and revival

Viewing 1 post (of 1 total)
  • You must be logged in to reply to this topic.