MySQL on Amazon Linux crashes frequently

[Udar Gromov]

Environment:
– Amazon t2.small Linux instance with 2GB of memory
– Apache 2.4
– MySQL 5.6

Problem:
Amazon Linux instance launched in October 2017.  Starting from December 2017 MySQL would crash and require a restart once every month.  Starting from mid-June 2018 MySQL would start crashing daily, then several times a day.

Solutions:
-01- [COMMON] Create and enable large swap file to allow access to virtual memory – that did not work. MySQL crashed within 6 hours of enabling swap file.

-02- [PROUD] Create a new section <IfModule mpm_prefork_module> in Apache “httpd.conf” file to limit simultaneous web users consuming all the memory.

Proud moment occurs when an obscure insurmountable problem slowly turns into an interesting research project, then into an action plan, and finally into a solution.

Please see, if this will be an interesting topic to discuss.

Linux commands used to diagnose and fix the problem:

Examine mysqld.log to see why MySQL is crashing.  Is it a memory problem?

tail /var/log/mysqld.log	Output last 10 lines of the log (works only if service is running)
ls /etc/init.d/	List all services
service mysqld restart	Restart MySQL
free -m	Display total/used/free memory info

 

RE: 2018-06-23 SA

[Udar Gromov]

Continue – 2 years later. Sorry for the delay.

Yes. this is a very interesting topic that might even deserves its own post.

Among other things you need to restrict number of simultaneous sessions for MySQL. This can be done by adding lines to your httpd.conf. That way, if you are under denial-of-service attack, your MySQL server will sit tight, limit new sessions and do not crash.

<IfModule mpm_prefork_module>
    ServerLimit          xxx
    StartServers         xxx
    MinSpareServers      xxx
    MaxSpareServers      xxx
    MaxClients           xxx
    MaxRequestsPerChild  xxx
</IfModule>

At the same time, you need to enable SWAP file to extend your virtual memory. You can use recently updated documentation from Amazon:

SWAP to a separate partition:
https://aws.amazon.com/premiumsupport/knowledge-center/ec2-memory-partition-hard-drive/

SWAP to a root file:
https://aws.amazon.com/premiumsupport/knowledge-center/ec2-memory-swap-file/

A word of advise

BLOCK size
When creating a SWAP file, create a lot of blocks of small size (512, 1024 or 2048). That way a lot of attackers and legitimate web sessions can get their small chunk of memory.

SWAP size
Also, set the SWAP size conservatively. If you only doing SWAP to accommodate spammers, let them sit and wait. Another matter, when your legitimate customers can’t get through. In that case, you need a whole different set of measures to prevent too many connections from a single IP address.

[Author]

Posts