- This topic has 1 reply, 1 voice, and was last updated 3 years, 7 months ago by
.
-
Posts
-
Environment:
– Amazon t2.small Linux instance with 2GB of memory
– Apache 2.4
– MySQL 5.6Problem:
Amazon Linux instance launched in October 2017. Starting from December 2017 MySQL would crash and require a restart once every month. Starting from mid-June 2018 MySQL would start crashing daily, then several times a day.Solutions:
-01- [COMMON] Create and enable large swap file to allow access to virtual memory – that did not work. MySQL crashed within 6 hours of enabling swap file.-02- [PROUD] Create a new section <IfModule mpm_prefork_module> in Apache “httpd.conf” file to limit simultaneous web users consuming all the memory.
Proud moment occurs when an obscure insurmountable problem slowly turns into an interesting research project, then into an action plan, and finally into a solution.
Please see, if this will be an interesting topic to discuss.
Linux commands used to diagnose and fix the problem:
Examine mysqld.log to see why MySQL is crashing. Is it a memory problem?
tail /var/log/mysqld.log
Output last 10 lines of the log (works only if service is running) ls /etc/init.d/
List all services service mysqld restart
Restart MySQL free -m
Display total/used/free memory info RE: 2018-06-23 SA
Continue – 2 years later. Sorry for the delay.
Yes. this is a very interesting topic that might even deserves its own post.
Among other things you need to restrict number of simultaneous sessions for MySQL. This can be done by adding lines to your httpd.conf. That way, if you are under denial-of-service attack, your MySQL server will sit tight, limit new sessions and do not crash.
<IfModule mpm_prefork_module> ServerLimit xxx StartServers xxx MinSpareServers xxx MaxSpareServers xxx MaxClients xxx MaxRequestsPerChild xxx </IfModule>At the same time, you need to enable SWAP file to extend your virtual memory. You can use recently updated documentation from Amazon:
SWAP to a separate partition:
https://aws.amazon.com/premiumsupport/knowledge-center/ec2-memory-partition-hard-drive/SWAP to a root file:
https://aws.amazon.com/premiumsupport/knowledge-center/ec2-memory-swap-file/A word of advise
BLOCK size
When creating a SWAP file, create a lot of blocks of small size (512, 1024 or 2048). That way a lot of attackers and legitimate web sessions can get their small chunk of memory.SWAP size
Also, set the SWAP size conservatively. If you only doing SWAP to accommodate spammers, let them sit and wait. Another matter, when your legitimate customers can’t get through. In that case, you need a whole different set of measures to prevent too many connections from a single IP address.
- You must be logged in to reply to this topic.