On a Wednesday evening this month, every WordPress site on one of the servers we host went down for about five minutes. An hour later it happened again. Traffic was normal. Nothing had crashed. The server had plenty of memory and spare processing power. Our uptime alarms told us the sites were down within a minute, but they couldn’t tell us why.
The reason turned out to be one setting and a few slow requests. We fixed it that night, then built a small tool so it can’t sneak up on us again. That tool is now free and open source: terraform-aws-php-fpm-monitoring, on GitHub.
What actually happened
WordPress runs on PHP. On a server that hosts several sites, PHP handles pages through a shared group of workers, and the group has a fixed size. Each visitor’s page takes a worker for a fraction of a second, then hands it back.
WordPress also runs background jobs: scheduled tasks, plugin syncs with outside services, update checks. Those take workers too. Most finish quickly. Some wait on another company’s server to answer, and while they wait they keep the worker.
That server’s pool had five workers. When a handful of background jobs got slow at the same moment, all five were busy. New visitors queued behind them, the queue timed out, and every site on the server stopped answering at once, even the ones that had nothing to do with the slow job.
The fix on the night was straightforward: a bigger pool (the server could easily afford it), a hard cap so no single request can hold a worker for more than a minute, and PHP’s own slow-request log switched on, so the next slow request is written down with the exact code it was stuck in.
Why the alarm couldn’t tell us
An uptime check asks one question: does the page load? When the answer is no, you know something is wrong, but not what. On a shared server, the answer is usually sitting in a log file nobody reads until after the outage.
PHP already knows the two things you need. Its error log records the moment every worker is busy (“server reached pm.max_children”). Its slow log records which site, which script and which plugin each slow request was stuck in. Both stay on the server unless something brings them to you.
What the module does
The module sets up four things in an AWS account, with Terraform:
- Collects the logs. It installs Amazon’s CloudWatch agent on each server through AWS Systems Manager, so there is nothing to log into and nothing to edit by hand. The agent ships PHP’s slow log and error log to CloudWatch.
- Warns when the pool is full. The moment every worker is busy, a message names the server. That usually arrives minutes before the uptime alarm, while there is still time to act.
- Names the slow code. Each new kind of slow request becomes one message: the site, how many times, the plugin or theme it was stuck in, and a link to the full entry. Repeats of the same thing are held back for half an hour, so a bad minute doesn’t flood the channel.
- Sends a daily summary. Every morning a CloudWatch Logs Insights query lists the slowest sites and plugins from the last 24 hours. On a clean day it says nothing.
Messages can go to Slack or Microsoft Teams through Amazon Q Developer, to any SNS topic, or straight to a Slack webhook.
One thing we learned shipping it
About twenty minutes after we turned it on, it reported a full pool on a server we had already fixed. The report was real, but old. The first time the agent sees a log file, it reads it from the beginning, so it sent along everything that had happened earlier that evening. The released version ignores anything more than 15 minutes old, which is the kind of detail you only find by running a tool on a real server.
Using it
If you run WordPress or any other PHP site on your own EC2 servers, turn on PHP-FPM’s slow log in the pool configuration, then add the module:
module "php_fpm_monitoring" {
source = "github.com/45squaredLLC/terraform-aws-php-fpm-monitoring?ref=v1.0.0"
servers = {
web1 = { instance_id = "i-0123456789abcdef0", name = "web1", php_version = "8.4" }
}
instance_role_names = ["web-servers-ssm-role"]
alert_topic_arn = aws_sns_topic.alerts.arn
}
The README covers the PHP settings, every option and the permissions it creates. It is MIT licensed, and issues and pull requests are welcome on GitHub.
Why we give this away
We host and look after WordPress sites for small businesses. The tools we build to do that well are the same ones we’d want any hosting provider to use, so when one is useful beyond our own servers, we publish it. This is the second, after our Amazon SES monitoring module.
If your sites go down and nobody can tell you why, send us a note through the form below.
