Nginx is an amazing web server – after years of working with Apache, nginx just feels more fluid and natural. It’s also fast as hell. But some of the things that make it fast can cause hidden issues if you’re not paying attention.
We have a setup that involves a Docker container running nginx that uses the proxy_pass directive to forward requests to several of our running services:
location /webservice {
proxy_pass http://webservice.mydomain.com/webservice;
}
Works like a charm. But when we recently moved our staging environment to AWS, we noticed that almost every night, our web server would just stop responding to those proxy requests. Relaunching the web server container would fix the issue, so it wasn’t a code thing.
Doing my random googling, I eventually ran into this article that described what we were seeing.
It turns out that AWS changes the IP addresses of its load balancers on a pretty regular basis, but nginx doesn’t (by default) honor the DNS TTL of the upstream server. In order to make it do so, you have to do this:
location /webservice {
set $webservice_upstream "webservice.mydomain.com";
resolver 8.8.8.8 valid=30s;
proxy_pass http://$webservice_upstream/webservice;
}
By setting the proxy pass URL to have a variable in it, nginx decides that it needs to re-lookup the DNS entry. It uses the resolver command to get DNS servers and (optionally) a TTL override. After we deployed this, things worked like a charm!
sdwr98 