Nginx upstream prematurely closed connection while reading response header from upstream, for large requests

Question 1

I am using nginx and node server to serve update requests. I get a gateway timeout when I request an update on large data. I saw this error from the nginx error logs :

2016/04/07 00:46:04 [error] 28599#0: *1 upstream prematurely closed connection while reading response header from upstream, client: 10.0.2.77, server: gis.oneconcern.com, request: "GET /update_mbtiles/atlas19891018000415 HTTP/1.1", upstream: "http://127.0.0.1:7777/update_mbtiles/atlas19891018000415", host: "gis.oneconcern.com"

I googled for the error and tried everything I could, but I still get the error.

My nginx conf has these proxy settings:

    ##
    # Proxy settings
    ##

    proxy_connect_timeout 1000;
    proxy_send_timeout 1000;
    proxy_read_timeout 1000;
    send_timeout 1000;

This is how my server is configured

server {
listen 80;

server_name gis.oneconcern.com;
access_log /home/ubuntu/Tilelive-Server/logs/nginx_access.log;
error_log /home/ubuntu/Tilelive-Server/logs/nginx_error.log;

large_client_header_buffers 8 32k;
location / {
    proxy_pass http://127.0.0.1:7777;
    proxy_redirect off;

    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
    proxy_set_header Host $http_host;
    proxy_cache_bypass $http_upgrade;
}

location /faults {
    proxy_pass http://127.0.0.1:8888;
    proxy_http_version 1.1;
    proxy_buffers 8 64k;
    proxy_buffer_size 128k;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
    proxy_set_header Host $host;
    proxy_cache_bypass $http_upgrade;
}

}

I am using a nodejs backend to serve the requests on an aws server. The gateway error shows up only when the update takes a long time (about 3-4 minutes). I do not get any error for smaller updates. Any help will be highly appreciated.

Node js code :

app.get("/update_mbtiles/:earthquake", function(req, res){
var earthquake = req.params.earthquake
var command = spawn(__dirname + '/update_mbtiles.sh', [ earthquake, pg_details ]);
//var output  = [];

command.stdout.on('data', function(chunk) {
//    logger.info(chunk.toString());
//     output.push(chunk.toString());
});

command.stderr.on('data', function(chunk) {
  //  logger.error(chunk.toString());
 //   output.push(chunk.toString());
});

command.on('close', function(code) {
    if (code === 0) {
        logger.info("updating mbtiles successful for " + earthquake);
        tilelive_reload_and_switch_source(earthquake);
        res.send("Completed updating!");
    }
    else {
        logger.error("Error occured while updating " + earthquake);
        res.status(500);
        res.send("Error occured while updating " + earthquake);
    }
});
});

function tilelive_reload_and_switch_source(earthquake_unique_id) {
tilelive.load('mbtiles:///'+__dirname+'/mbtiles/tipp_out_'+ earthquake_unique_id + '.mbtiles', function(err, source) {
    if (err) {
        logger.error(err.message);
        throw err;
    }
    sources.set(earthquake_unique_id, source); 
    logger.info('Updated source! New tiles!');
});
}

Thank you.

Question 2

I think that error from Nginx is indicating that the connection was closed by your nodejs server (i.e., "upstream"). How is nodejs configured?

Question 3

Nginx works as a front end server, which in this case proxies the requests to a node.js server. Therefore you need to setup an nginx config file for node.

This is what I have done in my Ubuntu box:

Create the file yourdomain.com at /etc/nginx/sites-available/:

vim /etc/nginx/sites-available/yourdomain.com

In it you should have something like:

# the IP(s) on which your node server is running. I chose port 3000.
upstream app_yourdomain {
    server 127.0.0.1:3000;
    keepalive 8;
}

# the nginx server instance
server {
    listen 80;
    listen [::]:80;
    server_name yourdomain.com www.yourdomain.com;
    access_log /var/log/nginx/yourdomain.com.log;

    # pass the request to the node.js server with the correct headers
    # and much more can be added, see nginx config options
    location / {
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header Host $http_host;
      proxy_set_header X-NginX-Proxy true;

      proxy_pass http://app_yourdomain/;
      proxy_redirect off;
    }
 }

If you want nginx (>= 1.3.13) to handle websocket requests as well, add the following lines in the location / section:

proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";

Once you have this setup you must enable the site defined in the config file above:

cd /etc/nginx/sites-enabled/ 
ln -s /etc/nginx/sites-available/yourdomain.com yourdomain.com

Create your node server app at /var/www/yourdomain/app.js and run it at localhost:3000

var http = require('http');

http.createServer(function (req, res) {
    res.writeHead(200, {'Content-Type': 'text/plain'});
    res.end('Hello Worldn');
}).listen(3000, "127.0.0.1");
console.log('Server running at http://127.0.0.1:3000/');

Test for syntax mistakes:

nginx -t

Restart nginx:

sudo /etc/init.d/nginx restart

Lastly start the node server:

cd /var/www/yourdomain/ && node app.js

Now you should see "Hello World" at yourdomain.com

One last note with regards to starting the node server: you should use some kind of monitoring system for the node daemon. There is an awesome tutorial on node with upstart and monit.

Question 4

I found the issue.

My [runserver] socket (app.sock) should be pointed under upstream django and my [wsserver] socket (django.sock) should be pointed under location /ws/ like so:

upstream django {
    server unix:/opt/django/app.sock;
}

server {
    listen 80 default_server;
    charset utf-8;
    client_max_body_size 20M;
    sendfile on;
    keepalive_timeout 0;
    large_client_header_buffers 8 32k;

location /media  {
    alias /opt/django/app/media/media;  
}

location /static {
    alias /opt/django/app/static;
}

location / {
    include /opt/django/uwsgi_params; 
}

location /ws/ {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_pass http://unix:/opt/django/django.sock;
        proxy_buffers 8 32k;
        proxy_buffer_size 64k;
    }
}

Question 5

It was related to the version of PHP. I have used latest version of nginx and slightly old version of PHP. The issue has been fixed by updating PHP to latest version.

Question 6

I had good results using node-http-proxy by nodejitsu. As stated in their readme, they seem to support WebSockets.

Example for WebSockets (taken from their GitHub readme):

var http = require('http'),
    httpProxy = require('http-proxy');

//
// Create an instance of node-http-proxy
//
var proxy = new httpProxy.HttpProxy();

var server = http.createServer(function (req, res) {
  //
  // Proxy normal HTTP requests
  //
  proxy.proxyRequest(req, res, {
    host: 'localhost',
    port: 8000
  })
});

server.on('upgrade', function(req, socket, head) {
  //
  // Proxy websocket requests too
  //
  proxy.proxyWebSocketRequest(req, socket, head, {
    host: 'localhost',
    port: 8000
  });
});

It's production usage should be no problem since it is used for nodejitsu.com. To run the proxy app as a daemon, consider using forever.

Nginx upstream prematurely closed connection while reading response header from upstream, for large requests

Answers

node.js,nginx,webserver