We Keep Coding

Cannot execute HTTP GET call twice on nodejs/express backend

Using the packages express and ````ftp``` I try to simply get files from an ftp and return
them by HTTP GET to the client requesting.
The first request goes through fine, but when I try calling it again I run into Exception:
Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
I've tried to use the solutions from Error: Can't set headers after they are sent to the client like having a return when sending, not setting , unfortunately none of the worked for me.
This is ALL the code:
const express = require('express');
const ftp = require('ftp');
const app = express();
const port = 3000;
const c = new ftp();
app.get('/files', (req, res) => {
c.on('ready', () => {
c.list((err, list) => {
c.end();
return res.setHeader("Content-Type", "application/json").status(200).send({data: list});
});
});
c.connect({
host: 'xxx',
user: 'xxx',
password: 'xxx',
});
});
app.listen(port, () => {
console.log(`Example app listening on port ${port}`);
});
I think it might be something with the c.list() callback, however I cannot for the love of god find what is wrong with it, as the res.send() does not get called twice anytime.

The problem is you have just one ftp object, and every request subscribes to (and never unsubscribes from) the ready event, and the ready event fires every time you call connect() , which you do for every request. So when the second request calls connect(), the event fires for both the first and second request. This leads to setHeader() being called a second time for the first request, hence the error.
Using once() instead of on() so that the event handler it only called once should resolve the issue, though there are probably better ways to write this code (use a promise API or promisify this one, only initialize a connection to the FTP server once instead of for every request).

Why is request.on data firing with a delay on NodeJS?

There is a simple web server that accepts data. Sample code below.
The idea is to track in real time how much data has entered the server and immediately inform the client about this. If you send a small amount of data, then everything works well, but if you send more than X data in size, then the on.data event on the server is triggered with a huge delay. I can see that data is transfering for 5 seconds already but on.data event is not trigerred.
on.data event seems to be triggered only when data is uploaded completely to the server, so that's why it works fine with small data (~2..20Mb), but with big data (50..200Mb) it doesnt work well.
Or maybe it is due to some kind of buffering..?
Do you have any suggestions why on.data triggered with delay and how to fix it?
const app = express();
const port = 3000;
// PUBLIC API
// upload file
app.post('/upload', function (request, response) {
request.on('data', chunk => {
// message appears with delay
console.log('upload on data', chunk.length);
// send message to the client about chunk.length
});
response.send({
message: `Got a POST request ${request.headers['content-length']}`
});
});
app.listen(port, () => {
console.log(`Example app listening at http://localhost:${port}`);
});

TLDR:
The delay that you are experiencing probably is the Queueing from Resource scheduling from the browser.
The Test
I did some tests with express, and then I found that it uses http to handle requests/response, so I used a raw http server listener to test this scenario, which has the same situation.
Backend code
This code, based on sample of Node transaction samples, will create a http server and give log of time on 3 situations:
When a request was received
When the first data event fires
When the end event fires
const http = require('http');
var firstByte = null;
var server = http.createServer((request, response) => {
const { headers, method, url } = request;
let body = [];
request.on('error', (err) => {
}).on('data', (chunk) => {
if (!firstByte) {
firstByte = Date.now();
console.log('received first byte at: ' + Date.now());
}
}).on('end', () => {
console.log('end receive data at: ' + Date.now());
// body = Buffer.concat(body).toString();
// At this point, we have the headers, method, url and body, and can now
// do whatever we need to in order to respond to this request.
if (url === '/') {
response.statusCode = 200;
response.setHeader('Content-Type', 'text/html');
response.write('<h1>Hello World</h1>');
}
firstByte = null;
response.end();
});
console.log('received a request at: ' + Date.now());
});
server.listen(8083);
Frontend code (snnipet from devtools)
This code will fire a upload to /upload which some array data, I filled the array before with random bytes, but then I removed and see that it did not have any affect on my timing log, so yes.. the upload content for now is just an array of 0's.
console.log('building data');
var view = new Uint32Array(new Array(5 * 1024 * 1024));
console.log('start sending at: ' + Date.now());
fetch("/upload", {
body: view,
method: "post"
}).then(async response => {
const text = await response.text();
console.log('got response: ' + text);
});
Now running the backend code and then running the frontend code I get some log.
Log capture (screenshots)
The Backend log and frontend log:
The time differences between backend and frontend:
Results
looking at the screenshoots and I get two differences between the logs:
The first, and most important, is the difference between frontend fetch start and backend request recevied, I got 1613ms which is "close" (1430ms) to Resource Scheduling in network timing tab, I think there are more things happening between the frontend fetch call and the node backend event, so I can't direct compare the times:
log.backendReceivedRequest - log.frontEndStart
1613
The second is the difference between receving data on backend, which I got
578ms, close to Request sent (585ms) in network timing tab:
log.backendReceivedAllData - log.backendReceivedFirstData
578
I also changed the frontend code to send different sizes of data and the network timing tab still matches the log
The thing that remains unknown for me is... Why does Google Chrome is queueing my fetch since I'm not running any more requests and not using the bandwidth of the server/host? I readed the conditions for Queueing but not found the reason, maybe is allocating the resources on disk, but not sure: https://developer.chrome.com/docs/devtools/network/reference/#timing-explanation
References:
https://nodejs.org/es/docs/guides/anatomy-of-an-http-transaction/
https://developer.chrome.com/docs/devtools/network/reference/#timing-explanation

I found a problem. It was in nginx config. Nginx was setup like a reverse proxy. By default proxy request buffering is enabled, so nginx grabs first whole request body and only then forwards it to nodejs, so that's why I saw delay.
https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_request_buffering

NodeJS reverse SSH tunnel: unable to bind to serveo.net:80

Context
Not long ago, I discovered a great service called Serveo. It allows me to expose my local apps to the Internet using reverse SSH tunneling.
e.g. Connections to https://abc.serveo.net get forwarded to http://localhost:3000 on my machine.
To do this, they require no client installation, and I can just type this in the command line:
ssh -R 80:localhost:3000 serveo.net
where 80 is the remote port on serveo.net to which I want to bind, and localhost:3000 is the local address for my app.
If I just type 80 on the left-hand side, Serveo will answer Forwarding HTTP traffic from https://xxxx.serveo.net where xxxx is an available subdomain, with https support.
However, if I type another port, like 59000, the app will be available through serveo.net:59000, but without SSL.
Problem
Now, I would like to do this with NodeJS, to automate things in a tool I'm building for my coworkers and my company's partners, so that they don't need to worry about it, nor to have an SSH client on their machine. I'm using the SSH2 Node module.
Here is an example of working code, using the custom port configuration (here, 59000), with an app listening on http://localhost:3000:
/**
* Want to try it out?
* Go to https://github.com/blex41/demo-ssh2-tunnel
*/
const Client = require("ssh2").Client; // To communicate with Serveo
const Socket = require("net").Socket; // To accept forwarded connections (native module)
// Create an SSH client
const conn = new Client();
// Config, just like the second example in my question
const config = {
remoteHost: "",
remotePort: 59000,
localHost: "localhost",
localPort: 3000
};
conn
.on("ready", () => {
// When the connection is ready
console.log("Connection ready");
// Start an interactive shell session
conn.shell((err, stream) => {
if (err) throw err;
// And display the shell output (so I can see how Serveo responds)
stream.on("data", data => {
console.log("SHELL OUTPUT: " + data);
});
});
// Request port forwarding from the remote server
conn.forwardIn(config.remoteHost, config.remotePort, (err, port) => {
if (err) throw err;
conn.emit("forward-in", port);
});
})
// ===== Note: this part is irrelevant to my problem, but here for the demo to work
.on("tcp connection", (info, accept, reject) => {
console.log("Incoming TCP connection", JSON.stringify(info));
let remote;
const srcSocket = new Socket();
srcSocket
.on("error", err => {
if (remote === undefined) reject();
else remote.end();
})
.connect(config.localPort, config.localPort, () => {
remote = accept()
.on("close", () => {
console.log("TCP :: CLOSED");
})
.on("data", data => {
console.log(
"TCP :: DATA: " +
data
.toString()
.split(/\n/g)
.slice(0, 2)
.join("\n")
);
});
console.log("Accept remote connection");
srcSocket.pipe(remote).pipe(srcSocket);
});
})
// ===== End Note
// Connect to Serveo
.connect({
host: "serveo.net",
username: "johndoe",
tryKeyboard: true
});
// Just for the demo, create a server listening on port 3000
// Accessible both on:
// http://localhost:3000
// https://serveo.net:59000
const http = require("http"); // native module
http
.createServer((req, res) => {
res.writeHead(200, {
"Content-Type": "text/plain"
});
res.write("Hello world!");
res.end();
})
.listen(config.localPort);
This works fine, I can access my app on http://serveo.net:59000. But it does not support HTTPS, which is one of my requirements. If I want HTTPS, I need to set the port to 80, and leave the remote host blank just like the plain SSH command given above, so that Servo assigns me an available subdomain:
// equivalent to `ssh -R 80:localhost:3000 serveo.net`
const config = {
remoteHost: "",
remotePort: 80,
localHost: "localhost",
localPort: 3000
};
However, this is throwing an error:
Error: Unable to bind to :80
at C:\workspace\demo-ssh2-tunnel\node_modules\ssh2\lib\client.js:939:21
at SSH2Stream.<anonymous> (C:\workspace\demo-ssh2-tunnel\node_modules\ssh2\lib\client.js:628:24)
at SSH2Stream.emit (events.js:182:13)
at parsePacket (C:\workspace\demo-ssh2-tunnel\node_modules\ssh2-streams\lib\ssh.js:3851:10)
at SSH2Stream._transform (C:\workspace\demo-ssh2-tunnel\node_modules\ssh2-streams\lib\ssh.js:693:13)
at SSH2Stream.Transform._read (_stream_transform.js:190:10)
at SSH2Stream._read (C:\workspace\demo-ssh2-tunnel\node_modules\ssh2-streams\lib\ssh.js:252:15)
at SSH2Stream.Transform._write (_stream_transform.js:178:12)
at doWrite (_stream_writable.js:410:12)
at writeOrBuffer (_stream_writable.js:394:5)
I've tried many things without any success. If anyone has an idea about what might be wrong in my example, I'll be really grateful. Thanks!

OpenSSH defaults to "localhost" for the remote host when it's not specified. You can also verify this by checking the debug output from the OpenSSH client by adding -vvv to the command line. You should see a line like:
debug1: Remote connections from LOCALHOST:80 forwarded to local address localhost:3000
If you mimic this by setting config.remoteHost = 'localhost' in your JS code you should get the same result as the OpenSSH client.

How to close an unbounded and piped stream request in node?

My node/express application has an endpoint that's proxying a stream of data from an internal service, which is using server-sent events. This means the internal service will continue to stream data in eternity until the connection closes.
It works well, but when the browser closes the connection to my node app, the piped connection to the internal service stays open, causing the internal service to have a lot of open/unused connections.
So I'm trying to force close the piped connection when the node connection closes, but can't seem to figure out how to do it.
Code looks something like this. Piping using the request/request library.
import request from 'request';
app.get('/stream', (req, res) => {
const stream = request.get({
url: 'https://internalservice.acme.com/stream'
})
stream.on('error', console.log);
stream.pipe(res);
// When browser closes...
req.on('close', () => {
// ...close connection to internal service
stream.destroy() // <-- doesn't work
});
});

When you're making a request in Node, there is the abort() method. It will close your request stream.
req.on('close', () => {
// ...close connection to internal service
stream.abort()
});

Access grpc stream variable for long-running process in Node

I am using Node.js to connect to a server using gRPC that performs a long running task.
The server sends a unidirectional stream to the client (the Node.js app) while the job is in progress. I need to implement a Stop button and am told that closing the gRPC stream will stop the job in progress.
This is currently my code:
let express = require('express'),
router = express.Router(),
grpc = require('grpc'),
srv = grpc.load(__dirname + '/job_handler.proto').ns;
let startJob = (jobID, parameters) => srv.createJob(jobID, parameters);
router.post('/jobs', (req, res) => {
let lengthyOperation = startJob(jobID, parameters);
lengthyOperation.on('data', (data) => {
console.log(`Data from lengthy operation: ${data}`);
});
lengthyOperation.on('end', () =>
console.log('Lengthy operation completed');
});
res.setHeader('Location', `/jobs/${jobID}`);
res.status(202).send();
});
As you can see, I send an HTTP 202 response to the client upon creating the job and it continues asynchronously in the background.
Questions:
How do I close the stream?
How do I access the lengthyOperation variable to do so?

The lengthyOperation object has a cancel method that cancels the call. So, when you want to stop the stream, just call lengthyOperation.cancel().
Note that when you do this, it will cause the call to end with an error. I would recommend adding a lengthyOperation.on('error', ...) handler to handle that error.