I'm having trouble wrapping my head around the pipe function shown in several Node.js examples for the net module.
var net = require('net');
var server = net.createServer(function (socket) {
socket.write('Echo server\r\n');
socket.pipe(socket);
});
Can anyone offer an explanation on how this works and why it's required?
The pipe() function reads data from a readable stream as it becomes available and writes it to a destination writable stream.
The example in the documentation is an echo server, which is a server that sends what it receives. The socket object implements both the readable and writable stream interface, so it is therefore writing any data it receives back to the socket.
This is the equivalent of using the pipe() method using event listeners:
var net = require('net');
net.createServer(function (socket) {
socket.write('Echo server\r\n');
socket.on('data', function(chunk) {
socket.write(chunk);
});
socket.on('end', socket.end);
});
pipe() reads from a readable stream and writes to a writeable stream, much like a Unix pipe. It does all "reasonable" things along the way with errors, end of files, if one side falls behind etc. Your particular example is slightly confusing because the socket is both readable and writeable.
An easier to understand example is in this SO question where you read from an http request and write to an http response.
There are 2 sockets per Server-Client Connection (2 endpoints). Socket binds IP Address:Port Number. The client gets assigned random port numbers, while the server has dedicated port number. This is the basic explanation of how socket works.
Piping is reserved for redirecting a readable stream to a writable stream.
What socket.pipe(socket) does?
It redirects all the data from the readable stream (server) to the writable stream (client). We can tweak this by adding event listeners as #hexacyanide has pointed out.
Consider the following request handler
var server = http.createServer(function(req, res){
console.log('Request for ' + req.url + ' by method ' + req.method);
if(req.method == 'GET'){
var fileurl;
if(req.url == '/')fileurl = '/index.html';
else {
fileurl = req.url;
}
}
var filePath = path.resolve('./public'+fileurl);
var fileExt = path.extname(filePath);
if(fileExt == '.html'){
fs.exists(filePath, function(exists){
if(!exists){
res.writeHead(404, {'Content-Type': 'text/html'});
res.end('<h1>Error 404' + filePath + 'not found </h1>');
//the end() method sends content of the response to the client
//and signals to the server that the response has been sent
//completely
return;
}
res.writeHead(200, {'Content-Type':'text/html'});
fs.createReadStream(filePath).pipe(res);
})
}
}
The fs.createReadStream method reads the file in the given file path (public/index.html) and pipe() writes it to the response (res) for client's view.
Related
There is a simple web server that accepts data. Sample code below.
The idea is to track in real time how much data has entered the server and immediately inform the client about this. If you send a small amount of data, then everything works well, but if you send more than X data in size, then the on.data event on the server is triggered with a huge delay. I can see that data is transfering for 5 seconds already but on.data event is not trigerred.
on.data event seems to be triggered only when data is uploaded completely to the server, so that's why it works fine with small data (~2..20Mb), but with big data (50..200Mb) it doesnt work well.
Or maybe it is due to some kind of buffering..?
Do you have any suggestions why on.data triggered with delay and how to fix it?
const app = express();
const port = 3000;
// PUBLIC API
// upload file
app.post('/upload', function (request, response) {
request.on('data', chunk => {
// message appears with delay
console.log('upload on data', chunk.length);
// send message to the client about chunk.length
});
response.send({
message: `Got a POST request ${request.headers['content-length']}`
});
});
app.listen(port, () => {
console.log(`Example app listening at http://localhost:${port}`);
});
TLDR:
The delay that you are experiencing probably is the Queueing from Resource scheduling from the browser.
The Test
I did some tests with express, and then I found that it uses http to handle requests/response, so I used a raw http server listener to test this scenario, which has the same situation.
Backend code
This code, based on sample of Node transaction samples, will create a http server and give log of time on 3 situations:
When a request was received
When the first data event fires
When the end event fires
const http = require('http');
var firstByte = null;
var server = http.createServer((request, response) => {
const { headers, method, url } = request;
let body = [];
request.on('error', (err) => {
}).on('data', (chunk) => {
if (!firstByte) {
firstByte = Date.now();
console.log('received first byte at: ' + Date.now());
}
}).on('end', () => {
console.log('end receive data at: ' + Date.now());
// body = Buffer.concat(body).toString();
// At this point, we have the headers, method, url and body, and can now
// do whatever we need to in order to respond to this request.
if (url === '/') {
response.statusCode = 200;
response.setHeader('Content-Type', 'text/html');
response.write('<h1>Hello World</h1>');
}
firstByte = null;
response.end();
});
console.log('received a request at: ' + Date.now());
});
server.listen(8083);
Frontend code (snnipet from devtools)
This code will fire a upload to /upload which some array data, I filled the array before with random bytes, but then I removed and see that it did not have any affect on my timing log, so yes.. the upload content for now is just an array of 0's.
console.log('building data');
var view = new Uint32Array(new Array(5 * 1024 * 1024));
console.log('start sending at: ' + Date.now());
fetch("/upload", {
body: view,
method: "post"
}).then(async response => {
const text = await response.text();
console.log('got response: ' + text);
});
Now running the backend code and then running the frontend code I get some log.
Log capture (screenshots)
The Backend log and frontend log:
The time differences between backend and frontend:
Results
looking at the screenshoots and I get two differences between the logs:
The first, and most important, is the difference between frontend fetch start and backend request recevied, I got 1613ms which is "close" (1430ms) to Resource Scheduling in network timing tab, I think there are more things happening between the frontend fetch call and the node backend event, so I can't direct compare the times:
log.backendReceivedRequest - log.frontEndStart
1613
The second is the difference between receving data on backend, which I got
578ms, close to Request sent (585ms) in network timing tab:
log.backendReceivedAllData - log.backendReceivedFirstData
578
I also changed the frontend code to send different sizes of data and the network timing tab still matches the log
The thing that remains unknown for me is... Why does Google Chrome is queueing my fetch since I'm not running any more requests and not using the bandwidth of the server/host? I readed the conditions for Queueing but not found the reason, maybe is allocating the resources on disk, but not sure: https://developer.chrome.com/docs/devtools/network/reference/#timing-explanation
References:
https://nodejs.org/es/docs/guides/anatomy-of-an-http-transaction/
https://developer.chrome.com/docs/devtools/network/reference/#timing-explanation
I found a problem. It was in nginx config. Nginx was setup like a reverse proxy. By default proxy request buffering is enabled, so nginx grabs first whole request body and only then forwards it to nodejs, so that's why I saw delay.
https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_request_buffering
I have this snippet of code:
app.post('/pst', function(req, res) {
var data = req.body.convo;
res.render('waiting.ejs'); //ADDED THIS
myFunc(data).then(result => {
res.render('success.ejs'); //THEN THIS
//---------------------------------
//clever way to send text file to client from the memory of the server
var fileContents = Buffer.from(result, 'ascii');
var readStream = new stream.PassThrough();
readStream.end(fileContents);
res.set('Content-disposition', 'attachment; filename=' + fileName);
res.set('Content-Type', 'text/plain');
readStream.pipe(res);
//--------------------------------------
}).catch( .....
The code i commented as 'clever way to send file from memory of the server' comes from this post:
Node Express.js - Download file from memory - 'filename must be a string'
What this does is is takes a string from the memory and serves it to the client as a .txt file.
This code used to work.
Then i decided to add the res.render('waiting.ejs'); line and i got this error:
Error: Can't set headers after they are sent.
I then experimented with adding another res.render() [in this case res.render('success.ejs');] before and after the code tht sends the .txt file to the client.
The error remained. Also, there is no redirect to success.ejs, in other words the res.render('success.ejs'); never worked, despite whether it is placed before ofr after that piece of code.
app.post('/pst', function(req, res) {
var data = req.body.convo;
myFunc(data).then(result => {
//---------------------------------
//clever way to send text file to client from the memory of the server
var fileContents = Buffer.from(result, 'ascii');
var readStream = new stream.PassThrough();
readStream.end(fileContents);
res.set('Content-disposition', 'attachment; filename=' + fileName);
res.set('Content-Type', 'text/plain');
readStream.pipe(res);
res.redirect(`/success`); //THEN THIS
//--------------------------------------
}).catch( .....
When you add middleware to express (which is built on connect) using the app.use method, you're appending items to Server.prototype.stack in connect.
When the server gets a request, it iterates over the stack, calling the (request, response, next) method.
The problem is, if in one of the middleware items writes to the response body or headers (it looks like it's either/or for some reason), but doesn't call response.end() and you call next() then as the core Server.prototype.handle method completes, it's going to notice that:
there are no more items in the stack, and/or
that response.headerSent is true.
So, it throws an error. But the error it throws is just this basic response (from the connect http.js source code:
res.statusCode = 404;
res.setHeader('Content-Type', 'text/plain');
res.end('Cannot ' + req.method + ' ' + req.url);
The problematic middleware sets the response header without calling response.end() and calls next(), which confuses express's server.
so you set the header through res.render() .Now if you will try to render again it will throw you an error.
app.get('/success',(req,res)=> {
res.render("container/index",{waiting:"waiting",......});
//handle your task then in client side index.ejs with appropriate setTimeout(()=>{},2000) for the waiting div , show waiting div for 2 seconds
});
//then your actual success gets render
You would have to check express.js source code (here):
res.render = function render(view, options, callback) {
var app = this.req.app;
var done = callback;
var opts = options || {};
var req = this.req;
var self = this;
// support callback function as second arg
if (typeof options === 'function') {
done = options;
opts = {};
}
// merge res.locals
opts._locals = self.locals;
// default callback to respond
done = done || function (err, str) {
if (err) return req.next(err);
self.send(str);
};
// render
app.render(view, opts, done);
};
You can see that when You use res.render() method, it will pass the done callback to app.render(...) (source code), it will then pass done to tryInitView etc.
At the end, it will invoke done callback with str in case of success or err in case of failure. It then triggers res.send() inside done callback which simply blocks You from setting headers after that.
res.render() function compiles your template, inserts locals there, and creates html output out of those two things. that's why error comes.
don't use it twice coz it send response.
Every few seconds, my code is writing a png to a http server. I'd like to include (through browser-side javascript or a header or something) some sort of marker that will make the browser automatically refresh every 500ms.
Alternatively, I'd like to just automatically update the page so the client doesn't need to refresh. How would I do either of these?
var arDrone = require('ar-drone');
var http = require('http');
console.log('Connecting png stream ...');
var client = arDrone.createClient();
var pngStream = client.getPngStream();
var lastPng;
pngStream
.on('error', console.log)
.on('data', function(pngBuffer) {
lastPng = pngBuffer;
});
client.config('video:video_channel', 0);
var server = http.createServer(function(req, res) {
if (!lastPng) {
res.writeHead(503);
res.end('Did not receive any png data yet.');
return;
}
res.writeHead(200, {'Content-Type': 'image/png'});
res.end(lastPng);
});
server.listen(8080, function() {
console.log('Serving latest png on port 8080 ...');
});
Look into using websockets to push content to the webpage. Socket.io does a really good job at making websockets easy. With socket.io you can push data to the webpage, and then handle that data in the webpage with javascript.
I'm creating node js application as a http server that communicate to tcp socket server, and the code look's like this:
var http = require('http');
var net = require('net');
var url = require('url') ;
const PORT=8080;
var client = new net.Socket();
client.connect(9000,'xxx.xxx.xxx.xxx', function(){
console.log('Connected');
});
function handleRequest(request, response){
var qo = url.parse(request.url,true).query;
if(typeof qo.q=="undefined"){
response.end("ERROR");
}else{
client.write(decodeURIComponent(qo.q));
client.on('data',function(data){
response.writeHead(200, {'Content-Type': 'text/html','Access-Control-Allow-Headers':'Content-Type,Access-Control-Allow-Headers,Access-Control-Allow-Origin,X-Powered-ByX-Powered-By','Access-Control-Allow-Origin': '*','X-Powered-By':'Poltak ganteng'});
response.end(data);
});
}
}
var server = http.createServer(handleRequest);
server.listen(PORT, function(){
console.log("Server listening on: http://localhost:%s", PORT);
});
i'm afraid that code not working properly to handle multiple request at a time. is there any way to handle multiple request and get correct response to requestor?
You want to use req/res with your socket connection, you want to send uid at each client.write end your other service responde with the same uid for conserve the relation one to one to req/res, you don't have other choice for guaranteed unicities.
The have an expensive resource as TCP client (or clients) that has to be shared among node requests, so socket-pool can help you on that.
I have problems streaming MP3 data via WebSocket with node.js and socket.io. Everything seems to work but decodeAudioData doesn't play fair with me.
This is my toy server:
var app = require('http').createServer(handler)
, io = require('socket.io').listen(app)
, fs = require('fs')
app.listen(8081);
function handler (req, res) {
res.writeHead(200, {
'Content-Type': 'text/html',
});
res.end('Hello, world!');
}
io.configure('development', function() {
io.set('log level', 1);
io.set('transports', [ 'websocket' ]);
});
io.sockets.on('connection', function (socket) {
console.log('connection established');
var readStream = fs.createReadStream("test.mp3",
{'flags': 'r',
'encoding': 'binary',
'mode': 0666,
'bufferSize': 64 * 1024});
readStream.on('data', function(data) {
console.log(typeof data);
console.log('sending chunk of data')
socket.send(data);
});
socket.on('disconnect', function () {
console.log('connection droped');
});
});
console.log('Server running at http://127.0.0.1:8081/');
The client receives the data as type string but I want to feed the data to decodeAudioData and it seems it doesn't like strings. The call to decodeAudioData results in the following error message:
Uncaught Error: SYNTAX_ERR: DOM Exception 12
I think decodeAudioData needs the data stored in an ArrayBuffer. Is there a way to convert the data?
This is the client code:
<script src="http://127.0.0.1:8081/socket.io/socket.io.js"></script>
<script>
var audioBuffer = null;
var context = null;
window.addEventListener('load', init, false);
function init() {
try {
context = new webkitAudioContext();
} catch(e) {
alert('Web Audio API is not supported in this browser');
}
}
function decodeHandler(buffer) {
console.log(data);
}
var socket = io.connect('http://127.0.0.1:8081');
socket.on('message', function (data) {
// HERE IS THE PROBLEM
context.decodeAudioData(data, decodeHandler, function(e) { console.log(e); });
});
</script>
I've found a way to stream MP3 data via Websockets myself.
One problem was the chunk size of the MP3 data. It seems that the Web Audio API needs to be fed with valid MP3 chunks to be able to decode the data. Probably not surprising. In my demo app I provide a set of MP3 chunk files.
Also the quality of the audio is not perfect. I have some subtle glitches. I was able to improve that by sending larger chunks of MP3 data but there are still tiny crackles.
EDIT: I managed to improve the audio quality. It seems the Web Audio method decodeAudioData isn't really designed to decode continuos chunks of MP3 data.
In your case context.decodeAudioData expects an ArrayBuffer of the binary data, I would suggest converting your chunk to a base64 string, then to an ArrayBuffer client-side for the most predictable results. This script should be a good starting point for the client-side decode from base64 of the chunked data.
Adding a line with data = Base64Binary.decodeArrayBuffer(data); right after getting your data (base-64 encoded string) would do the trick...
It seems that socket.io still doesn't support Binary transfer. So websocket.io can be used here.
https://github.com/LearnBoost/socket.io/issues/511#issuecomment-2370129
https://github.com/LearnBoost/socket.io/issues/680#issuecomment-3083490