We Keep Coding

Download functionality using streams for large files in NodeJS

I am trying to implement a download functionality using streams in NodeJS.
In the code I am trying to simulate an endpoint that sends data in chunks, something similar to paginated data, for example chunks in size of 5000. Or to make it further clear, we can send top and skip parameters to the endpoint to get a particular chunk of data. If no parameters are provided, it send the first 5000 entries.
There are 2 cases that I am trying to take care of:
When the user cancels the download from the browser, how do I handle the continuous fetching of data from the endpoint
When the user pauses the download from the browser, how do I pause the data fetching, and then resume once user resumes it
The first case can be taken care of using 'close' event of request. When the connection between the client and the server get cancelled, I disconnect.
If anyone has a better way of implementing this please suggest.
I am having trouble handling the second case when the user pauses.
If anyone could guide me through this, or even provide a better solution to the overall problem(incl. handling the chunks of data), it would be really helpful.
const {createServer} = require('http');
const {Transform} = require('stream');
const axios = require('axios');
var c = 0;
class ApiStream extends Transform {
constructor(apiCallback, res, req) {
super();
this.apiCallback = apiCallback;
this.isPipeSetup = false;
this.res = res;
this.req = req
}
//Will get data continuously
async start() {
let response;
try {
response = await this.apiCallback();
} catch (e) {
response = null;
}
if (!this.isPipeSetup) {
this.pipe(this.res);
this.isPipeSetup = true;
}
if (response) {
response = response.data
if (Array.isArray(response)) {
response.forEach((item) => {
this.push(JSON.stringify(item) + "\n");
});
} else if (typeof response === "object") {
this.push(JSON.stringify(response) + "\n");
} else if (typeof response === "string") {
this.push(response + "\n");
}
this.start()
}else{
this.push(null);
console.log('Stream ended')
}
}
}
const server = createServer(async (req, res, stream) => {
res.setHeader("Content-disposition", "attachment; filename=download.json");
res.setHeader("Content-type", "text/plain");
let disconnected = false;
const filestream = new ApiStream(async () => {
let response;
try {
if(disconnected){
console.log('Client connection closed')
return null;
}
c++;
response = await axios.get("https://jsonplaceholder.typicode.com/users");
//Simulate delay in data fetching
let z = 0;
if(c>=200) response = null;
while(z<10000){
let b = 0;
while(b<10000){
b+=0.5;
}
z +=0.5;
}
} catch (error) {
res.status(500).send(error);
}
if (response) {
return response;
}
return null;
}, res, req);
await filestream.start();
req.on('close', (err) => {
disconnected = true;
})
})
server.listen(5050, () => console.log('server running on port 5050'));

Ratchet PHP reconnect client who loose connection [duplicate]

var ws = new WebSocket('ws://localhost:8080');
ws.onopen = function () {
ws.send(JSON.stringify({
.... some message the I must send when I connect ....
}));
};
ws.onmessage = function (e) {
console.log('Got a message')
console.log(e.data);
};
ws.onclose = function(e) {
console.log('socket closed try again');
}
ws.onerror = function(err) {
console.error(err)
};
When I first connect to the socket, I must first send a message to the server to authenticate myself and subscribe to channels.
The problem I have is that sometimes the socket server is unreliable and that triggers the onerror and onclose events of the 'ws' object.
Question: What is a good design pattern that would allow me, whenever the socket closes or encounters an error, wait for 10 seconds and then reconnect to the socket server (and resend the initial message to the server)

Here is what I ended up with. It works for my purposes.
function connect() {
var ws = new WebSocket('ws://localhost:8080');
ws.onopen = function() {
// subscribe to some channels
ws.send(JSON.stringify({
//.... some message the I must send when I connect ....
}));
};
ws.onmessage = function(e) {
console.log('Message:', e.data);
};
ws.onclose = function(e) {
console.log('Socket is closed. Reconnect will be attempted in 1 second.', e.reason);
setTimeout(function() {
connect();
}, 1000);
};
ws.onerror = function(err) {
console.error('Socket encountered error: ', err.message, 'Closing socket');
ws.close();
};
}
connect();

This worked for me with setInterval, because client connection can be lost.
ngOnInit(): void {
if (window.location.protocol.includes('https')) {
this.protocol = 'wss';
}
this.listenChanges();
}
listenChanges(): void {
this.socket = new WebSocket(`${this.protocol}://${window.location.host}/v1.0/your/url`);
this.socket.onmessage = (event): void => {
// your subscription stuff
this.store.dispatch(someAction);
};
this.socket.onerror = (): void => {
this.socket.close();
};
this.socket.onopen = (): void => {
clearInterval(this.timerId);
this.socket.onclose = (): void => {
this.timerId = setInterval(() => {
this.listenChanges();
}, 10000);
};
};
}
Don't forget to call clearInterval when the socket has been opened.

This isn't explicitly a react question but here is a react style answer:
TLDR: You can use setInterval to periodically check the websocket connection status and try to re-connect if the connection is closed. https://developer.mozilla.org/en-US/docs/Web/API/WebSocket/readyState
class TestComponent extends React.Component {
constructor(props) {
super(props);
this.state = {};
this.connect = this.connect.bind(this);
}
componentDidMount() {
this.interval = setInterval(this.connect, 1000);
}
componentWillUnmount() {
if (this.ws) this.ws.close();
if (this.interval) clearInterval(this.interval);
}
connect() {
// https://developer.mozilla.org/en-US/docs/Web/API/WebSocket/readyState
if (this.ws === undefined || (this.ws && this.ws.readyState === 3)) {
this.ws = new WebSocket(`ws://localhost:8080`);
this.ws.onmessage = (e) => {
console.log(JSON.parse(e.data));
};
}
}
render() {
return <div>Hey!</div>;
}
}

I found that this package https://github.com/pladaria/reconnecting-websocket can solve the reconnection issues for Websocket connections. And it has the list of configurable options, one of them is reconnectionDelayGrowFactor which determines how fast the reconnection delay grows.

using async-await if socket closed or any error occurred on the server the client will try to connect automatically every 5 sec forever
have a look to my answer

UPDATED answer:
At last, (if you are not using java) I found you'd better implement your own "ping/pong" strategy. (if you are using java, please take a look at ping/pong "action type", I don't remember very clear... )
client sent "ping" to server every 5 seconds.
server should echo a "pong" to the client once it receive "ping".
client should reconnect server if doesn't receive "pong" in 5 seconds.
Don't rely on any third party libs.
WARNING: DO NOT use these tools: (reason: they are not reliable and not stable and works in a very limited way. )
check if the network is available: https://github.com/hubspot/offline
to re-connect: https://github.com/joewalnes/reconnecting-websocket

You can use a small library if you want - ReconnectingWebSocket
Add reconnecting-websocket.js in your script tag and
It is API compatible, so when you have:
var ws = new WebSocket('ws://....');
you can replace with:
var ws = new ReconnectingWebSocket('ws://....');

Try this:
const observable = Observable.create(
(obs: Observer<MessageEvent>) => {
this.ws.onmessage = obs.next.bind(obs);
this.ws.onerror = obs.error.bind(obs);
// this.ws.onclose = obs.complete.bind(obs);
this.ws.onclose = function () {
window.location.reload()
}
return this.ws.close.bind(this.ws);
});
const observer = {
next: (data: Object) => {
if (this.ws.readyState === WebSocket.OPEN) {
this.ws.send(JSON.stringify(data));
}
}
};
and component
getDatas() {
let url = environment.apiwebsocket
this.webSocketService.connect(url)
.subscribe(evt => {
let jsonObj = JSON.parse(evt.data)
});}

I used to have this somewhere in project:
let rc = new WebSocket(
'ws://'
+ window.location.host
+ `/ws/chat/${window.seen.pk}/`
)
now I switched to:
// ws create the websocket and returns it
function autoReconnect(ws_create){
let ws = ws_create();
function startReconnecting(){
let interval = setInterval(()=>{
console.log('trying')
ws = ws_create();
ws.onopen = () => {
console.log('stop');
ws.onclose = startReconnecting;
clearInterval(interval);
}
}, 3000);
}
ws.onclose = startReconnecting;
}
let rc;
autoReconnect(()=>{
rc = new WebSocket(
'ws://'
+ window.location.host
+ `/ws/chat/${window.seen.pk}/`
)
return rc;
});
test it by running and stop local host, it works fine. (btw I found it weird this question has been posted for a long time, but there is not a short and elegant solution)
the benefit of this method, is that it allows you to pass in an arrow function, so that you can assign variable to any outer scope.

Here's a simple version I use in my projects. It includes an incrementing wait timer for reconnects.
//wsURL - the string URL of the websocket
//waitTimer - the incrementing clock to use if no connection made
//waitSeed - used to reset the waitTimer back to default on a successful connection
//multiplier - how quickly you want the timer to grow on each unsuccessful connection attempt
const openSocket = (wsURL, waitTimer, waitSeed, multiplier) =>{
let ws = new WebSocket(wsURL);
console.log(`trying to connect to: ${ws.url}`);
ws.onopen = () => {
console.log(`connection open to: ${ws.url}`);
waitTimer = waitSeed; //reset the waitTimer if the connection is made
ws.onclose = () => {
console.log(`connection closed to: ${ws.url}`);
openSocket(ws.url, waitTimer, waitSeed, multiplier);
};
ws.onmessage = (message) => {
//do something with messge...
};
};
ws.onerror = () => {
//increaese the wait timer if not connected, but stop at a max of 2n-1 the check time
if(waitTimer < 60000) waitTimer = waitTimer * multiplier;
console.log(`error opening connection ${ws.url}, next attemp in : ${waitTimer/1000} seconds`);
setTimeout(()=>{openSocket(ws.url, waitTimer, waitSeed, multiplier)}, waitTimer);
}
}
openSocket(`ws://localhost:3000`, 1000, 1000, 2)

Alternatively you can explore socket.io. It offers this feature
on the client side you indicate reconnection: true
const io = require("socket.io-client");
const socket = io('ws://'+WS_REMOTE_ADDRESS,{
reconnection: true,
});

how to prevent new connection on every page refresh in sockjs

So, every time I refresh the page, it seems like sockjs is creating a new connection.
I am saving every message to my mongodb on every channel.onmessage, so if I refresh my page 7 times and send a message, I would save 7 messages of the same content into my mongodb.
This is very problematic because when I retrieve those messages when I go into the chat room, to see the log, I would see bunch of duplicate messages.
I want to keep track of all connections that are 'active', and if a user tries to make another connection, I want to terminate the old one so there is only one connection listening to each message at a time.
How do I do this ?
var connections = {};
//creating the sockjs server
var chat = sockjs.createServer();
//installing handlers for sockjs server instance, with the same url as client
chat.installHandlers(server, {prefix:'/chat/private'});
var multiplexer = new multiplexServer.MultiplexServer(chat);
var configChannel = function (channelId, userId, userName){
var channel = multiplexer.registerChannel(channelId);
channel.on('connection', function (conn) {
// console.log('connection');
console.log(connections);
connections[channelId] = connections[channelId] || {};
if (connections[channelId][userId]) {
//want to close the extra connection
} else {
connections[channelId][userId] = conn;
}
// }
// if (channels[channelId][userId]) {
// conn = channels[channelId][userId];
// } else {
// channels[channelId][userId] = conn;
// }
// console.log('accessing channel! ', channels[channelId]);
conn.on('new user', function (data, message) {
console.log('new user! ', data, message);
});
// var number = connections.length;
conn.on('data', function(message) {
var messageObj = JSON.parse(message);
handler.saveMessage(messageObj.channelId, messageObj.user, messageObj.message);
console.log('received the message, ', messageObj.message);
conn.write(JSON.stringify({channelId: messageObj.channelId, user: messageObj.user, message: messageObj.message }));
});
conn.on('close', function() {
conn.write(userName + ' has disconnected');
});
});
return channel;
};

The way I resolve a problem like yours was with a Closure and Promises, I don't know if that could help you. I let you the code that help me, this is with EventBus from Vertx:
window.Events = (function NewEvents() {
var eventBusUrl = $('#eventBusUrl').val();
var eventBus = null;
return new RSVP.Promise(function(resolve, reject) {
if(!eventBus) {
eventBus = new vertx.EventBus(eventBusUrl);
eventBus.onopen = function eventBusOpened() {
console.log('Event bus online');
resolve(eventBus);
}
eventBus.onclose = function() {
eventBus = null;
};
}
});
}());
And then in other script I call it in this way:
Events.then(function(eventBus) {
console.log("registering handlers for comments");
eventBus.registerHandler(address, function(incomingMessage) {
console.log(incomingMessage);
});
});
I hope this can help you.
Regards.

WebSocket: How to automatically reconnect after it dies

var ws = new WebSocket('ws://localhost:8080');
ws.onopen = function () {
ws.send(JSON.stringify({
.... some message the I must send when I connect ....
}));
};
ws.onmessage = function (e) {
console.log('Got a message')
console.log(e.data);
};
ws.onclose = function(e) {
console.log('socket closed try again');
}
ws.onerror = function(err) {
console.error(err)
};
When I first connect to the socket, I must first send a message to the server to authenticate myself and subscribe to channels.
The problem I have is that sometimes the socket server is unreliable and that triggers the onerror and onclose events of the 'ws' object.
Question: What is a good design pattern that would allow me, whenever the socket closes or encounters an error, wait for 10 seconds and then reconnect to the socket server (and resend the initial message to the server)

Here is what I ended up with. It works for my purposes.
function connect() {
var ws = new WebSocket('ws://localhost:8080');
ws.onopen = function() {
// subscribe to some channels
ws.send(JSON.stringify({
//.... some message the I must send when I connect ....
}));
};
ws.onmessage = function(e) {
console.log('Message:', e.data);
};
ws.onclose = function(e) {
console.log('Socket is closed. Reconnect will be attempted in 1 second.', e.reason);
setTimeout(function() {
connect();
}, 1000);
};
ws.onerror = function(err) {
console.error('Socket encountered error: ', err.message, 'Closing socket');
ws.close();
};
}
connect();

This worked for me with setInterval, because client connection can be lost.
ngOnInit(): void {
if (window.location.protocol.includes('https')) {
this.protocol = 'wss';
}
this.listenChanges();
}
listenChanges(): void {
this.socket = new WebSocket(`${this.protocol}://${window.location.host}/v1.0/your/url`);
this.socket.onmessage = (event): void => {
// your subscription stuff
this.store.dispatch(someAction);
};
this.socket.onerror = (): void => {
this.socket.close();
};
this.socket.onopen = (): void => {
clearInterval(this.timerId);
this.socket.onclose = (): void => {
this.timerId = setInterval(() => {
this.listenChanges();
}, 10000);
};
};
}
Don't forget to call clearInterval when the socket has been opened.

This isn't explicitly a react question but here is a react style answer:
TLDR: You can use setInterval to periodically check the websocket connection status and try to re-connect if the connection is closed. https://developer.mozilla.org/en-US/docs/Web/API/WebSocket/readyState
class TestComponent extends React.Component {
constructor(props) {
super(props);
this.state = {};
this.connect = this.connect.bind(this);
}
componentDidMount() {
this.interval = setInterval(this.connect, 1000);
}
componentWillUnmount() {
if (this.ws) this.ws.close();
if (this.interval) clearInterval(this.interval);
}
connect() {
// https://developer.mozilla.org/en-US/docs/Web/API/WebSocket/readyState
if (this.ws === undefined || (this.ws && this.ws.readyState === 3)) {
this.ws = new WebSocket(`ws://localhost:8080`);
this.ws.onmessage = (e) => {
console.log(JSON.parse(e.data));
};
}
}
render() {
return <div>Hey!</div>;
}
}

I found that this package https://github.com/pladaria/reconnecting-websocket can solve the reconnection issues for Websocket connections. And it has the list of configurable options, one of them is reconnectionDelayGrowFactor which determines how fast the reconnection delay grows.

using async-await if socket closed or any error occurred on the server the client will try to connect automatically every 5 sec forever
have a look to my answer

UPDATED answer:
At last, (if you are not using java) I found you'd better implement your own "ping/pong" strategy. (if you are using java, please take a look at ping/pong "action type", I don't remember very clear... )
client sent "ping" to server every 5 seconds.
server should echo a "pong" to the client once it receive "ping".
client should reconnect server if doesn't receive "pong" in 5 seconds.
Don't rely on any third party libs.
WARNING: DO NOT use these tools: (reason: they are not reliable and not stable and works in a very limited way. )
check if the network is available: https://github.com/hubspot/offline
to re-connect: https://github.com/joewalnes/reconnecting-websocket

You can use a small library if you want - ReconnectingWebSocket
Add reconnecting-websocket.js in your script tag and
It is API compatible, so when you have:
var ws = new WebSocket('ws://....');
you can replace with:
var ws = new ReconnectingWebSocket('ws://....');

Try this:
const observable = Observable.create(
(obs: Observer<MessageEvent>) => {
this.ws.onmessage = obs.next.bind(obs);
this.ws.onerror = obs.error.bind(obs);
// this.ws.onclose = obs.complete.bind(obs);
this.ws.onclose = function () {
window.location.reload()
}
return this.ws.close.bind(this.ws);
});
const observer = {
next: (data: Object) => {
if (this.ws.readyState === WebSocket.OPEN) {
this.ws.send(JSON.stringify(data));
}
}
};
and component
getDatas() {
let url = environment.apiwebsocket
this.webSocketService.connect(url)
.subscribe(evt => {
let jsonObj = JSON.parse(evt.data)
});}

I used to have this somewhere in project:
let rc = new WebSocket(
'ws://'
+ window.location.host
+ `/ws/chat/${window.seen.pk}/`
)
now I switched to:
// ws create the websocket and returns it
function autoReconnect(ws_create){
let ws = ws_create();
function startReconnecting(){
let interval = setInterval(()=>{
console.log('trying')
ws = ws_create();
ws.onopen = () => {
console.log('stop');
ws.onclose = startReconnecting;
clearInterval(interval);
}
}, 3000);
}
ws.onclose = startReconnecting;
}
let rc;
autoReconnect(()=>{
rc = new WebSocket(
'ws://'
+ window.location.host
+ `/ws/chat/${window.seen.pk}/`
)
return rc;
});
test it by running and stop local host, it works fine. (btw I found it weird this question has been posted for a long time, but there is not a short and elegant solution)
the benefit of this method, is that it allows you to pass in an arrow function, so that you can assign variable to any outer scope.

Here's a simple version I use in my projects. It includes an incrementing wait timer for reconnects.
//wsURL - the string URL of the websocket
//waitTimer - the incrementing clock to use if no connection made
//waitSeed - used to reset the waitTimer back to default on a successful connection
//multiplier - how quickly you want the timer to grow on each unsuccessful connection attempt
const openSocket = (wsURL, waitTimer, waitSeed, multiplier) =>{
let ws = new WebSocket(wsURL);
console.log(`trying to connect to: ${ws.url}`);
ws.onopen = () => {
console.log(`connection open to: ${ws.url}`);
waitTimer = waitSeed; //reset the waitTimer if the connection is made
ws.onclose = () => {
console.log(`connection closed to: ${ws.url}`);
openSocket(ws.url, waitTimer, waitSeed, multiplier);
};
ws.onmessage = (message) => {
//do something with messge...
};
};
ws.onerror = () => {
//increaese the wait timer if not connected, but stop at a max of 2n-1 the check time
if(waitTimer < 60000) waitTimer = waitTimer * multiplier;
console.log(`error opening connection ${ws.url}, next attemp in : ${waitTimer/1000} seconds`);
setTimeout(()=>{openSocket(ws.url, waitTimer, waitSeed, multiplier)}, waitTimer);
}
}
openSocket(`ws://localhost:3000`, 1000, 1000, 2)

Alternatively you can explore socket.io. It offers this feature
on the client side you indicate reconnection: true
const io = require("socket.io-client");
const socket = io('ws://'+WS_REMOTE_ADDRESS,{
reconnection: true,
});

Postgresql+Node.js queries start failing after a few tries

I have a node.js app with a postgresql database running on Heroku and for some reason select functions seem to break and start failing after many successful attempts for what seems to be no reason at all..
Example code:
pg.connect(database_url, function(err, client) {
if(client) {
var date = new Date();
var query = client.query('SELECT * FROM accounts',function(err, result) {
if(!err) {
res.send(JSON.stringify(result.rows));
accounts = JSON.stringify(result.rows);
} else {
res.send('failed');
}
});
} else {
res.send(JSON.stringify(err));
}
});
The above code works perfectly about 10 times and then out of nowhere seems to break.
How can i remedy this issue?

If it works 10 times, and fails on the 11'th then it's probably because the pool of connections has run out.
You should call done in your code after you have finished with the request, to send the database connection back to the pool.
pg.connect(database_url, function(err, client, done) {
if(client) {
var date = new Date();
var query = client.query('SELECT * FROM accounts',function(err, result) {
if(!err) {
res.send(JSON.stringify(result.rows));
accounts = JSON.stringify(result.rows);
} else {
res.send('failed');
}
done(); // call done to close the conection
});
} else {
res.send(JSON.stringify(err));
}
});
see the documentation for connect