I'm running on a fastify server and when I send a request, I get a ton of data from mongodb and start a for loop to process each item. This can take ~ 30 minutes. Each item is "processed" by sending to ffmpeg, redis pub->sub, and then a socket to the client.
// Streams controller
exports.renderStreams = async function (req, reply) {
const streams = await Stream.find({}).sort({ createdAt: -1 })//.limit(5)
const renderedStreams = renderStreams(this, streams);
return { success: true, streams: streams.length };
}
// renderStreams
const renderStreams = (fastify, streams = []) => {
const { redis } = fastify;
const channel = "streams";
for (let i = 0; i < streams.length; i++) {
setTimeout(async () => {
const stream = streams[i];
await renderStream(redis, channel, stream);
}, i * 200)
}
}
I am wondering in this for loop, how can I either "pause" it or stop it completely (or both?) via another request, maybe when I call /api/streams/stop.
How would this be possible?
You can use socket.io to do communications with your script while it is still running, so you would create a function to stop the loop and when you get the notification from socket.io you would run it. Documentation link: https://socket.io/docs/v4/
Related
I'm currently following this example and I'm tring to extend upon it with my own logic.
I can get the socket to connect, but I see that there is a problem in having the client reconnect continuously - Hence why I want to have a limit on the number of retries that the client can perform.
socket.pipe(
tap((data => console.log(data))),
retryWhen((errors) =>
errors.pipe(
take(this.retryCount),
delayWhen(val =>
timer(val * 1000)
)
)
)
).subscribe()
I looked up the documentation and figured that I could use the take() operator from rxjs. However, given that I set my retryCount to 5, the following scenario can be assumed:
Client connects to socket successfully
Client disconnects from the socket
Client reconnects after 4 retries
Client disconnects
Client retries 1 times, and stops retrying (take(5) has been reached)
Is there a way, in which I can "reset" the amount of times that take() will retry? i.e. so that every time the client disconnects from the socket, it will always have 5 retries?
The retryWhen logic from your example doesn't take into account all the possible intertwined flows of connect/disconnect/reconnect. Also the reset of the reconnection attempts doesn't happen.
Beside that, you're touching the inner Observables for open and close of the webSocket directly (which is not technically wrong), while you could put all the logic and handlers inside the subject's pipe and subscription code. I've forked your example from StackBlitz and adjusted it so that any connect / disconnect and reconnect events are handled properly.
As you can see in the screenshot below, it's trying to reconnect five time before it reaches the limit and quit.
The number of reconnect retries and the delay between each trial can be tweaked via this two constants:
const maxReconnectAttempts = 5; // -1 stands for 'forever'
const reconnectAttemptDelay = 1000; // in ms
As written in the comment, if you put const maxReconnectAttempts = -1 the reconnection trials will go on forever.
You can find below the whole code taken from index.js, or here in the forked StackBlitz repo for the full reference.
import { iif, of, throwError } from 'rxjs';
import { concatMap, delay, retryWhen, tap } from 'rxjs/operators';
import { webSocket } from 'rxjs/webSocket';
// get elements
const btn1 = document.getElementById('btn1');
const btn2 = document.getElementById('btn2');
const display = document.getElementById('app');
const msg = document.getElementById('msg');
// bind click actions
btn1.onclick = connect;
btn2.onclick = disconnect;
// create a webSocketSubject
const wsUrl = 'wss://www.gasnow.org/ws/gasprice';
const maxReconnectAttempts = 5; // -1 stands for 'forever'
const reconnectAttemptDelay = 1000; // in ms
let wsSubject$;
let wsSubscription;
let disconnected = false;
let reconnectRetryCount = 0;
// display live data on view
const bindDataToView = ({ data }) => {
console.log(data);
display.innerText = Number(data.rapid);
};
// connect/subscribe to websocket url
function connect() {
if (!this.wsSubject || this.wsSubject.closed) {
wsSubject$ = webSocket(wsUrl); // create a fresh instance
console.log(`Initializing WebSocket connection to ${wsUrl}...`);
wsSubscription = wsSubject$
.pipe(
retryWhen((errors) =>
errors.pipe(
concatMap((error) =>
iif(
() =>
maxReconnectAttempts !== -1 &&
reconnectRetryCount++ >= maxReconnectAttempts,
throwError('WebSocket reconnecting retry limit exceeded!'),
of(error).pipe(
tap(() => {
disconnected = true;
console.warn('Trying to reconnect to WebSocket server...');
}),
delay(reconnectAttemptDelay)
)
)
)
)
),
tap(() => {
if (disconnected) {
disconnected = false;
reconnectRetryCount = 0;
msg.innerText = 'Streaming ...';
console.log('Successfully reconnected to the WebSocket server.');
}
})
)
.subscribe(
(data) => {
msg.innerText = 'Streaming ...';
bindDataToView(data);
},
(err) => {
reconnectRetryCount = 0;
console.error(err);
},
() => {
reconnectRetryCount = 0;
msg.innerText = 'Connection closed';
console.warn('Connection to the WebSocket server was closed!');
}
);
}
}
// close websocket connection
function disconnect() {
if (wsSubject$) {
wsSubject$.complete(); // this will trigger closingObserver and closeObserver
wsSubject$.unsubscribe();
wsSubject$ = null;
disconnected = true;
reconnectRetryCount = 0;
if (wsSubscription) {
wsSubscription.unsubscribe();
wsSubscription = null;
}
console.log('Disconnected from the WebSocket server.');
}
}
So, for a course i'm taking, we're coding a UDP pinger in Javascript, using Node.js and Dgram. We've been given the following assignment:
Create the client code for an application. Your client should send 10 ping messages to the target UDP server. For each message, your client should calculate the round trip time from when the package is sent to when the response is received. Should a package be dropped along the way, the client is to handle this as well. This should be done by having the client wait 1 second for a response after sending each package. If no reply is received, the client should log accordingly (package lost, no response, timeout, etc.) and send a new package to try again. However, the total amount of packages sent should still only be 10. The client should also calculate a percentage of packages lost/no response received, and log this before connection is closed.
THis if course seems rather straight forward, and I thought so. I've been coding it for a while, and I'm almost finished, but I'm having issues with the aspect of making the client send a package, await response, and then act accordingly.
So far, what my code does is basically to send a ping, and when a pong is received, it sends another ping. What I can't figure out is how to make it log that a response wasn't received before sending the next package. In other words, I know how to make it react to a received response, I just don't know how to make it respond if no response is given within a set timeframe. I've tried playing around with if-statements and loops, as well as async functions, but I haven't made it work yet, so now I'm asking for help.
Code is here:
const dgram = require("dgram");
const ms = require("ms");
var client = dgram.createSocket("udp4");
const PORT = 8000;
const HOST = "localhost";
let today = "";
let t0 = "";
let t1 = "";
let RTT = "";
let sentPackages = "";
let receivedPackages = "";
const messageOutbound = Buffer.from("You Up?");
sendPackage();
const x = setInterval(sendPackage, 1000);
client.on("message", (message, remote) => {
receivedPackages++
today = new Date();
t1 = today.getTime();
console.log(
`Message from: ${remote.address}:${remote.port} saying: ${message}`
);
RTT = ms(t1 - t0, { long: true });
console.log(RTT);
const x = setInterval(sendPackage, 1000);
});
client.on('error', (err) => {
console.log(`server error:\n${err.stack}`);
server.close();
});
async function sendPackage() {
if (sentPackages < 10) {
client.send(messageOutbound, 0, messageOutbound.length, PORT, HOST, () => {
sentPackages++
let today = new Date();
t0 = today.getTime();
console.log(
`message has been sent to ${HOST}:${PORT}. Message sent at: ${t0}`
);
});
} else {
calculateLoss();
client.close();
}
};
function calculateLoss() {
let amountLost = sentPackages - receivedPackages;
let percentageLoss = amountLost / sentPackages * 100
console.log(amountLost);
console.log(percentageLoss +"% of packages lost");
};
I would use async / await to simply wait 1000ms / 1s between messages, then keep track of all messages in an array.
We identify messages with a uuid, so we can ensure that messages we receive can be matched to those we send.
We can then log all the required statistics afterwards:
const dgram = require("dgram");
const uuid = require('uuid');
const PORT = 8000;
const HOST = "localhost";
const client = dgram.createSocket("udp4");
// Array that keeps track of the messages we send
let messages = [];
// When we get a message, decode it and update our message list accordingly...
client.on("message", (messageBuffer, remote) => {
let receivedMessage = bufferToMessage(messageBuffer);
// Find the message we sent and set the response time accordingly.
let message = messages.find(message => message.uuid === (receivedMessage ||{}).uuid);
if (message) {
message.responseTimestamp = new Date().getTime();
}
});
client.on('error', (err) => {
console.log(`server error:\n${err.stack}`);
server.close();
});
function createMessage() {
return { uuid: uuid.v4() };
}
function messageToBuffer(message) {
return Buffer.from(JSON.stringify(message), "utf-8");
}
function bufferToMessage(buffer) {
try {
return JSON.parse(buffer.toString("utf-8"));
} catch (error) {
return null;
}
}
// Wait for timeout milliseconds
function wait(timeout) {
return new Promise(resolve => setTimeout(resolve, timeout));
}
function sendMessage(message, port, host) {
// Save the messages to our list...
messages.push(message);
console.log(`Sending message #${messages.length}...`);
// Set the time we send out message...
message.sentTimestamp = new Date().getTime();
let messageBuffer = messageToBuffer(message);
return new Promise((resolve, reject) => {
client.send(messageBuffer, 0, messageBuffer.length, port, host, (error, bytes) => {
if (error) {
reject(error);
} else {
resolve(bytes);
}
})
});
}
async function sendMessages(messageCount, port, host, timeout) {
for(let messageIndex = 0; messageIndex < messageCount; messageIndex++) {
let message = createMessage();
await sendMessage(message, port, host);
await wait(timeout);
if (message.responseTimestamp) {
console.log(`Response received after ${message.responseTimestamp - message.sentTimestamp} ms...`);
} else {
console.log(`No response received after ${timeout} ms...`);
}
}
logStatistics(messages);
}
function logStatistics(messages) {
let messagesSent = messages.length;
let messagesReceived = messages.filter(m => m.responseTimestamp).length;
let messagesLost = messagesSent - messagesReceived;
console.log(`Total messages sent: ${messagesSent}`);
console.log(`Total messages received: ${messagesReceived}`);
console.log(`Total messages lost: ${messagesLost} / ${(100*messagesLost / (messages.length || 1) ).toFixed(2)}%`);
if (messagesReceived > 0) {
console.log(`Average response interval:`, messages.filter(m => m.responseTimestamp).reduce((averageTime, message) => {
averageTime += (message.responseTimestamp - message.sentTimestamp) / messagesReceived;
return averageTime;
}, 0) + " ms");
}
}
sendMessages(10, PORT, HOST, 1000);
The function I would like this function to run by itself at time intervals. As it is now I have to visit the '/getCompanyInfo' path to trigger it. I would like it to run every minute as if I was visiting the '/getCompanyInfo' path each minute. The app is on Heroku and I would like the function to execute without any pages open.
The original function that is triggered by visiting the path.
const express = require('express');
const app = express();
/**
* getCompanyInfo ()
*/
app.get('/getCompanyInfo', function(req,res){
const companyID = oauthClient.getToken().realmId;
console.log(companyID)
const url = OAuthClient.environment.production ;
oauthClient.makeApiCall({url: url + 'v3/company/0000000000/salesreceipt/8?minorversion=41'})
.then(function(authResponse){
console.log("The response for API call is :"+JSON.parse(JSON.stringify(authResponse)));
res.send(authResponse);
})
.catch(function(e) {
console.error(e);
});
});
One of my attempts here was to put it in a function that executes each minute using node-schedule.
This one doesn't do anything other than print 'This will run once a minute.' to the console.
I tried removing
app.get(function(req,res){
and the
})
below it but that made the app (hosted on Heroku) fail to build.
const express = require('express');
const app = express();
var schedule = require('node-schedule');
var j = schedule.scheduleJob('* * * * *', function(){
console.log('This will run once a minute.');
app.get(function(req,res){
const companyID = oauthClient.getToken().realmId;
console.log(companyID)
const url = OAuthClient.environment.production ;
oauthClient.makeApiCall({url: url + 'v3/company/0000000000/salesreceipt/8?minorversion=41'})
.then(function(authResponse){
console.log("The response for API call is :"+JSON.parse(JSON.stringify(authResponse)));
res.send(authResponse);
})
.catch(function(e) {
console.error(e);
});
});
});
More Context:
It is inside an app I have on Heroku. I would like to set the app to make a requests for JSON data from the API every x time without me having to touch it.
app.get initializes api handler - e.g. this is your api route definition - the thing that will respond when you call GET /getCompanyInfo via web browser or some other client. You should not redefine it regularly with your scheduled action.
The failed build after you've removed the route handler is probably because of the res.send(authResponse); left behind.
You could have something like:
// function that will be used to get the data
const getCompanyInfo = (done) => {
const companyID = oauthClient.getToken().realmId
console.log(companyID)
const url = OAuthClient.environment.production
oauthClient.makeApiCall({url: url + 'v3/company/0000000000/salesreceipt/8?minorversion=41'})
.then((authResponse) => {
console.log("The response for API call is :"+JSON.parse(JSON.stringify(authResponse)))
done(authResponse)
})
.catch((e) => {
console.error(e)
})
}
// this will trigger the function regularly on the specified interval
const j = schedule.scheduleJob('* * * * *', () => {
getCompanyInfo((companyInfo) => {
// ...do whatever you want with the info
})
})
// this will return you the data by demand, when you call GET /getCompanyInfo via browser
app.get('/getCompanyInfo', function(req,res) {
getCompanyInfo((companyInfo) => {
res.send(companyInfo)
})
})
Heroku has an add on called Heroku Scheduler that does what you want. The node-schedule npm package might do the job, but as you mentioned, you probably aren't going to be able to see the execution/results/logs of your jobs that run every 24 hours without making some interface for it on your own.
For your issue, calling app.get doesn't make a lot of sense. That's just telling node about the route. Assuming you have your /getCompanyInfo route up and running, you just need to call it in your scheduled job, not re-register it every time.
You could also just do this (http being the http client you're using):
var j = schedule.scheduleJob('* * * * *', async function(){
console.log('This will run once a minute.');
const result = await http.get('/getCompanyInfo');
console.log(result);
});
I am currently trying to setup a server stream with the gRPC Node.js API. For that I want to achieve that when I write on server side to the stream that the client immediately receives the data event.
At the moment I don't receive anything on client side if I only call write on server side. However as soon as I call the end function on the server the client receives all data events.
To test this I used an endless while loop for writing messages on server side. Then the client does not receive messages (data events). If instead I use a for loop and call end afterwards the client receives all the messages (data events) when end is called.
My .proto file:
syntax = "proto3";
message ControlMessage {
enum Control {
Undefined = 0;
Start = 1;
Stop = 2;
}
Control control = 1;
}
message ImageMessage {
enum ImageType {
Raw = 0;
Mono8 = 1;
RGB8 = 2;
}
ImageType type = 1;
int32 width = 2;
int32 height = 3;
bytes image = 4;
}
service StartImageTransmission {
rpc Start(ControlMessage) returns (stream ImageMessage);
}
On the server side I implement the start function and try to endlessly write messages to the call:
function doStart(call) {
var imgMsg = {type: "Mono8", width: 600, height: 600, image: new ArrayBuffer(600*600)};
//for(var i = 0; i < 10; i++) {
while(true) {
call.write(imgMsg);
console.log("Message sent");
}
call.end();
}
I register the function as service in the server:
var server = new grpc.Server();
server.addService(protoDescriptor.StartImageTransmission.service, {Start: doStart});
On client side I generate an appropriate call and register the data and end event:
var call = client.Start({control: 0});
call.on('data', (imgMessage) => {
console.log('received image message');
});
call.read();
call.on('end', () => {console.log('end');});
I also tried to write the server side in python. In this case the node client instantly receives messages and not only after stream was ended on server side. So I guess this should be also possible for the server written with the Node API.
It seems that the problem was that the endless while loop is blocking all background tasks in node. A possible solution is to use setTimeout to create the loop. The following code worked for me:
First in the gRPC call store the call object in an array:
function doStart(call) {
calls.push(call);
}
For sending to all clients I use a setTimeout:
function sendToAllClients() {
calls.forEach((call) => {
call.write(imgMsg);
});
setTimeout(sendToAllClients, 10);
}
setTimeout(sendToAllClients, 10);
Helpful stackoverflow atricle: Why does a while loop block the event loop?
I was able to use uncork which comes from Node.js's Writable.
Here is an example. Pseudocode, but pulled from across a working implementation:
import * as grpc from '#grpc/grpc-js';
import * as proto from './src/proto/generated/organizations'; // via protoc w/ ts-proto
const OrganizationsGrpcServer: proto.OrganizationsServer = {
async getMany(call: ServerWritableStream<proto.Empty, proto.OrganizationCollection>) {
call.write(proto.OrganizationCollection.fromJSON({ value: [{}] }));
call.uncork();
// do some blocking stuff
call.write(proto.OrganizationCollection.fromJSON({ value: [{}] }));
call.uncork();
// call.end(), or client.close() below, at some point?
},
ping(call, callback) {
callback(null);
}
};
const client = new proto.OrganizationsClient('127.0.0.1:5000', grpc.credentials.createInsecure());
const stream = client.getMany(null);
stream.on('data', data => {
// this cb should run twice
});
export default OrganizationsGrpcServer;
//.proto
service Organizations {
rpc GetMany (google.protobuf.Empty) returns (stream OrganizationCollection) {}
}
message OrganizationCollection {
repeated Organization value = 1;
}
Versions:
#grpc/grpc-js 1.4.4
#grpc/proto-loader 0.6.7
ts-proto 1.92.1
npm 8.1.4
node 17
So I'm using Server.js for a middleware application that polls data every few seconds and emits the data to all clients. The problem I'm running into is that a new set of pollers are being created for every new socket connection, with each of those pollers getting and emiting to all clients (way too much data). I only want one poller that emits to all, is there a way to do this with Server.js using Socket.IO?
const server = require('server')
const { get, post, socket } = require('server/router')
socket('connect', socket => {
// New connection
setInterval(() => getData(socket), 60000),
});
function getData(socket) {
let sets = db.newSet()
if (typeof socket != 'undefined') {
socket.io.emit("set", sets);
}
return '';
}