We Keep Coding

Manage a long-running operation node.js

I am creating a telegram bot, which allows you to get some information about the destiny 2 game world, using the Bungie API. The bot is based on the Bot Framework and uses Telegram as a channel (as a language I am using JavaScript).
now I find myself in the situation where when I send a request to the bot it sends uses series of HTTP calls to the EndPoints of the API to collect information, format it and resubmit it via Adaptive cards, this process however in many cases takes more than 15 seconds showing in chat the message "POST to DestinyVendorBot timed out after 15s" (even if this message is shown the bot works perfectly).
Searching online I noticed that there doesn't seem to be a way to hide this message or increase the time before it shows up. So the only thing left for me to do is to make sure it doesn't show up. To do this I tried to refer to this documentation article. But the code shown is in C #, could someone give me an idea on how to solve this problem of mine or maybe some sample code?
I leave here an example of a call that takes too long and generates the message:
//Mostra l'invetraio dell'armaiolo
if (LuisRecognizer.topIntent(luisResult) === 'GetGunsmith') {
//Take more 15 seconds
const mod = await this.br.getGunsmith(accessdata, process.env.MemberShipType, process.env.Character);
if (mod.error == 0) {
var card = {
}
await step.context.sendActivity({
text: 'Ecco le mod vendute oggi da Banshee-44:',
attachments: [CardFactory.adaptiveCard(card)]
});
} else {
await step.context.sendActivity("Codice di accesso scaduto.");
await this.loginStep(step);
}
}

I have done something similar where you call another function and send the message once the function is complete via proactive message. In my case, I set up the function directly inside the bot instead of as a separate Azure Function. First, you need to save the conversation reference somewhere. I store this in conversation state, and resave it every turn (you could probably do this in onMembersAdded but I chose onMessage when I did it so it resaves the conversation reference every turn). You'll need to import const { TurnContext } = require('botbuilder') for this.
// In your onMessage handler
const conversationData = await this.dialogState.get(context, {});
conversationData.conversationReference = TurnContext.getConversationReference(context.activity);
await this.conversationState.saveChanges(context);
You'll need this for the proactive message. When it's time to send the API, you'll need to send a message (well technically that's optional but recommended), get the conversation data if you haven't gotten it already, and call the API function without awaiting it. If your API is always coming back around 15 seconds, you may just want a standard message (e.g. "One moment while I look that up for you"), but if it's going to be longer I would recommend setting the expectation with the user (e.g. "I will look that up for you. It may take up to a minute to get an answer. In the meantime you can continue to ask me questions."). You should be saving user/conversation state further down in your turn handler. Since you are not awaiting the call, the turn will end and the bot will not hang up or send the timeout message. Here is what I did with a simulation I created.
await dc.context.sendActivity(`OK, I'll simulate a long-running API call and send a proactive message when it's done.`);
const conversationData = await this.dialogState.get(context, {});
apiSimulation.longRunningRequest(conversationData.conversationReference);
// That is in a switch statement. At the end of my turn handler I save state
await this.conversationState.saveChanges(context);
await this.userState.saveChanges(context);
And then the function that I called. As this was just a simulation, I have just awaited a promise, but obviously you would call and await your API(s). Once that comes back you will create a new BotFrameworkAdapter to send the proactive message back to the user.
const request = require('request-promise-native');
const { BotFrameworkAdapter } = require('botbuilder');
class apiSimulation {
static async longRunningRequest(conversationReference) {
console.log('Starting simulated API');
await new Promise(resolve => setTimeout(resolve, 30000));
console.log('Simulated API complete');
// Set up the adapter and send the message
try {
const adapter = new BotFrameworkAdapter({
appId: process.env.microsoftAppID,
appPassword: process.env.microsoftAppPassword,
channelService: process.env.ChannelService,
openIdMetadata: process.env.BotOpenIdMetadata
});
await adapter.continueConversation(conversationReference, async turnContext => {
await turnContext.sendActivity('This message was sent after a simulated long-running API');
});
} catch (error) {
//console.log('Bad Request. Please ensure your message contains the conversation reference and message text.');
console.log(error);
}
}
}
module.exports.apiSimulation = apiSimulation;

Unable to get Notify data using Noble

Can't receive any notifications sent from the Server peripheral.
I am using ESP32 as Server with the "BLE_notify" code that you can find in the Arduino app (File> Examples ESP32 BLE Arduino > BLE_notify).
With this code the ESP32 starts notifying new messages every second once a Client connects.
The client used is a Raspberry Pi with Noble node library installed on it (https://github.com/abandonware/noble). this is the code I am using.
noble.on('discover', async (peripheral) => {
console.log('found peripheral:', peripheral.advertisement);
await noble.stopScanningAsync();
await peripheral.connectAsync();
console.log("Connected")
try {
const services = await peripheral.discoverServicesAsync([SERVICE_UUID]);
const characteristics = await services[0].discoverCharacteristicsAsync([CHARACTERISTIC_UUID])
const ch = characteristics[0]
ch.on('read', function(data, isNotification) {
console.log(isNotification)
console.log('Temperature Value: ', data.readUInt8(0));
})
ch.on('data', function(data, isNotification) {
console.log(isNotification)
console.log('Temperature Value: ', data.readUInt8(0));
})
ch.notify(true, function(error) {
console.log(error)
console.log('temperature notification on');
})
} catch (e) {
// handle error
console.log("ERROR: ",e)
}
});
SERVICE_UUID and CHARACTERISTIC_UUID are obviously the UUIDs coded in the ESP32.
This code sort of works, it can find Services and Characteristics and it can successfully connect to the peripheral, but it cannot receive messages notifications.
I also tried an Android app that works as client, from that app I can get all the messages notified by the peripheral once connected to it. So there is something missing in the noBLE client side.
I think there is something wrong in the on.read/on.data/notify(true) callback methods. Maybe these are not the methods to receive notifications from Server?
I also tried the subscribe methods but still not working.
The official documentation is not clear. Anyone could get it up and running? Please help.

on.read/on.data/ are event listeners. There is nothing wrong with them. They are invoked when there is a certain event.
For example adding characteristic.read([callback(error, data)]); would have invoked the on.read.
From the source:
Emitted when:
Characteristic read has completed, result of characteristic.read(...)
Characteristic value has been updated by peripheral via notification or indication, after having been enabled with
characteristic.notify(true[, callback(error)])

I resolve using the following two envs NOBLE_MULTI_ROLE=1 and NOBLE_REPORT_ALL_HCI_EVENTS=1 (see the documentation https://github.com/abandonware/noble)

IBM MQ cmit and rollback with syncpoint

Infra-Overview:
I have a setup where I am reading a set of messages from IBM MQ and processing those messages in k8 cluster env and sending it to the destination host.
Issue:
I observed that sometimes the flow of the messages is huge and before sending it to the destination host our pod gets failed and restarts, by this we are losing all the messages as we are following a read-and-delete approach from ibmmq example
Expected Solution:
I am looking for a solution where, until these messages are sent to the destination host, we don't lose the track of the messages.
What I tried:
We have a concept of unit of work in IBM MQ but since we can't expect a delay in reading and processing, I can't wait for a single message to get processed and then read the another message as it might have a major performance setback.
Code language:
NodeJs

As the comments suggest there are a number of ways to skin this cat, but you will need to use transactions.
As soon as you create the connection with the transaction option, the transaction scope begins. This gets closed and next transaction begins when you either commit or rollback.
So you should handle the messages in batches, that make sense to your application, and commit when the batch is complete. If your application is killed by k8s then all uncommitted read messages will get rolled back, via back out queue process to stop poison messages.
Section added to show sample code, and explanation of backout queues.
In your normal processing, if an app gets stopped before it has had time to process the message, you will want that message returned to the queue. So that the message is still available to be processed.
To enable this rollback you need to or in the MQC.MQPMO_SYNCPOINT into the get message options
gmo.Options |= MQC.MQGMO_SYNCPOINT
Then if all goes well, you can commit.
mq.Cmit(hConn, function(err) {
if (err) {
debug_warn('Error on commit', err);
} else {
debug_info('Commit was successful');
}
});
or rollback
mq.Back(hConn, function(err) {
if (err) {
debug_warn('Error on rollback', err);
} else {
debug_info('rollback was successful');
}
});
If you rollback, the message goes back to the queue. Which means it is also the next message that your app will read. This can generate a poison message loop. So you should also set up a backout queue with pass all context permissions for your app user and a backout threshold.
Say you set the threshold to 5. The message can be read 5 times, with rollback. Your app needs to check the threshold and decide that it is a poison message and move it off the queue.
To check the backout threshold (and the backout queue name) you can use the following code
// Remember to or in the Inquire option on the Open
openOptions |= MQC.MQOO_INQUIRE;
...
attrs = [ new mq.MQAttr(MQC.MQIA_BACKOUT_THRESHOLD),
new mq.MQAttr(MQC.MQCA_BACKOUT_REQ_Q_NAME) ];
mq.Inq(hObj, attrs, (err, selectors) => {
if (err) {
debug_warn('Error retrieving backout threshold', err);
} else {
debug_info('Attributes have been found');
selectors.forEach((s) => {
switch (s.selector) {
case MQC.MQIA_BACKOUT_THRESHOLD:
debug_info('Threshold is ', s.value);
break;
case MQC.MQCA_BACKOUT_REQ_Q_NAME:
debug_info('Backout queue is ', s.value);
break;
}
});
}
});
When getting the message your app can use mqmd.BackoutCount to check how often the message has been rolled back.
if (mqmd.BackoutCount >= threshold) {
...
}
What I have noticed, that if this is in the same application instance that is repeatedly calling rollback on the same message, then at the threshold a MQRC_HOBJ_ERROR error is thrown. Which your app can check for, and then discard the message.
If its a different app instance then it doesn't get the MQRC_HOBJ_ERROR error, so it can check the backout threshold and can discard the message, remembering to commit the discard action.
See https://github.com/ibm-messaging/mq-dev-patterns/tree/master/transactions/JMS/SE for more information.
As an alternative you could use keda - https://keda.sh - which works with k8s
to monitor your queue depth and scale according to the number of messages waiting to be processed, as opposed to CPU / memory consumption. That way you can scale up when there are lots of messages waiting to be processed, and slowly scale down then the queue becomes manageable. Here is a link to getting started - https://github.com/ibm-messaging/mq-dev-patterns/tree/master/Go-K8s - the example is for a Go app, but equally applies to Node.js

How to allow JavaScript to receive a message sent to a specific user (or self) using Signal R?

I previously asked this question but it was closed for duplication owing to this thread (SignalR - Sending a message to a specific user using (IUserIdProvider) *NEW 2.0.0*) - but this doesn't show the JavaScript as made clear in my title.
I have a WebForm application in ASP.Net that uses SignalR to push live data to the user logged in. The setup works perfectly, but realised I am broadcasting messages to all clients, which whilst it doesn't cause the wrong data to displayed to the logged in user, does cause the JavaScript function to get called for all users when just one has a data push.
I have amended the Hub code to broadcast to a specific user (User) and provided the User ID, and I have also tried Client with a Connection ID. Both fire off fine in the codebehind, but the javascript will not update the front end.
I believe it's because the JavaScript has not been modified to listen for a message sent to the user, but I'm not sure how I need to adapt the code to allow the message to be received.
The 2 tried lines in Hub here:
context.Clients.User(Me.Context.User.Identity.GetUserId()).ReceiveNotification(notifCount)
context.Clients.Client(Me.Context.ConnectionId).ReceiveNotification(notifCount)
JavaScript/jQuery function for the SignalR message here:
$(function () {
var nf = $.connection.notificationHub;
nf.client.receiveNotification = function (notifCount) {
// Update data
}
$.connection.hub.start().done(function () {
nf.server.sendNotifications();
}).fail(function (e) {
alert(e);
});
//$.connection.hub.start();
});

For calling back to the client (or self) you should use:
Clients.Caller.addContosoChatMessageToPage(name, message);
And for calling users you should use:
Clients.Client(Context.ConnectionId).addContosoChatMessageToPage(name, message);
Reference - docs

How to wait for the backend in Protractor?

I'm testing a web page where the user can send a message to another via a textinput. A POST request is then send on the server and the message is dumped on the disk in the var/mail/new folder.
After automatising the sending of the message in the page with Protractor I'm calling browser.waitForAngular() and browser.driver.sleep(4000) to leave time for the backend to write the mail on the disk.
After these calls the check of the email's presence fails. When looking in the Unix shell, I can confirm that the email was sent and also the next test marked with in Jasmine with it confirms the presence of the email.
Why is browser.driver.sleep(4000) not effective to wait for the backend to proceed? How can I correct the following code?
it("is possible to send a message", function() {
shared.loginContributor();
var mailsBeforeMessaging =
fs.readdirSync(browser.params.mail.queue_path + "/new");
console.log('mailsBeforeMessaging');
console.log(mailsBeforeMessaging.length);
console.log(fs.lstatSync(browser.params.mail.queue_path + "/new"));
var usersListing = new UserPages.UsersListing().get();
var annotatorPage = usersListing.getUserPage("annotator");
annotatorPage.sendMessage("title5", "content64");
exec("/tmp/check.sh");
// we expect the message widget to disappear
var button = element(by.css(".user-profile-info-button"));
console.log('waiting');
browser.wait(EC.elementToBeClickable(button), 5000);
console.log('waiting is finished');
expect(EC.elementToBeClickable(button)).toBeTruthy();
// wait for mail to be dumped on the disk?
browser.waitForAngular();
browser.driver.sleep(4000);
exec("/tmp/check.sh");
var mailsAfterMessaging =
fs.readdirSync(browser.params.mail.queue_path + "/new");
console.log('mailsAfterMessaging');
// ERROR: here the number of emails is NOT incremented
console.log(mailsAfterMessaging.length);
console.log(fs.lstatSync(browser.params.mail.queue_path + "/new"));
});
it("xyz", function() {
console.log(fs.lstatSync(browser.params.mail.queue_path + "/new"));
// here the number of emails is incremented
var mailsAfterMessaging =
fs.readdirSync(browser.params.mail.queue_path + "/new");
console.log('mailsAfterMessaging');
console.log(mailsAfterMessaging.length);
});

Most of the Protractor functions do not do anything. They queue something up to be done later, and return promise to do it. After an it block schedules a bunch of things to do, they actually start happening (via the promises they registered in the ControlFlow).
Your checks, however, are all executing immediately. So, they are happening before any of the protractor calls accomplish anything.
Use then to make the waiting and dependencies explicit in your test. Like this:
annotatorPage.sendMessage("title5", "content64").then(function() {
exec("/tmp/check.sh");
});
or:
browser.wait(EC.elementToBeClickable(button), 5000).then(function() {
console.log('wait-for-clickable has completed'); // B
});
console.log('wait-for-clickable has been scheduled'); // A
See the Protractor Control Flow documentation and the Webdriver JS API doc.
Its not you. This is a crazy API to learn because it does not act at all like anyone familiar with normal synchronous programming would expect.