We Keep Coding

Sending a message in smaller chunks

I am using DiscordJS and their API has a character limit and will reject message if limit is exceeded.
Through fetchData()I am successfully building assignedPrint, which is an array of messages that I would like to send over the API.
So I already have the array ready to go but I am also using an auto update feature (courtesy of WOK) where every set amount of time, array would be flushed refilled with fresh data and sent over again for editing the original message through the message.edit() method.
It works just fine but I am forseeing that my array might get bigger over time and sending a single message may break things because of the API max character limit.
const getText = () => {
return assignedPrint
+ greengo + 'Updating in ' + counter + 's...' + greenstop;
};
const updateCounter = async (message) => {
message.edit(getText());
counter -= seconds;
if (counter <= 0) {
counter = startingCounter;
// emptying the array before fetching again for the message edit
assignedPrint = [];
await fetchData();
}
setTimeout(() => {
updateCounter(message);
}, 1000 * seconds);
};
module.exports = async (bot) => {
await fetchData();
const message = await bot.channels.cache.get(tgt).send(getText());
updateCounter(message);
};
As you can see, the hurdle is that getText()includes everything.
I tried sending one element as a time using for (cont e of assignedPrint) and it worked but how can I edit every message upon refreshing data, knowing that new data may be added or some already sent data could be removed...
The easiest of course is to do it in one single message but again, it may hit the quota and cause a crash.
Thanks.

JavaScript function blocks web socket & causes sync issue & delay

I have a web socket that receives data from a web socket server every 100 to 200ms, ( I have tried both with a shared web worker as well as all in the main.js file),
When new JSON data arrives my main.js runs filter_json_run_all(json_data) which updates Tabulator.js & Dygraph.js Tables & Graphs with some custom color coding based on if values are increasing or decreasing
1) web socket json data ( every 100ms or less) -> 2) run function filter_json_run_all(json_data) (takes 150 to 200ms) -> 3) repeat 1 & 2 forever
Quickly the timestamp of the incoming json data gets delayed versus the actual time (json_time 15:30:12 vs actual time: 15:31:30) since the filter_json_run_all is causing a backlog in operations.
So it causes users on different PC's to have websocket sync issues, based on when they opened or refreshed the website.
This is only caused by the long filter_json_run_all() function, otherwise if all I did was console.log(json_data) they would be perfectly in sync.
Please I would be very very grateful if anyone has any ideas how I can prevent this sort of blocking / backlog of incoming JSON websocket data caused by a slow running javascript
function :)
I tried using a shared web worker which works but it doesn't get around the delay in main.js blocked by filter_json_run_all(), I dont thing I can put filter_json_run_all() since all the graph & table objects are defined in main & also I have callbacks for when I click on a table to update a value manually (Bi directional web socket)
If you have any ideas or tips at all I will be very grateful :)
worker.js:
const connectedPorts = [];
// Create socket instance.
var socket = new WebSocket(
'ws://'
+ 'ip:port'
+ '/ws/'
);
// Send initial package on open.
socket.addEventListener('open', () => {
const package = JSON.stringify({
"time": 123456,
"channel": "futures.tickers",
"event": "subscribe",
"payload": ["BTC_USD", "ETH_USD"]
});
socket.send(package);
});
// Send data from socket to all open tabs.
socket.addEventListener('message', ({ data }) => {
const package = JSON.parse(data);
connectedPorts.forEach(port => port.postMessage(package));
});
/**
* When a new thread is connected to the shared worker,
* start listening for messages from the new thread.
*/
self.addEventListener('connect', ({ ports }) => {
const port = ports[0];
// Add this new port to the list of connected ports.
connectedPorts.push(port);
/**
* Receive data from main thread and determine which
* actions it should take based on the received data.
*/
port.addEventListener('message', ({ data }) => {
const { action, value } = data;
// Send message to socket.
if (action === 'send') {
socket.send(JSON.stringify(value));
// Remove port from connected ports list.
} else if (action === 'unload') {
const index = connectedPorts.indexOf(port);
connectedPorts.splice(index, 1);
}
});
Main.js This is only part of filter_json_run_all which continues on for about 6 or 7 Tabulator & Dygraph objects. I wante to give an idea of some of the operations called with SetTimeout() etc
function filter_json_run_all(json_str){
const startTime = performance.now();
const data_in_array = json_str //JSON.parse(json_str.data);
// if ('DATETIME' in data_in_array){
// var milliseconds = (new Date()).getTime() - Date.parse(data_in_array['DATETIME']);
// console.log("milliseconds: " + milliseconds);
// }
if (summary in data_in_array){
if("DATETIME" in data_in_array){
var time_str = data_in_array["DATETIME"];
element_time.innerHTML = time_str;
}
// summary Data
const summary_array = data_in_array[summary];
var old_sum_arr_krw = [];
var old_sum_arr_irn = [];
var old_sum_arr_ntn = [];
var old_sum_arr_ccn = [];
var old_sum_arr_ihn = [];
var old_sum_arr_ppn = [];
var filtered_array_krw_summary = filterByProperty_summary(summary_array, "KWN")
old_sum_arr_krw.unshift(Table_summary_krw.getData());
Table_summary_krw.replaceData(filtered_array_krw_summary);
//Colour table
color_table(filtered_array_krw_summary, old_sum_arr_krw, Table_summary_krw);
var filtered_array_irn_summary = filterByProperty_summary(summary_array, "IRN")
old_sum_arr_irn.unshift(Table_summary_inr.getData());
Table_summary_inr.replaceData(filtered_array_irn_summary);
//Colour table
color_table(filtered_array_irn_summary, old_sum_arr_irn, Table_summary_inr);
var filtered_array_ntn_summary = filterByProperty_summary(summary_array, "NTN")
old_sum_arr_ntn.unshift(Table_summary_twd.getData());
Table_summary_twd.replaceData(filtered_array_ntn_summary);
//Colour table
color_table(filtered_array_ntn_summary, old_sum_arr_ntn, Table_summary_twd);
// remove formatting on fwds curves
setTimeout(() => {g_fwd_curve_krw.updateOptions({
'file': dataFwdKRW,
'labels': ['Time', 'Bid', 'Ask'],
strokeWidth: 1,
}); }, 200);
setTimeout(() => {g_fwd_curve_inr.updateOptions({
'file': dataFwdINR,
'labels': ['Time', 'Bid', 'Ask'],
strokeWidth: 1,
}); }, 200);
// remove_colors //([askTable_krw, askTable_inr, askTable_twd, askTable_cny, askTable_idr, askTable_php])
setTimeout(() => { askTable_krw.getRows().forEach(function (item, index) {
row = item.getCells();
row.forEach(function (value_tmp){value_tmp.getElement().style.backgroundColor = '';}
)}); }, 200);
setTimeout(() => { askTable_inr.getRows().forEach(function (item, index) {
row = item.getCells();
row.forEach(function (value_tmp){value_tmp.getElement().style.backgroundColor = '';}
)}); }, 200);
color_table Function
function color_table(new_arr, old_array, table_obj){
// If length is not equal
if(new_arr.length!=old_array[0].length)
console.log("Diff length");
else
{
// Comparing each element of array
for(var i=0;i<new_arr.length;i++)
//iterate old dict dict
for (const [key, value] of Object.entries(old_array[0][i])) {
if(value == new_arr[i][key])
{}
else{
// console.log("Different element");
if(key!="TENOR")
// console.log(table_obj)
table_obj.getRows()[i].getCell(key).getElement().style.backgroundColor = 'yellow';
if(key!="TIME")
if(value < new_arr[i][key])
//green going up
//text_to_speech(new_arr[i]['CCY'] + ' ' +new_arr[i]['TENOR']+ ' getting bid')
table_obj.getRows()[i].getCell(key).getElement().style.backgroundColor = 'Chartreuse';
if(key!="TIME")
if(value > new_arr[i][key])
//red going down
table_obj.getRows()[i].getCell(key).getElement().style.backgroundColor = 'Crimson';
}
}
}
}
Potential fudge / solution, thanks Aaron :):
function limiter(fn, wait){
let isCalled = false,
calls = [];
let caller = function(){
if (calls.length && !isCalled){
isCalled = true;
if (calls.length >2){
calls.splice(0,calls.length-1)
//remove zero' upto n-1 function calls from array/ queue
}
calls.shift().call();
setTimeout(function(){
isCalled = false;
caller();
}, wait);
}
};
return function(){
calls.push(fn.bind(this, ...arguments));
// let args = Array.prototype.slice.call(arguments);
// calls.push(fn.bind.apply(fn, [this].concat(args)));
caller();
};
}
This is then defined as a constant for a web worker to call:
const filter_json_run_allLimited = limiter(data => { filter_json_run_all(data); }, 300); // 300ms for examples
Web worker calls the limited function when new web socket data arrives:
// Event to listen for incoming data from the worker and update the DOM.
webSocketWorker.port.addEventListener('message', ({ data }) => {
// Limited function
filter_json_run_allLimited(data);
});
Please if anyone knows how websites like tradingview or real time high performance data streaming sites allow for low latency visualisation updates, please may you comment, reply below :)

I'm reticent to take a stab at answering this for real without knowing what's going on in color_table. My hunch, based on the behavior you're describing is that filter_json_run_all is being forced to wait on a congested DOM manipulation/render pipeline as HTML is being updated to achieve the color-coding for your updated table elements.
I see you're already taking some measures to prevent some of these DOM manipulations from blocking this function's execution (via setTimeout). If color_table isn't already employing a similar strategy, that'd be the first thing I'd focus on refactoring to unclog things here.
It might also be worth throwing these DOM updates for processed events into a simple queue, so that if slow browser behavior creates a rendering backlog, the function actually responsible for invoking pending DOM manipulations can elect to skip outdated render operations to keep the UI acceptably snappy.
Edit: a basic queueing system might involve the following components:
The queue, itself (this can be a simple array, it just needs to be accessible to both of the components below).
A queue appender, which runs during filter_json_run_all, simply adding objects to the end of the queue representing each DOM manipulation job you plan to complete using color_table or one of your setTimeout` callbacks. These objects should contain the operation to performed (i.e: the function definition, uninvoked), and the parameters for that operation (i.e: the arguments you're passing into each function).
A queue runner, which runs on its own interval, and invokes pending DOM manipulation tasks from the front of the queue, removing them as it goes. Since this operation has access to all of the objects in the queue, it can also take steps to optimize/combine similar operations to minimize the amount of repainting it's asking the browser to do before subsequent code can be executed. For example, if you've got several color_table operations that coloring the same cell multiple times, you can simply perform this operation once with the data from the last color_table item in the queue involving that cell. Additionally, you can further optimize your interaction with the DOM by invoking the aggregated DOM manipulation operations, themselves, inside a requestAnimationFrame callback, which will ensure that scheduled reflows/repaints happen only when the browser is ready, and is preferable from a performance perspective to DOM manipulation queueing via setTimeout/setInterval.

How to make dialogflow going while waiting for a process that takes more than the time limit?

I have created an API that takes user input and process it. However, the process takes more than 5 seconds (dialogflow limit).
How can I continue with other processes until this certain process is finished?
Or is it possible to return to the user any messages like "Please hold on a bit..." so it can restart the time.
var message = "hi"; //test purpose
async function GetCertain_info(agent) {
await uiFx();
agent.add("Here are the information on " + message);
}
async function uiFx() {
var {
ui
} = require('./uia.js');
return new Promise(function(resolve, reject) {
ui().then((msg) => {
console.log("Destination Message : " + msg)
message = msg;
resolve(message);
}).catch((msg) => {
console.log(msg)
message = msg;
reject(message);
})
});
}
Appreciate your help

Yes, it is possible to return to the user any messages like "Please hold on a bit…" by setting up a FollowupEvent.
You can extend the 5 seconds Intent limit up to 15 seconds by setting up multiple follow-up events. Currently, you can only set up 3 follow-up events one after another (which can extend the timeout up to 15 seconds).
Here's an example of how you can do it in the fulfillment:
function function1(agent){
//This function handles your intent fulfillment
//you can initialize your db query here.
//When data is found, store it in a separate table for quick search
//get current date
var currentTime = new Date().getTime();
while (currentTime + 4500 >= new Date().getTime()) {
/*waits for 4.5 seconds
You can check every second if data is available in the database
if not, call the next follow up event and do the
same while loop in the next follow-up event
(up to 3 follow up events)
*/
/*
if(date.found){
agent.add('your data here');//Returns response to user
}
*/
}
//add a follow-up event
agent.setFollowupEvent('customEvent1');
//add a default response (in case there's a problem with the follow-up event)
agent.add("This is function1");
}
let intentMap = new Map();
intentMap.set('Your intent name here', function1);;
agent.handleRequest(intentMap);

Web BLE Characteristic startNotifications sometimes doesn't bind

I'm using web BLE. I have based my code according to the example of the heart rate measurement.
Everything is working fine most of the time. But sometimes, even if the connection is successfully made, when I try to bind to the notification, it doesn't work.
The link is made in this function :
_startNotifications(characteristicUuid) {
let characteristic = this._characteristics.get(characteristicUuid);
console.log(characteristic);
return characteristic.startNotifications().then(() => characteristic);
}
When everything is OK, I can see in the console that BluetoothRemoteGATTCharacteristic has a value : DataView(2) {}
Otherwise, when it's not working it has a value : null
I would like to be able to retry automatically, if I detect that the value is null. But I'm not familiar with Promise (I think this is it) and console.log(characteristic.value) doesn't work here.
How would you approach this ?

What I ended up doing is "bypass" the issue. So it's a more algorithmic resolution than a pure Javascript one.
I didn't change the connection function, so it is still called like this :
device._startNotifications(some_uuid).then(handleHeartRateMeasurement)
I check everything in the handleHeartRateMeasurement function :
var ready = false;
function handleHeartRateMeasurement(heartRateMeasurement) {
console.log("Hold on...");
heartRateMeasurement.addEventListener("characteristicvaluechanged", event => {
// Everytime the value change, this should be triggered
// If it did not, variable "ready" will stay false
ready = true;
var value = device.parseValue(event.target.value);
// Do something with value here
});
var check = function(){
// If we have received data from characteristic, we are ready to go !
if(ready === false){
console.log("Device connected but not receiving data");
// Stop the current notification subscription
device.stopNotificationsHeartRateMeasurement();
// Start a new one
device._startNotifications(some_uuid).then(handleHeartRateMeasurement);
setTimeout(check, 1000); // check again in a 1s
}
else{
console.log("Device connected and receiving data");
}
}
setTimeout(() => {
check();
}, 500);
}

Discord.js - Getting users last activity?

I'm trying to find out if its possible to get the time/information of users last activity retrospectively using discord.js
Say I have something like
client.guilds.find('id', 'SERVER ID').fetchMembers().then(members => {
const role = members.roles.find('name', 'Newbies')
role.members.forEach(member => {
console.log(member.user.lastMessage) // null
})
})
Unless the member has posted, since the client is listening, the lastMessage is always null.
Is there a way to find the last activity? or a workaround, like a query to return all the users messages, which I can then take the most recent one from?
Effectively I want to know what date/time the user last posted so we can monitor non-contributing accounts.
Thanks

After looking thought the documentation I didn't find something neither so I came up with a manual search function.
Basically, it will scan every channels until finding a message from X user, or the end of the messages in the channel. It then compare the last messages of the users from every channels and print the last one.
It can be very long if the user hasn't write since a long time. Of course, you have to check lastMessage before trying this.
I would add a time limit maybe. Because if you have thousand of messages, the function will run eternally.
You can stop the function if the last message found is in the accepted time to not be kick/do whatever.
I made the search stop if the first message found in the pack of fetched messaged is older than the ban limit, however, if the first message is not older, remember that it means for the other, so we still need to check them (it can be avoided by checking the last message of the pack as well).
async function fetchMessageUser(chan, id, res) {
let option = {};
if (typeof res !== 'undefined'){
option = {before: res.id};
}
return await chan.fetchMessages(option)
.then(async msgs => {
if (msgs.size === 0){
return {continue: false, found: false};
};
if ((Date.now() - (msgs.first().createdTimestamp)) > 86400000 ) { // 1 day
return {continue: false, found: false};
}
let msgByAuthor = msgs.find(msg => {
return msg.author.id === id;
});
if (msgByAuthor === null){
return {continue: true, id: msgs.last().id};
} else {
return {continue: false, found: true, timestamp: msgByAuthor.createdTimestamp};
}
})
.catch(err => console.log('ERR>>', err));
}
client.on('message', async (msg) => {
let timestamp = [];
for (let [id, chan] of msg.guild.channels){
if (chan.type !== 'text'){ continue; }
let id = '587692527826763788'; // id of the user, here a non verified account
let res;
do {
res = await fetchMessageUser(chan, id, res);
} while(res.continue);
if (res.found) {
timestamp.push(res.timestamp);
}
}
console.log(timestamp);
let first = timestamp.sort((a,b) => (b-a))[0];
console.log(new Date(first));
});
A better variant would be to run it for an array of users, checking all 50 last messages from every channel, and associating each users with his most recent messages if he wrote one, and doing this until all the messages in all the channels are too old to avoid a kick/whatever. And then do something for all the users who don't have an associated messages.

I think what you need is one of Discord's built in features, namely: pruning. This feature will grab inactive members and lets you kick them. Luckily, discord.js has an API call for it and even lets you get the number of members first without actually kicking them by setting the dry parameter to true. The feature will also allow you to specify the amount of days a user has to be inactive.
Have a look at the docs: https://discord.js.org/#/docs/main/stable/class/Guild?scrollTo=pruneMembers
Hope that helps out!