I'm trying to add some functionality for rabbitmq with delay messages. Actually I need to get this message after 2 weeks. As I know we do not need any plugin. Also when this message invokes, how should I reschedule new x delay exchanger to invoke again over 2 weeks. Where shoul I added this x delay message.
config
"messageQueue": {
"connectionString": "amqp://guest:guest#localhost:5672?heartbeat=5",
"queueName": "history",
"exchange": {
"type": "headers",
"prefix": "history."
},
"reconnectTimeout": 5000
},
service:
import amqplib from 'amqplib'
import config from 'config'
import logger from './logger'
const {reconnectTimeout, connectionString, exchange: {prefix, type: exchangeType}, queueName} = config.messageQueue
const onConsume = (expectedMessages, channel, onMessage) => async message => {
const {fields: {exchange}, properties: {correlationId, replyTo}, content} = message
logger.silly(`consumed message from ${exchange}`)
const messageTypeName = exchange.substring(exchange.startsWith(prefix) ? prefix.length : 0)
const messageType = expectedMessages[messageTypeName]
if (!messageType) {
logger.warn(`Unexpected message of type ${messageTypeName} received. The service only accepts messages of types `, Object.keys(expectedMessages))
return
}
const deserializedMessage = messageType.decode(content)
const object = deserializedMessage.toJSON()
const result = await onMessage(messageTypeName, object)
if (correlationId && replyTo) {
const {type, response} = result
const encoded = type.encode(response).finish()
channel.publish('', replyTo, encoded, {correlationId})
}
}
const startService = async (expectedMessages, onMessage) => {
const restoreOnFailure = e => {
logger.warn('connection with message bus lost due to error', e)
logger.info(`reconnecting in ${reconnectTimeout} milliseconds`)
setTimeout(() => startService(expectedMessages, onMessage), reconnectTimeout)
}
const exchanges = Object.keys(expectedMessages).map(m => `${prefix}${m}`)
try {
const connection = await amqplib.connect(connectionString)
connection.on('error', restoreOnFailure)
const channel = await connection.createChannel()
const handleConsume = onConsume(expectedMessages, channel, onMessage)
const queue = await channel.assertQueue(queueName)
exchanges.forEach(exchange => {
channel.assertExchange(exchange, exchangeType, {durable: true})
channel.bindQueue(queue.queue, exchange, '')
})
logger.debug(`start listening messages from ${exchanges.join(', ')}`)
channel.consume(queue.queue, handleConsume, {noAck: true})
}
catch (e) {
logger.warn('error while subscribing for messages message', e)
restoreOnFailure(e)
}
}
export default startService
RabbitMQ has a plug-in for scheduling messages. You can use it, subject to an important design caveat which I explain below.
Use Steps
You must first install it:
rabbitmq-plugins enable rabbitmq_delayed_message_exchange
Then, you have to set up a delayed exchange:
Map<String, Object> args = new HashMap<String, Object>();
args.put("x-delayed-type", "direct");
channel.exchangeDeclare("my-exchange", "x-delayed-message", true, false, args);
Finally, you can set the x-delay parameter (where delay is in milliseconds).
byte[] messageBodyBytes = "delayed payload".getBytes();
AMQP.BasicProperties.Builder props = new AMQP.BasicProperties.Builder();
headers = new HashMap<String, Object>();
headers.put("x-delay", 5000);
props.headers(headers);
channel.basicPublish("my-exchange", "", props.build(), messageBodyBytes);
Two weeks is equal to (7*24*60*60*1000 = 604,800,000) milliseconds.
Important Caveat
As I explained in this answer, this is a really bad thing to ask the message broker to do.
It's important to keep in mind, when dealing with message queues, they perform a very specific function in a system: to hold messages while the processor(s) are busy processing earlier messages. It is expected that a properly-functioning message queue will deliver messages as soon as reasonable. Basically, the fundamental expectation is that as soon as a message reaches the head of the queue, the next pull on the queue will yield the message -- no delay.
Delay becomes a result of how a system with a queue processes messages. In fact, Little's Law offers some interesting insights into this. If you're going to stick an arbitrary delay in there, you really have no need of a message queue to begin with - all your work is scheduled up front.
So, in a system where a delay is necessary (for example, to join/wait for a parallel operation to complete), you should be looking at other methods. Typically a queryable database would make sense in this particular instance. If you find yourself keeping messages in a queue for a pre-set period of time, you're actually using the message queue as a database - a function it was not designed to provide. Not only is this risky, but it also has a high likelihood of hurting the performance of your message broker.
Related
I'm trying to save a value of the server being "Down" or "Up", it seems to be working fine, however, when I clear the console, then run the bot again, it looks to have reset the database fully, is there a way to make the database, work like a normal database would? (Save after restart)
I'm having a check to see if the server is up, then don't allow it to run the command and say it's already online.
I wasn't like this before, and I haven't used quick.db for a while, so sorry if I missed anything.
Code:
// Main requirements
const { ROLE_SSUPIN, CHANNEL_SSU, CHANNEL_ROLESEL, BOT_PREFIX } = require('../../config.json')
const commandName = 'ssu'
// Optional Requirements
const { MessageEmbed } = require('discord.js')
const { QuickDB } = require('quick.db');
const db = new QuickDB();
// Main Code
module.exports = async client => {
client.on('messageCreate', async message => {
if (message.author.bot) return;
if (!message.content.toLowerCase().startsWith(BOT_PREFIX)) return;
const command = message.content.split(' ')
const args = message.content.substring(1).split(' ')
if (command.toString().toLowerCase().replace(BOT_PREFIX, '').startsWith(commandName)) {
const ssuStatus = await db.get("ssu.status")
await db.set("ssu.status", "down");
console.log(ssuStatus)
if (!message.member.roles.cache.get('998320362234323026')) return message.reply("Only staff with `SSU Permissions` can use this command!")//.then(m => {setTimeout( function() { m.delete(), message.delete() }, 10000)});
if (!message.channel.id == CHANNEL_SSU) return message.reply(`Please use this command in <#${CHANNEL_SSU}>.`)
if (ssuStatus == 'up') return message.reply(`Server Status is already set to Online. Say \`${BOT_PREFIX}Server Status Set Offline\` if you believe this is a mistake.`)
message.channel.send(`
Start up done.
`)
.then(await db.set("ssu.status", "up"))
}
})
}
I had "await db.set("ssu.status", "down");" in the start of each function, so it was reseting it back to "Down", I didn't mean for it do that, if you have the same issue, make sure that you don't have anything that sets the database value in the start, as it might be the reason it's getting reset and you think it's wiping.
How to set MAX_ATTEMPT of the tasks in google cloud queue in code?
When I create new task, I want to set how many repetitions of a given task should be, can I do it from the code below?
I have google cloud queue like here:
const {CloudTasksClient} = require('#google-cloud/tasks');
const client = new CloudTasksClient();
async function createHttpTask() {
const project = 'my-project-id';
const queue = 'my-queue';
const location = 'us-central1';
const url = 'https://example.com/taskhandler';
const payload = 'Hello, World!';
const inSeconds = 180;
const parent = client.queuePath(project, location, queue);
const task = {
httpRequest: {
httpMethod: 'POST',
url,
},
};
if (payload) {
task.httpRequest.body = Buffer.from(payload).toString('base64');
}
console.log('Sending task:');
console.log(task);
const request = {parent: parent, task: task};
await client.createTask(request);
}
createHttpTask();
From the Google Cloud Documentation i see that I can do it from the console, for the whole queue - https://cloud.google.com/tasks/docs/configuring-queues#retry , but I want to set this dynamically for the tasks
thanks for any help!
Unfortunately, the answer is no. You can not set retry parameters on individual tasks. I make this claim by looking at this document:
REST Resource: projects.locations.queues
which is the REST API for creating task queues. In there, under the documentation for retryConfig we read:
For tasks created using Cloud Tasks: the queue-level retry settings apply to all tasks in the queue that were created using Cloud Tasks. Retry settings cannot be set on individual tasks.
I'm making a discord.js scheduling bot. I am using node-schedule for this. It's not throwing any errors but it's still not sending a message. What am I doing wrong, and how do I get rid of this issue? (thank you in advance)
My code is:
const Discord = require('discord.js');
const client = new Discord.Client();
const schedule = require('node-schedule');
client.once('ready', () => {
console.log('Ready!');
});
client.login('TOKEN IS HERE');
const rule = new schedule.RecurrenceRule();
rule.tz = 'EDT';
client.on('message', message => {
if (message.content === '!schedule 9pm meeting') {
message.channel.send('Alright. I will announce it for you, friend! :smiley:');
const job = schedule.scheduleJob('00 21 * * *', function () {
client.channels.cache.get("channel id is here").send("This is a test message. Does it work?");
});
}
});
You can't run the schedule.scheduleJob from inside the client.on function and expect the message to still exist. Discord API expects a response to a webhook within a specific time before it times out.
Also, if the bot runs on a cloud service, the node it runs on might be restarting once in a while, which messes up in-memory data like attaching cron jobs in node-schedule.
Persisting the data
You should probably get scheduled time by the user and persist the data in some sort of database. You should use database read\writes in order to save the data between your cloud provider restarts (unless you have a paid subscription).
Have a global cron job or interval
Since you can potentialy have thousands of scheduled meetings, it's better in your case to check for meetings withing a certain interval and send all the reminders at the same time.
Let's say a user can't give us a time more specific than a certain minute. Then we can check for reminders every minute, knowing we'll inform users before the meeting starts.
// Run checkForReminders every 60 seconds to scan DB for outstanding reminders
setInterval(checkForReminders, 60000);
// Parse reminder request, save to DB, DM confirmation to user
client.on('message', (msg) => {
createNewReminder(msg);
});
New reminders handling
const createNewReminder = (msg) => {
const formattedMessage = formatMessage(msg)
// If message isn't a remindme command, cease function execution
if (!formattedMessage.startsWith('!remindme')) {
return
}
// Determine if message contains a number to assign to intervalInteger
checkForNumbersInMessage(formattedMessage)
// Final format for message to be sent at reminderTime
const messageToDeliverToUser = formattedMessage.replace('!remindme', '')
// Set integer and verb values for moment.add() parameters
const intervalInteger = parseInt(checkForNumbersInMessage(formattedMessage))
const intervalVerb = setIntervalVerb(formattedMessage)
// Format time to send reminder to UTC timezone
const reminderTime = moment().utc().add(intervalInteger, intervalVerb).format('YYYY-MM-DD HH:mm:ss')
// Save reminder to DB
saveNewReminder(msg.author.id, msg.author.username, messageToDeliverToUser, reminderTime)
// Send embedded message to author & notify author in channel of remindertime request
const embed = createEmbed(messageToDeliverToUser, reminderTime)
msg.author.send(embed)
msg.channel.send(`${msg.author} - A reminder confirmation has been sent to your DMs. I will DM you again at the requested reminder time`)
}
Send a message to a guild or user later
In order to send a message later, save either the userId or guildId to the database, then, retrieve the user or guild from the discord client, and send the message.
const checkForReminders = () => {
db.serialize(() => {
// Select all messages older than the current dateTime
db.each("SELECT id, reminderTime, userID, message FROM reminders WHERE reminderTime < DATETIME()", (error, row) => {
if (error || !row) {
return console.log('Error or no row found')
}
// If reminders are found, fetch userIDs, then send the requested reminder through DM
client.users.fetch(row.userID).then((user) => {
user.send(`Hi, you asked to be reminded "${row.message}" - That's right now!`).catch(console.error)
console.log(`Message delivered: ${row.message}`)
console.log(`Message deleted successfully`)
// Delete message after DMing to user
db.run("DELETE FROM reminders WHERE id = ?", [row.id])
console.log('Message sent and removed successfully')
})
})
})
}
Code examples where taken from NathanDennis/discord-reminder-bot. check out the repository for a complete example. He comments on his code so it's easy to understand.
I'd like to be able to end a Google speech-to-text stream (created with streamingRecognize), and get back the pending SR (speech recognition) results.
In a nutshell, the relevant Node.js code:
// create SR stream
const stream = speechClient.streamingRecognize(request);
// observe data event
const dataPromise = new Promise(resolve => stream.on('data', resolve));
// observe error event
const errorPromise = new Promise((resolve, reject) => stream.on('error', reject));
// observe finish event
const finishPromise = new Promise(resolve => stream.on('finish', resolve));
// send the audio
stream.write(audioChunk);
// for testing purposes only, give the SR stream 2 seconds to absorb the audio
await new Promise(resolve => setTimeout(resolve, 2000));
// end the SR stream gracefully, by observing the completion callback
const endPromise = util.promisify(callback => stream.end(callback))();
// a 5 seconds test timeout
const timeoutPromise = new Promise(resolve => setTimeout(resolve, 5000));
// finishPromise wins the race here
await Promise.race([
dataPromise, errorPromise, finishPromise, endPromise, timeoutPromise]);
// endPromise wins the race here
await Promise.race([
dataPromise, errorPromise, endPromise, timeoutPromise]);
// timeoutPromise wins the race here
await Promise.race([dataPromise, errorPromise, timeoutPromise]);
// I don't see any data or error events, dataPromise and errorPromise don't get settled
What I experience is that the SR stream ends successfully, but I don't get any data events or error events. Neither dataPromise nor errorPromise gets resolved or rejected.
How can I signal the end of my audio, close the SR stream and still get the pending SR results?
I need to stick with streamingRecognize API because the audio I'm streaming is real-time, even though it may stop suddenly.
To clarify, it works as long as I keep streaming the audio, I do receive the real-time SR results. However, when I send the final audio chunk and end the stream like above, I don't get the final results I'd expect otherwise.
To get the final results, I actually have to keep streaming silence for several more seconds, which may increase the ST bill. I feel like there must be a better way to get them.
Updated: so it appears, the only proper time to end a streamingRecognize stream is upon data event where StreamingRecognitionResult.is_final is true. As well, it appears we're expected to keep streaming audio until data event is fired, to get any result at all, final or interim.
This looks like a bug to me, filing an issue.
Updated: it now seems to have been confirmed as a bug. Until it's fixed, I'm looking for a potential workaround.
Updated: for future references, here is the list of the current and previously tracked issues involving streamingRecognize.
I'd expect this to be a common problem for those who use streamingRecognize, surprised it hasn't been reported before. Submitting it as a bug to issuetracker.google.com, as well.
My bad — unsurprisingly, this turned to be an obscure race condition in my code.
I've put together a self-contained sample that works as expected (gist). It helped me tracking down the issue. Hopefully, it may help others and my future self:
// A simple streamingRecognize workflow,
// tested with Node v15.0.1, by #noseratio
import fs from 'fs';
import path from "path";
import url from 'url';
import util from "util";
import timers from 'timers/promises';
import speech from '#google-cloud/speech';
export {}
// need a 16-bit, 16KHz raw PCM audio
const filename = path.join(path.dirname(url.fileURLToPath(import.meta.url)), "sample.raw");
const encoding = 'LINEAR16';
const sampleRateHertz = 16000;
const languageCode = 'en-US';
const request = {
config: {
encoding: encoding,
sampleRateHertz: sampleRateHertz,
languageCode: languageCode,
},
interimResults: false // If you want interim results, set this to true
};
// init SpeechClient
const client = new speech.v1p1beta1.SpeechClient();
await client.initialize();
// Stream the audio to the Google Cloud Speech API
const stream = client.streamingRecognize(request);
// log all data
stream.on('data', data => {
const result = data.results[0];
console.log(`SR results, final: ${result.isFinal}, text: ${result.alternatives[0].transcript}`);
});
// log all errors
stream.on('error', error => {
console.warn(`SR error: ${error.message}`);
});
// observe data event
const dataPromise = new Promise(resolve => stream.once('data', resolve));
// observe error event
const errorPromise = new Promise((resolve, reject) => stream.once('error', reject));
// observe finish event
const finishPromise = new Promise(resolve => stream.once('finish', resolve));
// observe close event
const closePromise = new Promise(resolve => stream.once('close', resolve));
// we could just pipe it:
// fs.createReadStream(filename).pipe(stream);
// but we want to simulate the web socket data
// read RAW audio as Buffer
const data = await fs.promises.readFile(filename, null);
// simulate multiple audio chunks
console.log("Writting...");
const chunkSize = 4096;
for (let i = 0; i < data.length; i += chunkSize) {
stream.write(data.slice(i, i + chunkSize));
await timers.setTimeout(50);
}
console.log("Done writing.");
console.log("Before ending...");
await util.promisify(c => stream.end(c))();
console.log("After ending.");
// race for events
await Promise.race([
errorPromise.catch(() => console.log("error")),
dataPromise.then(() => console.log("data")),
closePromise.then(() => console.log("close")),
finishPromise.then(() => console.log("finish"))
]);
console.log("Destroying...");
stream.destroy();
console.log("Final timeout...");
await timers.setTimeout(1000);
console.log("Exiting.");
The output:
Writting...
Done writing.
Before ending...
SR results, final: true, text: this is a test I'm testing voice recognition This Is the End
After ending.
data
finish
Destroying...
Final timeout...
close
Exiting.
To test it, a 16-bit/16KHz raw PCM audio file is required. An arbitrary WAV file wouldn't work as is because it contains a header with metadata.
This: "I'm looking for a potential workaround." - have you considered extending from SpeechClient as a base class? I don't have credential to test, but you can extend from SpeechClient with your own class and then call the internal close() method as needed. The close() method shuts down the SpeechClient and resolves the outstanding Promise.
Alternatively you could also Proxy the SpeechClient() and intercept/respond as needed. But since your intent is to shut it down, the below option might be your workaround.
const speech = require('#google-cloud/speech');
class ClientProxy extends speech.SpeechClient {
constructor() {
super();
}
myCustomFunction() {
this.close();
}
}
const clientProxy = new ClientProxy();
try {
clientProxy.myCustomFunction();
} catch (err) {
console.log("myCustomFunction generated error: ", err);
}
Since it's a bug, I don't know if this is suitable for you but I have used this.recognizeStream.end(); several times in different situations and it worked. However, my code was a bit different...
This feed may be something for you:
https://groups.google.com/g/cloud-speech-discuss/c/lPaTGmEcZQk/m/Kl4fbHK2BQAJ
I am trying to write a chat application using WebRTC and I can send the messages over the dataChannel using a code like bellow one:
const peerConnection = new RTCPeerConnection();
const dataChannel =
peerConnection.createDataChannel("myLabel", dataChannelOptions);
dataChannel.onerror = (error) => {
console.log("Data Channel Error:", error);
};
dataChannel.onmessage = (event) => {
console.log("Got Data Channel Message:", event.data);
};
dataChannel.onopen = () => {
dataChannel.send("Hello World!");
};
dataChannel.onclose = () => {
console.log("The Data Channel is Closed");
};
with dataChannel.send() I can send data over the channel correctly. but I am wondering to know, is there any way to determine that the sent message is delivered to another side or not?
This simplest answer is: send a reply.
But you may not need to, if you use an ordered, reliable datachannel (which is the default).
An ordered reliable datachannel
With one of these, you determine a message was sent, by waiting for the bufferedAmount to go down:
const pc1 = new RTCPeerConnection(), pc2 = new RTCPeerConnection();
const channel = pc1.createDataChannel("chat");
chat.onkeypress = async e => {
if (e.keyCode != 13) return;
const before = channel.bufferedAmount;
channel.send(chat.value);
const after = channel.bufferedAmount;
console.log(`Queued ${after - before} bytes`);
channel.bufferedAmountLowThreshold = before; // set floor trigger and wait
await new Promise(r => channel.addEventListener("bufferedamountlow", r));
console.log(`Sent ${after - channel.bufferedAmount} bytes`);
chat.value = "";
};
pc2.ondatachannel = e => e.channel.onmessage = e => console.log(`> ${e.data}`);
pc1.onicecandidate = e => pc2.addIceCandidate(e.candidate);
pc2.onicecandidate = e => pc1.addIceCandidate(e.candidate);
pc1.oniceconnectionstatechange = e => console.log(pc1.iceConnectionState);
pc1.onnegotiationneeded = async e => {
await pc1.setLocalDescription(await pc1.createOffer());
await pc2.setRemoteDescription(pc1.localDescription);
await pc2.setLocalDescription(await pc2.createAnswer());
await pc1.setRemoteDescription(pc2.localDescription);
}
Chat: <input id="chat"><br>
Since the channel is reliable, it won't give up sending this message until it's been received.
Since the channel is ordered, it won't send a second message before this message has been received.
This lets you send a bunch of messages in a row without waiting on a reply. As long as the bufferedAmount keeps goes down, you know it's being sent and received.
In short, to determine a message was received, send a second message, or have the other side send a reply.
An unreliable datachannel
If you're using an unreliable datachannel, then sending a reply is the only way. But since there's no guarantee the reply will make it back, this may produce false negatives, causing duplicate messages on the receiving end.
Unreliable one way, reliable the other
Using the negotiated constructor argument, it's possible to create a datachannel that's unreliable in one direction, yet reliable in the other. This can be used to solve the unreliable reply, to avoid duplicate messages on the receiving (pc2) end.
dc1 = pc1.createDataChannel("chat", {negotiated: true, id: 0, maxRetransmits: 0});
dc2 = pc2.createDataChannel("chat", {negotiated: true, id: 0});