I have a node.js Function App which I'm trying to use to retrieve a specific message from a storage queue in order to delete it. It doesnt seem like there is a method to get a specific message by ID, so I'm trying to read all the messages, or at least 32 of them, then find the message to delete by ID.
My problem is I cant seem to figure out how to format the request to specify the number of messages to receive. I am trying to add the option "numberOfMessages" based on this documentation: https://learn.microsoft.com/en-us/javascript/api/#azure/storage-queue/queuereceivemessageoptions?view=azure-node-latest##azure-storage-queue-queuereceivemessageoptions-customheaders
Here is what I have right now that isn't working:
const response = await queueClient.receiveMessages({"numberOfMessages": 32});
context.log("response: " + JSON.stringify(response.receivedMessageItems));
context.log("received messages length: " + response.receivedMessageItems.length);
response is:
[Information] response: [{"messageId":"XXXXXX","insertedOn":"2022-08-28T06:12:45.000Z","expiresOn":"2022-09-04T06:12:45.000Z","popReceipt":"XXXXXXXXX","nextVisibleOn":"2022-08-30T14:57:37.000Z","dequeueCount":858,"messageText":"XXXXXXX"}]
[Information] received messages length: 1
How should I be formatting the option(s)???
edit:
Here is the complete code of index.js in my function app (with redactions):
module.exports = async function (context, req) {
context.log('JavaScript HTTP trigger function processed a request.');
const message = req.body;
const ackType = message.ackType;
if (ackType == 1){
const { QueueServiceClient, StorageSharedKeyCredential } = require("#azure/storage-queue");
const account = "MY_ACCOUNT_NAME";
const accountKey = "MY_ACCOUNT_KEY";
const sharedKeyCredential = new StorageSharedKeyCredential(account, accountKey);
const queueServiceClient = new QueueServiceClient(
`https://${account}.queue.core.windows.net`,
sharedKeyCredential,
{
retryOptions: { maxTries: 4 } // Retry options
}
);
const queueName = "MY_QUEUE_NAME";
const queueClient = queueServiceClient.getQueueClient(queueName);
const response = await queueClient.receiveMessages({"numofmessages": 32});
context.log("response: " + JSON.stringify(response));
context.log("receivedMessageItems: " + JSON.stringify(response.receivedMessageItems));
context.log("received messages length: " + response.receivedMessageItems.length);
}
context.res = {
status: 200 /* Defaults to 200 */
};
}
Here is the queue I am connecting to:
Related
I am currently trying to implement SendGrid Email Activity into my application. Whenever an event fires I am able to access the info I need. When trying to look at the message corresponding to the message_id I receive the error: NOT FOUND.
I am wondering if there is a solution for this or has it not yet been created on SendGrid and how would I go about accessing the message information based on an engagement or delivery event?
Thanks
const env = require('dotenv').config();
const client = require('#sendgrid/client');
client.setApiKey(process.env.SENDGRID_API_KEY);
const eventStatus = info.event;
const customerAccountNumber = info.customerAccountNumber;
const recipient = info.email;
const messageId = info.sg_message_id;
console.log("message ID:", messageId);
try {
const request = {};
request.method = 'GET';
request.url = '/v3/messages/' + messageId;
client.request(request)
.then(([response, body]) => {
console.log(response.statusCode);
if(response.statusCode == 200) {
const messageInfo = response.body;
const sender = messageInfo.from_email;
const subject = messageInfo.subject;
console.log(eventStatus, customerAccountNumber, sender, recipient, messageId, subject);
}
else {
console.dir("Here");
}
});
}
catch(e){
console.log(e);
}
return("Test");
I believe you're getting a HTTP 404 because the URL you're putting together is wrong. Given messageId = 'foobar123', the following code
request.url = '/v3/messages/' + messageId;
will produce the request URL "/v3/messages/foobar123".
Taking a quick peek at the docs, the URL should be something more like "/v3/messages?limit=10&query={your_query}", with options for {your_query} described here.
I'm assuming you're trying to make a query such as msg_id="filter0307p1las1-16816-5A023E36-1.0". You need to pass this as URL parameter query. You'll also need to URL encode it, since it contains = and " characters. Putting all of this together, I'd try something like:
const query = `msg_id="${messageId}"`;
const request = {};
request.method = 'GET';
request.url = `/v3/messages?query=${encodeURIComponent(query)}`;
client.request(request).then(...)
I am currently working on a weather station that gets data from OpenWeatherMap's API every 10 minutes.
Every 10 seconds the temperature is published via MQTT in the topic 'local/temperature', so that other systems (for example a heater or air conditioner) can do further actions depending on the temperature.
Every 10 minutes, parallel to the new data retrieval, the weather operations are also published, also via MQTT.
Publishing the data every 10 seconds is a requirement of the project, but not important for this case.
The problem I'm stuck on is this: My request to the API of OWM is done in an extra file, which contains a function that should return the data as an object. At the same time the data is stored in a file, so that in case of a network failure the last local status is saved and can still be used.
I already write into the file, the 'reading when offline' functionality will be added later on. I have also noticed that the assembleURL() function is actually unnecessary, but I haven't changed that yet.
I'm still relatively new in JavaScript / Nodejs, but I already have experience in Java and Python, so it may be that I have mixed in something from Java by mistake.
Can someone please explain to me why the object I return in openWeatherMapCall.js is undefined? I'm thankful for every hint.
My file weather-station.js that calls the function getData in openWeatherMapCall.js:
const mqtt = require('mqtt');
const owm = require('./lib/openWeatherMapCall');
const client = mqtt.connect('mqtt://localhost:1885');
const fs = require('fs');
const config = require('../config/config.json');
const owmConfig = config.owm;
let weatherData = owm.getData(owmConfig.city, owmConfig.owmapikey, owmConfig.lang, "metric");
console.log(weatherData); // -> it says undefined
setInterval(_ => {
weatherData = owm.getData(owmConfig.city, owmConfig.owmapikey, owmConfig.lang, "metric");
client.publish("local/condition", toString(weatherData.weatherID));
console.log('successful publish of wID ' + weatherData.weatherID);
}, 600000); //10 min
setInterval(_ => {
client.publish("local/temperature", toString(weatherData.celsius));
console.log('successful publish of ' + weatherData.celsius + ' celsius');
}, 30000); //10 sec
My OWM API call as openWeatherMapCall.js:
const fetch = require('node-fetch');
const fs = require('fs');
const util = require("util");
function assembleURL (city, apiKey, lang, units){
let url = "http://api.openweathermap.org/data/2.5/weather?q=" + city + "&units=" + units + "&lang=" + lang + "&appid=" + apiKey;
console.log("url: " + url);
return url;
}
function getData(city, apiKey, lang, units){
let url = assembleURL(city, apiKey, lang, units )
fetch(url)
.then(function(resp) { return resp.json() }) // Convert data to json
.then(function(data) {
var currentWeather = {
weather: data.weather[0].description,
weatherID: data.weather[0].id,
celsius: Math.round(parseFloat(data.main.temp)),
wind: data.wind.speed,
location: data.name
};
let toString = JSON.stringify(currentWeather);
fs.writeFile('../config/weather.json', toString, err => {
if (err) {
console.log('Error while writing', err)
} else {
console.log('Successful write')
}
})
return currentWeather;
})
.catch( err => {
console.log('caught it!',err);
});
}
module.exports = {getData};
Return fetch response from getData and use then on owm.getData as fetch returns a Promise.
function getData(city, apiKey, lang, units){
let url = assembleURL(city, apiKey, lang, units )
return fetch(url)....
}
And
owm.getData(owmConfig.city, owmConfig.owmapikey, owmConfig.lang, "metric").then((weatherData) => {
console.log(weatherData)
});
So, for a course i'm taking, we're coding a UDP pinger in Javascript, using Node.js and Dgram. We've been given the following assignment:
Create the client code for an application. Your client should send 10 ping messages to the target UDP server. For each message, your client should calculate the round trip time from when the package is sent to when the response is received. Should a package be dropped along the way, the client is to handle this as well. This should be done by having the client wait 1 second for a response after sending each package. If no reply is received, the client should log accordingly (package lost, no response, timeout, etc.) and send a new package to try again. However, the total amount of packages sent should still only be 10. The client should also calculate a percentage of packages lost/no response received, and log this before connection is closed.
THis if course seems rather straight forward, and I thought so. I've been coding it for a while, and I'm almost finished, but I'm having issues with the aspect of making the client send a package, await response, and then act accordingly.
So far, what my code does is basically to send a ping, and when a pong is received, it sends another ping. What I can't figure out is how to make it log that a response wasn't received before sending the next package. In other words, I know how to make it react to a received response, I just don't know how to make it respond if no response is given within a set timeframe. I've tried playing around with if-statements and loops, as well as async functions, but I haven't made it work yet, so now I'm asking for help.
Code is here:
const dgram = require("dgram");
const ms = require("ms");
var client = dgram.createSocket("udp4");
const PORT = 8000;
const HOST = "localhost";
let today = "";
let t0 = "";
let t1 = "";
let RTT = "";
let sentPackages = "";
let receivedPackages = "";
const messageOutbound = Buffer.from("You Up?");
sendPackage();
const x = setInterval(sendPackage, 1000);
client.on("message", (message, remote) => {
receivedPackages++
today = new Date();
t1 = today.getTime();
console.log(
`Message from: ${remote.address}:${remote.port} saying: ${message}`
);
RTT = ms(t1 - t0, { long: true });
console.log(RTT);
const x = setInterval(sendPackage, 1000);
});
client.on('error', (err) => {
console.log(`server error:\n${err.stack}`);
server.close();
});
async function sendPackage() {
if (sentPackages < 10) {
client.send(messageOutbound, 0, messageOutbound.length, PORT, HOST, () => {
sentPackages++
let today = new Date();
t0 = today.getTime();
console.log(
`message has been sent to ${HOST}:${PORT}. Message sent at: ${t0}`
);
});
} else {
calculateLoss();
client.close();
}
};
function calculateLoss() {
let amountLost = sentPackages - receivedPackages;
let percentageLoss = amountLost / sentPackages * 100
console.log(amountLost);
console.log(percentageLoss +"% of packages lost");
};
I would use async / await to simply wait 1000ms / 1s between messages, then keep track of all messages in an array.
We identify messages with a uuid, so we can ensure that messages we receive can be matched to those we send.
We can then log all the required statistics afterwards:
const dgram = require("dgram");
const uuid = require('uuid');
const PORT = 8000;
const HOST = "localhost";
const client = dgram.createSocket("udp4");
// Array that keeps track of the messages we send
let messages = [];
// When we get a message, decode it and update our message list accordingly...
client.on("message", (messageBuffer, remote) => {
let receivedMessage = bufferToMessage(messageBuffer);
// Find the message we sent and set the response time accordingly.
let message = messages.find(message => message.uuid === (receivedMessage ||{}).uuid);
if (message) {
message.responseTimestamp = new Date().getTime();
}
});
client.on('error', (err) => {
console.log(`server error:\n${err.stack}`);
server.close();
});
function createMessage() {
return { uuid: uuid.v4() };
}
function messageToBuffer(message) {
return Buffer.from(JSON.stringify(message), "utf-8");
}
function bufferToMessage(buffer) {
try {
return JSON.parse(buffer.toString("utf-8"));
} catch (error) {
return null;
}
}
// Wait for timeout milliseconds
function wait(timeout) {
return new Promise(resolve => setTimeout(resolve, timeout));
}
function sendMessage(message, port, host) {
// Save the messages to our list...
messages.push(message);
console.log(`Sending message #${messages.length}...`);
// Set the time we send out message...
message.sentTimestamp = new Date().getTime();
let messageBuffer = messageToBuffer(message);
return new Promise((resolve, reject) => {
client.send(messageBuffer, 0, messageBuffer.length, port, host, (error, bytes) => {
if (error) {
reject(error);
} else {
resolve(bytes);
}
})
});
}
async function sendMessages(messageCount, port, host, timeout) {
for(let messageIndex = 0; messageIndex < messageCount; messageIndex++) {
let message = createMessage();
await sendMessage(message, port, host);
await wait(timeout);
if (message.responseTimestamp) {
console.log(`Response received after ${message.responseTimestamp - message.sentTimestamp} ms...`);
} else {
console.log(`No response received after ${timeout} ms...`);
}
}
logStatistics(messages);
}
function logStatistics(messages) {
let messagesSent = messages.length;
let messagesReceived = messages.filter(m => m.responseTimestamp).length;
let messagesLost = messagesSent - messagesReceived;
console.log(`Total messages sent: ${messagesSent}`);
console.log(`Total messages received: ${messagesReceived}`);
console.log(`Total messages lost: ${messagesLost} / ${(100*messagesLost / (messages.length || 1) ).toFixed(2)}%`);
if (messagesReceived > 0) {
console.log(`Average response interval:`, messages.filter(m => m.responseTimestamp).reduce((averageTime, message) => {
averageTime += (message.responseTimestamp - message.sentTimestamp) / messagesReceived;
return averageTime;
}, 0) + " ms");
}
}
sendMessages(10, PORT, HOST, 1000);
I have a little problem, my firebase cloud function completes before I get the API key from calling the Google Secret Manager API. The API key is important as it makes an API call to get data from an external server to store the result of the API call in Google Cloud Storage.
Here is my code,
'use strict';
// Request Data From A URL
var request = require('request');
var https = require('https');
// Var Firebase Functions
var functions = require('firebase-functions');
const admin = require('firebase-admin');
// Initalise App
admin.initializeApp();
// init firebase admin and get the default project id
const projectId = admin.instanceId().app.options.projectId
const util = require('util');
// Imports the Google Storage client library
const {Storage} = require('#google-cloud/storage');
// Import the Secret Manager client and instantiate it:
const {SecretManagerServiceClient} = require('#google-cloud/secret-manager');
const secretClient = new SecretManagerServiceClient();
// Setting Timeout in Seconds - default is 1 second
// The maximum value for timeoutSeconds is 540, or 9 minutes. Valid values for memory are:
// 128MB, 256MB, 512MB, 1GB, 2GB
const runtimeOpts = {
timeoutSeconds: 300,
memory: '512MB'
}
let apikey = '';
// From were the data comes
// 1 = Shoreserver
var shipid = '1';
// Get the current date
var today = new Date();
var dd = String(today.getDate()).padStart(2, '0');
var mm = String(today.getMonth() + 1).padStart(2, '0'); //January is 0!
var yyyy = today.getFullYear();
today = '0000' + '-' + '00' + '-' + '00';
// Creates a storage client
const storage = new Storage({
projectId: projectId,
});
// Set Bucket Name
const bucket = storage.bucket('masterlog');
/**
* Delete a file from a storage bucket and download a new file from remote location to store in the bucket
*/
exports.getEmilyAPItoStorage = functions
.runWith(runtimeOpts)
.region('europe-west1')
.https.onRequest((req, res) => {
// Get Secret
***(async () => { apikey = await getSecret() })***
console.info(`ApiKey Secret: ${apikey}`);
// First we want to delete the current file, the filename is always the same.
// Delete files in the Bucket people
bucket.deleteFiles({
prefix: `people.json`
})
.catch( (err) => {
console.log(`Failed to delete people.json`);
});
// Start of the requesting different tables
// Table to get data from
var apitable = 'people';
// Set destination filename
const people = bucket.file('people.json');
var url = 'https://<URL>/api/' + shipid + '/' + apitable + '?apikey=' + apikey + '&syncdate=' + today;
// Set the options to make the request
var options = {
url: url,
strictSSL: false,
secureProtocol: 'TLSv1_method'
}
// Make a request for the API and store the file in Storage
request(options)
.pipe(people
.createWriteStream({sourceFormat: 'NEWLINE_DELIMITED_JSON'}))
.on('finish', function(error) {
if (error) {
console.log(error);
res.status(500).send(error);
} else {
console.log( "- done!")
res.status(200).send("OK");
}
});
// End Function with status code 200
// Set destination filename
const agents = bucket.file('agents.json');
// Table to get data from
var apitable = 'ports';
var url = 'https://emily.greenpeace.net/api/' + shipid + '/' + apitable + '?apikey=' + apikey + '&syncdate=' + today;
// Set the options to make the request
var options = {
url: url,
strictSSL: false,
secureProtocol: 'TLSv1_method'
}
// Make a request for the API and store the file in Storage
request(options)
.pipe(agents
.createWriteStream({sourceFormat: 'NEWLINE_DELIMITED_JSON'}))
.on('finish', function(error) {
if (error) {
console.log(error);
res.status(500).send(error);
} else {
console.log( "- done!")
res.status(200).send("OK");
}
});
// End Function with status code 200
async function getSecret() {
// Access the secret.
const resource_name = 'projects/' + projectId + '/secrets/emilyapikey/versions/latest';
let [version] = await secretClient.accessSecretVersion({name: resource_name})
console.info(`Found secret ${version.payload.data} with state ${version.state}`);
apikey = version.payload.data;
return apikey;
}
});
I can get the API key from Google Secrets Manager in the function getSecret(), the API Key is not available when I make the API key to my server. My expectation is that the getSecret would complete before it executes the rest of the code.
If someone has an insight her of what I'm missing I really interested to hear from you.
If you want to use async/await in any funciton, that function has to be declared async:
exports.getEmilyAPItoStorage = functions
.runWith(runtimeOpts)
.region('europe-west1')
.https.onRequest(async (req, res) => { ... })
Then you can await in the code in its body:
const apikey = await getSecret()
From a node.js server I'm trying to send data to a Pub/Sub topic and then fire a Cloud Function once it triggers.
I'm succesfully sending messages to Pub/Sub, however when I try to interpret the data in the function I get the error "No telemetry data was provided" - see function bellow.
I'm following this: https://github.com/GabeWeiss/GoogleIoTCoreApp/tree/master/gcf/telemetryToFirestore
Here's how I call it (this works):
const pubsub = new PubSub();
const topicName = 'topixXx';
const dataJ = JSON.stringify([{"temperature": 1}, {"bpm": 2}]);
const dataBuffer = Buffer.from(dataJ);
const messageId = pubsub.topic(topicName).publish(dataBuffer);
res.end(`Message ${messageId} published.`);
And here's the CGF (I've cut part of the function, i'ts not needed for the case):
const admin = "...";
admin.initializeApp();
var db = "..";
exports.telemetryToFirestore = (event, callback) => {
const pubsubMessage = event.data;
throw new Error("pubsubMessage.data: "+pubsubMessage.data);
if (!pubsubMessage.data) {
throw new Error('No telemetry data was provided!');
}
const payload = Buffer.from(pubsubMessage.data, 'base64').toString();
const telemetry = JSON.parse(payload);
db.collection(......)
callback(.....);
};