I am new to JS and async operations. In a router of nodeJS using express, I have aggregated some data from mongo using mongoose. The data is weather data collected from different sites every 15 minutes interval. I processed the data with mongoose aggregate pipeline to get hourly data and group by each site. But the data needs a further process to get periods where for example relative humidity over 90% and assign scores to each period so I wrote some synchronous functions that target each site (each geojson object).
Mongoose looks something like that:
module.exports.filteredData = function (collection, dateInput) {
return collection.aggregate([
{
$addFields :{
DateObj: {
$dateFromString: {
dateString: "$DateTime",
format: '%Y-%m-%d'
}
},
}
},
{
$addFields :{
NewDateTimes: {
$dateFromParts:{
'year': {$year: '$DateObj'},
'month':{$month: '$DateObj'},
'day':{$dayOfMonth: '$DateObj'},
'hour': {$toInt: "$Time"}
}
}
}
}
...
synchronouse functions:
const calcDSV = function(featuresJSON){
// featuresJSON
const SVscore = [];
const tuEval = featuresJSON.features.properties.TU90; // array
const obArr = featuresJSON.features.properties.OB; // array
const periodObj = getPeriods(tuEval);// get period position
const paramObj = getParams(periodObj, obArr); // get parameters
const periodDate = getPeriodDate(featuresJSON, periodObj);
const removeTime = periodDate.beginDate.map(x=>x.split('T')[0]);
let hourly = paramObj.hourCounts;
let avgTemps = paramObj.avgTemps;
for(let i = 0;i<hourly.length; i++){
let score = assignScore(avgTemps[i], hourly[i]);
SVscore.push(score);
}
// output sv score for date
const aggreScore = accumScore(removeTime, SVscore);
aggreScore.DSVdate = aggreScore.Date.map(x=>new Date(x));
featuresJSON.features.properties.periodSV = SVscore;
featuresJSON.features.properties.Periods = periodDate;
featuresJSON.features.properties.DSVscore = aggreScore;
return featuresJSON;
}
Now I am stuck on how to apply those function on each site return by the mongoose aggregate pipeline on a post request:
router.post('/form1', (req, res, next)=>{
const emdate = new Date(req.body.emdate);
const address = req.body.address;
const stationDataCursor = stationData.filteredData(instantData, emdate);
stationDataCursor.toArray((err, result)=>{
if(err){
res.status(400).send("An error occurred in Data aggregation")
};
res.json(result.map(x=>calcDSV.calcDSV(x)));
})
});
I tried in the callback:
stationDataCursor.toArray((err, result)=>{
if(err){
res.status(400).send("An error occurred in Data aggregation")
};
res.json(result.map(async (x)=>await calcDSV.calcDSV(x))));
})
and using then():
stationDataCursor.toArray().then((docArr)=>{
let newfeature = await docArr.map(async (x)=> await calcDSV.calcDSV(x))));
res.json(newfeature);
})
or make calcDSV() returns new promise
return new Promise((rej, res)=>{
resolve(featuresJSON);
})
I would expect to see all sites with a new feature added in the HTTP response output. But most of the time, I got ReferenceError: error is not defined.
I think I have figured it out:
after all, have to make all synchronous functions asynchronous by prepending async to those functions;
rewrite this part in the post router function, especially the array map part. I read from this. and in the map() gonna have try...catch... in it, otherwise it won't work.
await stationDataCursor.toArray().then(async (docArr)=>{
const newfeature = await Promise.all(docArr.map(async function(x){
try{
const feature = await calcDSV.calcDSV(x);
return feature
} catch(err){
console.log("Error happened!!! ", err);
}
}));
res.json(newfeature)
})
Hope it helps.
Related
i am trying to use mongodb to get some data for our analytics page, there's alot of data that's processing but for some reason the express returns empty array as response even when i use console log i see the data on terminal.
This is my endpoint
import Analytics from '../classes/Analytics';
import { Router } from 'express';
import role from '../middleware/role';
const router = Router();
router.use(role('admin'));
router.get('/analytics/weekly', async (req, res) => {
try {
const data = await Analytics.getWeekly();
return res.status(200).send({ data });
} catch (err) {
console.log(err);
return res.status(500).send({ message: 'Something went wrong, please try again later!' });
}
});
module.exports = router;
This is where all the magic happens for data variable
class Analytics {
static async getWeekly() {
// ignore this one day difference thing
const startDate = moment().subtract(1, 'days').startOf('week').format();
const endDate = moment().subtract(1, 'days').endOf('week').format();
try {
const orders = await Order.find(
{
sent_at: {
$gte: startDate,
$lte: endDate,
},
},
{ user: 1, created_at: 1, sent_at: 1 }
);
let counter = [];
for await (const item of orders) {
const date = moment(item.sent_at).format('YYYY-MM-DD HH');
if (!counter[date]) counter[date] = [];
if (!counter[date][item.user.username]) counter[date][item.user.username] = 0;
counter[date][item.user.username] += 1;
}
return counter;
} catch (err) {
console.log(err);
}
}
}
The point of the static method above is to fetch all orders and count how many times which user has handled the order.
Now when i console.log the data from the router endpoint i see the data on console perfectly just how i wanted it to be
'2021-07-03 22': [
Johnson: 10,
Mikaels: 15,
Vasquez: 24,
Blaskovich: 3
],
'2021-07-03 23': [
Johnson: 2,
Vasquez: 12,
Mikaels: 15,
Blaskovich: 5
]
The problem is when i make a request to my endpoint it returns an empty array []. What am i missing here?
Change this:
if (!counter[date]) counter[date] = [];
to this:
if (!counter[date]) counter[date] = {};
And this:
let counter = [];
to this:
let counter = {};
Your code here:
if (!counter[date][item.user.username]) counter[date][item.user.username] = 0;
Is adding a property to an array, not adding an array element. And JSON.stringify() ignores properties on an array. It only serializes actual array elements so when you try to send the JSON version of the array, it always appears empty.
So, when you call res.status(200).send({ data });, the .send() method serializes your object which contains a bunch of arrays that have no actual array elements, only properties.
By changing the arrays to be objects, then JSON.stringify() will serialize all those properties. Arrays and Objects have many things in common, but they are not the same. You should use the appropriate type for your situation.
In addition, change this:
} catch (err) {
console.log(err);
}
to this:
} catch (err) {
console.log(err);
throw err;
}
So that you are properly propagating errors back to the caller. Otherwise, you're just eating the error and returning undefined, both of which will cause problems for the caller.
I have the following function. I have a list of domains (very big list, more than 100000), I'm trying to put them in a foreach and resolveMx all of them and save the mx records in another database.
Edit, this is the complete function:
const dns = require('dns');
const {BigQuery} = require('#google-cloud/bigquery');
const bigquery = new BigQuery(project="smartiodomains");
const functions = require('firebase-functions');
exports.getMxRecords = functions.https.onRequest( async (req, res) => {
const query = "SELECT string_field_0 FROM smartiodomains.Domains.sk_domains_table";
const options = {
query: query,
location: 'US',
};
const [job] = await bigquery.createQueryJob(options);
const [rows] = await job.getQueryResults();
const datasetId = 'Domains';
const tableId = 'smartio_records';
var index = 0;
rows.forEach((row) => {
dns.resolveMx(row.string_field_0, function(error,addresses){
if(error){
const rows = [
{domain:row.string_field_0, mx_records: 'No data found.', priority: 'No data found.'}
];
// Insert data into a table
bigquery
.dataset(datasetId)
.table(tableId)
.insert(rows);
res.write("Something");
}else{
res.write("Something else");
addresses.forEach( address => {
const rows = [
{domain:row.string_field_0, mx_records: address.exchange, priority: address.priority}
];
// Insert data into a table
bigquery
.dataset(datasetId)
.table(tableId)
.insert(rows).then((foundErrors) => {
if (foundErrors && foundErrors.insertErrors != undefined) {
console.log('Error: ', err);
}
})
.catch((err) => {
console.error('ERROR:', err);
});
});
}
});
});
});
As #Doug Stevenson suggested i add a response (res.write("Something")). Now i have one error and a warning:
1.- Memory Limit exceeded
2.- TeenyStatisticsWarning: Possible excessive concurrent requests detected. 5000 requests in-flight, which exceeds the configured threshold of 5000. Use the TEENY_REQUEST_WARN_CONCURRENT_REQUESTS environment variable or the concurrentRequests option of teeny-request to increase or disable (0) this warning.
Old error:
With this implementation i got this error in the logs of GCF:
getMxRecordsp5ter5a8u17q { Error: queryMx ETIMEOUT marketingweb.sk
Sorry for my bad english. And thanks for any help.
An HTTP function requires that you send a response to the client after all of the asynchronous work is complete. The function terminates immediately after you send that response. Right now, you're not sending any response, so the function never terminates, and it will always time out. You should send a response after all the calls to dns.resolveMx are fully complete.
I'm trying to scrape a website with Puppeteer. I want to select the date of the last post inserted in my database and compare it to the dates taken by the scrape so I can see if the post is already in the database (using the date as the reference to see if it has been modified).
Here is my code:
const connection = mysql.createConnection({
host: 'localhost',
user: 'root',
password: '',
database: 'db_webcrawler_coches'
});
connection.connect((err) => {
if (err) throw err;
console.log('Connected!');
});
let lastpublishedDate;
let idCoches;
connection.query("SELECT id_coches, publish_date FROM coches ORDER BY publish_date DESC limit 1", function (err, row) {
if (err) throw err;
lastPublishedDate = row[0].publish_date;
idCoches = row[0].id_cochesNet;
console.log("Published in", lastPublishedDate);
console.log("Id Coches", idCoches);
});
const run = async () => {
try {
const options = {
headless: false,
};
...
const news = await page.evaluate(() => {
const idsList = [...document.querySelectorAll('div.mt-SerpList-item')].map(elem => elem.getAttribute("id")).filter(elem => elem.includes("#"))
const datePost = [...document.querySelectorAll('span.mt-CardAd-date')].map(elem => elem.innerText);
for(let i = 0; i < titlesCar.length; i++){
const finalDate = parsedDates[i];
if (finalDate > lastPublishedDate || idCoches !== idsList[i]){
console.log("Not repeated");
const carsList[i] = [
idsList[i],
parsedDates[i]
]
} else {
console.log("Repeated")
}
}
return carsList;
});
...
} catch (err) {
console.log(err);
await browser.close();
console.log("Browser Closed");
}
};
run();
As you can see I want to see if the date is the same or not as well as the id taken from the query. However, it appears an error that says Evaluation failed: ReferenceError: variable "lastPublishedDate" is not defined and I imagine that it will be the same with "idCoches". I wrote some console.logs to see when it crashes and it seems that it happens when reaches the function "news".
I'm not sure if it is because it is the scope or because of the function. What do you think I should do to make it work?
Could it be the scope?
Thank you!
EDIT: SOLVED!
I post it in the case that anyone faces a similar issue.
Indeed it was the scope, it is a problem related to Puppeteer. It seems that the function with page.evaluate() is unable to take any variable outside of it. To change it you need to add the page.evaluate in the following way:
await page.evaluate((variable_1, variable_2) => { /* ... */ }, variable_1, variable_2);
The callback to your Query probably does has not returned yet when the async function is run, so whatever your trying to reference is not defined.
I'm not sure if your mysql client supports promises, but if it does you could do something like this:
const run = async () => {
const row = await connection.query("SELECT id_coches, publish_date FROM coches ORDER BY publish_date DESC limit 1")
lastPublishedDate = row[0].publish_date;
idCoches = row[0].id_cochesNet;
...
}
If that does not work you could also run everything inside the callback of the query. Hope that helps.
First of all, please forgive me if this is a duplicate, I am new to coding and Javascript in general.
I have an async function that queries mongodb based on an objects passed in the function call. The function executes, and returns the results to a callback function which logs the results to the console, and then hangs. Ultimately, I want to take the results of the async query and then do something with them outside the original async function. I am not understanding why it hangs after it logs to the console.
const MongoClient = require('mongodb').MongoClient;
let fObj = {
field : {},
limit : 100
}
let cObj = {
dbName : 'myNewDatabase',
colName : 'newCollection'
}
async function findDoc(cObj,fObj) {
const url = 'mongodb://localhost:27017';
const client = new MongoClient(url, { useNewUrlParser: true });
try {
await client.connect();
const db = client.db(cObj.dbName);
const col = db.collection(cObj.colName);
console.log(`Connection Made to ${db.databaseName} database.`);
return await col.find(fObj.field).limit(fObj.limit).toArray();
client.close();
} catch (err) {
console.log(err.stack);
}
};
findDoc(cObj,fObj).then(function(result) {
console.log(result);
});
The code executes, and logs the results to the console, but then hangs. I have to ctrl-c out to get it to end. What am I missing?
I suppouse you're running your code with NodeJs. This implies that you have a promise hanging up, which keeps the server running. I assume this is because your connection to the DB is still open after you have found the document.
You need to move your client.close(); statement above the return statement, because it is never reached otherwise and your server will hang up forever.
Your code will look like this in the end:
const MongoClient = require('mongodb').MongoClient;
let fObj = {
field : {},
limit : 100
}
let cObj = {
dbName : 'myNewDatabase',
colName : 'newCollection'
}
async function findDoc(cObj,fObj) {
const url = 'mongodb://localhost:27017';
const client = new MongoClient(url, { useNewUrlParser: true });
try {
await client.connect();
const db = client.db(cObj.dbName);
const col = db.collection(cObj.colName);
console.log(`Connection Made to ${db.databaseName} database.`);
const result = await col.find(fObj.field).limit(fObj.limit).toArray();
client.close();
return result;
} catch (err) {
console.log(err.stack);
}
};
findDoc(cObj,fObj).then(function(result) {
console.log(result);
});
Also, I advise you to enclose your whole async function's body into the try clause. This way you will be able to effectively intercept any error. Imagine your new MongoClient failed to instantiate - you would end up with an uncaught error inside a promise, which isn't very nice.
I'm trying to return a value out of an array of values I get from querying Datastore.
results[0] have this content: {"prod_name":"Muffin","prod_price":3.99}.
I'd like to return via res.send only: 3.99
I've tried results[0].prod_price, or results[0]['prod_price'], I have tried saving results[0] as variable and trying to return prod_price, but nothing works.
Any help is appreciated.
My code is here:
const Datastore = require('#google-cloud/datastore');
const Storage = require('#google-cloud/storage');
// Instantiates a client
const datastore = Datastore();
const storage = new Storage();
exports.getprice = function getprice (req, res) {
const kind = datastore.createQuery("Dialogflow");
const filter = kind.filter("prod_name", req.body.queryResult.parameters['bakery_items']);
return query = datastore.runQuery(kind)
.then( (results) => {
const entities = results[0];
res.setHeader('Content-Type', 'application/json');
res.send(JSON.stringify({ "fulfillmentText": entities}));
})
.catch((err) => {
console.error(err);
res.status(500).send(err);
return Promise.reject(err);
});
};
I got it.
Actually I kept results instead of forcing results[0], and realized the output had an extra array, so to access the value, I had to do: results[0][0]['prod_price']
Thanks to JavaScript console.