I am trying to bring an array of strings from a database to a dropdown menu on a website I have created. I have everything working properly except for the final transfer of the data from the retrieval method to the website. Right now the data is in the form of a Promise, and I cannot for the life of me figure out how to get it to print out on my webpage. right now I'm just sending it to localhost:3000, I'm not at the point where I'm putting it into the dropdown yet. How would I do this ?
I've found very very little on this issue online and thus have been mainly just trying hack fixes that haven't really worked (tacking on the resolve() method, all() method). both of those resulted in syntax errors. All Var names/SQL queries have been changed btw. My latest attempt is below:
//code that sends the names to the webpage
app.get('/formfetch', function(req, res) {
const data = async() => {
let rawDat = await dbFormFetch.getNames();
return rawDat;
}
}
const hNs = data();
hNs.then((names) => {
if (names === null) {
res.end("Error: Names list came through as null.");
} else if (names.length > 0) {
resolve(names);
for (var i = 0; i < names.length; i++) {
res.end(names[i]);
}
res.status('200');
}
})
.catch((err) => {
res.status('404').json(err)
console.log("conversion of promise failed")
})
});
//the getNames() method (in a different file)
async function getNames() {
console.log("trying to get Names");
let query = `select NAME from NAMESTAB`;
console.log("query: " + query);
const binds = {};
const result = await database.simpleExecute(query, binds);
var results = [];
console.log("for loop in formfetch.js: ");
for (var i = 0; i < result.rows.length; i++) {
results[i] = i + ": " + result.rows[i].NAME+ ' \n';
}
return results;
}
The res.send method from the app.get function prints out "Made it to the Web server:" on my localhost. I checked the console, and I didn't see anything hidden in the html or something like that.
**Note: all of the data that should be in the promise is in the code (I can print it to console at any point in the code), but when I put it on the website it won't print. **
so big surprise here, I was doing it all wrong. Lesson of the day: read up on Promises and how they work before running and gunning your way through some async code. It's not as intuitive as you would hope.
// I only had made changes to the first of the two methods.
app.get('/formfetch', function(req, res) {
async function data() {
let rawDat = await dbFormFetch.getNames();
return rawDat;
}
data().then((Names) => {
if (Names === undefined) {
res.end("Error: Names list came through as null.");
} else if (Names.length > 0) {
res.setHeader('Content-Type', 'application/json');
res.status(200).json({ "names": Names });
}
})
.catch((err) => {
res.status('404').send("name retrieval failed in server.js module")
console.log(err)
console.log("conversion of promise failed")
})
});
when you use res.end() , it sets the header status and renders it immutable after calling this method, so it was the wrong thing to use. Instead of this, I used the setHeader() method to tell the website what kind of information I'm sending it, and then filled in the content by chaining the .json() method to the status() response I sent. I've never worked with promises before and I'm fairly new to NodeJS so this was a bit of a learning curve, but hopefully this helps people who are where I was yesterday. if you're new to promises, see this article and this article before you try to use this coding tool. you'll save yourself hours of debugging and error tracing.
Related
I'm trying to query some JIRA issues using a the jira-connector package.
I'm running into issues with the data returned not being defined until after everything else is executed in my code. I'm not sure if this is some issue with concurrency, but I can't for the life of me figure out where and how I'm messing up.
If inside the getJiraTimeEstimations function only call the getJiraTimeEstimate once it works just fine and I get access to the data to use further down in the program. It is when I'm trying to do it inside a map or foreach where I iterate over the Array.from(dataFromMaconomy.keys()) array that I seem to run into issues.
My understanding is that adding .then().catch() in the getJiraTimeEstimate function should be enough to stop it from continuing to run before all the calls are finished? Or am I misunderstanding how asynchronous calls work in Node and JS?
I've also tried converting it to an async getJiraTimeEstimations and adding an await infront of the search. But it doesn't seem to work either.
I am not populating the dataFromMaconomy array as I'm debugging. Which is what I was trying to do with the log statement. The log statement just prints undefined right now. But if I only call it with a single item from the rks array then it works fine.
function getJiraTimeEstimate(taskNumber, jiraClient) {
jiraClient.search.search({
jql: `id = ${taskNumber}`,
}).then((res) => res.issues[0].fields.timeoriginalestimate).catch((err) => err);
}
function getJiraTimeEstimations(dataFromMaconomy) {
const settings = JSON.parse(fs.readFileSync(path.join(__dirname, 'konfig.json'), 'utf8'));
const privateKeyData = fs.readFileSync(path.join(__dirname, settings.jira.consumerPrivateKeyFile), 'utf8');
const jira = new JiraClient({
host: settings.jira.server,
strictSSL: false, // Error: unable to verify the first certificate
rejectUnauthorized: false,
oauth: {
consumer_key: settings.jira.consumerKey,
private_key: privateKeyData,
token: settings.jira.accessToken,
token_secret: settings.jira.accessTokenSecret,
},
});
console.log('getting time estimations from Jira');
const dataFromMaconomyWithJira = [];
const rks = Array.from(dataFromMaconomy.keys());
rks.map((rk) => console.log(getJiraTimeEstimate(rk, jira)));
return dataFromMaconomyWithJira;
}
function generateData(){
const dataWithJira = getJiraTimeEstimations(convertedData);
// More functions where I use the data from getJiraTimeEstimations
// This gets run before all of the getJiraTimeEstimations have finished getting the data.
}
Giving your clarification in the comment, the getJiraTimeEstimate() function does not return anything. Try:
function getJiraTimeEstimate(taskNumber, jiraClient) {
return jiraClient.search.search({
jql: `id = ${taskNumber}`,
}).then((res) => res.issues[0].fields.timeoriginalestimate).catch((err) => err);
}
Also, you mentioned trying async / await but without luck. The async version of it would be:
async function getJiraTimeEstimate(taskNumber, jiraClient) {
try {
const res = await jiraClient.search.search({
jql: `id = ${taskNumber}`,
});
return res.issues[0].fields.timeoriginalestimate;
} catch (e) {
return e;
}
}
I have encountered a new error that has recently occurred. As part of a scheduled backup function, I explore a copy go each datastore and save in a bucket for later use. The following syntax has worked well over the last year, however, I suspect due to load, has now stopped working.
async function getDataStore(collection) {
return admin.firestore().collection(collection).get()
.then(querySnapshot => {
if (querySnapshot.size == 0) {
console.log('No matching request for: ' + collection);
return []
} else {
const data = [];
//Query snapshot line throwing snapshot too old error.
querySnapshot.forEach(doc => {
let data_temp = doc.data();
data_temp.doc_id = doc.id;
data.push(data_temp);
});
return data;
}
});
}
This now throws a code 9, the requested snapshot version is too old. One way I can think of overcoming this, would be to batch the request into smaller 'chunks' and save. however, I wanted to see if there was something more efficient I may be missing. Fresh eyes are always helpful.
Thanks
I come from C++, C, Python space and I'm new to react native / JS / back-end world.
I have some issues loading data from firebase. Here is what I want :
My Database :
users : uid : postids[]
posts : postids : content
I want to load the postids[] array from a user and then, load content of every postids[] in this array (according to every postids in the postids[] array).
Here is my code :
_getPostsFromDatabase() {
var docRef = firebase.firestore().collection("users").doc(firebase.auth().currentUser.uid);
return docRef.get().then(function(doc) {
if (doc.exists) {
return doc.data()["posts"];
}
}).catch(function(error) {
alert("Error getting document:", error);
});
}
_loadPosts() {
var new_posts = [];
this._getPostsFromDatabase()
.then(res => {
var i;
for (i = 0; i < res.length; i++) {
firebase.firestore().collection("posts").doc(res[i])
.onSnapshot(function(doc) {
new_posts.push(doc.data());
console.log(new_posts); --> This line print correct data
});
}
})
.catch(error => console.log(error));
console.log(new_posts); ---> This line print an empty array
}
componentDidMount() {
this._loadPosts()
}
So I want this behavior :
In componentDidMount I begine the routine --> this works
loadPosts is loading the postids[] array with _getPostsFromDatabase() function --> this works
Then, I make a for loop to push every object in an array to set the state at the end --> FAIL
At step 3, everything f... up, I made some console log to debug but there is a huge real time issue because evrything is printed randomly.
How can I get my new_posts filled array at the end of the for loop to setState. Maybe I'm wrong with this method, or if I'm not, I must have some issues with Async funtion ?
Is there an expert to help me understund better what is inside this kind of use case ?
Thanks
Basically the problem is that you are trying to perform an asynchronous code in a synchronous way.
You solution might be waiting for all promises to resolve.
_loadPosts() {
this._getPostsFromDatabase()
.then(res => {
let promises = res.map(id => {
return firebase.firestore().collection("posts").doc(id)
.get().then(doc => doc.data())
})
Promise.all(promises).then(res => {console.log(res);})
}
Your console will log before the for loop, that's the reason you are getting an empty array just include your console in the response just like this:
this._getPostsFromDatabase()
.then(res => {
var i;
for (i = 0; i < res.length; i++) {
firebase.firestore().collection("posts").doc(res[i])
.onSnapshot(function(doc) {
new_posts.push(doc.data());
console.log(new_posts); --> This line print correct data
});
}
console.log(new_posts); ---->Include here
})
Hope this helps!
I'm building an application using Node/Express/MongoDB (first time with all these) that will let me pull data from the DB and display it on an Express page. This is what the GET request looks like:
var str = "";
app.route('/view-reports').get(function(req, res) {
var cursor = collections.find({});
cursor.each(function(err, item) {
if (item != null) {
console.log(str);
str = str + "Substance: " + item.substance + "<br>";
}
if (err) {
console.log(err);
}
});
console.log(str);
res.send(str);
str = "";
});
I would expect this to return something like this:
Substance: a
Substance: b
Substance: c
However, the initial request does not return anything at all. The second request will return the above. If I enclose res.send(str) in an if conditional it simply will not load until a second request is made.
cursor.each() is asynchronous. That means it runs sometimes LATER, after your res.send(str), thus you get the previous version of str. You need to collect all the data first and then send your response only when you have all the data.
If you want all the data, then you could use promises and .toArray() like this:
app.route('/view-reports').get(function(req, res) {
collections.find({}).toArray().then(data => {
let result = data.map(item => {
return "Substance: " + item.substance + "<br>";
}).join("");
res.send(result);
}).catch(err => {
// database error
console.log(err);
res.sendStatus(500);
});
});
Note: This also wisely gets rid of the str variable which was outside the scope of the request and thus could easily lead to a concurrency bug when multiple requests were in flight at the same time (from different users).
Create a router specifically for substances and use it in app. Instead of breaks, you can create a ul, also, that processing should happen on the front end. Separate your concerns. The server shouldn't have to worry about any rendering and etc. One purpose per process.
The routers can be created per resource. Create a router for substances, for cats, for dogs. Each individual router has it's own get post delete and puts that allow you to modify that resource. app can use all the routers at once.
app.use(catRouter);
app.use(mooseRouter);
app.use(platypusRouter);
const { Router } = require('express');
const createError = require('http-errors');
let substanceRouter = new Router();
function buildElement(arr)
{
let start = '';
arr.forEach(val => {
if(!val) return;
start += `Substance : ${val}<br>`;
});
return start;
}
subtanceRouter.get('/endpoint/whatever', function(req, res, next) {
collectios.find({})
.then(results => {
if(!results) throw new Error('Resource Not Found');
let output = buildElement(results);
res.json(output);
next();
})
.catch(err => next(createError(404, err.message)));
})
app.use(substanceRouter);
Alternately we can write :
let output = results
.filter(sub => !!sub)
.join('<br>');
res.json(output);
But be advised this will add memory overhead, generates a completely new array to house the results, consuming at worst, O(n) memory.
I'm using this Gumroad-API npm package in order to fetch data from an external service (Gumroad). Unfortunately, it seems to use a .then() construct which can get a little unwieldy as you will find out below:
This is my meteor method:
Meteor.methods({
fetchGumroadData: () => {
const Gumroad = Meteor.npmRequire('gumroad-api');
let gumroad = new Gumroad({ token: Meteor.settings.gumroadAccessKey });
let before = "2099-12-04";
let after = "2014-12-04";
let page = 1;
let sales = [];
// Recursively defined to continue fetching the next page if it exists
let doThisAfterResponse = (response) => {
sales.push(response.sales);
if (response.next_page_url) {
page = page + 1;
gumroad.listSales(after, before, page).then(doThisAfterResponse);
} else {
let finalArray = R.unnest(sales);
console.log('result array length: ' + finalArray.length);
Meteor.call('insertSales', finalArray);
console.log('FINISHED');
}
}
gumroad.listSales(after, before, page).then(doThisAfterResponse); // run
}
});
Since the NPM package exposes the Gumorad API using something like this:
gumroad.listSales(after, before, page).then(callback)
I decided to do it recursively in order to grab all pages of data.
Let me try to re-cap what is happening here:
The journey starts on the last line of the code shown above.
The initial page is fetched, and doThisAfterResponse() is run for the first time.
We first dump the returned data into our sales array, and then we check if the response has given us a link to the next page (as an indication as to whether or not we're on the final page).
If so, we increment our page count and we make the API call again with the same function to handle the response again.
If not, this means we're at our final page. Now it's time to format the data using R.unnest and finally insert the finalArray of data into our database.
But a funny thing happens here. The entire execution halts at the Meteor.call() and I don't even get an error output to the server logs.
I even tried switching out the Meteor.call() for a simple: Sales.insert({text: 'testing'}) but the exact same behaviour is observed.
What I really need to do is to fetch the information and then store it into the database on the server. How can I make that happen?
EDIT: Please also see this other (much more simplified) SO question I made:
Calling a Meteor Method inside a Promise Callback [Halting w/o Error]
I ended up ditching the NPM package and writing my own API call. I could never figure out how to make my call inside the .then(). Here's the code:
fetchGumroadData: () => {
let sales = [];
const fetchData = (page = 1) => {
let options = {
data: {
access_token: Meteor.settings.gumroadAccessKey,
before: '2099-12-04',
after: '2014-12-04',
page: page,
}
};
HTTP.call('GET', 'https://api.gumroad.com/v2/sales', options, (err,res) => {
if (err) { // API call failed
console.log(err);
throw err;
} else { // API call successful
sales.push(...res.data.sales);
res.data.next_page_url ? fetchData(page + 1) : Meteor.call('addSalesFromAPI', sales);
}
});
};
fetchData(); // run the function to fetch data recursively
}