I'm using this Gumroad-API npm package in order to fetch data from an external service (Gumroad). Unfortunately, it seems to use a .then() construct which can get a little unwieldy as you will find out below:
This is my meteor method:
Meteor.methods({
fetchGumroadData: () => {
const Gumroad = Meteor.npmRequire('gumroad-api');
let gumroad = new Gumroad({ token: Meteor.settings.gumroadAccessKey });
let before = "2099-12-04";
let after = "2014-12-04";
let page = 1;
let sales = [];
// Recursively defined to continue fetching the next page if it exists
let doThisAfterResponse = (response) => {
sales.push(response.sales);
if (response.next_page_url) {
page = page + 1;
gumroad.listSales(after, before, page).then(doThisAfterResponse);
} else {
let finalArray = R.unnest(sales);
console.log('result array length: ' + finalArray.length);
Meteor.call('insertSales', finalArray);
console.log('FINISHED');
}
}
gumroad.listSales(after, before, page).then(doThisAfterResponse); // run
}
});
Since the NPM package exposes the Gumorad API using something like this:
gumroad.listSales(after, before, page).then(callback)
I decided to do it recursively in order to grab all pages of data.
Let me try to re-cap what is happening here:
The journey starts on the last line of the code shown above.
The initial page is fetched, and doThisAfterResponse() is run for the first time.
We first dump the returned data into our sales array, and then we check if the response has given us a link to the next page (as an indication as to whether or not we're on the final page).
If so, we increment our page count and we make the API call again with the same function to handle the response again.
If not, this means we're at our final page. Now it's time to format the data using R.unnest and finally insert the finalArray of data into our database.
But a funny thing happens here. The entire execution halts at the Meteor.call() and I don't even get an error output to the server logs.
I even tried switching out the Meteor.call() for a simple: Sales.insert({text: 'testing'}) but the exact same behaviour is observed.
What I really need to do is to fetch the information and then store it into the database on the server. How can I make that happen?
EDIT: Please also see this other (much more simplified) SO question I made:
Calling a Meteor Method inside a Promise Callback [Halting w/o Error]
I ended up ditching the NPM package and writing my own API call. I could never figure out how to make my call inside the .then(). Here's the code:
fetchGumroadData: () => {
let sales = [];
const fetchData = (page = 1) => {
let options = {
data: {
access_token: Meteor.settings.gumroadAccessKey,
before: '2099-12-04',
after: '2014-12-04',
page: page,
}
};
HTTP.call('GET', 'https://api.gumroad.com/v2/sales', options, (err,res) => {
if (err) { // API call failed
console.log(err);
throw err;
} else { // API call successful
sales.push(...res.data.sales);
res.data.next_page_url ? fetchData(page + 1) : Meteor.call('addSalesFromAPI', sales);
}
});
};
fetchData(); // run the function to fetch data recursively
}
Related
I've created some tests in Cypress to add and duplicate article in our Angular application. The code for test ArticlesTest.js
describe('Shop articles single station', () => {
const productManagementPage = new ProductManagementPage()
const shopArticlesPage = new ShopArticlesPage()
before(() => {
var credentials =
{
"username": "someusername#user.name",
"password": "strongPassword!"
}
cy.navigateToProductManagement(credentials)
})
beforeEach(() => {
productManagementPage.shopArticlesProgrammingButton().click()
shopArticlesPage.WaitUntilPageLoaded('/productmanagement/api/v1/articles/get', 'GetArticles')
})
it('Add article', () => {
var randomNumber = RandomDataGenerator.GenerateRandomInt(1000,4999)
var randomName = RandomDataGenerator.GenerateRandomString(20)
var randomPrice = RandomDataGenerator.GenerateRandomDecimal(1,99)
shopArticlesPage.newArticleButton().click()
shopArticlesPage.saveButton().should('be.disabled')
shopArticlesPage.undoButton().should('be.disabled')
shopArticlesPage.deleteButton().should('be.disabled')
shopArticlesPage.articlesList().should('not.exist')
shopArticlesPage.articleNumberTextBox().should('be.enabled')
shopArticlesPage.articleNumberTextBox().type(randomNumber)
shopArticlesPage.articleNameTextBox().type(randomName)
shopArticlesPage.articleUnitPriceTextBox().type(randomPrice)
shopArticlesPage.undoButton().should('be.enabled')
shopArticlesPage.saveButton().click()
shopArticlesPage.newArticleButton().should('exist')
shopArticlesPage.articlesList().should('exist')
shopArticlesPage.saveButton().should('be.disabled')
shopArticlesPage.undoButton().should('be.disabled')
})
it('Duplicate article', () => {
var articleNumber = RandomDataGenerator.GenerateRandomInt(51,65)
var newArticleNumber = RandomDataGenerator.GenerateRandomInt(1000, 4999)
var newArticleName = RandomDataGenerator.GenerateRandomString(20)
shopArticlesPage.articlesList().selectFromList(articleNumber)
const articleUnitPrice = shopArticlesPage.articleUnitPriceTextBox().invoke('text')
const vatCodeValue = shopArticlesPage.vatCodeDropDown().invoke('text')
const cardCodeValue = shopArticlesPage.cardCodeDropDown().invoke('text')
shopArticlesPage.duplicateArticleButton().click()
shopArticlesPage.WaitUntilPageLoaded()
shopArticlesPage.articleNumberTextBox().type(newArticleNumber)
shopArticlesPage.articleNameTextBox().type(newArticleName)
shopArticlesPage.saveButton().click()
shopArticlesPage.newArticleButton().should('be.enabled')
})
WaitUntilPageLoaded() method code is:
WaitUntilPageLoaded(path, alias) {
return cy.waitForRequestToComplete(path, alias)
}
which, in turn, is custom Cypress command:
Cypress.Commands.add('waitForRequestToComplete', (path, alias) => {
cy.intercept('POST', path).as(alias)
cy.wait('#' + alias).its('response.statusCode').should('be.ok')
})
In 1st beforeEach() run, there's no problem with intercepting GetArticles and waiting for it to complete.
The problem starts in 2nd test, as it looks like GetArticles is not intercepted, it's not called at all, though it's supposed to be. The problem doesn't exist when clicking through the application manually, and /articles/get is always invoked.
The test ends up with error message
Timed out retrying after 30000ms: cy.wait() timed out waiting 30000ms for the 1st request to the route: GetArticles. No request ever occurred.
I've also tried using other endpoint e.g. vatcodes/get, and it works perfectly. The problem occurs only for articles/get, but I don't see any trail that would tell my why this happens for articles endpoint.
What is the problem? Why Cypress "blocks" 2nd call to this endpoint? What's more interesting, the problem doesn't exist for GetFeatures alias, which is created in an identical way.
Make sure the network intercept is registered before the application makes the call.
it('is registered too late', () => {
cy.intercept('/todos').as('todos')
cy.visit('/')
cy.wait('#todos')
})
In our case, we need to register the intercept before visiting the page. Once the page is loaded, the application fetches the todo items, and everything is working as expected.
you can see this link: https://glebbahmutov.com/blog/cypress-intercept-problems/
If I'm reading the situation correctly, the last log image is the failing test.
There is no (xhr) 200 /productmanagement/api/v1/articles/get showing there.
It goes straight from api/v1/subscriptionfeatures/get to api/v1/vatcodes/get, but in the first test the api/v1/articles/get was between those two calls.
If it occurs later in the screenshot, add an increased timeout to catch it (the same intercept can use the longer timeout in both tests, but it won't delay the first test).
This may mean you have found a bug in the app - it seems that a "Duplicate" action should have the same POSTs as an "Add" action.
Have you resolved this?
I'm using this config:
Given('a GraphQL service error is thrown', () => {
cy.intercept({ method: 'POST', url: '/uat/graphql', times: 1 }, { forceNetworkError: true });
});
With times: 1. But the interception does not block the request now.
I found times in the docs.
The Problem:
I have been unable to use Firebase (Google) Cloud Functions to collect and utilize device tokens for the cloud messaging feature.
Context:
I am a self-taught android-Java developer and have no JavaScript experience. Despite that, I believe I have code that should work and am not sure what the problem is. To my understanding, it could be one of three things:
Somehow my Firebase Realtime Database references are being called incorrectly and I am not retrieving data as expected.
I may need to use Promises to wait for all calls to be made before proceeding, however I don't really understand how I would incorporate that into the code I have.
I may be using multiple return statements incorrectly (which I am also fuzzy on).
My error message on the Firebase Realtime Database console is as follows:
#firebase/database: FIREBASE WARNING: Exception was thrown by user callback. Error: Registration token(s) provided to sendToDevice() must be a non-empty string or a non-empty array.
at FirebaseMessagingError.FirebaseError [as constructor] (/srv/node_modules/firebase-admin/lib/utils/error.js:42:28)
at FirebaseMessagingError.PrefixedFirebaseError [as constructor] (/srv/node_modules/firebase-admin/lib/utils/error.js:88:28)
at new FirebaseMessagingError (/srv/node_modules/firebase-admin/lib/utils/error.js:254:16)
at Messaging.validateRegistrationTokensType (/srv/node_modules/firebase-admin/lib/messaging/messaging.js:729:19)
at Messaging.sendToDevice (/srv/node_modules/firebase-admin/lib/messaging/messaging.js:328:14)
at admin.database.ref.once.snapshot (/srv/index.js:84:12)
at onceCallback (/srv/node_modules/#firebase/database/dist/index.node.cjs.js:4933:51)
at /srv/node_modules/#firebase/database/dist/index.node.cjs.js:4549:22
at exceptionGuard (/srv/node_modules/#firebase/database/dist/index.node.cjs.js:698:9)
at EventList.raise (/srv/node_modules/#firebase/database/dist/index.node.cjs.js:9684:17)
The above indicates I am not retrieving data either at all or by the time the return is called. My JavaScript function code is:
'use strict';
const admin = require('firebase-admin');
admin.initializeApp();
exports.pushNotification = functions.database.ref('/Chat Messages/{chatId}/{pushID}').onCreate((snapshot, context) => {
const valueObject = snapshot.after.val();
return admin.database().ref(`/Chat Basics/${valueObject.chatKey}/Chat Users`).once('value', statusSnapshot => {
var index = 0;
var totalkeys = statusSnapshot.numChildren();
var msgIDs = [];
statusSnapshot.forEach(msg=>{
msgIDs.push(msg.key.toString());
if(index === totalkeys - 1){
const payload = {
notification : {
title: valueObject.userName,
body: valueObject.message,
sound: "default"
}
}
sendNotificationPayload(valueObject.uid, payload);
}
index++;
});
});
});
function sendNotificationPayload(uid, payload){
admin.database()
.ref(`/User Token Data/${uid}`)
.once('value', snapshot=> {
var tokens = [];
//if(!snapshot.exists())return;
snapshot.forEach(item =>{
tokens.push(item.val())
});
admin.messaging()
.sendToDevice(tokens, payload)
.then(res => {
return console.log('Notification sent')
})
.catch(err => {
return console.log('Error in sending notification = '+err)
});
});
}
This code is mostly inspired by what was said to be a working example here from another Stack Overflow question here. I have successfully tested sending a notification to a single device by manually copying a device token into my function, so the function does run to completion. My Java code seems to be irrelevant to the problem, so I have not added it (please ask in the comments if you would like it added for further context).
What I Have Tried:
I have tried implementing promises into my code, but I don't think I was doing it properly. My main reference for this was here. I have also looked at the documentation for literally everything related to this topic, however my knowledge of JS is not sufficient to really apply barebones examples to my code.
My Firebase Realtime Database Nodes:
#1: Loop through chat members to collect user IDs:
"Chat Basics" : {
"1607801501690_TQY41wIfArhHDxEisyupZxwyHya2" : {
"Chat Users" : {
"JXrclZuu1aOwEpCe6KW8vSDea9h2" : true,
"TQY41wIfArhHDxEisyupZxwyHya2" : true
},
#2: Collect user tokens from collected IDs (ignore that tokens are matching):
"User Token Data" : {
"JXrclZuu1aOwEpCe6KW8vSDea9h2" : "duDR3KH3i3I:APA91bH_LCeslZlqL8akYw-LrM9Dv__nx4nU1TquCS0j6bGF1tlIARcheREuNdX1FheC92eelatBC8LO4t6gt8liRdFHV-NDuNLa13oHYxKgl3JBPPlrMo5rB5XhH7viTo4vfYOMftRi",
"TQY41wIfArhHDxEisyupZxwyHya2" : "duDR3KH3i3I:APA91bH_LCeslZlqL8akYw-LrM9Dv__nx4nU1TquCS0j6bGF1tlIARcheREuNdX1FheC92eelatBC8LO4t6gt8liRdFHV-NDuNLa13oHYxKgl3JBPPlrMo5rB5XhH7viTo4vfYOMftRi"
}
Conclusion:
Concrete examples would be much appreciated, especially since I am crunching right now. Thanks for your time and help!
Update:
After some more testing, it looks like the problem is definitely due to my lack of understanding of promises in two areas. Firstly, only one user is collected before the final return is called. Secondly, the final return is called before the 2nd forEach() loop can store snapshot data to an array.
For this code then, how may I modify (or rebuild) it so that it collects all keys before proceeding to retrieve token data from all keys - ultimately before returning the notification?
Just as with every question I post, I managed to figure out how to do it (tentatively) a few hours later. Below is a full example of how to send a notification to chat users based on a message sent (although it does not yet exclude the sender) to a given chat. The order of operations are as such:
User message is saved and triggers event. Relevant data the message contains are:
username, chat key, message
These are retrieved, with (username + message) as the (title + body) of the
notification respectively, and the chat key is used for user id reference.
Loop through chat user keys + collect.
Loop through array of chat user keys to collect array of device tokens.
Send notification when complete.
The code:
//Use firebase functions:log to see log
exports.pushNotification = functions.database.ref('/Chat Messages/{chatId}/{pushId}').onWrite((change, context) => {
const valueObject = change.after.val();
return admin.database().ref(`/Chat Basics/${valueObject.chatKey}/Chat Users`).once('value', statusSnapshot => {
var index = 0;
var totalkeys = statusSnapshot.numChildren();
var msgIDs = [];
statusSnapshot.forEach(msg=>{
msgIDs.push(msg.key.toString());
if(index === totalkeys - 1){
const payload = {
notification : {
title: valueObject.userName,
body: valueObject.message,
sound: "default"
}
}
let promises = [];
var tokens = [];
for(let i=0; i < msgIDs.length; i++){
let userId = msgIDs[i];
let promise = admin.database().ref(`/User Token Data/${userId}`).once('value', snapshot=> {
tokens.push(snapshot.val());
})
promises.push(promise);
}
return Promise.all(promises).then(() => {
return admin.messaging().sendToDevice(tokens, payload);
});
}
index++;
return false;
});
});
});
I have encountered a new error that has recently occurred. As part of a scheduled backup function, I explore a copy go each datastore and save in a bucket for later use. The following syntax has worked well over the last year, however, I suspect due to load, has now stopped working.
async function getDataStore(collection) {
return admin.firestore().collection(collection).get()
.then(querySnapshot => {
if (querySnapshot.size == 0) {
console.log('No matching request for: ' + collection);
return []
} else {
const data = [];
//Query snapshot line throwing snapshot too old error.
querySnapshot.forEach(doc => {
let data_temp = doc.data();
data_temp.doc_id = doc.id;
data.push(data_temp);
});
return data;
}
});
}
This now throws a code 9, the requested snapshot version is too old. One way I can think of overcoming this, would be to batch the request into smaller 'chunks' and save. however, I wanted to see if there was something more efficient I may be missing. Fresh eyes are always helpful.
Thanks
Background: Been trying for the last 2 day to resolve this myself by looking at various examples from both this website and others and I'm still not getting it. Whenever I try adding callbacks or async/await I'm getting no where. I know this is where my problem is but I can't resolve it myself.
I'm not from a programming background :( Im sure its a quick fix for the average programmer, I am well below that level.
When I console.log(final) within the 'ready' block it works as it should, when I escape that block the output is 'undefined' if console.log(final) -or- Get req/server info, if I use console.log(ready)
const request = require('request');
const ready =
// I know 'request' is deprecated, but given my struggle with async/await (+ callbacks) in general, when I tried switching to axios I found it more confusing.
request({url: 'https://www.website.com', json: true}, function(err, res, returnedData) {
if (err) {
throw err;
}
var filter = returnedData.result.map(entry => entry.instrument_name);
var str = filter.toString();
var addToStr = str.split(",").map(function(a) { return `"trades.` + a + `.raw", `; }).join("");
var neater = addToStr.substr(0, addToStr.length-2);
var final = "[" + neater + "]";
// * * * Below works here but not outside this block* * *
// console.log(final);
});
// console.log(final);
// returns 'final is not defined'
console.log(ready);
// returns server info of GET req endpoint. This is as it is returning before actually returning the data. Not done as async.
module.exports = ready;
Below is an short example of the JSON that is returned by website.com. The actual call has 200+ 'result' objects.
What Im ultimately trying to achieve is
1) return all values of "instrument_name"
2) perform some manipulations (adding 'trades.' to the beginning of each value and '.raw' to the end of each value.
3) place these manipulations into an array.
["trades.BTC-26JUN20-8000-C.raw","trades.BTC-25SEP20-8000-C.raw"]
4) export/send this array to another file.
5) The array will be used as part of another request used in a websocket connection. The array cannot be hardcoded into this new request as the values of the array change daily.
{
"jsonrpc": "2.0",
"result": [
{
"kind": "option",
"is_active": true,
"instrument_name": "26JUN20-8000-C",
"expiration_timestamp": 1593158400000,
"creation_timestamp": 1575305837000,
"contract_size": 1,
},
{
"kind": "option",
"is_active": true,
"instrument_name": "25SEP20-8000-C",
"expiration_timestamp": 1601020800000,
"creation_timestamp": 1569484801000,
"contract_size": 1,
}
],
"usIn": 1591185090022084,
"usOut": 1591185090025382,
"usDiff": 3298,
"testnet": true
}
Looking your code we find two problems related to final and ready variables. The first one is that you're trying to console.log(final) out of its scope.
The second problem is that request doesn't immediately return the result of your API request. The reason is pretty simple, you're doing an asynchronous operation, and the result will only be returned by your callback. Your ready variable is just the reference to your request object.
I'm not sure about what is the context of your code and why you want to module.exports ready variable, but I suppose you want to export the result. If that's the case, I suggest you to return an async function which returns the response data instead of your request variable. This way you can control how to handle your response outside the module.
You can use the integrated fetch api instead of the deprecated request. I changed your code so that your component exports an asynchronous function called fetchData, which you can import somewhere and execute. It will return the result, updated with your logic:
module.exports = {
fetchData: async function fetchData() {
try {
const returnedData = await fetch({
url: "https://www.website.com/",
json: true
});
var ready = returnedData.result.map(entry => entry.instrument_name);
var str = filter.toString();
var addToStr = str
.split(",")
.map(function(a) {
return `"trades.` + a + `.raw", `;
})
.join("");
var neater = addToStr.substr(0, addToStr.length - 2);
return "[" + neater + "]";
} catch (error) {
console.error(error);
}
}
}
I hope this helps, otherwise please share more of your code. Much depends on where you want to display the fetched data. Also, how you take care of the loading and error states.
EDIT:
I can't get responses from this website, because you need an account as well as credentials for the api. Judging your code and your questions:
1) return all values of "instrument_name"
Your map function works:
var filter = returnedData.result.map(entry => entry.instrument_name);
2)perform some manipulations (adding 'trades.' to the beginning of each value and '.raw' to the end of each value.
3) place these manipulations into an array. ["trades.BTC-26JUN20-8000-C.raw","trades.BTC-25SEP20-8000-C.raw"]
This can be done using this function
const manipulatedData = filter.map(val => `trades.${val}.raw`);
You can now use manipulatedData in your next request. Being able to export this variable, depends on the component you use it in. To be honest, it sounds easier to me not to split this logic into two separate components - regarding the websocket -.
I am using firebase functions and have successfully run a sanitize function.
The problem I am having is my firebase database looks like this.
The ABC123 and the 0l692lPD6EfqUZ4Y4xiCOVmnNmC2 are both sets automatically when a post is created so I would not know what they will be.
I can hard code the first ABC123 and set {postID} for the second one and it runs successfully. But if I set both to posteId it fails.
below is my code. I need to be able to change the ABC123 to postId because I will not know what it will be in my index.js code.
exports.sanitizePost = functions.database
.ref('/posts/ABC123/question/{postId}')
.onWrite(event => {
const post = event.data.val()
if (post.sanitized) {
return
}
console.log("Sanitizing new post " + event.params.pushId)
console.log(post)
post.sanitized = true
post.question = sanitize(post.question)
const promise = event.data.ref.set(post)
return promise
})
function sanitize(s) {
var sanitizedText = s
sanitizedText = sanitizedText.replace(/\bstupid\b/ig, "wonderful")
return sanitizedText
}
I need .ref('/posts/ABC123/question/{postId}') to be .ref('/posts/{postID}/question/{postId}') or somthing that works.
You cannot have the same parameter twice in a path. But you can have two parameters of different names, e.g.
exports.sanitizePost = functions.database
.ref('/posts/{questionId}/question/{postId}')