Advice from a Parse developer forum said to "limit saveAll to 75 objects unless one wants saveAll to make its own batches" which by default are 20 objects. And to put this in a promise chain.
I need to do a saveAll promise chain where I don't know how many promises I need.
How would this be done?
I have an Array of Arrays. The sub arrays are all length 75. I need all the indexes of the master array to be saveAll in a Promise each.
var savePromises = []; // this will collect save promises
while((partition=partitionedArray.pop()) != null){
savePromises.push(Parse.Object.saveAll(partition, {
success: function(objs) {
// objects have been saved...
},
error: function(error) {
// an error occurred...
status.error("something failed");
}
}));
}
return Parse.Promise.when(savePromises);
}).then(function() {
// Set the job's success status
status.success("successful everything");
A nice way to do this is to build the chain of promises recursively. If you've already batched the objects that need saving into batches, then some of the work is done already.
// assume batches is [ [ unsaved_object0 ... unsaved_object74 ], [ unsaved_object75 ... unsaved_object149 ], ... ]
function saveBatches(batches) {
if (batches.length === 0) { return Parse.Promise.as(); }
var nextBatch = batches[0];
return Parse.Object.saveAll(nextBatch).then(function() {
var remainingBatches = batches.slice(1, batches.length);
return saveBatches(remainingBatches);
});
}
EDIT - To call this, just call it and handle the promise it returns...
function doAllThoseSaves() {
var batches = // your code to build unsaved objects
// don't save them yet, just create (or update) e.g....
var MyClass = Parse.Object.extend("MyClass")
var instance = new MyClass();
// set, etc
batches = [ [ instance ] ]; // see? not saved
saveBatches(batches).then(function() {
// the saves are done
}, function(error) {
// handle the error
});
}
EDIT 2 - At some point, the transactions you want to won't fit under the burst limit of the free tier, and spread out (somehow) won't fit within the timeout limit.
I've struggled with a similar problem. In my case, it's a rare, admin-facing migration. Rare enough and invisible to the end user, to have made me lazy about a solid solution. This is kind of a different question, now, but a few ideas for a solid solution could be:
see underscore.js _.throttle(), running from the client, to spread the transactions out over time
run your own node server that throttles calls into parse similarly (or the equal) to _.throttle().
a parse scheduled job that runs frequently, taking a small bite at a time (my case involves an import file, so I can save it quickly initially, open it in the job, count the number of objects that I've created so far, scan accordingly into the file, and do another batch)
my current (extra dumb, but functional) solution: admin user manually requests N small batches, taking care to space those requests ("one mississippi, two mississippi, ...") between button presses
heaven forbid - hire another back-end, remembering that we usually get what we pay for, and parse -- even at the free-tier -- is pretty nice.
Related
I am trying to resolve the array of promises together. Not sure how to do it. Let me share the pseudo code for it.
async function sendNotification(user, notificationInfo) {
const options = {
method: 'POST',
url: 'http://xx.xx.xx:3000/notification/send',
headers:
{ 'Content-Type': 'application/json' },
body:
{ notificationInfo, user },
json: true,
};
console.log('sent');
return rp(options);
}
I have wrapped the sendNotification method in another method which returns the promise of rp(request-promise) module.
Next i am pushing this sendNotification method in array of promise , something like this
const notificationWorker = [];
for (const key3 in notificationObject) {
if(notificationObject[key3].users.length > 0) {
notificationWorker.push(sendNotification(notificationObject[key3].users, notificationObject[key3].payload)); // problem is notification are going as soon as i am pushing in notificationWorker array.
}
}
// task 1 - send all notifications
const result = await Promise.all(notificationWorker); // resolving all notification promises together
// task 2 - update values in db , after sending all notifications
const result2 = await Promise.all(updateWorker); // update some values in db
In above code , my problem is notifications are going as soon as i am pushing it in notificationWorker array. I want all notifications to go together, when i run await Promise.all(notificationWorker)
Not sure , how to achieve what i am trying?
I understood the question partially , but then i feel this is difference between nodejs working concurrently and we trying to achieve parallelism , isn't that so.
Nodejs just switching between the tasks by , and not actually parallely doing it.Child Process might help you in that case.
So for eg. if you go through a snippet
function done(i){
try{
return new Promise((resolve,reject)=>{
console.log(i);
resolve("resolved " + i + "th promise");
})
}catch(e){
return null;
}
}
let promises = [];
for(let i=0;i < 100000; i++){
promises.push(done(i));
}
So console starts even when you dont call Promise.all right ? this was your question but infact Promise.all should not suffice your thing , should go by spwaning child processes to achieve parallelism to some extent.
The point i am trying to make it you are potraying the question to do something like first generate array of promises and start all of them once when Promise.all is called but in my opinion Promise.all also will be running concurrently not giving you what you want to achieve.
Something like this - https://htayyar.medium.com/multi-threading-in-javascript-with-paralleljs-10e1f7a1cf32 || How to create threads in nodejs
Though most of these cases show up when we need to do a cpu intensive task etc but here we can achieve something called map reduce to distribute you array of users in parts and start that array to loop and send notifications.
All of the solutions, i am presenting is to achieve some kind of parallelism but i dont think sending array of huge amount of users would ever be done easily (with less resources - instance config etc) at same instant
I'm developing a SAPUI5 deployed to a mobile device which is experiencing performance issues.
The below function returns a promise,
It iterates through categories and questions using underscore _each loops,
It then performs a READ on each question in turn, before finally updating the view Model and resolving the promise.
Is there any issue with doing it this way and can it be further optimised?
_getAnswers: function() {
return new Promise(function(resolve, reject) {
// Loop through Categories for questions.
_.each(oViewData.categories, function(result, index) {
// For each Category, read Answers for each Question
_.each(result, function(resultInner, indexInner) {
// Read AnswerSet on QuestionId
surveyModel.read("/AnswerSet", {
filters: [
new Filter("QuestionId", FilterOperator.EQ, resultInner.QuestionId)
],
success: function(oData) {
oData.results = _.sortBy(oData.results, 'AnswerId');
// Populate Answer Array for Question
var oAnswersArray = [];
_.each(oData.results, function(resultInnerInner, indexInnerInner) {
oAnswersArray.push(resultInnerInner);
});
// Check what the current Answer is for the Question.
_.each(oAnswersArray, function(answerData, answerIndex) {
if (oViewData.categories[resultInner.CategoryId].questions[a].AnswerId === oAnswersArray[answerIndex].AnswerId) {
oAnswersArray[answerIndex].Selected = true;
}
});
// Write back the Answer Array to the viewModel
oViewData.categories[resultInner.CategoryId].questions[a].answers = oAnswersArray;
oViewModel.setData(oViewData);
// Go to next Question in the Loop.
a++;
// resovle Promise and continue.
resolve(true);
},
error: function(oError) {}
});
});
});
});
}
In computer science there is a term called "Big O' Notation". This is used to measure the time your algorithm takes given the amount of data. In general, nested loops makes the computation time go up.
What you are doing is known as O(N^2) where each nested loop increments the exponent.
Please read this: https://rob-bell.net/2009/06/a-beginners-guide-to-big-o-notation/
If I were you, I would change how you are accessing the data and look into a better structure that does not involve nested loops.
i.e. Don't load the items in the categories until a category is clicked.
EDIT:
It looks like you are returning your callback in every iteration of the second loop. This is not optimal either.
I am using npm ws module (or actually the wrapper called isomorphic-ws) for websocket connection.
NPM Module: isomorphic-ws
I use it to receive some array data from a websocket++ server running on the same PC. This data is then processed and displayed as a series of charts.
Now the problem is that the handling itself takes a very long time. I use one message to calculate 16 charts and for each of them I need to calculate a lot of logarithms and other slow operations and all that in JS. Well, the whole refresh operation takes about 20 seconds.
Now I could actually live with that, but the problem is that when I get a new request it is processed after the whole message handler is finished. And if I get several requests in the meantime, all of them shall be processed as they came in. And so the requests are there queued and the current state gets more and more outdated as the time goes on...
I would like to have a way of detecting that there is another message waiting to be processed. If that is the case I could just stop the current handler at any time and start over... So when using npm ws, is there a way of telling that there is another message waiting in to be processed?
Thanks
You need to create some sort of cancelable job wrapper. It's hard to give a concrete suggestion without seeing your code. But it could be something like this.
const processArray = array => {
let canceled = false;
const promise = new Promise((resolve, reject) => {
// do something with the array
for(let i = 0; i < array.length; i++) {
// check on each iteration if the job has been canceled
if(canceled) return reject({ reason: 'canceled' });
doSomething(array[i])
}
resolve(result)
})
return {
cancel: () => {
cancel = true
},
promise
}
}
const job = processArray([1, 2, 3, ...1000000]) // huge array
// handle the success
job.promise.then(result => console.log(result))
// Cancel the job
job.cancel()
I'm sure there are libraries to serve this exact purpose. But I just wanted to give a basic example of how it could be done.
PREFACE
So it seems I've coded myself into a corner again. I've been teaching myself how to code for about 6 months now and have a really bad habit of starting over and changing projects so please, if my code reveals other bad practices let me know, but help me figure out how to effectively use promises/callbacks in this project first before my brain explodes. I currently want to delete it all and start over but I'm trying to break that habit. I did not anticipate the brain melting difficulty spike of asynchronicity (is that the word? spellcheck hates it)
The Project
WorldBuilder - simply a CLI that will talk to a MySQL database on my machine and mimic basic text game features to allow building out an environment quickly in my spare time. It should allow me to MOVE [DIRECTION] LOOK at and CREATE [ rooms / objects ].
Currently it works like this:
I use Inquirer to handle the prompt...
request_command: function(){
var command = {type:"input",name:"command",message:"CMD:>"}
inquirer.prompt(command).then(answers=>{
parser.parse(answers.command);
}).catch((error)=>{
console.log(error);
});
}`
The prompt passed whatever the user types to the parser
parse: function(text) {
player_verbs = [
'look',
'walk',
'build',
'test'
]
words = text.split(' ');
found_verb = this.find(player_verbs, words)[0];
if (found_verb == undefined) {
console.log('no verb found');
} else {
this.verbs.execute(found_verb, words)
}
}
The parser breaks the string into words and checks those words against possible verbs. Once it finds the verb in the command it accesses the verbs object...
(ignore scope mouthwash, that is for another post)
verbs: {
look: function(cmds) {
// ToDo: Check for objects in the room that match a word in cmds
player.look();
},
walk: function(cmds) {
possible_directions = ['north','south','east','west'];
direction = module.exports.find(possible_directions, cmds);
player.walk(direction[0]);
},
build: function(cmds) {
// NOTE scope_mouthwash exists because for some
// reason I cannot access the global constant
// from within this particular nested function
// scope_mouthwash == menus
const scope_mouthwash = require('./menus.js');
scope_mouthwash.room_creation_menu();
},
execute: function(verb, args) {
try{
this[verb](args);
}catch(e){
throw new Error(e);
}
}
}
Those verbs then access other objects and do various things but essentially it breaks down to querying the database an unknown number of times to either retrieve info about an object and make a calculation or store info about an object. Currently my database query function returns a promise.
query: function (sql) {
return new Promise(function(resolve, reject){
db.query(sql, (err, rows)=>{
if (err) {
reject(err);
}
else {
resolve(rows);
}
});
});
},
The Problem
Unfortunately I jumped the gun and started using promises before I fully understood when to use them and I believe I should have used callbacks in some of these situations but I just don't quite get it yet. I solved 'callback hell' before I had to experience 'callback hell' and now am trying to avoid 'promise hell'. My prompt used to call itself in a loop after it triggered the required verbs but this whole approach broke down when I realized I'd get prompt messages in the middle of other prompt cycles for room building and such.
Should my queries be returning promises or should I rewrite them to use callback functions handled by whichever verb calls the query? How should I determine when to prompt the user again in the situation of having an unknown number of asynchronous processes?
So, put in a different way, my question is..
Given the parameters of my program how should I be visualizing and managing the asynchronous flow of commands, each of which may chain to an unknown number of database queries?
Possible Solution directions that have occurred to me..
Create an object that keeps track of when there are pending promises and
simply prompts the user when all promises are resolved or failed.
Rewrite my program to use callback functions where possible and force a known number of promises. If I can get all verbs to work off callback functions I might be able to say something like Prompt.then(resolve verb).then(Prompt)...
Thank you for bearing with me, I know that was a long post. I know enough to get myself in trouble and Im pretty lost in the woods right now.
I recently did a lot of coding in AngularJS. After some time it started to feel comfortable with it and also got really productive. But unfortunately there is this one thing I don't understand:
Within my project I need to get data through $http.get and a RESTful API server. This is where I started to stumble first. After implementing promise ($q.defer etc and .then) at functions which are processing data that's necessary to continue, I thought I conquered the problem.
But in this code:
$scope.getObservationsByLocations = function() {
var promise = $q.defer();
var locationCount = 0;
angular.forEach($scope.analysisData, function(loc) { // for each location
$http.get($scope.api + 'Device?_format=json', { // get all devices
params: {
location: loc.location.id
}
}).then(function (resultDevices) {
var data = angular.fromJson(resultDevices);
promise.resolve(data);
// for each device in this location
angular.forEach(angular.fromJson(resultDevices).data.entry.map(function (dev) {
http.get($scope.api + 'Observation?_format=json', { // get all observations
params: {
device: dev.resource.id
}
}).then(function (resultObservations) {
var observations = angular.fromJson(resultObservations);
// for each obervation of that device in this location
angular.forEach(observations.data.entry.map(function(obs) {
$scope.analysisData[locationCount].observations.push({observation: obs.resource});
}));
})
}))
});
locationCount++
});
return promise.promise
};
I can't understand in which order the commands are executed. Since I use the Webstorm IDE and it's debugging feature, it would be more accurate to say I don't know why the commands are executed in an order I don't understand.
Thinking simple, everything included in the forEach have to be executed before the return is reached, because $http.get's are connected through .then's. But following the debugging information, the function iterates over locationCount++ and even returns the promise before it goes deeper (meaning after the first .then() ).
What's that all about? Did I misunderstood this part of the AngularJS concept?
Or is this just really bad practice and I should reach out for a different solution?
If the context is important/interesting: Objects are based on i.e. https://www.hl7.org/fhir/2015May/location.html#5.15.3
With JavaScript you can create only single thread applications, although e.g. here they say it is not guaranteed to be like that.
But we are talking about the real world and real browsers, so your code sample is running as a single thread (by the way the same thread is also used for rendering your CSS and HTML, at least in Firefox).
When it comes to an asynchronous call
$http.get($scope.api + 'Device?_format=json', {
it says "hey, I can do that later". And it waits with that because it must go on with the current thread.
Then, once the current task is done with return it finally can start getting the remote data.
Proof? Check this fiddle:
console.log(1);
for (var i=0;i<1000000;i++) setTimeout(function(){
console.log(2);
},0);
console.log(3);
You see the spike with the for loop? This is the moment when it registers the setTimeout asynchronous calls. Still 3 is printed before 2 because the task is not done until the 3 is printed.
The $http.get is asynchronous, so depending on (among other things) how large the fetched data is, the time it takes to 'complete' the get is variable. Hence why there is no saying in what order they will be completed