How can I perform a bulk find or create with ember.js? This would be simple to do synchronously (foreach... continue if exists). But working with ember's asynchronous store creates lots of overhead in keeping track of the state of the operation.
Specifically, I have a variable to keep track of the number of objects waiting to be processed (createIfNotExistTaskCounter), so I can check when the store has finished working on all of the objects to be saved. And I use an array to keep track of the items stored so far (createIfNotExistQueue) - I can't let the store handle this task, because I can't count on an item being found after it has been saved.
Here's my best solution below (also on JS Bin). Is there an easier way to do this?
App = Ember.Application.create({});
App.LSAdapter = DS.LSAdapter.extend({
namespace: 'whitespace'
});
App.Store = DS.Store.extend({
adapter: App.LSAdapter
});
App.Fruit = DS.Model.extend({
name: DS.attr("string")
});
App.IndexRoute = Ember.Route.extend({
createIfNotExistTaskCounter: 0, // store number of items waiting to be processed
createIfNotExistQueue: [], // store a list of the items being added, to prevent duplicate adds
setupController: function(controller) {
/* This is a simplified version of a real task I'm trying to acomplish. The code adds a list of objects to the store, only creating them if they don't exist. After the list has been processed, the contents of the store are shown.
To achieve this end I've used a counter and a queue to keep track of the operations' state. Is there a simpler way to do this? These overheads seem excessive for such a straightforward bulk insert operation.
*/
var fruitToStore = ["apple", "pear", "banana", "apple"],
store = this.get('store');
this.set('createIfNotExistTaskCounter', fruitToStore.length);
for(var i=0; i<fruitToStore.length; i++) {
this.createIfNotExist(fruitToStore[i]);
}
},
createListener: function() {
if(this.get('createIfNotExistTaskCounter') !== 0) return;
this.get('store').find('fruit').then(function(results) {
// should only print three fruits, one of each type
for (var i = 0; i < results.content.length; i++) {
console.log(results.content[i].get('name'));
};
});
}.observes('createIfNotExistTaskCounter'),
createIfNotExist: function(f) {
var store = this.get('store'),
queue = this.get('createIfNotExistQueue'),
that = this;
// prevent duplicate records being created by adding every (distinct) item to a queue
// the queue is used because there seems to be no way to tell if an item is already (asynchonously) being found / created / saved
if(queue.indexOf(f) !== -1) {
that.decrementProperty('createIfNotExistTaskCounter');
return;
}
queue.push(f);
// find or create
store.find('fruit', {name: f}).then(function(results) {
// found...
if(results.get('length') !== 0) {
that.decrementProperty('createIfNotExistTaskCounter');
return;
}
// ...else create
var fruit = store.createRecord('fruit', {name: f});
fruit.save().then(function() {
that.decrementProperty('createIfNotExistTaskCounter');
}, function() {
console.log("save failed!");
});
});
}
});
If you return a promise from a then callback, you can create a chain of promises that behaves quite like a queue.
First you start with an already resolved callback, then you keep replacing it with a "then"-able object.
queue: new Ember.RSVP.resolve,
addToQueue: function() {
this.queue = this.queue.then(function() {
return new Ember.RSVP.Promise(function(resolve, reject){
// something that eventually calls resolve
})
})
}
Here's my updated JSBin of your code: http://jsbin.com/OtoZowI/2/edit?html,console
There is probably a way to make that much smaller if you can work out a way to return the existing find / save promises instead of creating a new one. I played with it a bit but I need to get back to work :P
Also, you can collect together a bunch of promises with RSVP.all and resolve only once they're all resolved. Depending on your actual code, this might be a much cleaner solution - do all the finds, wait until they're all resolved, then create the missing objects.
The RSVP docs have a good example of that here: https://github.com/tildeio/rsvp.js/blob/master/README.md#arrays-of-promises
Related
I have a web socket that receives data from a web socket server every 100 to 200ms, ( I have tried both with a shared web worker as well as all in the main.js file),
When new JSON data arrives my main.js runs filter_json_run_all(json_data) which updates Tabulator.js & Dygraph.js Tables & Graphs with some custom color coding based on if values are increasing or decreasing
1) web socket json data ( every 100ms or less) -> 2) run function filter_json_run_all(json_data) (takes 150 to 200ms) -> 3) repeat 1 & 2 forever
Quickly the timestamp of the incoming json data gets delayed versus the actual time (json_time 15:30:12 vs actual time: 15:31:30) since the filter_json_run_all is causing a backlog in operations.
So it causes users on different PC's to have websocket sync issues, based on when they opened or refreshed the website.
This is only caused by the long filter_json_run_all() function, otherwise if all I did was console.log(json_data) they would be perfectly in sync.
Please I would be very very grateful if anyone has any ideas how I can prevent this sort of blocking / backlog of incoming JSON websocket data caused by a slow running javascript
function :)
I tried using a shared web worker which works but it doesn't get around the delay in main.js blocked by filter_json_run_all(), I dont thing I can put filter_json_run_all() since all the graph & table objects are defined in main & also I have callbacks for when I click on a table to update a value manually (Bi directional web socket)
If you have any ideas or tips at all I will be very grateful :)
worker.js:
const connectedPorts = [];
// Create socket instance.
var socket = new WebSocket(
'ws://'
+ 'ip:port'
+ '/ws/'
);
// Send initial package on open.
socket.addEventListener('open', () => {
const package = JSON.stringify({
"time": 123456,
"channel": "futures.tickers",
"event": "subscribe",
"payload": ["BTC_USD", "ETH_USD"]
});
socket.send(package);
});
// Send data from socket to all open tabs.
socket.addEventListener('message', ({ data }) => {
const package = JSON.parse(data);
connectedPorts.forEach(port => port.postMessage(package));
});
/**
* When a new thread is connected to the shared worker,
* start listening for messages from the new thread.
*/
self.addEventListener('connect', ({ ports }) => {
const port = ports[0];
// Add this new port to the list of connected ports.
connectedPorts.push(port);
/**
* Receive data from main thread and determine which
* actions it should take based on the received data.
*/
port.addEventListener('message', ({ data }) => {
const { action, value } = data;
// Send message to socket.
if (action === 'send') {
socket.send(JSON.stringify(value));
// Remove port from connected ports list.
} else if (action === 'unload') {
const index = connectedPorts.indexOf(port);
connectedPorts.splice(index, 1);
}
});
Main.js This is only part of filter_json_run_all which continues on for about 6 or 7 Tabulator & Dygraph objects. I wante to give an idea of some of the operations called with SetTimeout() etc
function filter_json_run_all(json_str){
const startTime = performance.now();
const data_in_array = json_str //JSON.parse(json_str.data);
// if ('DATETIME' in data_in_array){
// var milliseconds = (new Date()).getTime() - Date.parse(data_in_array['DATETIME']);
// console.log("milliseconds: " + milliseconds);
// }
if (summary in data_in_array){
if("DATETIME" in data_in_array){
var time_str = data_in_array["DATETIME"];
element_time.innerHTML = time_str;
}
// summary Data
const summary_array = data_in_array[summary];
var old_sum_arr_krw = [];
var old_sum_arr_irn = [];
var old_sum_arr_ntn = [];
var old_sum_arr_ccn = [];
var old_sum_arr_ihn = [];
var old_sum_arr_ppn = [];
var filtered_array_krw_summary = filterByProperty_summary(summary_array, "KWN")
old_sum_arr_krw.unshift(Table_summary_krw.getData());
Table_summary_krw.replaceData(filtered_array_krw_summary);
//Colour table
color_table(filtered_array_krw_summary, old_sum_arr_krw, Table_summary_krw);
var filtered_array_irn_summary = filterByProperty_summary(summary_array, "IRN")
old_sum_arr_irn.unshift(Table_summary_inr.getData());
Table_summary_inr.replaceData(filtered_array_irn_summary);
//Colour table
color_table(filtered_array_irn_summary, old_sum_arr_irn, Table_summary_inr);
var filtered_array_ntn_summary = filterByProperty_summary(summary_array, "NTN")
old_sum_arr_ntn.unshift(Table_summary_twd.getData());
Table_summary_twd.replaceData(filtered_array_ntn_summary);
//Colour table
color_table(filtered_array_ntn_summary, old_sum_arr_ntn, Table_summary_twd);
// remove formatting on fwds curves
setTimeout(() => {g_fwd_curve_krw.updateOptions({
'file': dataFwdKRW,
'labels': ['Time', 'Bid', 'Ask'],
strokeWidth: 1,
}); }, 200);
setTimeout(() => {g_fwd_curve_inr.updateOptions({
'file': dataFwdINR,
'labels': ['Time', 'Bid', 'Ask'],
strokeWidth: 1,
}); }, 200);
// remove_colors //([askTable_krw, askTable_inr, askTable_twd, askTable_cny, askTable_idr, askTable_php])
setTimeout(() => { askTable_krw.getRows().forEach(function (item, index) {
row = item.getCells();
row.forEach(function (value_tmp){value_tmp.getElement().style.backgroundColor = '';}
)}); }, 200);
setTimeout(() => { askTable_inr.getRows().forEach(function (item, index) {
row = item.getCells();
row.forEach(function (value_tmp){value_tmp.getElement().style.backgroundColor = '';}
)}); }, 200);
color_table Function
function color_table(new_arr, old_array, table_obj){
// If length is not equal
if(new_arr.length!=old_array[0].length)
console.log("Diff length");
else
{
// Comparing each element of array
for(var i=0;i<new_arr.length;i++)
//iterate old dict dict
for (const [key, value] of Object.entries(old_array[0][i])) {
if(value == new_arr[i][key])
{}
else{
// console.log("Different element");
if(key!="TENOR")
// console.log(table_obj)
table_obj.getRows()[i].getCell(key).getElement().style.backgroundColor = 'yellow';
if(key!="TIME")
if(value < new_arr[i][key])
//green going up
//text_to_speech(new_arr[i]['CCY'] + ' ' +new_arr[i]['TENOR']+ ' getting bid')
table_obj.getRows()[i].getCell(key).getElement().style.backgroundColor = 'Chartreuse';
if(key!="TIME")
if(value > new_arr[i][key])
//red going down
table_obj.getRows()[i].getCell(key).getElement().style.backgroundColor = 'Crimson';
}
}
}
}
Potential fudge / solution, thanks Aaron :):
function limiter(fn, wait){
let isCalled = false,
calls = [];
let caller = function(){
if (calls.length && !isCalled){
isCalled = true;
if (calls.length >2){
calls.splice(0,calls.length-1)
//remove zero' upto n-1 function calls from array/ queue
}
calls.shift().call();
setTimeout(function(){
isCalled = false;
caller();
}, wait);
}
};
return function(){
calls.push(fn.bind(this, ...arguments));
// let args = Array.prototype.slice.call(arguments);
// calls.push(fn.bind.apply(fn, [this].concat(args)));
caller();
};
}
This is then defined as a constant for a web worker to call:
const filter_json_run_allLimited = limiter(data => { filter_json_run_all(data); }, 300); // 300ms for examples
Web worker calls the limited function when new web socket data arrives:
// Event to listen for incoming data from the worker and update the DOM.
webSocketWorker.port.addEventListener('message', ({ data }) => {
// Limited function
filter_json_run_allLimited(data);
});
Please if anyone knows how websites like tradingview or real time high performance data streaming sites allow for low latency visualisation updates, please may you comment, reply below :)
I'm reticent to take a stab at answering this for real without knowing what's going on in color_table. My hunch, based on the behavior you're describing is that filter_json_run_all is being forced to wait on a congested DOM manipulation/render pipeline as HTML is being updated to achieve the color-coding for your updated table elements.
I see you're already taking some measures to prevent some of these DOM manipulations from blocking this function's execution (via setTimeout). If color_table isn't already employing a similar strategy, that'd be the first thing I'd focus on refactoring to unclog things here.
It might also be worth throwing these DOM updates for processed events into a simple queue, so that if slow browser behavior creates a rendering backlog, the function actually responsible for invoking pending DOM manipulations can elect to skip outdated render operations to keep the UI acceptably snappy.
Edit: a basic queueing system might involve the following components:
The queue, itself (this can be a simple array, it just needs to be accessible to both of the components below).
A queue appender, which runs during filter_json_run_all, simply adding objects to the end of the queue representing each DOM manipulation job you plan to complete using color_table or one of your setTimeout` callbacks. These objects should contain the operation to performed (i.e: the function definition, uninvoked), and the parameters for that operation (i.e: the arguments you're passing into each function).
A queue runner, which runs on its own interval, and invokes pending DOM manipulation tasks from the front of the queue, removing them as it goes. Since this operation has access to all of the objects in the queue, it can also take steps to optimize/combine similar operations to minimize the amount of repainting it's asking the browser to do before subsequent code can be executed. For example, if you've got several color_table operations that coloring the same cell multiple times, you can simply perform this operation once with the data from the last color_table item in the queue involving that cell. Additionally, you can further optimize your interaction with the DOM by invoking the aggregated DOM manipulation operations, themselves, inside a requestAnimationFrame callback, which will ensure that scheduled reflows/repaints happen only when the browser is ready, and is preferable from a performance perspective to DOM manipulation queueing via setTimeout/setInterval.
I'm trying to keep track of an increment of a certain reactive value in Meteor. If the current value has increased by 1 or more, I want something to happen. I do have two problems:
First: I don't know how I can make an if-statement of this function.
Second: I don't know how I can keep track of the increases.
This is the code I have now, using the Mongo.Collection cars (which is from an API):
api = DDP.connect('url');
const currentCars = new Meteor.Collection('cars', api);
const newCars = cars.find().count()
if (Meteor.isClient) {
Template.currentCars.helpers({
carsInCity: function() {
return currentCars.find(
{
location: "city"
}).count();
},
})
}
So there's a current amount of cars in the city. Everytime when there is one more car, I want something to happen in the code. But how on earth can I do that? Maybe by keeping track of when the database has been updated?
A fairly straight-forward solution would be to store the current amount of data in that collection, then run a reactive computation to see if anything changed.
Something like this:
let currentCarsCount = cars.find().count()
Tracker.autorun(function checkForAddedCars() {
// cars.find() is our reactive source
const newCarsCount = cars.find().count()
if(newCarsCount > currentCarsCount) {
currentCarsCount = newCarsCount
// There's new cars, handle them now
// ...
}
})
You may also want to use a template-level autorun so that you don't have to manage stopping checkForAddedCars. You could also store currentCarsCount as a state on the template instance instead of as a hoisted loner.
For example:
Template.currentCars.onCreated(function() {
const templateInstance = this;
// equivalent:
const templateInstance = Template.instance();
templateInstance.currentCarsCount = cars.find().count();
templateInstance.autorun(function checkForAddedCars() {
// cars.find() is our reactive source
const newCarsCount = cars.find().count();
if(newCarsCount > templateInstance.currentCarsCount) {
templateInstance.currentCarsCount = newCarsCount;
// There's new cars, handle them now
// ...
}
});
});
It would also allow you to access currentCarsCount from other places in the template code.
I am tryng to create posts using a for loop, but when i look at Parse database only the last object of my array get's stored. this is the code i wrote.
var Reggione = Parse.Object.extend("Reggione");
var creaReggione = new Reggione();
var selectobject = $('#searcharea')[0];
for (var i = 2; i < selectobject.length; i++) {
creaReggione.set("name", selectobject.options[i].text);
creaReggione.save();
Thanks, Bye.
Do this by creating an array of new objects, then save them together...
var newObjects = [];
for (var i = 2; i < selectobject.length; i++) {
creaReggione.set("name", selectobject.options[i].text);
newObjects.push(creaReggione);
// ...
}
Parse.Object.saveAll(newObjects);
Remember, if you want something to happen after saveAll completes (like call response.success() if you're in cloud code), then you should use that promise as follows...
Parse.Object.saveAll(newObjects).then(function(result) {
response.success(result);
}, function(error) {
response.error(error);
});
In extension to danhs answer, the reason this does not work is because only one transaction can happen at a time from the JS client to Parse.
Therefore in your loop the first call to .save() is made and the object is saved to Parse at some rate (asynchrounously), in that time the loop continues to run and skips over your other save calls, these objects are NOT queued to be saved. As Danh pointed out, you must use Parse's batch operations to save multiple objects to the server in one go, to do this you can:
var newObjects = [];
for (var i = 2; i < selectobject.length; i++) {
creaReggione.set("name", selectobject.options[i].text);
newObjects.push(creaReggione);
// ...
}
Parse.Object.saveAll(newObjects);
Hope this helps, I'd also recommend taking a look at Parse's callback functions on the save method to get more details on what happened (you can check for errors and success callbacks here to make debugging a little easier)
An example of this would be to extend the previous call with:
Parse.Object.saveAll(newObjects, {
success: function(messages) {
console.log("The objects were successfully saved...")
},
error: function(error) {
console.log("An error occurred when saving the messages array: %s", error.message)
}
})
I hope this is of some help to you
I have an array of objects containing a users info.
var names=[{name: 'yoda',
address:'123 Compton',
email:'yodalicious#force.com'},
{name: 'darth vader',
address:'69 harlem',
email:'elevader#force.com'},{....}]
this is a list that can range from 100 to 1000 users.
The problem is that I need to request/pull information from the web and DB about each user and create a new array containing more details about each user. this will be 3 sources. Now each pull takes anywhere from 10 ms to 1 min.
that is where I jump to promises. ("native-promise-only"), as an example:
require("native-promise-only");
function getFile(file) {
return new Promise(function(resolve){
fakeAjax(file,resolve);
});
}
//Recursive call
function recursivePromise(originalArray, newArray){
if(isEmpty(originalArray) ){
store(newArray);
display(newArray);
}
else{
var currentItem = originalArray[0];
var p1 = getFile(currentItem.name);
var p2 = getFile(currentItem.address );
var p3 = getFile(currentItem.email);
var newDataItem={};
p1.then(function(msg){
//TODO check status
newDataItem.nameinfo= msg;
return p2;
})
.then(function(msg){
//TODO check status
newDataItem.addressinfo= msg
return p3;
})
.then(function(msg){
newDataItem.emailinfo= msg
newArray.add(newDataItem);
recursivePromise(originalArray.shift(), newArray)
});
}
}
var new_array=[];
recursivePromise(names, new_array);
this is a rough code, I have something similar and it works! somewhat. But a bug in me tells me that i might be setting up a future failure. I am doing it recursively because the position of the item in 'names' is important. so they need to be processed in order.
Issues as I see them :
Coding errors :
fruits comes out of nowhere; presumably originalArray.shift() was intended.
array.shift() removes the 0th element from the array and returns the removed element, not the array.
originalArray would be destroyed, which is not a good idea.
.add() is not an array method, presumably .push() was intended.
Pattern issues :
The p1.then(...).then(...) chain is a rather cumbersome way to build newDataItem; Promise.all(p1, p2, p3).then(...) would be more readable, with newDataItem built in a single expression.
Recursion is not necessary; the process would be more simply coded as ...; Promise.all(arrayOfPromises).then(function(newArray) {...}); (that's a second use of Promise.all()); although the promises will settle in any order they like, newArray will be congruent with originalArray.
newArray can be generated internally, without needing to pass in [].
function process(originalArray) {
var promises = originalArray.map(function(currentItem) {
var p1 = getFile(currentItem.name);
var p2 = getFile(currentItem.address);
var p3 = getFile(currentItem.email);
return Promise.all([p1, p2, p3]).then(function(results) {
return {
nameinfo: results[0],
addressinfo: results[1],
emailinfo: results[2]
};
});
});
return Promise.all(promises).then(function(newArray) {
display(newArray);
return newArray;
});
}
process(names).then(function(newArray) {
store(newArray);
});
display(newArray) and store(newArray) could both be executed internally or externally. I split them above to demonstrate the two possibilities.
Here is a simple task I would like to accomplish on Parse.com with Cloud Code.
The task consists to delete a Unit and what is related to it.
One Unit has several Sentences related to it and each Sentence has one or more Translations.
So when the task is performed, the Unit as well as the Sentence and Translations should be deleted.
I have a strong feeling I should be using Promises (and chain them up) in order to do what I want in a good manner.
Below is the code I wrote, but it works only partially (The translations are deleted, not the rest).
Parse.Cloud.define("deleteUnitAndDependencies", function(request, response) {
var unitListQuery = new Parse.Query("UnitList");
unitListQuery.equalTo("objectId", request.params.unitID);
unitListQuery.equalTo("ownerID", request.params.userID);
unitListQuery.find().then(function(resUnit) {
var sentenceListQuery = new Parse.Query("SentenceList");
sentenceListQuery.equalTo("unit", resUnit[0]);
return sentenceListQuery.find();
}).then(function(resSentence) {
var translatListQuery = new Parse.Query("TranslatList");
for (i = 0; i < resSentence.length; i++) {
var query = new Parse.Query("TranslatList");
query.equalTo("sentence", resSentence[i]);
translatListQuery = Parse.Query.or(translatListQuery, query);
}
return translatListQuery.find();
}).then(function(resTranslat) {
for (iT = 0; iT < resTranslat.length; iT++) {
resTranslat[iT].destroy({});
}
});
});
I surely need to add some lines of code like:
resSentence[x].destroy({});
and:
resUnit[0].destroy({});
The problem is that I do not quite see where is the adequate place for that.
Collect the objects to be deleted then use Parse.Object.destroyAll(someArray); to delete all at once.
In cases like this I like to use a scope variable to hold things for later use.
var scope = {
sentences: [],
units: []
};
// later inside then block...
scope.sentences.push(resSentence[i]);
// ...now we have them collected safely
.then(function() {
return Parse.Object.destroyAll(scope.sentences);
})