I am looking for the most performant solution for sending structured data to the client in the Meteor framework on a request.
THE ISSUE:
Sometimes, before you send data from the database to the client, you want to add some server side generated additional information beeing sent to the client (i.e. security credentials for many objects). This data can be time-critical (i.e. due to an expiration timestamp) and therefore should not be stored in the db. Also, this data sometimes cannot be processed on the client side (i.e. due to security reasons). In many cases, this data will be structurally related to actual database data, but also very much related to a single request, since you might want to have it discarded and re-generated on a new request.
YOU CAN (at least by design..):
create a second collection, store and publish your request-related data there, accepting the write-overhead, and then i.e. in the Meteor.myTemplate.destroyed=function(){...} remove the data again accepting another write-overhead.
store each entry in a session variable, but then you also have to take care
of deleting it later on (Meteor.myTemplate.destroyed=function(){...}), this is my favourite right now, but I am running into problems with storing large objects there.
store this data in the dom (i.e. in the attributes or data fields of hidden or visible elements)
generate this data from the dom with Meteor.call('method',arguments,callback(){...}) by storing the appropriate arguments in the dom and injecting them back with i.e. jQuery in the callback(){...}.
YOU CAN'T: (by design!!)
use transformations within Meteor.publish("name",function(){...}) on the server
use a Meteor.call() within a transformation on a Template.variable=function(){return collection.find(...)} (also not if you have a corresponding Meteor.method() on the client for guessing the result!).
Again, what I am looking for is the best performing solution for this.
To address the transform issue, which I care about because I think in terms of smart models doing things rather than a bunch of anonymous functions, here is an example of server transform reaching the client (not reactive as is, through a call, not a publish, but illustrative of the point under discussion of server transforms).
You will get this:
each image local
data: LOLCATZ RULZ
transform:
data: LOLCATZ RULZ
transform:
each image server transformed
data: LOLCATZ RULZ
transform: XYZ
data: LOLCATZ RULZ
transform: XYZ
from:
<template name='moritz'>
<h3>each image local</h3>
<dl>
{{#each images}}
<dt>data: {{caption}}</dt>
<dd>transform: {{secretPassword}}</dd>
{{/each}}
</dl>
<h3>each image server transformed</h3>
<dl>
{{#each transformed}}
<dt>data: {{caption}}</dt>
<dd>transform: {{secretPassword}}</dd>
{{/each}}
</dl>
</template>
if (Meteor.isServer) {
Images = new Meteor.Collection('images', {
transform: function (doc) {
doc.secretPassword = 'XYZ'
return doc
}
});
Images.allow({
insert: function (userid, doc) {
return true;
}
});
if (Images.find().count() < 1) {
Images.insert({ caption: 'LOLCATZ RULZ'});
}
Meteor.publish('images', function () {
return Images.find();
})
Meteor.methods({
'transformed': function() {
return Images.find().fetch();
}
})
}
else {
Images = new Meteor.Collection('images');
imageSub = Meteor.subscribe('images');
Template.moritz.helpers({
'images': function () {
console.log(Images.find().count() + ' images')
return Images.find();
},
'transformed': function () {
// Should be separated, call should be in route for example
Meteor.call('transformed', function(err,data){
Session.set('transformed', data);
});
return Session.get('transformed');
}
});
}
Have a look at Meteor Streams, you can send something directly to the client from the server without having to use collections on the client.
You could do something with the message you get (I'm using an example from the meteor-streams site):
Client
chatStream.on('message', function(message) {
if(message.expiry > new Date()) {
//Do something with the message (not being read from a collection)
}
});
Even though you do this with some intent on not having it stored, be wary that simple tools (Chrome Inspector) can peak into the Network/Websocket tab (even if its encrypted via SSL) and see the raw data being passed through.
While i'm not sure of your intentions, if this is for security in any scenario never trust whatever data you get from the client.
Your first line would seem ideally answered by 'Mongo collections' but then I read a few conflicting ideas and conclusions I'm not sure I agree with. (For example, why can't you do those two things? Because it must happen on the server?) The most confusing statement for me is:
This data can be time-critical (i.e. due to an expiration timestamp) and therefore should not be stored in the db.
I don't understand what assumptions go into that conclusion, but if you consider storing data in the dom to be plausible, per points 2 and three above, you would seem to be open to systems much less performant than a mongo db.
You know you can, from the server, publish a second collection of the additional server-generated calculations, which you generate on the fly from the data, and put that together with the data once on the client. Kind of like a parent-child relationship: as you display the client document, pull up the additional data from the server from the dynamic collection and work it into your templates.
Jim Mack created a nice example here that proves, how well it works to store db data as well as additional "transform" properties in a Session variable.
Unfortunately this example lacks reactivity and does not perform the desired "transformations" after Meteor's magic re-render. So I grabbed his cool code and added back the reactivity, it's slim code that works very well, but will be outperformed by Jim Mack's example in terms of efficiency.
lolz.html
<head>
<title>lolz</title>
</head>
<body>
{{>myItems}}
</body>
<template name="myItems">
<h3>Reactive Item List with additional properties</h3>
<button id="add">add</button>
<button id="remove">remove</button>
<dl>
{{#each items}}
<dt>data: {{caption}}</dt>
<dd>added property: {{anotherProp _id}}</dd>
{{/each}}
</dl>
</template>
lolz.js
items = new Meteor.Collection('Items');
if (Meteor.isServer) {
items.allow({
insert: function (userid, doc) {
return true;
},
remove: function(userid,doc){
return true;
}
});
while(items.find().count()>0){
items.remove(items.findOne()._id);
}
while (items.find().count() < 3) {
items.insert({caption: 'LOLCATZ RULZ'});
}
Meteor.publish('Items', function () {
return items.find();
});
Meteor.methods({
'getAdditionalProps': function() {
additionalProps={};
items.find().forEach(function(doc){
additionalProps[doc._id]=reverse(doc.caption);
});
return additionalProps;
}
});
function reverse(s){ // server side operation, i.e. for security reasons
return s.split("").reverse().join("");
};
}
if (Meteor.isClient){
Meteor.subscribe('Items');
Meteor.startup(function(){
getAdditionalProps();
itemsHandle=items.find().observe({
added : function(doc){
getAdditionalProps();
},
removed : function(doc){
getAdditionalProps();
},
changed : function(docA,docB){
getAdditionalProps();
}
});
});
Template.myItems.rendered=function(){
console.log(new Date().getTime());
};
Template.myItems.items=function(){
return items.find();
}
Template.myItems.anotherProp=function(id){
return Session.get('additionalProps')[id];
}
Template.myItems.events({
'click #add':function(e,t){
items.insert({caption: 'LOLCATZ REACTZ'});
},
'click #remove':function(e,t){
items.remove(items.findOne()._id);
}
});
}
function getAdditionalProps(){
setTimeout(function(){
Meteor.call('getAdditionalProps',function(error,props){
Session.set('additionalProps',props);
});
},0);
}
Related
I have a pub that wraps and external API. Client subs the external api pub. There is a 'ACTIVATE' button they can push to activate a billing method. Button calls an update method that updates the collection. The pub updates the external api. Simulation runs and updates the client collection. Button changes to 'DEACTIVATE' as expected. This is where the issue comes in. The external api takes some time to return with the updated doc. Within 100-200ms of the button turning to 'DEACTIVATE' it will flip back to 'ACTIVATE' and then 500ms latter back to 'DEACTIVATE' where it should be assuming there were no issues with the external api.
I'm sure I could come up with some hacky solution to deal with this in the client but wondering if there is a way to tell the simulation/client collection that the pub is slow and to not update quite as often? Thus, giving the pub/external api more time to complete it's updates.
This turned out to be really simple.
Client side simulation alone is not enough. The trick is to do server side simulation as well. To accomplish this first setup a hook to the Meteor.publish this object something like this.
_initServer() {
if (Meteor.isServer) {
console.log(`Server initializing external collection "${this.name}"`)
let self = this
Meteor.publish(this.name, function (selector, options) {
check(selector, Match.Optional(Match.OneOf(undefined, null, Object)))
check(options, Match.Optional(Match.OneOf(undefined, null, Object)))
self.publication = this
self._externalApi.fetchAll()
.then((docs)=>docs.forEach((doc)=>this.added(self.name, doc._id, doc)))
.then(()=>this.ready())
// todo handle error
.catch((error)=>console.error(`${self.name}._initServer: self._externalApi.fetchAll`, error))
})
}
}
Then in your update function you can simulate on both the client and server like so:
this.update = new ValidatedMethod({
name: `${self.name}.update`,
validate: (validators && validators.update) ? validators.update : self.updateSchema.validator({clean: true}),
run(doc) {
console.log(`${self.name}.update `, doc)
if (Meteor.isServer && self._externalApi.update) {
// server side simulation
self.changed(doc)
self._externalApi.update(doc._id, doc)
.then(self.changed)
.catch((error)=>handleError(`${self.name}.update`, 'externalApi.update', error))
} else {
// client side simulation
self.collection.update(doc._id, {$set: doc})
}
},
})
Apologizes if this is over simplified these examples are from a large library we use for external api's.
I am trying to remove an item from $firebaseArray (boxes).
The remove funcion:
function remove(boxJson) {
return boxes.$remove(boxJson);
}
It works, however it is immediately added back:
This is the method that brings the array:
function getBoxes(screenIndex) {
var boxesRef = screens
.child("s-" + screenIndex)
.child("boxes");
return $firebaseArray(boxesRef);
}
I thought perhaps I'm holding multiple references to the firebaseArray and when one deletes, the other adds, but then I thought firebase should handle it, no?
Anyway I'm lost on this, any idea?
UPDATE
When I hack it and delete twice (with a timeout) it seems to work:
function removeForce(screenIndex, boxId) {
setTimeout(function () {
API.removeBox(screenIndex, boxId);
}, 1000);
return API.removeBox(screenIndex, boxId);
}
and the API.removeBox:
function removeBox(screenIndex, boxId) {
var boxRef = screens
.child("s-" + screenIndex)
.child("boxes")
.child(boxId);
return boxRef.remove();
}
When you remove something from firebase it is asynchronous. Per the docs the proper way to remove an item is from firebase, using AngularFire is:
var obj = $firebaseObject(ref);
obj.$remove().then(function(ref) {
// data has been deleted locally and in the database
}, function(error) {
console.log("Error:", error);
});
$remove() ... Removes the entire object locally and from the database. This method returns a promise that will be fulfilled when the data has been removed from the server. The promise will be resolved with a Firebase reference for the exterminated record.
Link to docs: https://www.firebase.com/docs/web/libraries/angular/api.html#angularfire-firebaseobject-remove
The most likely cause is that you have a security rules that disallows the deletion.
When you call boxes.$remove Firebase immediately fires the child_removed event locally, to ensure the UI is updated quickly. It then sends the command to the Firebase servers to check it and update the database.
On the server there is a security rule that disallows this deletion. The servers send a "it failed" response back to the client, which then raises a child_added event to fix the UI.
Appearantly I was saving the items again after deleting them. Clearly my mistake:
function removeSelected(boxes) {
var selectedBoxes = Selector.getSelectedBoxes(boxes);
angular.forEach(selectedBoxes, function (box) {
BoxManager.remove(box);
});
Selector.clearSelection(boxes, true);
}
In the clearSelection method I was updating a field on the boxes and saved them again.
Besides the obvious mistake this is a lesson for me on how to work with Firebase. If some part of the system keeps a copy of your deleted item, saving it won't produce a bug but revive the deleted item.
For those, who have the similar issue, but didn't solve it yet.
There are two methods for listening events: .on() and .once(). In my case that was the cause of a problem.
I was working on a migration procedure, that should run once
writeRef
.orderByChild('text_hash')
.equalTo(addItem.text_hash)
.on('value', val => { // <--
if (!val.exists()) {
writeRef.push(addItem)
}
});
So the problem was exactly because of .on method. It fires each time after a data manipulation from FB's console.
Changing to .once solved that.
In an effort to prevent certain objects from being created, I set a conditional in that type of object's beforeSave cloud function.
However, when two objects are created simultaneously, the conditional does not work accordingly.
Here is my code:
Parse.Cloud.beforeSave("Entry", function(request, response) {
var theContest = request.object.get("contest");
theContest.fetch().then(function(contest){
if (contest.get("isFilled") == true) {
response.error('This contest is full.');
} else {
response.success();
});
});
Basically, I don't want an Entry object to be created if a Contest is full. However, if there is 1 spot in the Contest remaining and two entries are saved simultaneously, they both get added.
I know it is an edge-case, but a legitimate concern.
Parse is using Mongodb which is a NoSQL database designed to be very scalable and therefore provides limited synchronisation features. What you really need here is mutual exclusion which is unfortunately not supported on a Boolean field. However Parse provides atomicity for counters and array fields which you can use to enforce some control.
See http://blog.parse.com/announcements/new-atomic-operations-for-arrays/
and https://parse.com/docs/js/guide#objects-updating-objects
Solved this by using increment and then doing the check in the save callback (instead of fetching the object and checking a Boolean on it).
Looks something like this:
Parse.Cloud.beforeSave("Entry", function(request, response) {
var theContest = request.object.get("contest");
theContest.increment("entries");
theContest.save().then(function(contest) {
if (contest.get("entries") > contest.get("maxEntries")) {
response.error('The contest is full.');
} else {
response.success();
}
});
}
Using Meteor, I'd like to understand the most efficient way to use JQuery UI's Autocomplete with large volumes of server-side data.
I have two working proposals and would like to hear opinions on the differences and if there are any better ways to do the same thing.
Using pub/sub:
// Server
Meteor.publish("autocompleteData", function (theSearchTerm) {
var query = {
name: { $regex: theSearchTerm, $options: 'i'}
};
return MyData.find(query, options);
});
// Client
Template.myTemplate.rendered = function() {
initAutocomplete($(this.find('.my.autocomplete')));
};
var initAutocomplete = function(element){
element.customAutocomplete({
source: function(request, callback){
var sub = Meteor.subscribe('autocompleteData', request.term, function(){
var results = MyData.find({}, {limit: 50}).fetch();
sub.stop();
callback(results);
});
},
select: function(event, ui){
// Do stuff with selected value
}
});
};
Using remote functions (Meteor.Methods):
// Server
Meteor.methods({
getData: function(theSearchTerm) {
var query = {
name: { $regex: theSearchTerm, $options: 'i'}
};
return MyData.find(query, {limit: 50}).fetch();
});
});
// Client
Template.myTemplate.rendered = function() {
initAutocomplete($(this.find('.my.autocomplete')));
};
var initAutocomplete = function(element){
element.customAutocomplete({
source: function(request, callback){
Meteor.call('getData', request.term, function(err, results){
callback(results);
});
},
select: function(event, ui){
// Do stuff with selected value
}
});
};
Which, if either, is the the most efficient way to setup a server-side autocomplete using Meteor with a large dataset?
For what it's worth, I'll offer a few of my thoughts on the subject. As a disclaimer, I'm just a Meteor enthusiast and not an expert, so please correct me if I've said something faulty.
To me, it seems like a potential advantage of pub/sub in cases like these is that data is cached. So when subscribing to the same record set, lookup will be near instantaneous since the client can search the local cache instead of asking the server for data again (publication is smart enough not to push repeated data to the client).
However, the advantage is lost here since you're stopping the subscription, so every time the user types the same search term, data is again pushed to the client (at least, the cursor's added event fires again for every document). In this case I would expect the pub/sub to be on nearly equal footing with Meteor.call.
If you want to cache the data of pub/sub, one way is to take out the sub.stop(). But unless your users have the tendency to search similar terms, caching the data is actually worse since with every letter the user types more data will be stored on the client, perhaps never to be seen again (unless searching is such a prominent feature in your app that the user would benefit from this?).
Overall, I see no compelling advantage with using pub/sub over Meteor methods, though I'm not versed in Meteor well enough to offer more specific advantages/disadvantages between the two. I personally think Meteor methods looks cleaner though.
If you're trying to implement a search feature though, I personally like the easy-search package, which supports this type of server-side search with autocomplete. In any case, I hope you get your question resolved! I'm curious to know the answer too.
I've been trying to do Meteor's leaderboard example, and I'm stuck at the second exercise, resetting the scores. So far, the furthest I've got is this:
// On server startup, create some players if the database is empty.
if (Meteor.isServer) {
Meteor.startup(function () {
if (Players.find().count() === 0) {
var names = ["Ada Lovelace",
"Grace Hopper",
"Marie Curie",
"Carl Friedrich Gauss",
"Nikola Tesla",
"Claude Shannon"];
for (var i = 0; i < names.length; i++)
Players.insert({name: names[i]}, {score: Math.floor(Random.fraction()*10)*5});
}
});
Meteor.methods({
whymanwhy: function(){
Players.update({},{score: Math.floor(Random.fraction()*10)*5});
},
}
)};
And then to use the whymanwhy method I have a section like this in if(Meteor.isClient)
Template.leaderboard.events({
'click input#resetscore': function(){Meteor.call("whymanwhy"); }
});
The problem with this is that {} is supposed to select all the documents in MongoDB collection, but instead it creates a new blank scientist with a random score. Why? {} is supposed to select everything. I tried "_id" : { $exists : true }, but it's a kludge, I think. Plus it behaved the same as {}.
Is there a more elegant way to do this? The meteor webpage says:
Make a button that resets everyone's score to a random number. (There
is already code to do this in the server startup code. Can you factor
some of this code out and have it run on both the client and the
server?)
Well, to run this on the client first, instead of using a method to the server and having the results pushed back to the client, I would need to explicitly specify the _ids of each document in the collection, otherwise I will run into the "Error: Not permitted. Untrusted code may only update documents by ID. [403]". But how can I get that? Or should I just make it easy and use collection.allow()? Or is that the only way?
I think you are missing two things:
you need to pass the option, {multi: true}, to update or it will only ever change one record.
if you only want to change some fields of a document you need to use $set. Otherwise update assumes you are providing the complete new document you want and replaces the original.
So I think the correct function is:
Players.update({},{$set: {score: Math.floor(Random.fraction()*10)*5}}, {multi:true});
The documentation on this is pretty thorough.