Using Meteor, I'd like to understand the most efficient way to use JQuery UI's Autocomplete with large volumes of server-side data.
I have two working proposals and would like to hear opinions on the differences and if there are any better ways to do the same thing.
Using pub/sub:
// Server
Meteor.publish("autocompleteData", function (theSearchTerm) {
var query = {
name: { $regex: theSearchTerm, $options: 'i'}
};
return MyData.find(query, options);
});
// Client
Template.myTemplate.rendered = function() {
initAutocomplete($(this.find('.my.autocomplete')));
};
var initAutocomplete = function(element){
element.customAutocomplete({
source: function(request, callback){
var sub = Meteor.subscribe('autocompleteData', request.term, function(){
var results = MyData.find({}, {limit: 50}).fetch();
sub.stop();
callback(results);
});
},
select: function(event, ui){
// Do stuff with selected value
}
});
};
Using remote functions (Meteor.Methods):
// Server
Meteor.methods({
getData: function(theSearchTerm) {
var query = {
name: { $regex: theSearchTerm, $options: 'i'}
};
return MyData.find(query, {limit: 50}).fetch();
});
});
// Client
Template.myTemplate.rendered = function() {
initAutocomplete($(this.find('.my.autocomplete')));
};
var initAutocomplete = function(element){
element.customAutocomplete({
source: function(request, callback){
Meteor.call('getData', request.term, function(err, results){
callback(results);
});
},
select: function(event, ui){
// Do stuff with selected value
}
});
};
Which, if either, is the the most efficient way to setup a server-side autocomplete using Meteor with a large dataset?
For what it's worth, I'll offer a few of my thoughts on the subject. As a disclaimer, I'm just a Meteor enthusiast and not an expert, so please correct me if I've said something faulty.
To me, it seems like a potential advantage of pub/sub in cases like these is that data is cached. So when subscribing to the same record set, lookup will be near instantaneous since the client can search the local cache instead of asking the server for data again (publication is smart enough not to push repeated data to the client).
However, the advantage is lost here since you're stopping the subscription, so every time the user types the same search term, data is again pushed to the client (at least, the cursor's added event fires again for every document). In this case I would expect the pub/sub to be on nearly equal footing with Meteor.call.
If you want to cache the data of pub/sub, one way is to take out the sub.stop(). But unless your users have the tendency to search similar terms, caching the data is actually worse since with every letter the user types more data will be stored on the client, perhaps never to be seen again (unless searching is such a prominent feature in your app that the user would benefit from this?).
Overall, I see no compelling advantage with using pub/sub over Meteor methods, though I'm not versed in Meteor well enough to offer more specific advantages/disadvantages between the two. I personally think Meteor methods looks cleaner though.
If you're trying to implement a search feature though, I personally like the easy-search package, which supports this type of server-side search with autocomplete. In any case, I hope you get your question resolved! I'm curious to know the answer too.
Related
Goal: a dynamically generated list from external source.
I've set up a simple angular app that gets a list of events from an external JSON source. I want the list to update when events are added from the external source. It's currently working, but I have one problem and three questions:
1) I'm currently rewriting the list every 15 seconds. How do I just add to the end of the list without rewriting the list? (problem and question)
2) Is there another, better way to keep up to date with the external list? I'm trying to follow "RESTful" techniques, does that mean I should rely on the client side code to poll every so many seconds the way I'm doing? (best practice question)
3) Is setting the timeout in the controller best practice? Because it's controlling the action on the page?(best practice/comprehension question)
var eventModule = angular.module('eventModule', []);
eventModule.controller('eventControlller',
function($scope, $timeout, eventList) {
$scope.events = eventList.getAllEvents().success(
function(events) {$scope.events = events});
var poll = function() {
$timeout(function() {
$scope.events = eventList.getAllEvents().success(
function(events) {$scope.events = events});
poll();
}, 15000);
};
poll();
});
eventModule.factory('eventList', function($http) {
var url = "http://localhost/d8/events/request";
return {
getAllEvents: function() {
return $http.get(url);
}
};
});
If the list is an array, and you want to add new members to it, there are a few different ways. One way is to use the prototype.concat() function, like so:
function(events) {
$scope.events = $scope.events.concat(events)
});
If you cannot use that then you can go for loops solution:
function concatenateEvents(events) {
events.forEach(function(element) {
events.push(element);
}
}
Regarding the best ways to update the list, it depends on your requirements. If 15 seconds is not too long for you, then you can keep this logic, but if you need to speed up the response time, or even make it real time, then you need to emulate server-push architecture, which is different than the default web architecture, which is request-response architecture. Basically you may want to explore web sockets, and/or long polling, or reverse ajax, or comet... has many names. Web sockets is the recommended solution, others are only in case you have to use some non-compatible browsers.
Regarding the third question, I honestly don't know. Truly it doesn't feel good to control the UI from within your controller, but as I don't really know what your app is supposed to be doing, I don't know whether this is actually a bad way to do it.
Hope this helps!
EDIT - forgot to add another important point: You don't need to assign the eventList.getAllEvents() to $scope.events, as you are doing that in the callback handler function.
Perhaps you can modify your controller to something like this:
eventModule.controller('eventControlller', function($scope, $timeout, eventList) {
eventList.getAllEvents().success(
function(events) {
$scope.events = events
});
var poll = function() {
$timeout(function() {
eventList.getAllEvents().success(
function(events) {$scope.events = events});
poll();
}, 15000);
};
poll();
});
In an effort to prevent certain objects from being created, I set a conditional in that type of object's beforeSave cloud function.
However, when two objects are created simultaneously, the conditional does not work accordingly.
Here is my code:
Parse.Cloud.beforeSave("Entry", function(request, response) {
var theContest = request.object.get("contest");
theContest.fetch().then(function(contest){
if (contest.get("isFilled") == true) {
response.error('This contest is full.');
} else {
response.success();
});
});
Basically, I don't want an Entry object to be created if a Contest is full. However, if there is 1 spot in the Contest remaining and two entries are saved simultaneously, they both get added.
I know it is an edge-case, but a legitimate concern.
Parse is using Mongodb which is a NoSQL database designed to be very scalable and therefore provides limited synchronisation features. What you really need here is mutual exclusion which is unfortunately not supported on a Boolean field. However Parse provides atomicity for counters and array fields which you can use to enforce some control.
See http://blog.parse.com/announcements/new-atomic-operations-for-arrays/
and https://parse.com/docs/js/guide#objects-updating-objects
Solved this by using increment and then doing the check in the save callback (instead of fetching the object and checking a Boolean on it).
Looks something like this:
Parse.Cloud.beforeSave("Entry", function(request, response) {
var theContest = request.object.get("contest");
theContest.increment("entries");
theContest.save().then(function(contest) {
if (contest.get("entries") > contest.get("maxEntries")) {
response.error('The contest is full.');
} else {
response.success();
}
});
}
I use jquery autocomplete and it look like my code, to call the plugin, not good. is there any more simple way to call jquery autocomplete
js
$(document).ready(function(){
$("#m_occupation").autocomplete("search/moccupation.php", {
selectFirst: true
});
$("#foccupation").autocomplete("search/f_occupation.php", {
selectFirst: true
});
$("#g_address").autocomplete("search/g_address.php", {
selectFirst: true
});
$("#relationship").autocomplete("search/relationship.php", {
selectFirst: true
});
});
What you've got isn't really terrible. If you're only ever initializing these autocompletes one time, then it's pretty readable overall, although you do have some repetition.
Cache your jQuery objects for future use. In your snippet above, you only reference each jQuery object (e.g., $("#m_occupation") ) once, but in a real webapp, there's a pretty good chance you'll use it more. Caching helps reduce the number of jQuery finding operations, and is a good practice to adopt even though it's not likely to increase performance for a user noticeably.
Cache your options objects. You're repeating your option declaration multiple times; just declare a single autocompleteOptions object and be done with it.
Refactor initialization into a function. If you're really feeling like the autocomplete initialization is ugly, or complex, or subject to frequent edits, make it a single function. Future global edits to initialization can be made one time rather than multiple times.
A redo of your code taking those into account would look like:
var initAutocomplete = function($el, dataUrl) {
var autocompleteOptions = { selectFirst: true };
$el.autocomplete(dataUrl, autocompleteOptions);
};
$(document).ready(function(){
var $m_occupation, $foccupation, $g_address, $relationship;
$m_occupation = $('#m_occupation');
initAutocomplete($m_occupation, "search/moccupation.php");
$foccupation = $('#foccupation');
initAutocomplete($foccupation, "search/f_occupation.php");
$g_address = $('#g_address');
initAutocomplete($g_address, "search/g_address.php");
$relationship = $('#relationship');
initAutocomplete($relationship, "search/relationship.php");
});
You could technically optimize further by using a single string to represent the DOM ID and URL from which you gather the data, but in my experience, that breaks down in maintenance. So I wouldn't couple those too tightly.
I need to keep track of a counter of a collection with a huge number of documents that's constantly being updated. (Think a giant list of logs). What I don't want to do is to have the server send me a list of 250k documents. I just want to see a counter rising.
I found a very similar question here, and I've also looked into the .observeChanges() in the docs but once again, it seems that .observe() as well as .observeChanges() actually return the whole set before tracking what's been added, changed or deleted.
In the above example, the "added" function will fire once per every document returned to increment a counter.
This is unacceptable with a large set - I only want to keep track of a change in the count as I understand .count() bypasses the fetching of the entire set of documents. The former example involves counting only documents related to a room, which isn't something I want (or was able to reproduce and get working, for that matter)
I've gotta be missing something simple, I've been stumped for hours.
Would really appreciate any feedback.
You could accomplish this with the meteor-streams smart package by Arunoda. It lets you do pub/sub without needing the database, so one thing you could send over is a reactive number, for instance.
Alternatively, and this is slightly more hacky but useful if you've got a number of things you need to count or something similar, you could have a separate "Statistics" collection (name it whatever) with a document containing that count.
There is an example in the documentation about this use case. I've modified it to your particular question:
// server: publish the current size of a collection
Meteor.publish("nbLogs", function () {
var self = this;
var count = 0;
var initializing = true;
var handle = Messages.find({}).observeChanges({
added: function (id) {
count++;
if (!initializing)
self.changed("counts", roomId, {nbLogs: count});
},
removed: function (id) {
count--;
self.changed("counts", roomId, {nbLogs: count});
}
// don't care about moved or changed
});
// Observe only returns after the initial added callbacks have
// run. Now return an initial value and mark the subscription
// as ready.
initializing = false;
self.added("counts", roomId, {nbLogs: count});
self.ready();
// Stop observing the cursor when client unsubs.
// Stopping a subscription automatically takes
// care of sending the client any removed messages.
self.onStop(function () {
handle.stop();
});
});
// client: declare collection to hold count object
Counts = new Meteor.Collection("counts");
// client: subscribe to the count for the current room
Meteor.subscribe("nbLogs");
// client: use the new collection
Deps.autorun(function() {
console.log("nbLogs: " + Counts.findOne().nbLogs);
});
There might be some higher level ways to do this in the future.
I am looking for the most performant solution for sending structured data to the client in the Meteor framework on a request.
THE ISSUE:
Sometimes, before you send data from the database to the client, you want to add some server side generated additional information beeing sent to the client (i.e. security credentials for many objects). This data can be time-critical (i.e. due to an expiration timestamp) and therefore should not be stored in the db. Also, this data sometimes cannot be processed on the client side (i.e. due to security reasons). In many cases, this data will be structurally related to actual database data, but also very much related to a single request, since you might want to have it discarded and re-generated on a new request.
YOU CAN (at least by design..):
create a second collection, store and publish your request-related data there, accepting the write-overhead, and then i.e. in the Meteor.myTemplate.destroyed=function(){...} remove the data again accepting another write-overhead.
store each entry in a session variable, but then you also have to take care
of deleting it later on (Meteor.myTemplate.destroyed=function(){...}), this is my favourite right now, but I am running into problems with storing large objects there.
store this data in the dom (i.e. in the attributes or data fields of hidden or visible elements)
generate this data from the dom with Meteor.call('method',arguments,callback(){...}) by storing the appropriate arguments in the dom and injecting them back with i.e. jQuery in the callback(){...}.
YOU CAN'T: (by design!!)
use transformations within Meteor.publish("name",function(){...}) on the server
use a Meteor.call() within a transformation on a Template.variable=function(){return collection.find(...)} (also not if you have a corresponding Meteor.method() on the client for guessing the result!).
Again, what I am looking for is the best performing solution for this.
To address the transform issue, which I care about because I think in terms of smart models doing things rather than a bunch of anonymous functions, here is an example of server transform reaching the client (not reactive as is, through a call, not a publish, but illustrative of the point under discussion of server transforms).
You will get this:
each image local
data: LOLCATZ RULZ
transform:
data: LOLCATZ RULZ
transform:
each image server transformed
data: LOLCATZ RULZ
transform: XYZ
data: LOLCATZ RULZ
transform: XYZ
from:
<template name='moritz'>
<h3>each image local</h3>
<dl>
{{#each images}}
<dt>data: {{caption}}</dt>
<dd>transform: {{secretPassword}}</dd>
{{/each}}
</dl>
<h3>each image server transformed</h3>
<dl>
{{#each transformed}}
<dt>data: {{caption}}</dt>
<dd>transform: {{secretPassword}}</dd>
{{/each}}
</dl>
</template>
if (Meteor.isServer) {
Images = new Meteor.Collection('images', {
transform: function (doc) {
doc.secretPassword = 'XYZ'
return doc
}
});
Images.allow({
insert: function (userid, doc) {
return true;
}
});
if (Images.find().count() < 1) {
Images.insert({ caption: 'LOLCATZ RULZ'});
}
Meteor.publish('images', function () {
return Images.find();
})
Meteor.methods({
'transformed': function() {
return Images.find().fetch();
}
})
}
else {
Images = new Meteor.Collection('images');
imageSub = Meteor.subscribe('images');
Template.moritz.helpers({
'images': function () {
console.log(Images.find().count() + ' images')
return Images.find();
},
'transformed': function () {
// Should be separated, call should be in route for example
Meteor.call('transformed', function(err,data){
Session.set('transformed', data);
});
return Session.get('transformed');
}
});
}
Have a look at Meteor Streams, you can send something directly to the client from the server without having to use collections on the client.
You could do something with the message you get (I'm using an example from the meteor-streams site):
Client
chatStream.on('message', function(message) {
if(message.expiry > new Date()) {
//Do something with the message (not being read from a collection)
}
});
Even though you do this with some intent on not having it stored, be wary that simple tools (Chrome Inspector) can peak into the Network/Websocket tab (even if its encrypted via SSL) and see the raw data being passed through.
While i'm not sure of your intentions, if this is for security in any scenario never trust whatever data you get from the client.
Your first line would seem ideally answered by 'Mongo collections' but then I read a few conflicting ideas and conclusions I'm not sure I agree with. (For example, why can't you do those two things? Because it must happen on the server?) The most confusing statement for me is:
This data can be time-critical (i.e. due to an expiration timestamp) and therefore should not be stored in the db.
I don't understand what assumptions go into that conclusion, but if you consider storing data in the dom to be plausible, per points 2 and three above, you would seem to be open to systems much less performant than a mongo db.
You know you can, from the server, publish a second collection of the additional server-generated calculations, which you generate on the fly from the data, and put that together with the data once on the client. Kind of like a parent-child relationship: as you display the client document, pull up the additional data from the server from the dynamic collection and work it into your templates.
Jim Mack created a nice example here that proves, how well it works to store db data as well as additional "transform" properties in a Session variable.
Unfortunately this example lacks reactivity and does not perform the desired "transformations" after Meteor's magic re-render. So I grabbed his cool code and added back the reactivity, it's slim code that works very well, but will be outperformed by Jim Mack's example in terms of efficiency.
lolz.html
<head>
<title>lolz</title>
</head>
<body>
{{>myItems}}
</body>
<template name="myItems">
<h3>Reactive Item List with additional properties</h3>
<button id="add">add</button>
<button id="remove">remove</button>
<dl>
{{#each items}}
<dt>data: {{caption}}</dt>
<dd>added property: {{anotherProp _id}}</dd>
{{/each}}
</dl>
</template>
lolz.js
items = new Meteor.Collection('Items');
if (Meteor.isServer) {
items.allow({
insert: function (userid, doc) {
return true;
},
remove: function(userid,doc){
return true;
}
});
while(items.find().count()>0){
items.remove(items.findOne()._id);
}
while (items.find().count() < 3) {
items.insert({caption: 'LOLCATZ RULZ'});
}
Meteor.publish('Items', function () {
return items.find();
});
Meteor.methods({
'getAdditionalProps': function() {
additionalProps={};
items.find().forEach(function(doc){
additionalProps[doc._id]=reverse(doc.caption);
});
return additionalProps;
}
});
function reverse(s){ // server side operation, i.e. for security reasons
return s.split("").reverse().join("");
};
}
if (Meteor.isClient){
Meteor.subscribe('Items');
Meteor.startup(function(){
getAdditionalProps();
itemsHandle=items.find().observe({
added : function(doc){
getAdditionalProps();
},
removed : function(doc){
getAdditionalProps();
},
changed : function(docA,docB){
getAdditionalProps();
}
});
});
Template.myItems.rendered=function(){
console.log(new Date().getTime());
};
Template.myItems.items=function(){
return items.find();
}
Template.myItems.anotherProp=function(id){
return Session.get('additionalProps')[id];
}
Template.myItems.events({
'click #add':function(e,t){
items.insert({caption: 'LOLCATZ REACTZ'});
},
'click #remove':function(e,t){
items.remove(items.findOne()._id);
}
});
}
function getAdditionalProps(){
setTimeout(function(){
Meteor.call('getAdditionalProps',function(error,props){
Session.set('additionalProps',props);
});
},0);
}