I spent so much time on my meteor app, and now that users are coming in, it's so slow! Almost unusable, specially on mobile.
I am having to revamp all my code and try to pass all logic to the server so the client is not so overloaded.
However, I am running into many issues. I've tried all sort of solutions but nothing is working.
Anyways, here is the issue at hand. When I try to use server side methods, an infinite loop gets triggered that will console.log on the server thousands of times.
Here is the code
on templates/question.js
Template.questionItem.helpers({
editPermission: function(){
var question = this
Meteor.call('checkForEditPermission', question, function(error, result){
console.log(result)
Session.set('editPermissionVar', result)
});
return Session.get('editPermissionVar')
}
});
on server/question.js
if (Meteor.isServer) {
Meteor.startup(function() {
Meteor.methods({
checkForEditPermission: function(question) {
if (Meteor.user() && (question.user_id == Meteor.userId())) {
return true
}
}
});
});
}
This is a very simple helper. I have dozens of other, more complex helpers, but I can't figure out how to put them server side. I was taught to do things on the client, never realizing I was overloading it.
Not sure if this helps, but what I had before (the client only helper) is much simpler:
editPermission: function(){
if (currentUser && (this.user_id == currentUser._id)) {
return true
}
}
Doesn't need meteor.call.
The question is, how do you deal with calling methods from client and using them as helpers. I have tried many things, like reactive vars and session variables, but nothing seems to work. I know this particular example is very simple, and wouldn't take much computation, but I think the principle is the same with more complex helpers. The question is how to move them to the server.
Thanks
The question is, how do you deal with calling methods
How I deal is Meteor.call wich makes changes in database and just displaying data with cursors.
So for example as you have some template like:
Template.exammple.events({'click .sth' : function(event){
Meteor.call("serveside", {some: data});
return false;
});
In 'serverside' I got all callculations or DB queries/inserts/updates etc
and helper:
Template.example.helpers({
example : function(){
return dbCollectionName.find();
});
In this way I do not need to use Meteor.call for pulling data/results. Mostly I save results in its raw shape and just addjusting diplaying after pulling them from with cursor.
MongoDB works perfect with meteor, and if you need some aggregation you can use this extension: Meteor Aggregate
Related
I have a node js web app that is using handlebars. Users are asking me to let them register their own handlebars helpers.
I'm quite hesitant about letting them do it... but I'll give it a go if there is a secure way of doing it so.
var Handlebars = require("handlebars");
var fs = require("fs");
var content = fs.readFileSync("template.html", "utf8");
//This helper will be posted by the user
var userHandlebarsHelpers = "Handlebars.registerHelper('foo', function(value) { return 'Foo' + value; });"
//eval(userHandlebarsHelpers); This I do not like! Eval is evil
//Compile handlebars with user submitted Helpers
var template = Handlebars.compile(content);
var handleBarContent = template({ foo: bar });
//Save compiled template and some extra code.
Thank you in advance!
Because helpers are just Javascript code, the only way you could safely run arbitrary Javascript from the outside world on your server is if you either ran it an isolated sandbox process or you somehow sanitized the code before you ran it.
The former can be done with isolated VMs and external control over the process, but that makes it quite a pain to have helper code in some external process as you now have to develop ways to even call it and pass data back and forth.
Sanitizing Javascript to be safe from running exploits on your server is a pretty much impossible task when your API set is as large as node.js. The browser has a very tightly controlled set of things that Javascript can do to keep the underlying system safe from what browser Javascript can do. node.js has none of those safeguards. You could put code in one of these helpers to erase the entire hard drive of the server or install multiple viruses or pretty much whatever evil exploit you wanted to code. So, running arbitrary Javascript will simply not be safe.
Depending upon the exact problems that need to be solved, one can something develop a data driven approach where, instead of code, the user provides some higher level set of instructions (map this to that, substitute this with that, replace this with that, display from this set of data, etc...) that is not actually Javascript, but rather some non-executable meta data. That is much more feasible to make safe because you control all the code that acts on this meta data so you just have to make sure that the code that processes the meta data isn't capable of being tricked into doing something evil.
Following #jfriend00 input and after some serious testing I found a way to do it using nodejs vm module.
Users will input their helpers with this format:
[[HBHELPER 'customHelper' value]]
value.replace(/[0-9]/g, "");
[[/HBHELPER]]
[[HBHELPER 'modulus' index mod result block]]
if(parseInt(index) % mod === parseInt(result))
block.fn(this);
[[/HBHELPER]]
//This will throw an error when executed "Script execution timed out."
[[HBHELPER 'infiniteLoop' value]]
while(1){}
[[/HBHELPER]]
I translate that block into this and execute it:
Handlebars.registerHelper('customHelper', function(value) {
//All the code is executed inside the VM
return vm.runInNewContext('value.replace(/[0-9]/g, "");', {
value: value
}, {
timeout: 1000
});
});
Handlebars.registerHelper('modulus', function(index, mod, result, block) {
return vm.runInNewContext('if(parseInt(index) % mod === parseInt(result)) block.fn(this);', {
index: index,
mod: mod,
result: result,
block: block
}, {
timeout: 1000
});
});
Handlebars.registerHelper('infiniteLoop', function(value) {
//Error
return vm.runInNewContext('while(1){}', {
value: value
}, {
timeout: 1000
});
});
I made multiple tests so far, trying to delete files, require modules, infinite loops. Everything is going perfectly, all those operations failed.
Running the handlebar helper callback function in a VM is what made this work for me, because my main problem using VM's and running the whole code inside was adding those helpers to my global Handlebars object.
I'll update if I found a way to exploit it.
I am trying to create a game with meteor. Since many people told me to use mongo db (because it's vanilla, fast and reactive) I realised, that I would need to "listen" to the mongo db update, in order to be able to respond to the recived code, and make changes to the DOM.
Can I use the Meteor Trackers like this:
var handle = Tracker.autorun(function () {
handleEvent(
collection.find({}, {sort: {$natural : 1}, limit: 1 }) // find last element in collection
);
});
What you are looking for is the observe and observerChanges functions of cursors. See here: http://docs.meteor.com/#/full/observe
You could use the tracker, but I think observing the cursor is more scalable.
Since in your example you seem to be interested only in responding to the last added object, here is a skeleton on how you could do that with observeChanges:
var cursor = Collection.find();
cursor.observeChanges({
added: function(id, object) {
// This code runs when a new object "object" was added to collection.
}
});
cursor.observer() seems to be exactly what I was looking for.
My resolution looks like this:
collection.find({}).observe({
addedAt: function (document, atIndex, before) {
handleEvent(
document
);
}
});
The only "problem" I realised is, that while testing it seemed like the event got fired twice. (but htat will probably go into another thread. sometime soon)
(This is (I think) because of the latency compensation, where the object gets inserted in the client db, then the server executes his method, and then sends the new collection to the client, where the "added" event is triggered again. right?)
I think Tracker.autorun only work for client, so your code can run on the client but not server. May be if you want to update your client side collection, you can autorun subscription instead.
I'm new to Meteor.js, so hopefully this is an inability on my part rather than a limitation of the platform because it's pretty amazing otherwise.
What I'm trying to achieve is pretty straightforward: run Javascript whenever a template helper gets updated with new data (but not from the db!).
A simple example scenario could be this: A user makes a request to get some images. But rather than the images just "popping up", they should be hidden and fadein once they've been fully loaded (among other things like positioning them, etc.).
In other words, right after the helper receives new data, a function should run to do something with that data (that can not be done on the server before it is actually rendered).
If the data is from a collection, it's quite easy to achieve this with a subscribe callback.
However, there seems to be no callback for once a helper has rendered the new data.
Yes, it's possible to add a timeout of a few ms, but that's not a clean or reliable solution in my mind, because you obviously never know exactly how long it will need to render.
I've searched through dozens of seemingly related posts, but was not able to find anything that could be considered a "standard" way of achieving this...
Here's a bit of (simplified) example code to illustrate the scenario:
var images = [];
//When showImages is updated with new data from the images array...
Template.gallery.helpers({
showImages: function () {
return images;
}
});
//...this function should fire
function doMagicWork () {
...
}
//Because firing it on the on click event would be too soon,
//as the helper hasn't rendered yet
Template.gallery.events({
"click #fetch_images": function (event) {
Meteor.call("getImagesFromServer", function(error, result) {
images = result.content;
});
}
});
There is a pending feature for adding animation/transition support for UI changes (referenced here)
As an interim solution, you can use Blaze UI hooks. There are quite a few packages that use them. example here and here
In general, , Meteor way is to reduce the amount of boiler plate code. Smooth transition is something of a pattern rather than an individual thing for element and should be treated as such as per meteor philosophy.
I read a lot about Express / SocketIO and that's crazy how rarely you get some other example than a "Hello" transmitted directly from the app.js. The problem is it doesn't work like that in the real world ... I'm actually desperate on a logic problem which seems far away from what the web give me, that's why I wanted to point this out, I'm sure asking will be the solution ! :)
I'm refactoring my app (because there were many mistakes like using the global scope to put libs, etc.) ; Let's say I've got a huge system based on SocketIO and NodeJS. There's a loader in the app.js which starts the socket system.
When someone join the app it require() another module : it initializes many socket.on() which are loaded dynamically and go to some /*_socket.js files in a folder. Each function in those modules represent a socket listener, then it's way easier to call it from the front-end, might look like this :
// Will call `user_socket.js` and method `try_to_signin(some params)`
Queries.emit_socket('user.try_to_signin', {some params});
The system itself works really well. But there's a catch : the module that will load all those files which understand what the front-end has sent also transmit libraries linked with req/res (sessions, cookies, others...) and must do it, because the called methods are the core of the app and very often need those libraries.
In the previous example we obviously need to check if the user isn't already logged-in.
// The *_socket.js file looks like this :
var $h = require(__ROOT__ + '/api/helpers');
module.exports = function($s, $w) {
var user_process = require(__ROOT__ + '/api/processes/user_process')($s, $w);
return {
my_method_called: function(reference, params, callback) {
// Stuff using $s, $w, etc.
}
}
// And it's called this way :
// $s = services (a big object)
// $w = workers (a big object depending on $s)
// They are linked with the req/res from the page when they are instantiated
controller_instance = require('../sockets/'+ controller_name +'_socket')($s, $w);
// After some processes ...
socket_io.on(socket_listener, function (datas, callback) {
// Will call the correct function, etc.
$w.queries.handle_socket($w, controller_name, method_name, datas);
});
The good news : basically, it works.
The bad news : every time I refresh the page, the listeners double themselves because they are in a loop called on page load.
Below, this should have been one line :
So I should put all the socket.on('connection'...) stuff outside the page loading, which means when the server starts ... Yes, but I also need the req/res datas to be able to load the libraries, which I get only when the page is loaded !
It's a programing logic problem, I know I did something wrong but I don't know where to go now, I got this big system which "basically" works but there's like a paradox on the way I did it and I can't figure out how to resolve this ... It's been a couple of hours I'm stuck.
How can I refacto to let the possibility to get the current libraries depending on req/res within a socket.on() call ? Is there a trick ? Should I think about changing completely the way I did it ?
Also, is there another way to do what I want to do ?
Thank you everyone !
NOTE : If I didn't explain well or if you want more code, just tell me :)
EDIT - SOLUTION : As seen above we can use sockets.once(); instead of sockets.on(), or there's also the sockets.removeAllListeners() solution which is less clean.
Try As Below.
io.sockets.once('connection', function(socket) {
io.sockets.emit('new-data', {
channel: 'stdout',
value: data
});
});
Use once instead of on.
This problem is similar as given in the following link.
https://stackoverflow.com/questions/25601064/multiple-socket-io-connections-on-page-refresh/25601075#25601075
I am having trouble getting my head round how i can use sinon to mock a call to postgres which is required by the module i am testing, or if it is even possible.
I am not trying to test the postgres module itself, just my object to ensure it is working as expected, and that it is calling what it should be calling in this instance.
I guess the issue is the require setup of node, in that my module requires the postgres module to hit the database, but in here I don't want to run an integration test I just want to make sure my code is working in isolation, and not really care what the database is doing, i will leave that to my integration tests.
I have seen some people setting up their functions to have an optional parameter to send the mock/stub/fake to the function, test for its existence and if it is there use it over the required module, but that seems like a smell to me (i am new at node so maybe this isn't).
I would prefer to mock this out, rather then try and hijack the require if that is possible.
some code (please note this is not the real code as i am running with TDD and the
function doesn't do anything really, the function names are real)
TEST SETUP
describe('#execute', function () {
it('should return data rows when executing a select', function(){
//Not sure what to do here
});
});
SAMPLE FUNCTION
PostgresqlProvider.prototype.execute = function (query, cb) {
var self = this;
if (self.connection === "")
cb(new Error('Connection can not be empty, set Connection using Init function'));
if (query === null)
cb(new Error('Invalid Query Object - Query Object is Null'))
if (!query.buildCommand)
cb(new Error("Invalid Query Object"));
//Valid connection and query
};
It might look a bit funny to wrap around the postgres module like this but there are some design as this app will have several "providers" and i want to expose the same API for them all so i can use them interchangeably.
UPDATE
I decided that my test was too complicated, as i was looking to see if the connect call had been made AND then returning data, which smelt to me, so i stripped it back and put it into two tests:
The Mock Test
it('should call pg.connect when a valid Query object is parsed', function(){
var mockPg = sinon.mock(pg);
mockPg.expects('connect').once;
Provider.init('ConnectionString');
Provider.execute(stubQueryWithBuildFunc, null, mockPg);
mockPg.verify();
});
This works (i think) as without the postgres connector code it fails, with it passes (Boom :))
Issue now is with the second method, which i am going to use a stub (maybe a spy) which is passing 100% when it should fail, so i will pick that up in the morning.
Update 2
I am not 100% happy with the test, mainly because I am not hijacking the client.query method which is the one that hits the database, but simply my execute method and forcing it down a path, but it allows me to see the result and assert against it to test behaviour, but would be open to any suggested improvements.
I am using a spy to catch the method and return null and a faux object with contains rows, like the method would pass back, this test will change as I add more Query behaviour but it gets me over my hurdle.
it('should return data rows when a valid Query object is parsed', function(){
var fauxRows = [
{'id': 1000, 'name':'Some Company A'},
{'id': 1001, 'name':'Some Company B'}
];
var stubPg = sinon.stub(Provider, 'execute').callsArgWith(1, null, fauxRows);
Provider.init('ConnectionString');
Provider.execute(stubQueryWithBuildFunc, function(err, rows){
rows.should.have.length(2);
}, stubPg);
stubPg.called.should.equal.true;
stubPg.restore();
});
Use pg-pool: https://www.npmjs.com/package/pg-pool
It's about to be added to pg anyway and purportedly makes (mocking) unit-testing easier... from BrianC ( https://github.com/brianc/node-postgres/issues/1056#issuecomment-227325045 ):
Checkout https://github.com/brianc/node-pg-pool - it's going to be the pool implementation in node-postgres very soon and doesn't rely on singletons which makes mocking much easier. Hopefully that helps!
I very explicitly replace my dependencies. It's probably not the best solution but all the other solutions I saw weren't that great either.
inject: function (_mock) {
if (_mock) { real = _mock; }
}
You add this code to the module under test. In my tests I call the inject method and replace the real object. The reason why I don't 100% like it is because you have to add extra code only for testing.
The other solution is to read the module file as a string and use vm to manually load the file. When I investigated this I found it a little to complex so I went with just using the inject function. It's probably worth investigating this approach though. You can find more information here.