Is there a javascript library that will allow me to express object predicates in a DSL similar to MongoDB's query language? For the sake of clarity in a large program, I'd like to be able to say:
var obj = {
a: 1,
b: 'abcdefg'
}, qry = {
a: { $gt: 0 },
b: /^abc/
};
if(query(qry).matches(obj)) {
// do something appropriate since
}
instead of:
var obj = {
a: 1,
b: 'abcdefg'
};
if(obj.a>0 && qry.b.test(obj.b)) {
// do something appropriate
}
I'm using Node.js, so anything on NPM would be great. It would be an added bonus if the library can select objects out of an array as well as just matching individual objects.
I reviewed these two related questions, but they weren't particularly helpful for my situation:
Implementing goMongoDB-like Query expression object evaluation
Evaluating MongoDB-like JSON Queries in PHP
OK I found the answer: Sift.js
Now for the long answer: This has been asked and answered before. The salient points are:
Use Sift if you really want Mongo syntax
If you want to be more mainstream, use Underscore.js like everyone else. It has heaps of handy functions in addition to the fact that it basically does what sift does with a slightly different syntax.
You may not need any library at all - modern browsers support many useful functions directly on the Array prototype, like filter() for example.
As a final note, mongodb-riff appears to be trying to do something similar but currently the page states clearly that it doesn't work - perhaps it's abandoned. But his readme is at least of value :-), he mentions sift and Query Engine which looks more mature, though too complicated for me!
Personally I'm going to go with Underscore because now that I've looked into it for the first time, I realise that it has heaps of handy stuff I need, plus I really only wanted to do simple functions like what would be _.find() in Underscore. But I guess if you want to do more complicated mongo-like queries, you'll do it in less LOC with Sift.
Check out Mingo
I implemented it after finding no suitable alternative.
It is still actively being developed but is usable. Test coverage is not complete.
Usable from both browser and nodejs
[Edit]
This library is now the most complete implementation of MongoDB's query language for the frontend.
https://github.com/mirek/node-json-criteria library does exactly that - evaluates criteria queries on JSON objects using MongoDB query format.
lodash.js
You want to use lodash.js. Its a drop in replacement for underscore.js.
The performance is twice as fast.
http://lodash.com/
The closest I could find was linq.js, which is LINQ for JavaScript. Hopefully this will be of some help to you.
Related
Disclaimer: My experience is mainly development with statically typed languages, and even if I understand how dynamic types works, I'm not very familiar with the common practices, neither I'm very aware of the usual tips and tricks yet.
I recently started to work in a project where we use serverless and deploy some lambdas in AWS with javascript. After deep diving the existing code I found out a common practice that shocked me a bit:
When a lambda is invoked and it receives a JSON payload, there is an initial pre-processing, converting all the values in the object to strings.
const mapValues = require('lodash/mapValues')
const escape = require('validator/lib/escape')
...
const body = mapValues(requestBody, value => escape('' + value))
...
Then after that pre-processing, the real work begins (validation, processing, invoking other services, etc...)
My first thoughts this are:
Benefit: Helps to reduce the cognitive load of the possible different types, enabling to assume that everything will be always a string.
Downside: Extra complexity dealing with non-string values, such as numbers.
Q1: Are there any other benefits and pitfalls to using this approach?
Q2: Could this be considered a bad practice? If so, why? (facts, not opinions please)
Thanks in advance! :)
Interesting question. I suggest you to ask the author if they 're available. Here are some of my thoughts:
I argue having string values does not reduce but increase cognitive load, as you have to watch out to convert them when doing basic arithmetics. I am sure all of us have seen at least once 1+1 being 11.
Also, handling every data as strings is a so bad practice it even has its own mock name: stringly typed. This is the cookbook example of that.
The only valid reason I can possibly imagine is to prevent various injection attacks against the service. In some scenarios, if the user is allowed to send in arbitrary json, it is possible to make the service execute code paths that it would not do normally. (something whacky like {"__proto__":[],"length":1,"0":"foo"} coerects to "foo" but is typeof object, this could bypass some flawed validation logic. mongodb is also prone to some attacks alike). But even in this case, proper validation would be much better than converting every value to string.
Hi I am trying to convert my database from Postgres to Mongo and changing from Ruby to Javascript, I am having trouble converting the following query.
name_array = RoomRaw.pluck(:name).uniq()
Any help is appreciated.
Edit:
Also is it possible to do the sum of a row? ie.
energy_ac = RoomRaw.where(timestamp: (beginning_of_last_full_hour..end_of_last_full_hour), name: name).last.sum(:energy_ac)
It sounds like you'd like to use a few of these functions without having to re-invent the wheel. While you may be able to write these functions yourself using only JavaScript, I recommend you learn how these functions work overall.
If you're looking to be able to use these types of functions without learning the inner workings, I recommend the lodash library. It's got exactly what you need.
We would like to exchange PO files with translators, and convert these to i18next's native JSON format. This sounds pretty straightforward using the i18next-conv utility.
However, i18next expects more or less special keys; for example the dot has special meaning with regard to i18next namespaces. In contrast, gettext PO files are intended to carry source strings (in the original language) for their message IDs.
We know that message IDs can be arbitrary, and can thus be mapped to i18next keys directly, but we would like to use source strings and use PO files as they were intended for various reasons.
The main reason is that all the translation tools we would like to use, and probably those of all our translators, expect this. Using symbolic keys would make translating a real pain. In any case, we figured from the debates around this that this is mainly a matter of opinion; we kind of made ours, and we would like to put this restriction as a requirement for this question.
Is it really a bad idea to use source strings as i18next keys from a technical standpoint? How hard is it to escape them? Is there anything else than the dot and namespaces that we should care about?
If we determine that we want to keep using symbolic keys, is there an alternative to i18next-conv that can generate i18next JSON translation files from PO files using source strings as message IDs? We understand that we would most likely need to maintain a separate mapping between the symbolic names and the original language strings, and we're prepared to do so.
Moreover, we wonder about the general workflow. How is the original PO file generated? How are the translation files maintained?
If we use source strings as keys in i18next, what are the best tools to extract strings from the codebase? xgettext doesn't seem to support Javascript.
If we use symbolic keys in i18next, how can we best generate the original PO file? Is writing a POT file by hand a good practice?
Again, if we use symbolic keys, how can we easily invalidate translations whenever we update the original language strings? Are there tools for that?
We understand these questions are very basic, but we were a bit surprised at how little information we could find about i18next-gettext integration. The i18next-conv tool exists and works perfectly as advertised, but is it actually useful? Do people actually use it? If so, are our questions relevant?
Finally, are our expectations about the maturity of the system a little too high?
if you like to use source strings as keys just change the
nsseparator = ':::'
keyseparator = '::'
so . and : could be used inside the key without fear.
You could try using https://github.com/cheton/i18next-text. It allows you using i18next translation without having the key as strings, and you do not need to worry about i18n key naming. Furthermore, you can also register the i18n helper with Handlebars.
Following is a simple example:
var i18n = require('i18next');
// extends i18n object to provide a new _() method
i18n._ = require('i18next-text')._;
i18n._('Save your time and work more efficiently.');
Check out the demo on JSFiddle.
I think regex is pretty fast and the third option is confusing. What do you think?
http://jqfundamentals.com/book/ch09s12.html
// old way
if (type == 'foo' || type == 'bar') { ... }
// better
if (/^(foo|bar)$/.test(type)) { ... }
// object literal lookup
if (({ foo : 1, bar : 1 })[type]) { ... }
I'll humbly disagree with Rebecca Murphey and vote for simplicity, for the first option.
I think regex is pretty fast
Machine code is even faster, but we don't use it.
the third option is confusing
It's only confusing if you're unfamiliar with the trick. (And for people not used to seeing regex to compare two strings, second option will be even more confusing.)
I just made a rudimentary benchmark and I'm honestly not sure how she got those results...
http://jsbin.com/uzuxi4/2/edit
Regex seems to scale the best, but the first is by far the fastest on all modern browsers. The last is excruciatingly slow. I understand the complexity theory between the three, but in practice, it doesn't seem that she's correct.
Let alone the fact that the first also has the best readability, it also seems to be the fastest. I even nested loops to take advantage of any browser caching of literal tables or constants (to no avail).
Edit:
It appears that when an object is explicitly created, she is indeed correct, however: http://jsbin.com/uzuxi4/4/edit
function __hash() {
...
var type = 'bar';
var testobj = { foo : 1, bar : 1 };
var c = 0;
for (i = 0; i < 1000; i++) {
if (testobj[type]) {
for (j = 0; j < 10000; j++) {
if (testobj[type]) { c++; }
}
}
}
...
}
We see that once the object has an internal reference, the seek time drops to about 500 ms which is probably the plateau. Object key lookup may be the best for larger data-sets, but in practice I don't really see it as a viable option for every-day use.
The first option involves
potentially two string compares.
The second option involves a parse each time.
The third option does a simple hash of the string and then a hash table look
up, which is the most efficient in this case, in terms of the amount of work that needs to be done.
The third option also scales better than the other two as more alternative strings are added, because the first two are O(n) and the third is O(1) in the average case.
If we want to talk about which option is prettier / more maintainable, that's a whole separate conversation.
The first case should really be done with === to avoid any type coercions, but depending on the number of alternatives you need to check it can become O(N), however depending on your code most JS engines will be able to a simple pointer check for the comparison.
In the second case you use a RegExp, and while RegExps are very fast, they tend to be slower for simple equality decisions than more direct equality comparisons. Simple string comparisons like yours are likely to be a pointer compare in a modern JS engine, but if you use a regexp the regexp must read every character.
The third case is more tricky -- if you do have a lot of values to check it may be faster, especially if you cache the object rather than repeatedly recreating it as it will simply be a hash lookup -- the exact performance of the lookup depends on the engine though.
I suspect a switch statement would beat the object literal case though.
Out of curiosity I made a test (which you can see here), the fastest approach (in a webkit nightly at least) seems to be a switch statement, followed by if, followed by the object, with regexp's last.
Just wanted to weigh in here and remind everyone that this is an open-source book with contributions from many people! The section being discussed, indeed, is based on content provided by a community member. If you have suggestions for improving the section, by all means, please open an issue on the repository, or better, fork the repo and send me a pull request :)
That said, I have just set up a jsPerf test (http://jsperf.com/string-tests), and at least in Chrome, the results are the opposite of what the book says. I've opened an issue on the book, and will try to deal with this in the near future.
Finally, two things:
I want to echo what another commenter said: perf optimizations are fun to talk about, and while there are some that really do matter, many don't. It's important to keep perspective on how much -- or little -- of a difference stuff like this makes.
I also want to echo the commenter who said, essentially, that readability is in the eyes of the beholder. Something confusing to one person may be perfectly clear to another. I do believe we should strive for readability, but I think there's a happy medium. Reading code that was a bit perplexing to me at first opened my eyes to a lot of great techniques; I'd have hated if it had been written so the complete newb that I was at the time could understand it.
The object literal lookup is optimized with a hash lookup which only requires one logical check instead of n. In a longer list you also won't have to repeat "type == " a zillion times.
For simplicity and readability the first will win every time. It might not be as fast, but who cares unless it is in a heavily run loop.
Good compilers should optimize things like this away.
During my routine work, i happened to write the chained javascript function which is something like LINQ expression to query the JSON result.
var Result = from(obj1).as("x").where("x.id=5").groupby("x.status").having(count("x.status") > 5).select("x.status");
It works perfectly and give the expected result.
I was wondering this looks awesome if the code is written like this (in a more readable way)
var Result = from obj1 as x where x.status
groupby x.status having count(x.status) > 5
select x.status;
is there a way to achieve this??
Cheers
Ramesh Vel
No. JavaScript doesn't support this.
But this looks quite good too:
var Result = from(obj1)
.as("x")
.where("x.id=5")
.groupby("x.status")
.having(count("x.status") > 5)
.select("x.status");
Most people insist on trying to metaprogram from inside their favorite language. That doesn't work if the language doesn't support metaprogramming well; other answers have observed that JavaScript does not.
A way around this is to do metaprogramming from outside the language, using
program transformation tools. Such tools can parse source code, and carry out arbitrary transformations on it (that's what metaprogramming does anyway) and then spit the revised program.
If you have a general purpose program transformation system, that can parse arbitrary languages, you can then do metaprogramming on/with whatever language you like. See our DMS Software Reengineering Toolkit for such a tool, that has robust front ends for C, C++, Java, C#, COBOL, PHP, and ECMAScript and a number of other programming langauges, and has been used for metaprogramming on all of these.
In your case, you want to extend the JavaScript grammar with new syntax for SQL queries, and then transform them to plain JavaScript. (This is a lot like Intentional Programming)
DMS will easily let you build a JavaScript dialect with additional rules, and then you can use its program transformation capabilities to produce the equivalent standard Javascript.
Having said, that, I'm not a great fan of "custom syntax for every programmer on the planet" which is where Intentional Programming leads IMHO.
This is a good thing to do if there is a large community of users that would find this valuable. This idea may or may not be one of them; part of the problem is you don't get to find out without doing the experiment, and it might fail to gain enough social traction to matter.
although not quite what you wanted, it is possible to write parsers in javascript, and just parse the query (stored as strings) and then execute it. e.g.,using libraries like http://jscc.jmksf.com/ (no doubt there are others out there) it shouldnt be too hard to implement.
but what you have in the question looks great already, i m not sure why you'd want it to look the way you suggested.
Considering that this question is asked some years ago, I will try to add more to it based on the current technologies.
As of ECMAScript 6, metaprogramming is now supported in a sense via Symbol, Reflect and Proxy objects.
By searching on the web, I found a series of very interesting articles on the subject, written by Keith Kirkel:
Metaprogramming in ES6: Symbols and why they're awesome
In short, Symbols are new primitives that can be added inside an object (without practically being properties) and are very handy for passing metaprogramming properties to it among others. Symbols are all about changing the behavior of existing classes by modifying them (Reflection within implementation).
Metaprogramming in ES6: Part 2 - Reflect
In short, Reflect is effectively a collection of all of those “internal methods” that were available exclusively through the JavaScript engine internals, now exposed in one single, handy object. Its usage is analogous to the Reflection capabilities of Java and C#. They are used to discover very low level information about your code (Reflection through introspection).
Metaprogramming in ES6: Part 3 - Proxies
In short, Proxies are handler objects, responsible for wrapping objects and intercepting their behaviors through traps (Reflection through intercession).
Of course, these objects provide specific metaprogramming capabilities, much more restrictive compared to metaprogramming languages, but still can provide handy ways of basic metaprogramming, mainly through Reflection practices, in fact.
In the end, it is worth mentioning that there is some worth-noticing ongoing research work on staged metaprogramming in JavaScript.
Well, in your code sample:
var Result = from(obj1)
.as("x")
.where("x.id=5")
.groupby("x.status")
.having(count("x.status") > 5)
.select("x.status");
The only problem I see (other than select used as an identifier) is that you embed a predicate as a function argument. You'd have to make it a function instead:
.having(function(x){ return x.status > 5; })
JavaScript has closures and dynamic typing, so you can do some really nifty and elegant things in it. Just letting you know.
In pure JS no you can not. But with right preprocessor it is possible.
You can do something similar with sweet.js macros or (God forgive me) GPP.
Wat you want is to change the javascript parser into an SQL parser. It wasn't created to do that, the javascript syntax doesn't allow you to.
What you have is 90% like SQL (it maps straight onto it), and a 100% valid javascript, which is a great achievement. My answer to the question in the title is: YES, metaprogramming is possible, but NO it won't give you an SQL parser, since it's bound to use javascript grammar.
Maybe you want something like JSONPath if you've got JSON data. I found this at http://www.json.org/. Lots of other tools linked to from there if it's not exactly what you need.
(this is being worked on as well: http://docs.dojocampus.org/dojox/json/query)