I'm playing around with the idea of creating a global search that allows me to find any model in any of a number of collections by any of the model's attributes. For example:
I have the following collections:
Users
Applications
Roles
I don't know ahead of time what attributes each User, Applicaion and Role will have but for illustration purposes lets say I have:
User.name
User.last_name
User.email
Application.title
Application.description
Role.name
Role.description
Now, lets say I create a model called Site with a method called search. I want Site.search(term) to search through all the items in each collection where term matches any of the attributes. In essence, a global model search.
How would you suggest I approach this? I can brute-force it by iterating through all the collections' models and each model's attributes but that seems bloated and inefficient.
Any suggestions?
/// A few minutes later...
Here's a bit of code I tried just now:
find: function(query) {
var results = {}; // variable to hold the results
// iterate over the collections
_.each(["users", "applications", "roles"], _.bind(function(collection){
// I want the result to be grouped by type of model so I add arrays to the results object
if ( !_.isUndefined(results[collection]) || !_.isArray(results[collection]) ) {
results[collection] = [];
}
// iterate over the collection's models
_.each(this.get(collection).models, function(model){
// iterate over each model's attributes
_.each(model.attributes, function(value){
// for now I'm only considering string searches
if (_.isString(value)) {
// see if `query` is in the attribute's string/value
if (value.indexOf(query) > -1) {
// if so, push it into the result's collection arrray
results[collection].push(model);
}
};
});
});
// a little cleanup
results[collection] = _.compact(results[collection]);
// remove empty arrays
if (results[collection].length < 1) {
delete results[collection];
}
},this));
// return the results
return results;
}
This yields the expected result and I suppose it works fine but it bothers me that I'm iterating over three arrays. there may not be another solution but I have a feeling there is. If anyone can suggest one, thank you! Meanwhile I'll keep researching.
Thank you!
I would strongly discourage you from doing this, unless you have a very limited set of data and performance is not really a problem for you.
Iteration over everything is a no-no if you want to perform search. Search engines index data and make the process feasible. It is hard to build search, and there is no client-side library that does that effectively.
Which is why everybody is doing searching on the server. There exist easy (or sort of) to use search engines such as solr or the more recent and my personal preference elasticsearch. Presumably you already store your models/collections on the server, it should be trivial to also index them. Then searching becomes a question of making a REST call from your client.
Related
I have certain requirements , I wanted to do the following in quickest way as possible.
I have 1000's of objects like below
{id:1,value:"value1"} . . {id:1000,value:"value1000"}
I want to access above objects by id
I want to clean the objects Lesser than certain id every few minutes (Because it generates 1000's of objects every second for my high frequency algorithm)
I can clean easily by using this.
myArray = myArray.filter(function( obj ) {
return obj.id > cleanSize;
});
I can find the object by id using
myArray.find(x => x.id === '45');
Problem is here , I feel that find is little slower when there is larger sets of data.So I created some objects of object like below
const id = 22;
myArray["x" + id] = {};
myArray["x" + id] = { id: id, value:"test" };
so I can access my item by id easily by myArray[x22]; , but problem is i am not able find the way to remove older items by id.
someone guide me better way to achieve the three points I mentioned above using arrays or objects.
The trouble with your question is, you're asking for a way to finish an algorithm that is supposed to solve a problem of yours, but I think there's something fundamentally wrong with the problem to begin with :)
If you store a sizeable amount of data records, each associated with an ID, and allow your code to access them freely, then you cannot have another part of your code dump some of them to the bin out of the blue (say, from within some timer callback) just because they are becoming "too old". You must be sure nobody is still working on them (and will ever need to) before deleting any of them.
If you don't explicitly synchronize the creation and deletion of your records, you might end up with a code that happens to work (because your objects happen to be processed quickly enough never to be deleted too early), but will be likely to break anytime (if your processing time increases and your data becomes "too old" before being fully processed).
This is especially true in the context of a browser. Your code is supposed to run on any computer connected to the Internet, which could have dozens of reasons to be running 10 or 100 times slower than the machine you test your code on. So making assumptions about the processing time of thousands of records is asking for serious trouble.
Without further specification, it seems to me answering your question would be like helping you finish a gun that would only allow you to shoot yourself in the foot :)
All this being said, any JavaScript object inherently does exactly what you ask for, provided you're okay with using strings for IDs, since an object property name can also be used as an index in an associative array.
var associative_array = {}
var bob = { id:1456, name:"Bob" }
var ted = { id:2375, name:"Ted" }
// store some data with arbitrary ids
associative_array[bob.id] = bob
associative_array[ted.id] = ted
console.log(JSON.stringify(associative_array)) // Bob and Ted
// access data by id
var some_guy = associative_array[2375] // index will be converted to string anyway
console.log(JSON.stringify(some_guy)) // Ted
var some_other_guy = associative_array["1456"]
console.log(JSON.stringify(some_other_guy)) // Bob
var some_AWOL_guy = associative_array[9999]
console.log(JSON.stringify(some_AWOL_guy)) // undefined
// delete data by id
delete associative_array[bob.id] // so long, Bob
console.log(JSON.stringify(associative_array)) // only Ted left
Though I doubt speed will really be an issue, this mechanism is about as fast as you will ever get JavaScript to run, since the underlying data structure is a hash table, theoretically O(1).
Anything involving array methods like find() or filter() will run in at least O(n).
Besides, each invocation of filter() would waste memory and CPU recreating the array to no avail.
I'm working with EmberJS and SailsJS. Now I've been asked to make a statistics page and handle the filtering process in the SailsJS.
I have a model for departments and another model for requests the relationship between these models is (request belongsTo department). For some reason my manager prevented me to make a (hasMany) relationship.
Now what I want to do is to loop through all the departments and store them in new Object, inside that loop I want to loop through all the requests by Using Request.count({where : { department : department.id }}) and get the number of requests for each department in the departments Array Of Object.
I tried to do it as I explained but the problem is when I log the department inside the request loop it gives me the result as I imagined but when I add (.id) it shows me 'undefined'.
Here is my code:
Department.find().then((departments) => {
report.departments = departments;
Request.count({ department : departments.id}).exec(function countMe(err, count) {
console.log(count);
})
})
PS: if there's any other approach for this task please tell me, I'm kind of a beginner.
Your departments object in the .then callback is an array of department objects. To get an id you'd need to do something like departments[0].id.
I might not recommend using .count because that means a separate trip to the database for each department. Sails waterline may have some way for you to count up Requests grouped by department id, but just to get a steamroller working example, I would first just get everything and do some processing in your code:
Department.find().then((departments) => {
Request.find().then((requests) => {
var requestCounts = {}; // we'll store the counts in this object
for (var idx = 0; idx < requests.length; idx++) {
if (!requestCounts[requests[idx].department]) {
requestCounts[requests[idx].department] = 0;
}
requestCounts[requests[idx].department]++;
}
// use requestCounts...
});
});
Creating a separate object like that may not be what you want to do, but something like this should serve whatever purpose you have. Notice, the code I wrote did not require finding all the department objects, but whatever you are doing likely will...
If you're interested in efficiency, then once you get this working, you can see if there is some way you can query the database to directly get the counts you want instead of this in-code processing. But I would start with something simple like this.
EDIT:
It sounds like you may be able to use something like:
Request.find().groupBy('department').sum('count').exec(function (err, results){
console.log(results);
});
But I'm finding conflicting reports on whether this works with sails-mongo, so take this as a "maybe this will work" recommendation.
I assume Request has a property department which is an ObjectId related to departments.
The quick one is to correct the following:
the 2nd line does not make sense as reports is not used
do
let departmentIds = _.map(departments, 'id');
and use departmentIds in the count query object
However, you are anyway finding all departments in the initial query, I would assume you count all the Requests anyway. Finding the departments first only makes sense in case you have query object there limiting the number of returned departments.
In addition, if you know the departmentIds in question you might not need to query them. Otherwise you might use a projection to just return the departmentIds instead of all the properties of all the departments
How it is
I have an array of objects called vm.queued_messages (vm is set to this in my controller), and vm.queued_messages is used in ng-repeat to display a list of div's.
When I make an API call which changes the underlying model in the database, I have the API call return a fresh list of queued messages, and in my controller I set the variable vm.queued_messages to that new value, that fresh list of queued messages.
vm.queued_messages = data; // data is the full list of new message objects
The problem
This "full replacement" of vm.queued_messages worked exactly as I wanted, at first. But what I didn't think about was the fact that even objects in that list which had no changes to any properties were leaving and new objects were taking their place. This made no different to the display because the new objects had identical keys and values, they were technically different objects, and thus the div's were secretly leaving and entering every time. THIS MEANS THERE ARE MANY UNWANTED .ng-enter's AND .ng-leave's OCCURRING, which came to my attention when I tried to apply an animation to these div's when they entered or left. I would expect a single div to do the .ng-leave animation on some click, but suddenly a bunch of them did!
My solution attempt
I made a function softRefreshObjectList which updates the keys and values (as well as any entirely new objects, or now absent objects) of an existing list to match those of a new list, WITHOUT REPLACING THE OBJECTS, AS TO MAINTAIN THEIR IDENTITY. I matched objects between the new list and old list by their _id field.
softRefreshObjectList: function(oldObjs, newObjs) {
var resultingObjList = [];
var oldObjsIdMap = {};
_.each(oldObjs, function(obj) {
oldObjsIdMap[obj._id] = obj;
});
_.each(newObjs, function(newObj) {
var correspondingOldObj = oldObjsIdMap[newObj._id];
if (correspondingOldObj) {
// clear out the old obj and put in the keys/values from the new obj
for (var key in correspondingOldObj) delete correspondingOldObj[key];
for (var key in newObj) correspondingOldObj[key] = newObj[key];
resultingObjList.push(correspondingOldObj);
} else {
resultingObjList.push(newObj);
};
});
return resultingObjList;
}
which works for certain things, but with other ng-repeat lists I get odd behavior, I believe because of the delete's and values of the objects being references to other controller variables. Before continuing down this rabbit hole, I want to make this post in case I'm thinking about this wrong, or there's something I'm missing.
My question
Is there a more appropriate way to handle this case, which would either make it easier to handle, or bypass my issue altogether?
Perhaps a way to signal to Angular that these objects are identified by their _id instead of their reference, so that it doesn't make them leave and enter as long as the _id doesn't change.
Or perhaps a better softRefreshObjectList function which iterates through the objects differently, if there's something fishy about how I'm doing it.
Thanks to Petr's comment, I now know about track by for ng-repeat. It's where you can specify a field in your elements that "identifies" that element, so that angular can know when that element really is leaving or entering. In my case, that field was _id, and adding track by message._id to my ng-repeat (ng-repeat="message in ctrl.queued_messages track by message._id") solved my issue perfectly.
Docs here. Search for track by.
I have a pretty big array of JSON objects (its a music library with properties like artist, album etc, feeding a jqgrid with loadonce=true) and I want to implement lucene-like (google-like) query through whole set - but locally, i.e. in the browser, without communication with web server. Are there any javascript frameworks that will help me?
Go through your records, to create a one time index by combining all search
able fields in a single string field called index.
Store these indexed records in an Array.
Partition the Array on index .. like all a's in one array and so on.
Use the javascript function indexOf() against the index to match the query entered by the user and find records from the partitioned Array.
That was the easy part but, it will support all simple queries in a very efficient manner because the index does not have to be re-created for every query and indexOf operation is very efficient. I have used it for searching up to 2000 records. I used a pre-sorted Array. Actually, that's how Gmail and yahoo mail work. They store your contacts on browser in a pre-sorted array with an index that allows you to see the contact names as you type.
This also gives you a base to build on. Now you can write an advanced query parsing logic on top of it. For example, to support a few simple conditional keywords like - AND OR NOT, will take about 20-30 lines of custom JavaScript code. Or you can find a JS library that will do the parsing for you the way Lucene does.
For a reference implementation of above logic, take a look at how ZmContactList.js sorts and searches the contacts for autocomplete.
You might want to check FullProof, it does exactly that:
https://github.com/reyesr/fullproof
Have you tried CouchDB?
Edit:
How about something along these lines (also see http://jsfiddle.net/7tV3A/1/):
var filtered_collection = [];
var query = 'foo';
$.each(collection, function(i,e){
$.each(e, function(ii, el){
if (el == query) {
filtered_collection.push(e);
}
});
});
The (el == query) part of course could/should be modified to allow more flexible search patterns than exact match.
I recently discovered (sadly) that WebSQL is no longer being supported for HTML5 and that IndexedDB will be replacing it instead.
I'm wondering if there is any way to query or search through the entries of an IndexedDB in a similar way to how I can use SQL to search for an entry satisfying multiple conditions.
I've seen that I can search through IndexedDB using one condition with the KeyRange. However, I can't seem to find any way to search two or more columns of data without grabbing all the data from the database and doing it with for loops.
I know this is a new feature that's barely implemented in the browsers, but I have a project that I'm starting and I'm researching the different ways I could do it.
Thank you!
Check out this answer to the same question. It is more detailed than the answer I give here. The keypath parameter to store.createIndex and IDBKeyRange methods can be an array. So, crude example:
// In onupgradeneeded
var store = db.createObjectStore('mystore');
store.createIndex('myindex', ['prop1','prop2'], {unique:false});
// In your query section
var transaction = db.transaction('mystore','readonly');
var store = transaction.objectStore('mystore');
var index = store.index('myindex');
// Select only those records where prop1=value1 and prop2=value2
var request = index.openCursor(IDBKeyRange.only([value1, value2]));
// Select the first matching record
var request = index.get(IDBKeyRange.only([value1, value2]));
Let's say your SQL Query is something like:
SELECT * FROM TableName WHERE Column1 = 'value1' AND Column2 = 'value2'
Equivalent Query in JsStore library:
var Connection = new JsStore.Instance("YourDbName");
Connection.select({
From: "YourTableName"
Where: {
Column1: 'value1',
Column2: 'value2'
},
OnSuccess:function (results){
console.log(results);
},
OnError:function (error) {
console.log(error);
}
});
Now, if you are wondering what JsStore is, let me tell you it is a library to query IndexedDB in a simplified manner. Click here to learn more about JsStore
I mention some suggestions for querying relationships in my answer to this question, which may be of interest:
Conceptual problems with IndexedDB (relationships etc.)
As to querying multiple fields at once, it doesn't look like there's a native way to do that in IndexedDB (I could be wrong; I'm still new to it), but you could certainly create a helper function that used a separate cursor for each field, and iterated over them to see which records met all the criteria.
Yes, opening continuous key range on an index is pretty much that is in indexedDB. Testing for multiple condition is not possible in IndexedDB. It must be done on cursor loop.
If you find the solution, please let me know.
BTW, i think looping cursor could be very fast and require less memory than possible with Sqlite.
I'm a couple of years late, but I'd just like to point out that Josh's answer works on the presumption that all of the "columns" in the condition are part of the index's keyPath.
If any of said "columns" exist outside the the index's keyPath, you will have to test the conditions involving them on each entry which the cursor created in the example iterates over. So if you're dealing with such queries, or your index isn't unique, be prepared to write some iteration code!
In any case, I suggest you check out BakedGoods if you can represent your query as a boolean expression.
For these types of operations, it will always open a cursor on the focal objectStore unless you're performing a strict equality query (x ===? y, given x is an objectStore or index key), but it will save you the trouble writing your own cursor iteration code:
bakedGoods.getAll({
filter: "keyObj > 5 && valueObj.someProperty !== 'someValue'",
storageTypes: ["indexedDB"],
complete: function(byStorageTypeResultDataObj, byStorageTypeErrorObj){}
});
Just for the sake of complete transparency, BakedGoods is maintained by moi.