I recently discovered (sadly) that WebSQL is no longer being supported for HTML5 and that IndexedDB will be replacing it instead.
I'm wondering if there is any way to query or search through the entries of an IndexedDB in a similar way to how I can use SQL to search for an entry satisfying multiple conditions.
I've seen that I can search through IndexedDB using one condition with the KeyRange. However, I can't seem to find any way to search two or more columns of data without grabbing all the data from the database and doing it with for loops.
I know this is a new feature that's barely implemented in the browsers, but I have a project that I'm starting and I'm researching the different ways I could do it.
Thank you!
Check out this answer to the same question. It is more detailed than the answer I give here. The keypath parameter to store.createIndex and IDBKeyRange methods can be an array. So, crude example:
// In onupgradeneeded
var store = db.createObjectStore('mystore');
store.createIndex('myindex', ['prop1','prop2'], {unique:false});
// In your query section
var transaction = db.transaction('mystore','readonly');
var store = transaction.objectStore('mystore');
var index = store.index('myindex');
// Select only those records where prop1=value1 and prop2=value2
var request = index.openCursor(IDBKeyRange.only([value1, value2]));
// Select the first matching record
var request = index.get(IDBKeyRange.only([value1, value2]));
Let's say your SQL Query is something like:
SELECT * FROM TableName WHERE Column1 = 'value1' AND Column2 = 'value2'
Equivalent Query in JsStore library:
var Connection = new JsStore.Instance("YourDbName");
Connection.select({
From: "YourTableName"
Where: {
Column1: 'value1',
Column2: 'value2'
},
OnSuccess:function (results){
console.log(results);
},
OnError:function (error) {
console.log(error);
}
});
Now, if you are wondering what JsStore is, let me tell you it is a library to query IndexedDB in a simplified manner. Click here to learn more about JsStore
I mention some suggestions for querying relationships in my answer to this question, which may be of interest:
Conceptual problems with IndexedDB (relationships etc.)
As to querying multiple fields at once, it doesn't look like there's a native way to do that in IndexedDB (I could be wrong; I'm still new to it), but you could certainly create a helper function that used a separate cursor for each field, and iterated over them to see which records met all the criteria.
Yes, opening continuous key range on an index is pretty much that is in indexedDB. Testing for multiple condition is not possible in IndexedDB. It must be done on cursor loop.
If you find the solution, please let me know.
BTW, i think looping cursor could be very fast and require less memory than possible with Sqlite.
I'm a couple of years late, but I'd just like to point out that Josh's answer works on the presumption that all of the "columns" in the condition are part of the index's keyPath.
If any of said "columns" exist outside the the index's keyPath, you will have to test the conditions involving them on each entry which the cursor created in the example iterates over. So if you're dealing with such queries, or your index isn't unique, be prepared to write some iteration code!
In any case, I suggest you check out BakedGoods if you can represent your query as a boolean expression.
For these types of operations, it will always open a cursor on the focal objectStore unless you're performing a strict equality query (x ===? y, given x is an objectStore or index key), but it will save you the trouble writing your own cursor iteration code:
bakedGoods.getAll({
filter: "keyObj > 5 && valueObj.someProperty !== 'someValue'",
storageTypes: ["indexedDB"],
complete: function(byStorageTypeResultDataObj, byStorageTypeErrorObj){}
});
Just for the sake of complete transparency, BakedGoods is maintained by moi.
Related
I have certain requirements , I wanted to do the following in quickest way as possible.
I have 1000's of objects like below
{id:1,value:"value1"} . . {id:1000,value:"value1000"}
I want to access above objects by id
I want to clean the objects Lesser than certain id every few minutes (Because it generates 1000's of objects every second for my high frequency algorithm)
I can clean easily by using this.
myArray = myArray.filter(function( obj ) {
return obj.id > cleanSize;
});
I can find the object by id using
myArray.find(x => x.id === '45');
Problem is here , I feel that find is little slower when there is larger sets of data.So I created some objects of object like below
const id = 22;
myArray["x" + id] = {};
myArray["x" + id] = { id: id, value:"test" };
so I can access my item by id easily by myArray[x22]; , but problem is i am not able find the way to remove older items by id.
someone guide me better way to achieve the three points I mentioned above using arrays or objects.
The trouble with your question is, you're asking for a way to finish an algorithm that is supposed to solve a problem of yours, but I think there's something fundamentally wrong with the problem to begin with :)
If you store a sizeable amount of data records, each associated with an ID, and allow your code to access them freely, then you cannot have another part of your code dump some of them to the bin out of the blue (say, from within some timer callback) just because they are becoming "too old". You must be sure nobody is still working on them (and will ever need to) before deleting any of them.
If you don't explicitly synchronize the creation and deletion of your records, you might end up with a code that happens to work (because your objects happen to be processed quickly enough never to be deleted too early), but will be likely to break anytime (if your processing time increases and your data becomes "too old" before being fully processed).
This is especially true in the context of a browser. Your code is supposed to run on any computer connected to the Internet, which could have dozens of reasons to be running 10 or 100 times slower than the machine you test your code on. So making assumptions about the processing time of thousands of records is asking for serious trouble.
Without further specification, it seems to me answering your question would be like helping you finish a gun that would only allow you to shoot yourself in the foot :)
All this being said, any JavaScript object inherently does exactly what you ask for, provided you're okay with using strings for IDs, since an object property name can also be used as an index in an associative array.
var associative_array = {}
var bob = { id:1456, name:"Bob" }
var ted = { id:2375, name:"Ted" }
// store some data with arbitrary ids
associative_array[bob.id] = bob
associative_array[ted.id] = ted
console.log(JSON.stringify(associative_array)) // Bob and Ted
// access data by id
var some_guy = associative_array[2375] // index will be converted to string anyway
console.log(JSON.stringify(some_guy)) // Ted
var some_other_guy = associative_array["1456"]
console.log(JSON.stringify(some_other_guy)) // Bob
var some_AWOL_guy = associative_array[9999]
console.log(JSON.stringify(some_AWOL_guy)) // undefined
// delete data by id
delete associative_array[bob.id] // so long, Bob
console.log(JSON.stringify(associative_array)) // only Ted left
Though I doubt speed will really be an issue, this mechanism is about as fast as you will ever get JavaScript to run, since the underlying data structure is a hash table, theoretically O(1).
Anything involving array methods like find() or filter() will run in at least O(n).
Besides, each invocation of filter() would waste memory and CPU recreating the array to no avail.
I have jQuery autocomplete field that has to search through several thousand items, populated from an IndexedDB query (using the idb wrapper). The following is the autocomplete function called when the user begins typing in the box. hasKW() is a function that finds keywords.
async function siteAutoComplete(request, response) {
const db = await openDB('AgencySite');
const hasKW = createKeyWordFunction(request.term);
const state = "NY";
const PR = 0;
const agency_id = 17;
const range = IDBKeyRange.bound([state, PR, agency_id], [state, PR, agency_id || 9999999]);
let cursor = await db.transaction('sites').store.index("statePRAgency").openCursor(range);
let result = [];
while (cursor) {
if (hasKW(cursor.value.name)) result.push({
value: cursor.value.id,
label: cursor.value.name
});
cursor = await cursor.continue();
}
response(result);
}
My question is this: I'm not sure if the cursor is making everything slow. Is there a way to get all database rows that match the query without using a cursor? Is building the result array slowing me down? Is there a better way of doing this? Currently it takes 2-3s to show the autocomplete list.
I hope this will be useful to someone else. I removed the cursor and just downloaded the whole DB into a javascript array and then used .filter. The speedup was dramatic. It took 2300ms using the way above and about 21ms using this:
let result = await db.transaction('sites').store.index("statePRAgency").getAll();
response(result.filter(hasKW));
You probably want to use an index, where by the term index, I mean a custom built one that represents a search engine index. You cannot easily and efficiently perform "startsWith" style queries over one of indexedDB's indices because they are effectively whole value (or least lexicographic).
There are many ways to create the search engine index I am suggesting. You probably want something like a prefix-tree, also known informally as a trie.
Here is a nice article by John Resig that you might find helpful: https://johnresig.com/blog/javascript-trie-performance-analysis/. Otherwise, I suggest searching around on Google for trie implementations and then figuring out how to represent a similar data structure within an indexedDb object store or indexdDb index on an object store.
Essentially, insert the data first without the properties used by the index. Then, in an "indexing step", visit each object and index its value, and set the properties used by the indexedDb index. Or do this at time of insert/update.
From there, you probably want to open a connection shortly after page load and keep it open for the entire duration of the page. Then query against the index every time a character is typed (probably want to rate limit this call to refrain from querying more than n/second, perhaps using some kind of debounce helper function).
On the other hand, I might be a bit rusty on this one, but maybe you can create an index on the string prop, then use a lower bound that is the entered characters. A string that is lesser length than another string that contains it is present earlier in lexicographic order. So maybe it is actually that easy. You would also need to impose an upper bound that contains the entered characters thus far concatenated with some kind of sentinel value that can never realistically exist in the data, something silly like ZZZZZ.
Try this out in the browser's console:
indexedDB.cmp('test', 'tasting'); // 1
indexedDB.cmp('test', 'testing'); // -1
indexedDB.cmp('test', 'test'); // 0
You essentially want to experiment with a query like this:
const sentinel = 'ZZZ';
const index = store.index('myStore');
const bounds = IDBKeyRange.bound(value, value + sentinel);
const request = index.get(bounds);
You might need to tweak the sentinel, experiment with other parameters to IDBKeyRange.bound (the inclusive/exclusive flags), probably need to store the value in homogenized case so that the search is case insensitive, avoid every sending a query when nothing has been typed, etc.
I want to query object from Parse DB through javascript, that has only 1 of some specific relation object. How can this criteria be achieved?
So I tried something like this, the equalTo() acts as a "contains" and it's not what I'm looking for, my code so far, which doesn't work:
var query = new Parse.Query("Item");
query.equalTo("relatedItems", someItem);
query.lessThan("relatedItems", 2);
It seems Parse do not provide a easy way to do this.
Without any other fields, if you know all the items then you could do the following:
var innerQuery = new Parse.Query('Item');
innerQuery.containedIn('relatedItems', [all items except someItem]);
var query = new Parse.Query('Item');
query.equalTo('relatedItems', someItem);
query.doesNotMatchKeyInQuery('objectId', 'objectId', innerQuery);
...
Otherwise, you might need to get all records and do filtering.
Update
Because of the data type relation, there are no ways to include the relation content into the results, you need to do another query to get the relation content.
The workaround might add a itemCount column and keep it updated whenever the item relation is modified and do:
query.equalTo('relatedItems', someItem);
query.equalTo('itemCount', 1);
There are a couple of ways you could do this.
I'm working on a project now where I have cells composed of users.
I currently have an afterSave trigger that does this:
const count = await cell.relation("members").query().count();
cell.put("memberCount",count);
This works pretty well.
There are other ways that I've considered in theory, but I've not used
them yet.
The right way would be to hack the ability to use select with dot
notation to grab a virtual field called relatedItems.length in the
query, but that would probably only work for me because I use PostGres
... mongo seems to be extremely limited in its ability to do this sort
of thing, which is why I would never make a database out of blobs of
json in the first place.
You could do a similar thing with an afterFind trigger. I'm experimenting with that now. I'm not sure if it will confuse
parse to get an attribute back which does not exist in its schema, but
I'll find out, by the end of today. I have found that if I jam an artificial attribute into the objects in the trigger, they are returned
along with the other data. What I'm not sure about is whether Parse will decide that the object is dirty, or, worse, decide that I'm creating a new attribute and store it to the database ... which could be filtered out with a beforeSave trigger, but not until after the data had all been sent to the cloud.
There is also a place where i had to do several queries from several
tables, and would have ended up with a lot of redundant data. So I wrote a cloud function which did the queries, and then returned a couple of lists of objects, and a few lists of objectId strings which
served as indexes. This worked pretty well for me. And tracking the
last load time and sending it back when I needed up update my data allowed me to limit myself to objects which had changed since my last query.
I'm playing around with the idea of creating a global search that allows me to find any model in any of a number of collections by any of the model's attributes. For example:
I have the following collections:
Users
Applications
Roles
I don't know ahead of time what attributes each User, Applicaion and Role will have but for illustration purposes lets say I have:
User.name
User.last_name
User.email
Application.title
Application.description
Role.name
Role.description
Now, lets say I create a model called Site with a method called search. I want Site.search(term) to search through all the items in each collection where term matches any of the attributes. In essence, a global model search.
How would you suggest I approach this? I can brute-force it by iterating through all the collections' models and each model's attributes but that seems bloated and inefficient.
Any suggestions?
/// A few minutes later...
Here's a bit of code I tried just now:
find: function(query) {
var results = {}; // variable to hold the results
// iterate over the collections
_.each(["users", "applications", "roles"], _.bind(function(collection){
// I want the result to be grouped by type of model so I add arrays to the results object
if ( !_.isUndefined(results[collection]) || !_.isArray(results[collection]) ) {
results[collection] = [];
}
// iterate over the collection's models
_.each(this.get(collection).models, function(model){
// iterate over each model's attributes
_.each(model.attributes, function(value){
// for now I'm only considering string searches
if (_.isString(value)) {
// see if `query` is in the attribute's string/value
if (value.indexOf(query) > -1) {
// if so, push it into the result's collection arrray
results[collection].push(model);
}
};
});
});
// a little cleanup
results[collection] = _.compact(results[collection]);
// remove empty arrays
if (results[collection].length < 1) {
delete results[collection];
}
},this));
// return the results
return results;
}
This yields the expected result and I suppose it works fine but it bothers me that I'm iterating over three arrays. there may not be another solution but I have a feeling there is. If anyone can suggest one, thank you! Meanwhile I'll keep researching.
Thank you!
I would strongly discourage you from doing this, unless you have a very limited set of data and performance is not really a problem for you.
Iteration over everything is a no-no if you want to perform search. Search engines index data and make the process feasible. It is hard to build search, and there is no client-side library that does that effectively.
Which is why everybody is doing searching on the server. There exist easy (or sort of) to use search engines such as solr or the more recent and my personal preference elasticsearch. Presumably you already store your models/collections on the server, it should be trivial to also index them. Then searching becomes a question of making a REST call from your client.
I have a pretty big array of JSON objects (its a music library with properties like artist, album etc, feeding a jqgrid with loadonce=true) and I want to implement lucene-like (google-like) query through whole set - but locally, i.e. in the browser, without communication with web server. Are there any javascript frameworks that will help me?
Go through your records, to create a one time index by combining all search
able fields in a single string field called index.
Store these indexed records in an Array.
Partition the Array on index .. like all a's in one array and so on.
Use the javascript function indexOf() against the index to match the query entered by the user and find records from the partitioned Array.
That was the easy part but, it will support all simple queries in a very efficient manner because the index does not have to be re-created for every query and indexOf operation is very efficient. I have used it for searching up to 2000 records. I used a pre-sorted Array. Actually, that's how Gmail and yahoo mail work. They store your contacts on browser in a pre-sorted array with an index that allows you to see the contact names as you type.
This also gives you a base to build on. Now you can write an advanced query parsing logic on top of it. For example, to support a few simple conditional keywords like - AND OR NOT, will take about 20-30 lines of custom JavaScript code. Or you can find a JS library that will do the parsing for you the way Lucene does.
For a reference implementation of above logic, take a look at how ZmContactList.js sorts and searches the contacts for autocomplete.
You might want to check FullProof, it does exactly that:
https://github.com/reyesr/fullproof
Have you tried CouchDB?
Edit:
How about something along these lines (also see http://jsfiddle.net/7tV3A/1/):
var filtered_collection = [];
var query = 'foo';
$.each(collection, function(i,e){
$.each(e, function(ii, el){
if (el == query) {
filtered_collection.push(e);
}
});
});
The (el == query) part of course could/should be modified to allow more flexible search patterns than exact match.