I have a 10MB JSON file of the following structure (10k entries):
{
entry_1: {
description: "...",
offset: "...",
value: "...",
fields: {
field_1: {
offset: "...",
description: "...",
},
field_2: {
offset: "...",
description: "...",
}
}
},
entry_2:
...
...
...
}
I want to implement an autocomplete input field that will fetch suggestions from this file, as fast as possible while searching multiple attributes.
For example, finding all entry names,field names and descriptions that contain some substring.
Method 1:
I tried to flatten the nesting into an array of strings:
"entry_1|descrption|offset|value|field1|offset|description",
"entry_1|descrption|offset|value|field2|offset|description",
"entry2|..."
and perform case insensitive partial string match, query took about 900ms.
Method 2
I tried Xpath-based JSON querying (using defiant.js).
var snapshot = Defiant.getSnapshot(DATA);
found = JSON.search(snapshot, '//*[contains(fields, "substring")]');
query took about 600ms (just for a single attribute, fields).
Are there other options that will get me to sub 100ms? I have control of the file format so I can turn it into XML or any other format, the only requirement is speed.
Since you are trying to search for a substring of values it is not a good idea to use indexeddb as suggested. You can try flattening the values of the fields to text where fields seperated by :: and each key in the object is a line in the text file:
{
key1:{
one:"one",
two:"two",
three:"three"
},
key2:{
one:"one 2",
two:"two 2",
three:"three 2"
}
}
Will be:
key1::one::two::three
key2::one 2::two 2::three
Then use regexp to search for text after the keyN:: part and store all keys that match. Then map all those keys to the objects. So if key1 is the only match you'd return [data.key1]
Here is an example with sample data of 10000 keys (search on laptop takes couple of milliseconds but have not tested when throttling to mobile):
//array of words, used as value for data.rowN
const wordArray = ["actions","also","amd","analytics","and","angularjs","another","any","api","apis","application","applications","are","arrays","assertion","asynchronous","authentication","available","babel","beautiful","been","between","both","browser","build","building","but","calls","can","chakra","clean","client","clone","closure","code","coherent","collection","common","compiler","compiles","concept","cordova","could","created","creating","creation","currying","data","dates","definition","design","determined","developed","developers","development","difference","direct","dispatches","distinct","documentations","dynamic","easy","ecmascript","ecosystem","efficient","encapsulates","engine","engineered","engines","errors","eslint","eventually","extend","extension","falcor","fast","feature","featured","fetching","for","format","framework","fully","function","functional","functionality","functions","furthermore","game","glossary","graphics","grunt","hapi","has","having","help","helps","hoisting","host","how","html","http","hybrid","imperative","include","incomplete","individual","interact","interactive","interchange","interface","interpreter","into","its","javascript","jquery","jscs","json","kept","known","language","languages","library","lightweight","like","linked","loads","logic","majority","management","middleware","mobile","modular","module","moment","most","multi","multiple","mvc","native","neutral","new","newer","nightmare","node","not","number","object","objects","only","optimizer","oriented","outside","own","page","paradigm","part","patterns","personalization","plugins","popular","powerful","practical","private","problem","produce","programming","promise","pure","refresh","replace","representing","requests","resolved","resources","retaining","rhino","rich","run","rxjs","services","side","simple","software","specification","specifying","standardized","styles","such","support","supporting","syntax","text","that","the","their","they","toolkit","top","tracking","transformation","type","underlying","universal","until","use","used","user","using","value","vuejs","was","way","web","when","which","while","wide","will","with","within","without","writing","xml","yandex"];
//get random number
const rand = (min,max) =>
Math.floor(
(Math.random()*(max-min))+min
)
;
//return object: {one:"one random word from wordArray",two:"one rand...",three,"one r..."}
const threeMembers = () =>
["one","two","three"].reduce(
(acc,item)=>{
acc[item] = wordArray[rand(0,wordArray.length)];
return acc;
}
,{}
)
;
var i = -1;
data = {};
//create data: {row0:threeMembers(),row1:threeMembers()...row9999:threeMembers()}
while(++i<10000){
data[`row${i}`] = threeMembers();
}
//convert the data object to string "row0::word::word::word\nrow1::...\nrow9999..."
const dataText = Object.keys(data)
.map(x=>`${x}::${data[x].one}::${data[x].two}::${data[x].three}`)
.join("\n")
;
//search for someting (example searching for "script" will match javascript and ecmascript)
// i in the regexp "igm" means case insensitive
//return array of data[matched key]
window.searchFor = search => {
const r = new RegExp(`(^[^:]*).*${search}`,"igm")
,ret=[];
var result = r.exec(dataText);
while(result !== null){
ret.push(result[1]);
result = r.exec(dataText);
}
return ret.map(x=>data[x]);
};
//example search for "script"
console.log(searchFor("script"));
Let's suppose I have an associative array like this one
var client1={
"id":"1"
"category":"Interiorism",
"photo1":"img/ClientCorp/photoClient1.jpg",
"photo2":"img/ClientCorp/photoClient2.jpg",
"photo3":"img/ClientCorp/photoClient3.jpg",
"photo4":"img/ClientCorp/photoClient4.jpg",
};
var client2={
.
.
.
};
allClients=[client1, client2..., clientx];
I want to set up a function that pushs the photo keys in an empty array. The problem is that not all the clients have the same number of photos, so I am using 'for'. Here is the function I wrote
function photoKeys()
{
var keyList=Object.keys(allClients[id]);
var numKey=parseInt(listaKeys.length);
var photoAlbum=[]; //here I want to put the photo URL's
for (i=2; i<=numFotos; i++)
{
????????????
}
}
Here is the problem, how I can write the photo object from the client array whith the i var from the 'for' function?
I tried this but didn't work
for (i=2; i<=numFotos; i++)
{
photoAlbum.push(allClients[id].photo+'i');
}
Your current code would be parsed like this:
photoAlbum.push(allClients[id].photo + 'i');
It would try to evaluate allClients[id].photo and then append the string i. You need to access the property name using bracket notation instead of dot notation.
You also have the symbol and string part backward, photo is the string and i is your index variable.
photoAlbum.push(allClients[id]['photo' + i]);
The big thing to understand is that client in your example isn't an array, it's an object.
var client1={
"id":"1"
"category":"Interiorism",
"photo1":"img/ClientCorp/photoClient1.jpg",
"photo2":"img/ClientCorp/photoClient2.jpg",
"photo3":"img/ClientCorp/photoClient3.jpg",
"photo4":"img/ClientCorp/photoClient4.jpg",
};
You can acquire an object's keys as an array using Object.keys(client1), or you can loop through all an object's keys using for...in syntax.
If you want to feed an arbitrary number (numFotos) of property values from your object into an array called photoAlbum, you can use the following syntax:
var i = 0;
for(var key in client1){
photoAlbum.push(client1[key]);
if(++i >= numFotos){
break; // break out of the loop if i equals or exceeds numFotos
}
}
First of all you should be accessing the photo paths like:
photoAlbum.push(allClients[id]['photo' + i]);
But i would really recommend you to change the format of your client object to something like this:
var client1 = {
"id" :"1"
"category" :"Interiorism",
"photos" : [
"img/ClientCorp/photoClient1.jpg",
"img/ClientCorp/photoClient2.jpg",
...
]
};
Or this, if you need to store those "photo1", "photo2" ids:
var client2 = {
"id" :"1"
"category" :"Interiorism",
"photos" : [
{
"id" : "photo1",
"path" :"img/ClientCorp/photoClient1.jpg"
},
...
]
};
Then you can iterate them way easier like this:
for(var i = 0; i < allClients[id].photos.length; i++){
photoAlbum.push(allClients[id].photos[i]);
//or this for the second format:
//photoAlbum.push(allClients[id].photos[i].path);
}
I have the following data
persons = [
{
"age":20,
"parameter1":94,
"name":"Foobarin"
},
{
"age":33,
"parameter1":49,
"name":"Johan"
}
]
I want to create an advanced live search which recognizes patterns. An examples could be "foo a20 p94" which would get me the first object.
a20 - Search where age is 20
p94 - Search where parameter1 is 94
and then if there is any other text which does not have any set prefix then test that against the name value.
All the values (except name which is case-insensitive) is of type integer. I want to limit the prefixes to predefined such as a, p and not age20.
The data-sets is around 400.
I've created a basic live search which searches all variables in the object, but now I do not know where to continue. Any ideas?
It's not foolproof but as a first-pass this is what I'd start with, I'll try to talk through it in pseudo.
First declare a propertyMatrix, a simple object-map which can point "prefixes" to the actual property names which exist within person. The searchPersons function accepts a single string (query) value and consists of two main parts:
The query string is split on whitespace characters into an array of "tokens". Each token is also an array of exactly 2 length, containing each token name and token value. At this stage it attempts find the predetermined prefix - if no such entry exists the token name: name is assumed.
A filter is then applied to the persons array. For each person we iterate over the tokens array and make an appropriate comparison, if any single check fails we return false (thus excluding the person from the results).
var propertyMatrix = {
'a': 'age',
'p': 'parameter1'
},
searchPersons = function(query){
var tokens = query.split(/\s+/).map(function(t){
t = t.toLowerCase();
var i = t.match(/\d+$/), p;
if(i && i.length) {
p = t.substring(0, t.indexOf(i));
if(p in propertyMatrix)
return [propertyMatrix[p], parseInt(i, 10)];
}
return ['name', t];
}),
result = persons.filter(function(p){
for(var i=0, l=tokens.length; i<l; i++){
var token = tokens[i][0], value = tokens[i][1];
if(!(token in p)) return false;
if(token === 'name'){
if(p.name.toLowerCase().indexOf(value)<0) return false;
} else if(p[token] !== value) return false;
}
return true;
});
return result;
};
fiddle