optimizing JSON querying performance in javascript - javascript
I have a 10MB JSON file of the following structure (10k entries):
{
entry_1: {
description: "...",
offset: "...",
value: "...",
fields: {
field_1: {
offset: "...",
description: "...",
},
field_2: {
offset: "...",
description: "...",
}
}
},
entry_2:
...
...
...
}
I want to implement an autocomplete input field that will fetch suggestions from this file, as fast as possible while searching multiple attributes.
For example, finding all entry names,field names and descriptions that contain some substring.
Method 1:
I tried to flatten the nesting into an array of strings:
"entry_1|descrption|offset|value|field1|offset|description",
"entry_1|descrption|offset|value|field2|offset|description",
"entry2|..."
and perform case insensitive partial string match, query took about 900ms.
Method 2
I tried Xpath-based JSON querying (using defiant.js).
var snapshot = Defiant.getSnapshot(DATA);
found = JSON.search(snapshot, '//*[contains(fields, "substring")]');
query took about 600ms (just for a single attribute, fields).
Are there other options that will get me to sub 100ms? I have control of the file format so I can turn it into XML or any other format, the only requirement is speed.
Since you are trying to search for a substring of values it is not a good idea to use indexeddb as suggested. You can try flattening the values of the fields to text where fields seperated by :: and each key in the object is a line in the text file:
{
key1:{
one:"one",
two:"two",
three:"three"
},
key2:{
one:"one 2",
two:"two 2",
three:"three 2"
}
}
Will be:
key1::one::two::three
key2::one 2::two 2::three
Then use regexp to search for text after the keyN:: part and store all keys that match. Then map all those keys to the objects. So if key1 is the only match you'd return [data.key1]
Here is an example with sample data of 10000 keys (search on laptop takes couple of milliseconds but have not tested when throttling to mobile):
//array of words, used as value for data.rowN
const wordArray = ["actions","also","amd","analytics","and","angularjs","another","any","api","apis","application","applications","are","arrays","assertion","asynchronous","authentication","available","babel","beautiful","been","between","both","browser","build","building","but","calls","can","chakra","clean","client","clone","closure","code","coherent","collection","common","compiler","compiles","concept","cordova","could","created","creating","creation","currying","data","dates","definition","design","determined","developed","developers","development","difference","direct","dispatches","distinct","documentations","dynamic","easy","ecmascript","ecosystem","efficient","encapsulates","engine","engineered","engines","errors","eslint","eventually","extend","extension","falcor","fast","feature","featured","fetching","for","format","framework","fully","function","functional","functionality","functions","furthermore","game","glossary","graphics","grunt","hapi","has","having","help","helps","hoisting","host","how","html","http","hybrid","imperative","include","incomplete","individual","interact","interactive","interchange","interface","interpreter","into","its","javascript","jquery","jscs","json","kept","known","language","languages","library","lightweight","like","linked","loads","logic","majority","management","middleware","mobile","modular","module","moment","most","multi","multiple","mvc","native","neutral","new","newer","nightmare","node","not","number","object","objects","only","optimizer","oriented","outside","own","page","paradigm","part","patterns","personalization","plugins","popular","powerful","practical","private","problem","produce","programming","promise","pure","refresh","replace","representing","requests","resolved","resources","retaining","rhino","rich","run","rxjs","services","side","simple","software","specification","specifying","standardized","styles","such","support","supporting","syntax","text","that","the","their","they","toolkit","top","tracking","transformation","type","underlying","universal","until","use","used","user","using","value","vuejs","was","way","web","when","which","while","wide","will","with","within","without","writing","xml","yandex"];
//get random number
const rand = (min,max) =>
Math.floor(
(Math.random()*(max-min))+min
)
;
//return object: {one:"one random word from wordArray",two:"one rand...",three,"one r..."}
const threeMembers = () =>
["one","two","three"].reduce(
(acc,item)=>{
acc[item] = wordArray[rand(0,wordArray.length)];
return acc;
}
,{}
)
;
var i = -1;
data = {};
//create data: {row0:threeMembers(),row1:threeMembers()...row9999:threeMembers()}
while(++i<10000){
data[`row${i}`] = threeMembers();
}
//convert the data object to string "row0::word::word::word\nrow1::...\nrow9999..."
const dataText = Object.keys(data)
.map(x=>`${x}::${data[x].one}::${data[x].two}::${data[x].three}`)
.join("\n")
;
//search for someting (example searching for "script" will match javascript and ecmascript)
// i in the regexp "igm" means case insensitive
//return array of data[matched key]
window.searchFor = search => {
const r = new RegExp(`(^[^:]*).*${search}`,"igm")
,ret=[];
var result = r.exec(dataText);
while(result !== null){
ret.push(result[1]);
result = r.exec(dataText);
}
return ret.map(x=>data[x]);
};
//example search for "script"
console.log(searchFor("script"));
Related
How to iterate over only specific keys from a JSON array object using javascript
I am having a below json array and now I need to iterate over the json object to retrieve two values of fields ServicePort And ServiceAddress and form a final output as {"MyIp" : "http://IP:Port"} from my json array object. var bodyObject = [ { "ServiceAddress": "10.X.X.125", "ServiceConnect": {}, "ServicePort": 80 }, { "ServiceAddress": "10.X.X.126", "ServiceConnect": {}, "ServicePort": 80 } ]; I have tried as below to iterate for (var key in bodyObject ) { if (bodyObject.hasOwnProperty(key)) { console.log(bodyObject[key].ServiceAddress); console.log(bodyObject[key].ServicePort); } } How can I form a output final output like {"MyIp" : "http://IP:Port"} from my json array object each hitting giving me a diffrent Ip's from my above JSON list dynamically. Can someone help on this please
I think you're asking how to create a new array with a single object with a MyIp property whose value is the combination of ServiceAddress and ServicePort. map is the idiomatic way to do that, perhaps with some destructuring to pick out the properties from each object and a template literal to build the resulting string: const result = bodyObject.map(({ServiceAddress, ServicePort}) => { return {MyIp: `http://${ServiceAddress}:${ServicePort}`}; }); or with a concise-form arrow function: const result = bodyObject.map(({ServiceAddress, ServicePort}) => ({MyIp: `http://${ServiceAddress}:${ServicePort}`}) ); (You need the () around the object literal because otherwise it looks like the full function body form of arrow function to the parser.) Live Example: const bodyObject = [ { "ServiceAddress": "10.X.X.125", "ServiceConnect": {}, "ServicePort": 80 }, { "ServiceAddress": "10.X.X.126", "ServiceConnect": {}, "ServicePort": 80 } ]; const result = bodyObject.map(({ServiceAddress, ServicePort}) => ({MyIp: `http://${ServiceAddress}:${ServicePort}`}) ); console.log(result); That has a fair number of newish JavaScript features in it, so just for clarity here's a version without destructuring or a template literal: const result = bodyObject.map(element => { return {MyIp: "http://" + element.ServiceAddress + ":" + element.ServicePort}; });
Javascript String splitting and Organizing based on first letter
I am currently working a project that I have to use js and php to retrieve data from the gateway. Now that i have retrieved it, but the data is not organised: {"timestamp":1526524809413,"data":[ {"_id":"rJeixnNtpG","data":"N11B00074","raw": [78,49,49,66,48,48,48,55,52],"timestamp":1525398515116}, {"_id":"HkzognEYpf","data":"N11E00000","raw": [78,49,49,69,48,48,48,48,48],"timestamp":1525398515479}, {"_id":"BJxXp4t6M","data":"N11A00029","raw": [78,49,49,65,48,48,48,50,57],"timestamp":1525398807747} As you can see there are three types of data: the one starts with B(N11B00074), E(N11E00000) and A(N11A00029), followed by the 5 digits which is the data i wanted to split from the string while categorised by the type(B, E and A). I have three tables in my web page and want to put the data into them based on the types: Like B being humidity table, A being temperature table and E being pH readings table. So far i only managed to list them out in a table. Is there a way that I can seperate the string and put them into an array based on their types?
You can use reduce to group objects in an array: const input={"timestamp":1526524809413,"data":[{"_id":"rJeixnNtpG","data":"N11B00074","raw":[78,49,49,66,48,48,48,55,52],"timestamp":1525398515116},{"_id":"HkzognEYpf","data":"N11E00000","raw":[78,49,49,69,48,48,48,48,48],"timestamp":1525398515479},{"_id":"BJxXp4t6M","data":"N11A00029","raw":[78,49,49,65,48,48,48,50,57],"timestamp":1525398807747}]} const arranged = input.data.reduce((accum, obj) => { const { data } = obj; const type = data[3]; const digits = data.slice(5); if (!accum[type]) accum[type] = []; accum[type].push({ ...obj, digits }); return accum; }, {}); console.log(arranged); // If you want an array and not an object: console.log(Object.values(arranged));
If you want to group the array into an object. You can use reduce. You can get the fourth character of the string by using charAt let arr = {"timestamp":1526524809413,"data":[{"_id":"rJeixnNtpG","data":"N11B00074","raw": [78,49,49,66,48,48,48,55,52],"timestamp":1525398515116}, {"_id":"HkzognEYpf","data":"N11E00000","raw": [78,49,49,69,48,48,48,48,48],"timestamp":1525398515479}, {"_id":"BJxXp4t6M","data":"N11A00029","raw":[78,49,49,65,48,48,48,50,57],"timestamp":1525398807747}]}; let result = arr.data.reduce((c, v) => { let l = v.data.charAt(3); //Get the 4th chatacter c[l] = c[l] || []; c[l].push(v); return c; }, {}); console.log( result );
Replace object keys with values in given string pattern
I have a task to replace all keys in string pattern with their values. The input is something like that: [ '{ "name": "John", "age": 13 }', "My name is #{name} and I am #{age}-years-old" ] And the output is this: 'My name is John and I am 13-years-old'. So I come up with this: function FillTemplate() { if (arguments.length < 2 || arguments.length > 7) { console.log('The input objects should be at least 1 and lesser than 7!'); } for (let i = 0; i <= arguments.length - 2; i += 1) { JSON.parse(arguments[i]); for (let j = 0; j < Object.keys(arguments[i]).length; i += 1) { let currentKey = Object.keys(arguments[i])[j]; console.log(currentKey); } } } I have a problem when i console.log(currentKey) i got only zeros but my idea is take the first object in the input then json.parse it next take all the keys in that object and with one loop take every single key separately and replace it in the pattern string with regex pattern. But this Object.keys return only zeros to me. Where is the problem?
Here you go: <script> var foo = { "name" : "John", "age" : 13 } var string = "My name is #{name} and I am #{age}-years-old"; // Extract all templates (#{name}, #{age}, ...) var matches = string.match(/#{[a-zA-Z]+?}/g); if ( matches ) { matches.forEach(function(templateStringToReplace) { // Strip the special characters to dynamically get the indices of the object templateString = templateStringToReplace.replace(/#|{|}/g, ""); string = string.replace(templateStringToReplace, foo[templateString]) }); } alert(string);
Try the other way around, parse the template string first, then loop over the keys you need so you can reference them directly in the object. Also, I have no idea what you're trying to do with the arguments object. // Our source array containing a data string and a template string var source = [ '{"name": "John", "age": 13 }', 'My name is #{name} and I am #{age}-years-old' ], // helper function to grab all the parameters from a template string parseTemplate = function parseTemplate( template ) { var regex = /#\{(.+?)\}/g, parameters = [], nextParameter; do { // this regexp will grab the next series of characters surrounded by #{} nextParameter = regex.exec(template); if (nextParameter) parameters.push(nextParameter[1]); } // as long as there are parameters left, keep searching while (nextParameter); return parameters; }, // population function, uses parseTemplate to get the parameters and then adds them to the template populateTemplate = function populate( template, data ) { var parametersToSaturate = parseTemplate(template); // for each parameter found, repalce that parameter in the string with the value from the data return parametersToSaturate.reduce(function( saturatedTemplate, parameter ) { return saturatedTemplate.replace('#{' + parameter + '}', data[parameter] || ('#{' + parameter + '}')); }, template); }, result = populateTemplate( source[1], JSON.parse(source[0]) ); console.log(result); As long as you keep the array returned from parseTemplate is the same order, you can reuse any parameter as many times in a string as you want. Any #{val} parameter not found in the data will just remain. If you have multiple objects, you can just loop over them. sources.forEach(function( source ) { console.log(populateTemplate( source[1], JSON.parse(source[0]) )); }); If your browser supports it, you can use actual JS template strings: https://developer.mozilla.org/nl/docs/Web/JavaScript/Reference/Template_literals
How to make fuzzy search using Lodash java-script library?
I want to make fuzzy search from array of objects using query string . Search may be nested in search object like the following example ; var data = [ { "id":"1", "name":"Ali", "BOD":"29/10/2055", "type":"primary", "email":"b#v.com", "mobile":"0100000000", "notes":["note1" ,"note2" ,"note3"] }, { "id":"2", "name":"Tie", "BOD":"29/10/2055", "type":"primary", "email":"b#v.net", "mobile":"0100000000", "notes":["note4" ,"note5" ,"note6"] } ]; function search(query){ // search code go here } // search examples search('.net'); //expected data[1] search('ali'); //expected data[0] search('0110'); //expected data[0],data[1]
I made simple solution it is not that optimal but it works . this done without score of fuzzy search for nested searching . var data = [ { "id":"1", "name":"Ali", "BOD":"29/10/2055", "type":"primary", "email":null, "mobile":"010100000000", "notes":["note1" ,"note2.nett" ,"note3"] }, { "id":"2", "name":"Tie", "BOD":"29/10/2055", "type":"primary", "email":"b#v.net", "mobile":"0100000000", "notes":["note4" ,"note5" ,"note6"] } ]; /** * query: query string to match with * dataArray: data array variable, array or opject to search it **/ function search(query,dataArray){ // search code go here //console.log(query); var matched = []; //init null values if(!dataArray)dataArray={}; if(!query)query=''; dataArray.forEach(function(obj,index){ for(var key in obj){ if(!obj.hasOwnProperty(key) || !obj[key]) continue; if(obj[key].toString().indexOf(query) !== -1) { matched.push(obj ); } } }); return matched ; } // search examples . console.log("found",search('.net',data).length);//expected data[0] data[1] console.log("found",search('Ali',data).length);//expected data[0] console.log("found",search('0116',data).length);//expected data[0],data[1]
Client-side full-text search on array of objects
I have the following example JavaScript array of objects and need to enable users to search on it using words/phrases, returning the objects: var items = []; var obj = { index: 1, content: "This is a sample text to search." }; items.push(obj); obj = { index: 2, content: "Here's another sample text to search." }; items.push(obj); It's probably efficient to use jQuery's $.grep to perform the search, such as this for a single word: var keyword = "Here"; var results = $.grep(items, function (e) { return e.content.indexOf(keyword) != -1; }); However, how do you search for a phrase in the content field of the objects? For example, searching for the phrase another text won't work using indexOf, because the two words aren't next to each other. What's an efficient way to perform this search in jQuery?
You can use vanilla JS if you're stuck. It does use filter and every which won't work in older browsers, but there are polyfills available. var items = []; var obj = { index: 1, content: "This is a sample text to search." }; items.push(obj); obj = { index: 2, content: "Here's another sample text to search." }; items.push(obj); function find(items, text) { text = text.split(' '); return items.filter(item => { return text.every(el => { return item.content.includes(el); }); }); } console.log(find(items, 'text')) // both objects console.log(find(items, 'another')) // object 2 console.log(find(items, 'another text')) // object 2 console.log(find(items, 'is text')) // object 1 (Edit: updated to use includes, and a slightly shorter arrow function syntax).
if you use query-js you can do this like so var words = phrase.split(' '); items.where(function(e){ return words.aggregate(function(state, w){ return state && e.content.indexOf(w) >= 0; }); },true); if it should just match at least one change the && to || and true to false