I apologize there is going to be a lot of attachments to explain this.
I am pulling a JSON object, storing it in SQLite, then later retrieve it and trying to compare it with a newer pull to check for updates. This works perfectly in simple, one level JSON. The trouble is with nested JSON.
I console.log the datasets before comparison (the first array(4) is the new set, the second array(4) you see in this picture is the old set, previously retrieved from the Same API, stored in SQLite and then retrieved on future loading
datasets
they look identical and both are valid JSON objects according to jsonlint.com,
I was able to loop through the new JSON and "collapse" it to a single level array so I can use it later to compare. With this code I can see that the "links" array is correct
new links array
newData.forEach(m=>{
console.log('new data loop 1');
Object.entries(m as Record<string, string[]>).forEach(([key, value]) => {
console.log('new data loop 2');
if (key == 'links'){
console.log('new value');
console.log(value);
for (let i=0; i< value.length; i++){
newData2.push(value[i]);
}
// newData2.push(value);
}
})
})
However, when I tried to do the same loop for the old JSON object (stored in SQLite then retrieved), Chrome debugger thinks the "links" array has a length of zero, which from my research it could just mean it is recongzied as an object and therefore no length, never mind it worked fine for the new JSON object.
old links array
The issue:
4. I have no way to loop the old JSON object, here are three different ways to try to loop it and none of them makes into the loop.
oldData.forEach(m2=>{
console.log('old data loop 1');
Object.entries(m2 as Record<string, string[]>).forEach(([key2, value2]) => {
console.log('old data loop 2');
if (key2 == 'links'){
console.log('value');
console.log(value2);
for (var ii=0; ii< value2.length; ii++){
console.log('old data loop 2.1');
}
console.log('old data loop 2.5');
value2.forEach(m3=>{
console.log('old data loop 3');
Object.entries(m3).forEach(([key3, value3]) => {
console.log('old data loop 4');
console.log(key3 + '...' + value3);
});
})
for (var key in value2) {
console.log('old data loop 4');
if (value2.hasOwnProperty(key) ){
console.log( "key:"+key+", val:"+value2[key] );
}
}
}
})
})
The result looks something like this, so you can tell none of the attempts to loop the old "links" object works.
no loop happening
5. so there is obviously something about the old nested JSON object that is causing problems with the looping. I really appreciate any help as I have worked on this off and on for days and can't figure this out. The ultimate goal is to compare the two nested JSON objects so if there are other ways I am open to that too.
I solved this. The problem had nothing to do with how JSON objects is looped. It was an async problem. I did something after I load the array and I forgot to put that in a Promise so that even though Chrome debugger shows me the array had content, in real time it was empty!
Thanks #Serge, I did take your advice and just use string.search function now instead of trying to flatten the arrays manually then compare.
Related
I currently save a bunch of objects (thousands) into the chrome.storage.local and then when on a specific web page checking whether specific IDs on the web page are in fact saved in local storage.
Here's a pseudo code
Bakcground script:
var storage = chrome.storage.local;
var json = '[{"kek1": {"aaa": "aaaValue", "bbb": "bbbValue", "ccc": "cccValue"}},{"kek2": {"ddd": "dddValue", "eee": "eeeValue", "fff": "fffValue"}}]';
var jsonParsed = JSON.parse(json);
jsonParsed.forEach(function(object) {
storage.set(object);
});
Content script (when on a specific page):
ids.forEach(function(id) {
storage.get(id, function(result){
if(!isEmpty(result)) {
//we found it, nice, now flag it as found
}
});
});
function isEmpty(obj) {
for(var key in obj) {
if(obj.hasOwnProperty(key))
return false;
}
return true;
}
Which is easy and nice since I only have to do storage.get(id, ...
Unfortunately, I save a lot of stuff in storage, some of it I need to be removing periodically, which then becomes a hustle since I have to loop through all the objects and determining whether that particular object needs to be removed or it needs to remain.
So i decided I would do like these "parent object". Ie one object for settings, containing an array of objects with different settings the user would save. One object for the stuff that needs to be removed, containing an array objects. Etc
Like so - all relevant info that I want to remove periodically will be under one key "test" (temp name):
var json = '{"test":[{"kek1": {"aaa": "aaaValue", "bbb": "bbbValue", "ccc": "cccValue"}},{"kek2": {"ddd": "dddValue", "eee": "eeeValue", "fff": "fffValue"}}]}';
I know how to access the nested objects and their values:
var jsonParsed = JSON.parse(json);
jsonParsed.test[0].kek1.aaa
But I don't know how I would easily check for the keys saved in the storage since I would have to specify the "element number" ([i]).
Do I just do a for loop itterating over the array like so?
for (i = 0; i < jsonParsed.test.length; i++) {
var getKey = Object.keys(jsonParsed.test[i]);
if (getKey[0] == 'theKeyImLookingFor') {
//do stuff
}
}
To me that feels like non ideal solution since the for loop would have to run for each of the ids on the page and there could sometimes be close to 4000 of them. (4000 for loops back to back)
Is it a good idea to save a single object holding an array of thousands of other objects?
Am I doing it wrong or is this the way to go?
But I don't know how I would easily check for the keys saved in the storage
Use the standard Array methods like find or findIndex:
const i = arrayOfObjects.findIndex(o => 'someKey' in o);
Is it a good idea to save a single object holding an array of thousands of other objects?
It's a bad idea performance-wise.
What you probably need here is an additional value in the storage that would contain an array with ids of other values in the storage that need to be processed in some fashion e.g. expired/removed. It's basically like a database index so you would update it every time when writing an individual object. Since it contains only the ids, updating it is cheaper than rewriting the entire data.
Also, instead of performing lots of calls to the API, do just a single call:
// writing
chrome.storage.local.set(Object.assign({}, ...arrayOfObjects));
// reading
chrome.storage.local.get(arrayOfIds, data => {
for (const id of arrayOfIds) {
const value = data[id];
if (value !== undefined) {
// ok
}
}
});
I'm trying to write a function to append an array with a for loop. The data I'm trying to append is in a JSON repsonse. However, I keep getting x10 Array[] in the web console instead of one array with all the data. When I run console.log(dates[0]) I get returned in the web console "undefined". This tells me the data isn't even making it into the array. When I run console.log(time) I return x10 peices of data from the JSON I want but of course its not in the array. Any ideas? Thanks.
function mostRecent(time) {
var dates=[];
for (var i = 0; i < time.length; i++) {
dates.push(dates[i]);
}
return console.log(dates);
}
You are pushing dates[i] with every loop cycle. And since dates array keeps being empty, you are actually pushing undefined.
Just replace dates.push(dates[i]) with dates.push(time[i]).
Note: You should return dates instead of console.log.
This is the JSON I'm working with:
https://data.cityofnewyork.us/resource/xx67-kt59.json?$where=camis%20=%2230112340%22
I'd be dynamically making the queries using different data, so it'll possibly change.
What I'm essentially trying to do is to somehow organize the elements within this array into different arrays based on inspection_date.
So for each unique inspection_date value, those respective inspections would be put into its own collection.
If I knew the dates beforehand, I could easily iterate through each element and just push into an array.
Is there a way to dynamically create the arrays?
My end goal is to be able to display each group of inspections (based on inspection date) using Angular 5 on a webpage. I already have the site up and working and all of the requests being made.
So, I'm trying to eventually get to something like this. But of course, using whatever dates in the response from the request.
2016-10-03T00:00:00
List the inspections
2016-04-30T00:00:00
List the inspections
2016-04-12T00:00:00
List the inspections
Just for reference, here's the code I'm using:
ngOnInit() {
this.route.params.subscribe(params => {
this.title = +params['camis']; // (+) converts string 'id' to a number
this.q.getInpectionsPerCamis(this.title).subscribe((res) => {
this.inspectionList = res;
console.log(res);
});
// In a real app: dispatch action to load the details here.
});
}
I wish I could give you more info, but at this point, I'm just trying to get started.
I wrote this in jQuery just because it was faster for me, but it should translate fairly well to Angular (I just don't want to fiddle with an angular app right now)
Let me know if you have any questions.
$(function() {
let byDateObj = {};
$.ajax({
url: 'https://data.cityofnewyork.us/resource/xx67-kt59.json?$where=camis%20=%2230112340%22'
}).then(function(data) {
//probably do a check to make sure the data is an array, im gonna skip that
byDateObj = data.reduce(function(cum, cur) {
if (!cum.hasOwnProperty(cur.inspection_date)) cum[cur.inspection_date] = [];
//if the cumulative array doesn't have the inspection property already, add it as an empty array
cum[cur.inspection_date].push(cur);
//push to inspection_date array.
return cum;
//return cumulatie object
}, byDateObj);
//start with an empty object by default;
console.log(byDateObj);
}, console.error);
});
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
I am trying to filter out some attributes from an array in D3. The array contains the values of a csv file.
This all worked well for a small csv file doing it like this:
d3.csv("foods.csv", function(data) {
data.forEach(function(v){ delete v.name });
data.forEach(function(v){ delete v.created_at });
});
This is what the first array looks like:
But when I try to do it for a bigger csv file I get an error saying : "devtools was disconnected from the page. once page is reloaded devtools will automatically reconnect".
This is what the 2nd array looks like.
Why is this not working for the 2nd array? Is the array too big or should I try to address the values recursively because I already tried doing it like this:
function deleteCitation(v) {
if(Object.prototype.toString.call(v) === '[object Array]' ) {
v.forEach(deleteCitation);
}
else {
delete v.citation;
}
}
d3.csv("compounds_foods.csv", function(data) {
data.forEach(deleteCitation);
print(data);
});
I never loaded an CSV with 740 thousand rows. However, I believe you have some alternatives:
Use a row conversion function, or an accessor function:
d3.csv("foods.csv", deleteCitation, function(data) {
//the rest of the code
And then declare the conversion function:
function deleteCitation(d){
delete d.name;
delete d.created_at;
return d;
}
I didn't benchmarked it, maybe the conversion function takes the same time that your forEach (they do pretty much the same thing), but I believe that it's worth to check if this is quicker than calling your deleteCitation function for each object inside the data array.
The second alternative is simpler: don't remove those two properties, just leave them there and don't use them!
When you load an CSV to your data array you don't have to use all the properties in each object for your visualisation. You can simply ignore them. It's possible that you waste more processing time manipulating that huge array than simply leaving those extra two objects there.
The third alternative is the logical one: as there is absolutely no way you're gonna use 740k objects in a dataviz, consider filtering/reducing/cropping this CSV before sending it to the client side.
I'm passing an associative array (id => val) using Ajax and receiving it with jQuery's $.getJSON which read the data properly and prepared the object. There is, however, very annoying sorting issue.
It appears that on Chrome and IE the data becomes sorted by the id part of the associate array. So if the array should be (5=> 'xxx', 3 => 'fff') it actually becomes (3 => 'fff',5=> 'xxx'). On FireFox it works as expected, i.e. not sorted.
Any ideas?
You can add a leading 0 for all integer indexes.
var json = { '05' => 'xxx', '03' => 'fff' };
Seems the best way is to avoid associative arrays at all. When you want to send an associate array simply send it as two separate arrays - one of keys and one of values. Here's the PHP code to do that:
$arWrapper = array();
$arWrapper['k'] = array_keys($arChoices);
$arWrapper['v'] = array_values($arChoices);
$json = json_encode($arWrapper);
and the simple JavaScript code to do whatever you'd like with it
for (i=0; i < data['k'].length; i++) {
console.log('key:' + data['k'][i] + ' val:' + data['v'][i]);
}
Another option is to return the data as an array of objects. That will ensure that the objects stay in the order that you return them.
Edit:
Basically, for each key > value pair, push it to a new array and json_encode that array.