I am trying to filter out some attributes from an array in D3. The array contains the values of a csv file.
This all worked well for a small csv file doing it like this:
d3.csv("foods.csv", function(data) {
data.forEach(function(v){ delete v.name });
data.forEach(function(v){ delete v.created_at });
});
This is what the first array looks like:
But when I try to do it for a bigger csv file I get an error saying : "devtools was disconnected from the page. once page is reloaded devtools will automatically reconnect".
This is what the 2nd array looks like.
Why is this not working for the 2nd array? Is the array too big or should I try to address the values recursively because I already tried doing it like this:
function deleteCitation(v) {
if(Object.prototype.toString.call(v) === '[object Array]' ) {
v.forEach(deleteCitation);
}
else {
delete v.citation;
}
}
d3.csv("compounds_foods.csv", function(data) {
data.forEach(deleteCitation);
print(data);
});
I never loaded an CSV with 740 thousand rows. However, I believe you have some alternatives:
Use a row conversion function, or an accessor function:
d3.csv("foods.csv", deleteCitation, function(data) {
//the rest of the code
And then declare the conversion function:
function deleteCitation(d){
delete d.name;
delete d.created_at;
return d;
}
I didn't benchmarked it, maybe the conversion function takes the same time that your forEach (they do pretty much the same thing), but I believe that it's worth to check if this is quicker than calling your deleteCitation function for each object inside the data array.
The second alternative is simpler: don't remove those two properties, just leave them there and don't use them!
When you load an CSV to your data array you don't have to use all the properties in each object for your visualisation. You can simply ignore them. It's possible that you waste more processing time manipulating that huge array than simply leaving those extra two objects there.
The third alternative is the logical one: as there is absolutely no way you're gonna use 740k objects in a dataviz, consider filtering/reducing/cropping this CSV before sending it to the client side.
Related
I apologize there is going to be a lot of attachments to explain this.
I am pulling a JSON object, storing it in SQLite, then later retrieve it and trying to compare it with a newer pull to check for updates. This works perfectly in simple, one level JSON. The trouble is with nested JSON.
I console.log the datasets before comparison (the first array(4) is the new set, the second array(4) you see in this picture is the old set, previously retrieved from the Same API, stored in SQLite and then retrieved on future loading
datasets
they look identical and both are valid JSON objects according to jsonlint.com,
I was able to loop through the new JSON and "collapse" it to a single level array so I can use it later to compare. With this code I can see that the "links" array is correct
new links array
newData.forEach(m=>{
console.log('new data loop 1');
Object.entries(m as Record<string, string[]>).forEach(([key, value]) => {
console.log('new data loop 2');
if (key == 'links'){
console.log('new value');
console.log(value);
for (let i=0; i< value.length; i++){
newData2.push(value[i]);
}
// newData2.push(value);
}
})
})
However, when I tried to do the same loop for the old JSON object (stored in SQLite then retrieved), Chrome debugger thinks the "links" array has a length of zero, which from my research it could just mean it is recongzied as an object and therefore no length, never mind it worked fine for the new JSON object.
old links array
The issue:
4. I have no way to loop the old JSON object, here are three different ways to try to loop it and none of them makes into the loop.
oldData.forEach(m2=>{
console.log('old data loop 1');
Object.entries(m2 as Record<string, string[]>).forEach(([key2, value2]) => {
console.log('old data loop 2');
if (key2 == 'links'){
console.log('value');
console.log(value2);
for (var ii=0; ii< value2.length; ii++){
console.log('old data loop 2.1');
}
console.log('old data loop 2.5');
value2.forEach(m3=>{
console.log('old data loop 3');
Object.entries(m3).forEach(([key3, value3]) => {
console.log('old data loop 4');
console.log(key3 + '...' + value3);
});
})
for (var key in value2) {
console.log('old data loop 4');
if (value2.hasOwnProperty(key) ){
console.log( "key:"+key+", val:"+value2[key] );
}
}
}
})
})
The result looks something like this, so you can tell none of the attempts to loop the old "links" object works.
no loop happening
5. so there is obviously something about the old nested JSON object that is causing problems with the looping. I really appreciate any help as I have worked on this off and on for days and can't figure this out. The ultimate goal is to compare the two nested JSON objects so if there are other ways I am open to that too.
I solved this. The problem had nothing to do with how JSON objects is looped. It was an async problem. I did something after I load the array and I forgot to put that in a Promise so that even though Chrome debugger shows me the array had content, in real time it was empty!
Thanks #Serge, I did take your advice and just use string.search function now instead of trying to flatten the arrays manually then compare.
Bit of a lengthy one so those of you who like a challenge (or I'm simply not knowledgeable enough - hopefully it's an easy solution!) read on!
(skip to the actual question part to skip the explanation and what I've tried)
Problem
I have a site that has a dataset that contains an object with multiple objects inside. Each of those objects contains an array, and within that array there are multiple objects. (yes this is painful but its from an API and I need to use this dataset without changing or modifying it.) I am trying to filter the dataset based of the key-value pairs in the final object. However, I have multiple filters being executed at once.
Example of Path before looping which retrieves the key-value pair needed for one hall.
["Hamilton Hall"]["Hire Options"][2].Commercial
After Looping Path of required key-value pair for all halls, not just one (the hall identifier is stored):
[0]["Hire Options"][2].Commercial
Looping allows me to check each hall for a specific key-value pair (kind of like map or forEach, but for an object).
After getting that out of the way back to the question.
How would I go about filtering which of the looped objects are displayed?
What I have Tried
(userInput is defined elsewhere - this happens on a btn click btw)
let results = Object.keys(halls);
for (key of results) {
let weekend = [halls[ `${key}` ][ 'Hire Options' ][4][ 'Weekend function' ]];
if(userInput == weekend) {
outputAll([halls[ `${key}` ]]);
}
}
That filters it fine. However, I run into an issue here. I want to filter by multiple queries, and naturally adding an AND into the if statement doesn't work. I also dont want to have 10 if statements (I have 10+ filters of various data types I need to sort by).
I have recently heard of ternary operators, but do not know enough about them to know if that is the correct thing to do? If so, how? Also had a brief loook at switches, but doesnt seem to look like what I want (correct me if I am wrong.)
Actual Question minus the babble/explanation
Is there a way for me to dynamically modify an if statements conditions? Such as adding or removing conditions of an if statement? Such as if the filter for 'a' is set to off, remove the AND condition for 'a' in the if statement? This would mean that the results would only filter with the active filters.
Any help, comments or 'why haven't you tried this' remark are greatly appreciated!
Thanks!
Just for extra reference, here is the code for retrieving each of the objects from the first object as it loops through them:
(Looping Code)
halls = data[ 'Halls' ];
let results = Object.keys(halls);
for (key of results) {
let arr = [halls[ `${key}` ]];
outputAll(arr);
}
You can use Array.filter on the keys array - you can structure the logic for a match how you like - just make it return true if a match is found and the element needs to be displayed.
let results = Object.keys(halls);
results.filter(key => {
if (userInput == halls[key]['Hire Options'][4]['Weekend function']) {
return true;
}
if (some other condition you want to match) {
return true;
}
return false;
}).forEach(key => outputAll([halls[key]]));
This is the JSON I'm working with:
https://data.cityofnewyork.us/resource/xx67-kt59.json?$where=camis%20=%2230112340%22
I'd be dynamically making the queries using different data, so it'll possibly change.
What I'm essentially trying to do is to somehow organize the elements within this array into different arrays based on inspection_date.
So for each unique inspection_date value, those respective inspections would be put into its own collection.
If I knew the dates beforehand, I could easily iterate through each element and just push into an array.
Is there a way to dynamically create the arrays?
My end goal is to be able to display each group of inspections (based on inspection date) using Angular 5 on a webpage. I already have the site up and working and all of the requests being made.
So, I'm trying to eventually get to something like this. But of course, using whatever dates in the response from the request.
2016-10-03T00:00:00
List the inspections
2016-04-30T00:00:00
List the inspections
2016-04-12T00:00:00
List the inspections
Just for reference, here's the code I'm using:
ngOnInit() {
this.route.params.subscribe(params => {
this.title = +params['camis']; // (+) converts string 'id' to a number
this.q.getInpectionsPerCamis(this.title).subscribe((res) => {
this.inspectionList = res;
console.log(res);
});
// In a real app: dispatch action to load the details here.
});
}
I wish I could give you more info, but at this point, I'm just trying to get started.
I wrote this in jQuery just because it was faster for me, but it should translate fairly well to Angular (I just don't want to fiddle with an angular app right now)
Let me know if you have any questions.
$(function() {
let byDateObj = {};
$.ajax({
url: 'https://data.cityofnewyork.us/resource/xx67-kt59.json?$where=camis%20=%2230112340%22'
}).then(function(data) {
//probably do a check to make sure the data is an array, im gonna skip that
byDateObj = data.reduce(function(cum, cur) {
if (!cum.hasOwnProperty(cur.inspection_date)) cum[cur.inspection_date] = [];
//if the cumulative array doesn't have the inspection property already, add it as an empty array
cum[cur.inspection_date].push(cur);
//push to inspection_date array.
return cum;
//return cumulatie object
}, byDateObj);
//start with an empty object by default;
console.log(byDateObj);
}, console.error);
});
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
I have been trying to implement a basic Keras model generated in Python into a website using the Keras.js library. Now, I have the model trained and exported into the model.json, model_weights.buf, and model_metadata.json files. Now, I essentially copied and pasted test code from the github page to see if the model would load in browser, but unfortunately I am getting errors. Here is the test code. (EDIT: I fixed some errors, see below for remaining ones.)
var model = new KerasJS.Model({
filepaths: {
model: 'dist/model.json',
weights: 'dist/model_weights.buf',
metadata: 'dist/model_metadata.json'
},
gpu: true
});
model.ready()
.then(function() {
console.log("1");
// input data object keyed by names of the input layers
// or `input` for Sequential models
// values are the flattened Float32Array data
// (input tensor shapes are specified in the model config)
var inputData = {
'input_1': new Float32Array(data)
};
console.log("2 " + inputData);
// make predictions
return model.predict(inputData);
})
.then(function(outputData) {
// outputData is an object keyed by names of the output layers
// or `output` for Sequential models
// e.g.,
// outputData['fc1000']
console.log("3 " + outputData);
})
.catch(function(err) {
console.log(err);
// handle error
});
EDIT: So I changed my program around a little to be compatible with JS 5 (that was a stupid mistake on my part), and now I have encountered a different error. This error is caught and then is logged. The error I get is: Error: predict() must take an object where the keys are the named inputs of the model: input. I believe this problem arises because my data variable is not in the correct format. I thought that if my model took in a 28x28 array of numbers, then data should also be a 28x28 array so that it could correctly "predict" the right output. However, I believe I am missing something and that is why the error is being thrown. This question is very similar to mine, however it is in python and not JS. Again, any help would be appreciated.
Ok, so I figured out why this was happening. There were two problems. First, the data array needs to be flattened, so i wrote a quick function to take the 2D input and "flatten" it to be a 1D array of length 784. Then, because I used a Sequential model, the key name of the data should not have been 'input_1', but rather just 'input'. This got rid of all the errors.
Now, to get the output information, we simply can store it in an array like this: var out = outputData['output']. Because I used the MNIST data set, out was a 1D array of length 10 that contained probabilities of each digit being the user-written digit. From there, you can simply find the number with the highest probability and use that as a prediciton for the model.
I am attempting to load some json data into a custom JQuery function. There is probably an easy solution to this by I am admittedly learning as I go. Please advise if I am just going about it the wrong way.
As of now I am just appending it into the body to ensure that I am actually finding the data properly. I was trying to use append() to add it into the function as well but was not getting any where.
$.getJSON("designs/new.json",function(result){
$.each(result, function(i, field){
$("body").append(field['hex'] + " ");
});
});
Essentially what I'd like to do is replace the hex data values for this colorPicker function with the hex codes from my json data above. I've been messing with it for a while and can't seem to find the right approach.
$.fn.colorPicker.defaults.colors = ['000', '000', 'fff', 'fff'];
I appreciate the help in advance!
Try using alert(JSON.stringify(result)) to see if your json data is loaded properly. This code should work:
$.getJSON("t.json", function (result) {
//alert(JSON.stringify(result));
$.fn.colorPicker.defaults.colors = result.colors;
});
where t.json has the following content:
{
"colors": ['00a', '00b', 'ffa', 'ffb']
}
If I understand correctly:
1) Your JSON returns an array of objects (presumably four?) that each have a "hex" property.
2) You want to set the color picker's default colors based on these four hex properties.
So, assuming that my assumptions are correct and your array is properly ordered, you could do something like this:
$.getJSON("designs/new.json",function(result){
$.fn.colorPicker.defaults.colors = $.map(result, function(i) {
return i.hex;
});
});
I've mapped the hex properties to an array and assigned it directly to the colors. Again, this is assuming you have an array of four objects in your result object, each with a hex property.