How to work with big json file (in vue project) - javascript

What I'm building
There is this game Path of Exile, that has a passive tree that i'm recreating in my Vue application with some help from Pixi Js. The game developers release a json with the data for this passive tree every expansion for people to work with.
The Tree: https://www.pathofexile.com/passive-skill-tree
The json: shorturl.at/gBSV5 (Don't forget to press the "View raw" link)
My problem
I got the tree component working, but since the json is quite large the app.js is also very large when i build for production. This is not that great for the performance and i'm looking to improve this.
Vue Project
My tree component looks something like shown below. So i'm using the json file to do various things so i can draw the tree on my canvas correctly.
import json from "#/data/data.json"
import * as PIXI from 'pixi.js'
export default {
name: "Tree",
data() {
return {
treeData: {},
};
},
mounted() {
this.treeData = json
this.drawTree()
},
methods: {
drawTree () {
// Calculate size of the tree
// Make some containers
// Loop all nodes on the tree and draw them
// Calculate connections between nodes and draw them
// Some more stuff...
}
}
}
The solution?
I'm accessing the data from the json all the time in my code, so i don't really know where i can win performance if in the end i need all the data and this all will be in my app.js. So how should i go about working with this data? Is it maybe better to store it in a (mongoDB) database or would that result in the same problem?

Related

Where to store remote data in Svelte for responsive data vizualisations with Svelte and D3 (best practices)

Disclaimer: I am really not well experienced with neither svelte, nor D3, nor general JavaScript patterns. But I really like it, so I really want to learn it and already invested quite some time. Still, this feels like a super basic question that annoys me a lot. I hope it is not too confusing and someone might has an idea.
It is basically about how to setup a simple graph (let it be a bar chart) in an efficient, reproducible and "best-practice" way. I guess my main concern is on how to pass around the data and use it for different tasks. E.g. I think it might be a good idea to separate out the construction of the scales (using d3) in a separate component. However, this component needs access to the data (and probably also access to the, in the best case resposive width, of the chart-container).
However, also the bars, which are in another component, need access to the data in order to know how do draw the rectangles.
A general misunderstanding (i guess that is the right word) I have with JavaScript is that I do not understand how to fetch data asynchronously (using e.g. the browsers fetchor D3's csvmethod). I simply can not fetch the data and then pass it as prop to another component. Because what I would be passing would be a promise...
So I have this very basic REPL that kind of shows a bit this know I have in my head: https://svelte.dev/repl/398f4c21b7a9409a9811fd8e38703a36?version=3.44.1
It looks like this. In the App.html I fetch the data that I want to use for multiple purposes. However I cannot "get it out" of that component.
<script>
import Chart from "./Chart.svelte"
const url = "https://api.github.com/search/repositories?q=stars:>100000";
async function getData(){
let response = await fetch(url)
let data = await response.json()
console.log(data)
}
//async function getDataLocal(){
// let data = await d3.csv(<path_to_data>)
// return await data
// }
let data = await getData()
</script>
<Chart {data}>Do Something with the data. Make the chart, build the scales, ....</Chart>
So the main questions are:
Are there any ressources on how to learn building sustainable graphics with remote data, svelte and a bit of D3. I already watched many many youtube videos and I guess I will rewatch the one from Matthias Stahl;)
Is it a good idea to use stores in such a case to store the data
And a little more specific: As the data is (probably) fixed, however the dimension arent't: What is a good way/place to let the app know to recalculate the scales etc.
There are 3 separate concerns here:
fetching, storing and retrieving data (aka the data source layer)
manipulating/transforming data (aka the business logic layer)
displaying data (aka the presentation layer)
I will leave the last part aside as it solely concerns D3 (if that is your visualization library of choice) and there are plenty of resources available online on this topic, and I will instead focus on what seems to be the heart of your question, i.e. how to fetch data in Svelte, where to store it, how to pass it around to components, and how to manipulate the data.
1. Asynchronous queries in Svelte
Your first inquiry is about how to deal with asynchronous requests. You cannot use await statements at the root level of the <script> section of a Svelte file, meaning the following reactive statement would generate an error:
// will fail
$: data = await getData(url)
However, you can call an asynchronous function that will handle the assignment. Reactivity will still work and your component will re-render when the url is changed and the new data retrieved:
// will work
$: updateData(url)
async function updateData(url) {
data = await getData(url)
}
Here is a working example based on the REPL in your question
2. Using stores
As you could see from the above example, you had to pass the data to your <Header> and <Chart> components for it to be used in either:
<Header {data}>GitHub Lookup</Header>
<Chart {data}/>
But what if you want to use your Chart somewhere else in your application? What if you have another component that wants to make use of the same data?
Obviously you do not want to fetch the same data over & over (unless the request itself has changed). You also want to avoid passing the data around as a prop everywhere in your app. You will want to make the data available only to these components that will use it.
This is where stores come in handy. Stores can be subscribed to by any component. A writable store will allow its contents to be updated, while a readable store will be -as the name implies- read-only.
A store need not be complex. The following is a very basic writable store:
import { writable } from 'svelte/store'
export const githubStore = writable(null) // initialized with a null value
All you have to do then is interact with your store.
Updating the store in your App component:
import { githubStore as data } from './githubStore.js' // import the store we defined above under the name 'data'
.
.
.
async function updateData(url) {
$data = await getData(url) // using the $ shorthand to access the store (assigning a new value will update the store content)
}
Using (i.e. subscribing to) the store in your components:
import { githubStore as data } from './githubStore.js' // import the store we defined above under the name 'data'
.
.
.
// using the $ shorthand to access the store
{#each $data.items as item (item.id)}
<li><a href={item.html_url}>{item.full_name}</a> [{item.stargazers_count}⭐]</li>
{/each}
Read here for details on using the $ reactive syntax with stores
Now that your child components are subscribing to the store where you stored your data, you do not need to pass that data as a prop any more:
<Header>GitHub Lookup</Header>
<Chart />
Here is an updated version of the REPL above, using stores
3. Further considerations
When you want to start manipulating or transforming data that has been put into a store, derived stores come in handy. When the data in your original store is updated, the derived store will automatically update itself based on the changes to the original.
You can also build on the provided readable/writable stores by adding your own functionality and custom methods. These are slightly more advanced topics but would come in handy where data manipulation is concerned.
Finally, D3 will provide its own data manipulation methods, so it will be up to you to decide how much manipulation you handle directly in Svelte, and how much you delegate to D3. I would probably leave everything connected to visualization (scaling, zooming, etc.) on the D3 side, and have the actual pre-visualization manipulation of data (i.e. the business logic) on the Svelte side (or better yet, directly on the back-end if you have access to that!).

Organizing a big helper file full of exports

Apologies if this question is too open-ended.
I have a big "helper" file filled with useful functions. They're all exported functions, so
exports.getFirstNameLastName = nameString => { ... }
This file is getting to be pretty big and has enough functions in it that I feel that it can be divided up into smaller, categorized files (e.g. parsingHelper.js, webHelper.js, etc).
I'm coming from a heavily object oriented software dev background. In something like C# I'd create a static class file for each of these categories (parsing, web, etc), then simply do the one import (using ParserHelpers;), then do ParserHelpers.GetFirstNameLastName(...); instead of importing each and every function I end up using.
My question is, is there a way to organize all my helper functions in a similar manner? I'm trying to reduce the number of individually exported/imported items and trying to split this big file into smaller files.
Would rather not use additional packages if I don't have to (using ES6).
Yes! There is a way to do something like that!
// file users.js
exports.users = {
getFirstName: () => {},
getLastName: () => {},
// Add as many as you want
}
// file categories.js
exports.categories = {
getCategoryById: () => {},
getCategoryName: () => {}
}
// You can use them separately or you can create another file an union all them:
// file helpers.js
import users from './users'
import categories from './categories'
exports helpers = {
users,
categories
}
// This way you can do something like:
import helpers from './helpers.js'
helper.users.getFirstName()

Map JSON Object To Something Worthwhile in React

I was wondering if I could get a little advice on pulling in some JSON data in my React app. I have a good amount of experience in front end development, but unfortunately have minimal knowledge when it comes to pulling in data from remote resources.
Here is my sample json data
{
"hello": {
"yo" : "random",
"hey" : "cool"
}
}
This data sits locally and I can pull it in just fine, but I am wondering if there is an easier way to get the data into the format I want. My code is below of what I am doing,
grabData() {
axios.get('data.json')
.then(function(response) {
var info = response.data.hello;
info = Object.values(info);
this.setState({ info : info })
});
}
For those unfamiliar, I am using an ES6 feature to turn the object into an array so I can use the map method in a child component that will receive this array.
However, I think this method is flawed, as it completely strips the object of its keys, which I would like to keep intact to use in the map method. Any help would be greatly appreciated.
PS. Is it really necessary for me to use "response.data.hello" to access the object?

Redux data structuring

I'm trying to build a complex fully-dynamic app with Redux. I mean my App has lots of dynamic-generated forms with generated fields-components on-the-fly. I want to store in my Redux-store visual data about my components too. But how should i do it without mixing real data with visual component data?
For example if i have structure like this
Store {
visual: {...deeply nested visual-data-tree...},
data: {...deeply-nested real-data-tree...}
}
It is hard to render component because i need to search visual data first, then react component "value" in two trees.
But if have a structure similar to this:
Store {
form {
visual: {...form visual data...},
data: {
//Ok here the form "data" - widgets. Or it must to be visual? :)
widget1 {
visual: {type:"ComboBox", opened: true},
data: 1
}
}
}
}
You see the problem, now i have visual data inside real data of Form widget.
(form - data - widget1 - visual)
Visual data inside the real data is out of the concept.
How do you guys solve same problems of mixing data?
Really sorry for my poor english. I hope i clearly explained the problem.
Isn't the distinction superficial? I think a more important rule is that the data in the state should be normalized. For example, if you have Combobox widget letting you choose users, your data shape better be
{
chosenUserId: 10, // Good!
users: {
10: { name: 'Alice' }
}
rather than
{
chosenUser: { name: 'Alice' }, // Bad!
users: {
10: { name: 'Alice' }
}
If the data is duplicated in the state tree, it's hard to update it correctly and avoid inconsistencies.
As long as you keep the data normalized, I see no real need to divide visual and data. You might want to have top-level entity cache that looks like a database (e.g. entities which includes users, posts, or whatever data objects your app uses), but other than that, go with whatever state shape feels most comfortable when retrieving the relevant state.

Making RESTful API call from React.js

I am doing a POC for isomorphic JavaScript application to render HTML from the server side. The POC is working with simple HTML, but I want to make an API call and get the JSON response and send to the render function. I tried various ways but it is not working.
What am I missing? I am very new to React.js.
loadCategoriesFromServer: function() {
var self = this;
// get walking directions from central park to the empire state building
var http = require("http");
url = "api url here";
var request = http.get(url, function (response) {
// data is streamed in chunks from the server
// so we have to handle the "data" event
var buffer = "",
data,
route;
response.on("data", function (chunk) {
buffer += chunk;
});
response.on("end", function (err) {
data = JSON.parse(buffer);
//console.log(data.d);
//console.log(data.d.Items);
self.setState({
categories: data.d.Items
});
});
});
}, // load from server end
getInitialState: function() {
return { categories: [] };
},
componentWillMount: function() {
console.log("calling load categories")
this.loadCategoriesFromServer();
},
render: function () {
//console.log("data");
//console.log(this.state.categories);
var postNodes = this.state.categories.map(function (cat) {
console.log(cat);
});
return (
<div id="table-area">
//i want to paint the data here..
</div>
)
}
});
Fetching inside of component using componentWillMount is not a right place, in case when you need to render server side. You need to somehow move it out form component, and pass actual data as props after it is fetched - for example as #JakeSendar suggested in his answer.
I have some experience doing isomorphic app with React, and the main problem I faced is how to wait until all data would be loaded before first render
As #FakeRainBrigand already mentioned in comments, there is not only one way to do this, and it depends from your requirements.
There is few ways to do build an isomorphic app, the some interesting from my perspective is: https://github.com/webpack/react-starter and http://fluxible.io/
But, the most elegant way to do this, as I figured out for myself - is to organise asynchronous rendering for react components, in particular using RxJS.
In general my application is structured as following:
views - React components without any logic (just a view)
models - Observables with current state (initial data is loaded using superagent, then combined with other models and/or actions results).
In simple case it is something like:
Rx.Observable.defer(fetchData).concat(updatesSubject).shareReplay()
actions(or intents) - Observers used to collects user input, do something, and dispatch action results to subscribers models and/or other actions. In simple case something like:
updatesSubject = new Rx.Subject();
action = new Rx.Subject();
action.switchMap(asyncRequest).subscribe(updatesSubject)
components - Observables(stream of virtual DOM elements) combined from models, other components and actions (I have a note about this, explaining how and why to create Observable React elements with RxJS), also now I am planning to add partial components (tuple from: react component, observables, observers, and properties. partially filled with using DI)
router - component responsible to handling location changes,
in general main feature is to map location changes to stream of virtual DOM elements and meta information. But in details, it is bit more complicated in my case(url generation, active url highlighting, handling scrolls when navigating, also it has possibility of nested routes and multiple views)
All this is assembled together using DI container, in my case similar to angular2 DI container, but a lot simplified for my specific needs.
Components, models and actions are created using DI.
On server side application is like this:
var rootInjector = new Injector();
// setup server specific providers
rootInjector.provide(..., ...)
app.get('/*', function(req,res){
var injector = rootInjector.createChild();
// setup request specific providers
injector.provide(..., ...);
injector.get(Router)
.first()
.subscribe(function(routingResult){
res.render('app', {
title: routingResult.title,
content: React.renderToString(routingResult.content)
});
});
}
and similar on client side:
var rootInjector = new Injector();
// setup server specific providers
// actually this is omitted in my case because default providers are client side
rootInjector.provide(..., ...)
contentElement = document.getElementById('#content');
rootInjector.get(Router)
.subscribe(function(routingResult){
document.title = routingResult.title;
React.render(routingResult.content, contentElement)
});
In comparison to flux, it is more declarative and more powerful way to organise app. And in case of isomorphic app - for me, it looks much better that various hacks with flux. But of course there is drawbacks... - it is more complicated.
Likely later, I will opensource all this, but for now - it is not quite ready to be published.
UPD1:
Original answer is a bit outdated(later I plan to update it), and I have some progress in this area.
Links to code mentioned above, already opensourced:
DI container: di1
Container for react componentns(connecting view to observables and obsrvers): rx-react-container
Starter template, for implementing isomorphic widgets, using RxJS and React, and libraries above: Reactive Widgets
About complete application(work still in progress, and documentation there is not quite good, but in general it should be clear):
Router built especially for isomophic reactive applications router1 and react components to use it router1-react
Application template with router and all libraries mentioned above: router1-app-template
React's renderToString method (for rendering components on the server) is synchronous. Therefore, any sort of async task, such as your api request, will still be pending by the time the component has rendered.
There are a couple of ways you can go about fixing this, depending on whether or not you want to fetch your data on the server or client.
If you choose to fetch the data on the server, first move your api-request logic outside of your component. Then, render your component in the callback, passing the fetched-data as a prop. It would look something like this:
response.on("end", function (err) {
var data = JSON.parse(buffer);
var markup = React.renderToString(Component({categories: data}));
});
Inside your component, you'd be able to access the data via this.props.categories.
The other option is to handle the api request on the client. You would make an AJAX request in componentDidMount, and set the component's state from the fetched data. It would look very similar to what you have now, the key difference being that your request logic would live in componentDidMount (async, called on the client) rather than componentWillMount (not async, called on the server).
You should use superagent, works really good for me, also you are missing the most important part, you should use flux to fetch data from a server, flux is the way that facebook strongly recommended, it's pretty easy to use flux architecture.

Categories