We Keep Coding

Where to store remote data in Svelte for responsive data vizualisations with Svelte and D3 (best practices)

Disclaimer: I am really not well experienced with neither svelte, nor D3, nor general JavaScript patterns. But I really like it, so I really want to learn it and already invested quite some time. Still, this feels like a super basic question that annoys me a lot. I hope it is not too confusing and someone might has an idea.
It is basically about how to setup a simple graph (let it be a bar chart) in an efficient, reproducible and "best-practice" way. I guess my main concern is on how to pass around the data and use it for different tasks. E.g. I think it might be a good idea to separate out the construction of the scales (using d3) in a separate component. However, this component needs access to the data (and probably also access to the, in the best case resposive width, of the chart-container).
However, also the bars, which are in another component, need access to the data in order to know how do draw the rectangles.
A general misunderstanding (i guess that is the right word) I have with JavaScript is that I do not understand how to fetch data asynchronously (using e.g. the browsers fetchor D3's csvmethod). I simply can not fetch the data and then pass it as prop to another component. Because what I would be passing would be a promise...
So I have this very basic REPL that kind of shows a bit this know I have in my head: https://svelte.dev/repl/398f4c21b7a9409a9811fd8e38703a36?version=3.44.1
It looks like this. In the App.html I fetch the data that I want to use for multiple purposes. However I cannot "get it out" of that component.
<script>
import Chart from "./Chart.svelte"
const url = "https://api.github.com/search/repositories?q=stars:>100000";
async function getData(){
let response = await fetch(url)
let data = await response.json()
console.log(data)
}
//async function getDataLocal(){
// let data = await d3.csv(<path_to_data>)
// return await data
// }
let data = await getData()
</script>
<Chart {data}>Do Something with the data. Make the chart, build the scales, ....</Chart>
So the main questions are:
Are there any ressources on how to learn building sustainable graphics with remote data, svelte and a bit of D3. I already watched many many youtube videos and I guess I will rewatch the one from Matthias Stahl;)
Is it a good idea to use stores in such a case to store the data
And a little more specific: As the data is (probably) fixed, however the dimension arent't: What is a good way/place to let the app know to recalculate the scales etc.

There are 3 separate concerns here:
fetching, storing and retrieving data (aka the data source layer)
manipulating/transforming data (aka the business logic layer)
displaying data (aka the presentation layer)
I will leave the last part aside as it solely concerns D3 (if that is your visualization library of choice) and there are plenty of resources available online on this topic, and I will instead focus on what seems to be the heart of your question, i.e. how to fetch data in Svelte, where to store it, how to pass it around to components, and how to manipulate the data.
1. Asynchronous queries in Svelte
Your first inquiry is about how to deal with asynchronous requests. You cannot use await statements at the root level of the <script> section of a Svelte file, meaning the following reactive statement would generate an error:
// will fail
$: data = await getData(url)
However, you can call an asynchronous function that will handle the assignment. Reactivity will still work and your component will re-render when the url is changed and the new data retrieved:
// will work
$: updateData(url)
async function updateData(url) {
data = await getData(url)
}
Here is a working example based on the REPL in your question
2. Using stores
As you could see from the above example, you had to pass the data to your <Header> and <Chart> components for it to be used in either:
<Header {data}>GitHub Lookup</Header>
<Chart {data}/>
But what if you want to use your Chart somewhere else in your application? What if you have another component that wants to make use of the same data?
Obviously you do not want to fetch the same data over & over (unless the request itself has changed). You also want to avoid passing the data around as a prop everywhere in your app. You will want to make the data available only to these components that will use it.
This is where stores come in handy. Stores can be subscribed to by any component. A writable store will allow its contents to be updated, while a readable store will be -as the name implies- read-only.
A store need not be complex. The following is a very basic writable store:
import { writable } from 'svelte/store'
export const githubStore = writable(null) // initialized with a null value
All you have to do then is interact with your store.
Updating the store in your App component:
import { githubStore as data } from './githubStore.js' // import the store we defined above under the name 'data'
.
.
.
async function updateData(url) {
$data = await getData(url) // using the $ shorthand to access the store (assigning a new value will update the store content)
}
Using (i.e. subscribing to) the store in your components:
import { githubStore as data } from './githubStore.js' // import the store we defined above under the name 'data'
.
.
.
// using the $ shorthand to access the store
{#each $data.items as item (item.id)}
<li><a href={item.html_url}>{item.full_name}</a> [{item.stargazers_count}⭐]</li>
{/each}
Read here for details on using the $ reactive syntax with stores
Now that your child components are subscribing to the store where you stored your data, you do not need to pass that data as a prop any more:
<Header>GitHub Lookup</Header>
<Chart />
Here is an updated version of the REPL above, using stores
3. Further considerations
When you want to start manipulating or transforming data that has been put into a store, derived stores come in handy. When the data in your original store is updated, the derived store will automatically update itself based on the changes to the original.
You can also build on the provided readable/writable stores by adding your own functionality and custom methods. These are slightly more advanced topics but would come in handy where data manipulation is concerned.
Finally, D3 will provide its own data manipulation methods, so it will be up to you to decide how much manipulation you handle directly in Svelte, and how much you delegate to D3. I would probably leave everything connected to visualization (scaling, zooming, etc.) on the D3 side, and have the actual pre-visualization manipulation of data (i.e. the business logic) on the Svelte side (or better yet, directly on the back-end if you have access to that!).

How to improve data handling with meteor

Still on my meteor app, i'd like to now how to improve my data handling from minimongo.
Used to SQL / PHP, I'd like to know how to find() an object from my minimongo collection only one time, and access each of its properties with helpers, without having to re-access the collection each time.
Until now, what I did was something like that :
Template.profile.helpers({
name: function(e, tmpl){
return Meteor.users.FindOne({_id: Meteor.userId()}.profile.name;
},
phone: function(e, tmpl){
return Meteor.users.FindOne({_id: Meteor.userId()}.profile.phone;
}
[...]
});
But it's getting boring and i guess there must be a more efficient way to deal with it, something where I could load my entire users information only one time, and then display with a helper taking one parameter, to display the data like that : {{data name}}, {{data phone}}
With only one helper like that :
Template.profile.helpers({
data: function(aString){
if (aString == "phone)
return [...].phone;
}
}
[...]
});
Of course, I can use a session value, but I'm not sure it's as relevant as I could do.
Another thing : how to end a Meteor session ? Because with PHP, the session ends at the closure of the browser, and cookies stay for a given duration, but with meteor session, i never have to reconnect as day after day, my logs seems to stay.
Would someone guide me through this or give me some good habits / tips ? I'm still reading the doc, but it's quite huge :S
Thanks you !

One thing to note here -- you're not actually sending a request to the server when you call Mongo.Collection.find() on the client -- that's getting handled by the miniMongo instance on the client, so it's not really that inefficient. Also, in this particular instance, the current user is always available as Meteor.user() -- no need to use Meteor.users.find({_id: Meteor.userId()}); (see docs for more info)
However, if you really wanted to cache that value, you could do something like this:
// JS
var data; // scoped so it's available to the whole file
Template.profile.onCreated({
data = Meteor.users.findOne({_id: Meteor.userId()}); // above scope makes this new definition available to the whole file
});
Then, to do what you're describing with the string arguments, you can do something like...
Template.profile.helpers({
data: function(aString) {
if (data) { return data[aString]; }
}
});
Perhaps a better option even is to send the Template a data context -- something like this in your HTML file:
{{> profile currentUser}} <!-- currentUser returns Meteor.user() -->
Which is directly available to your helpers as this:
Template.profile.helpers({
data: function(aString) {
return this[aString];
}
});
You can also pass data contexts through IronRouter if you're using that for your routing.
As for the Meteor session, the model is different than the model for PHP. In PHP, you store session information on the server, and every time you access the server, the browser (or client, more generally) sends along the session ID so it can look up anything pertaining to your session (see this question for a potentially better explanation). Meteor keeps track of clients that are connected to it, so the session will stay open as long as your browser is open. If you need to manipulate Meteor user sessions, take a look at this question.

I recommend finding the collection helpers package on Atmosphere. This will enable you to write currentUser.name in a template and it'll automatically return users name helper function returns.
There is no problem working like this. It doesn't matter if the function is called multiple times. It won't hurt your apps performance.
PHP and meteor sessions are different. A meteor session lasts for as long as browser window remains open or the page is refreshed. You are right that meteor sessions are not the way to go for your problem.

Passing array via ReactiveVar in Meteor

I have a Meteor method that returns all user accounts on my application
returnUsers: function(){
return Meteor.users.find().fetch();
}
I'm using new ReactiveVar to pass the return value of the Meteor method into my template helper:
Template.listViewTemplate.created = function (){
var self = this;
self.myAsyncValue = new ReactiveVar("Waiting for response from serv er...");
Meteor.call('returnUsers', function (err, users) {
if (err)
console.log(err);
else
self.myAsyncValue.set(users);
});
}
Template.listViewTemplate.helpers({
userCollection: function(){
return Template.instance().myAsyncValue.get();
}
});
But when I go to render the users into the view, I get a console error that reads
{{#each}} currently only accepts arrays
When I render without the #each iterator, using
<ul id='usersList'>
{{userCollection}}
</ul>
the output on my web-page accurately reflects the number of users (2), but reads
[object Object],[object Object]
I'm pretty sure that there is some funkiness going on here because I'm using a global Meteor collection (Meteor.users.find().fetch(), as opposed to having defined my own collection), but I'm not sure how to get around it.
I want to display a list of all users so the current user can click another user and share a document with them--not sure how to get around this.

You don't need to use a reactive variable for this. The function at Template.listViewTemplate.created is not container in an autorun, which means: It won't get recomputed.
The best approach for your scenario is: Use a variable to get the status ( loading, loaded, error) and another variable to save the array itself attach to self. Reactivity is cool but you should only use it when needed.
About:
[object Object],[object Object]
This is happening because you're not extracting any value form the object provided nor looping using {{#each}}.
Your solutions for listing users is dangerous and inefficient. You're sending to the client all the fields from the user collection, including login tokens.
The best approach is to create a subscription that send only the necessaries fields like: _id, info.firstName. You should also have some criteria to the list users and use pagination. Consider also a search feature for such purpose.

ReactiveVar doesn't like arrays. You could install the ReactiveArray package which should accomplish exactly what you want.
Update
Based on comment of mper
In the latest versions of Meteor you can put an array in a ReactiveVar.
Tested on
meteor#1.6.0
reactive-var#1.0.11

I have several remarks about your question:
Do not fetch
You don't need .fetch() on your method. When you call find() on collections, such as Meteor.users a cursor is returned. The template (and #each in particular) can iterate through cursors. Cursors are usually better because you don't load the entire collection into memory at once - fetch does.
Meteor collections are reactive
Meteor collections are already reactive, meaning that if they change, they will trigger changes on your templates as well. So, you don't need to use a ReactiveVar to wrap your collection.
Query your local database
You don't need to use a method to get the users and in fact, you shouldn't, because usually you want to make queries to the database stored locally, not make calls to the server. Just call Meteor.users.find() directly in your template helper. You can (and should) control what is available locally through subscriptions.
Use #each with else
You can use the following in your template:
{{#each userCollection}}
...
{{else}}
Waiting for response from server...
{{/each}}
If userCollection is empty, the template will render the else block, just like you wanted.
Summarizing
Delete your method and onCreated with everything inside, change whatever is inside your template helper to only return Meteor.users.find() and use {{#each userCollection}}...{{else}}Waiting for response from server...{{/else}}
By the way
In the latest versions of Meteor you can put an array in a ReactiveVar.

Template.onCreated(function(){}) only gets run once and meteor methods only run once
You need reactivity here.
Collections sre reactive meaning pub/sub.
You need to create a publish function that allows certain users to fetch other users in the database. So all uses with maybe if the currentUser has permission to read all user info. Id limit the fields too.

Where does data returned by ember-data 'live'?

ya'll I have a bit of a structural/procedural question for ya.
So I have a pretty simple ember app, trying to use ember-data and I'm just not sure if I'm 'doing it right'. So the user hits my index template, I grab their location coordinates and encode a hash of it (that part works). Then on my server I have a db that stores 'tiles' named after there hash'd coords (if i hit my #/tiles/H1A2S3H4E5D route I get back properly formatted JSON).
What I would like to happen next, if to display each of the returned tiles to the user on the bottom of the first page (like in a partial maybe? if handlebars does that).
I have a DS.Model for the tiles, if I hard code the Hash'd cords into a App.find(H1A2S3H4E5D); I can see my server properly responding to the query. However, I cannot seem to be able to figure out how to access the returned JSON object, or how to display it to the user.
I did watch a few tutorial videos but they all seem to be outdated with the old router.
Mainly I would like to know:
1. Where does the information returned by App.find(); live & how to access it?
2. what is the 'correct' way to structure my templates/views to handle this?
3. how should I pass that id (the hash'd coords) to App.find? as a global variable? or is there a better way?
the biggest problem(to me) seems to be that the id I search by doesn't exist until the user hit the page tho first time. (since its dynamically generated) so I can't just grab it when the page loads.
I can post a fiddle if required, but I'm looking for more of a conceptual/instructional answer rather then some one to just write my code for me

I'm still learning a lot with Ember as well, but this is my understanding. When you follow the guides and the tutorials out there, you'll have something like this:
App.TileController = Ember.ObjectController.extend();
App.TileRoute = Ember.Route.extend({
setupController: function(controller) {
controller.set('content', App.Tile.find(MYHASH));
}
});
What it does is set the special content object to the result. So since we're declaring an object controller, and calling find with a parameter, it knows that a single result is expected. So a view & template that follow the naming convention of Tile will be loaded. And in there you can access properties on the Tile object:
<p>{{lat}}</p><p>{{lng}}</p>
I have to admit that this feels a bit mystical at times. The core to it is all in the naming convention. You need to be pretty specific in how you name all your various controllers, routes, etc. Once that's nailed down, it's a matter of binding what data you want to the controller's content.
1) Aside from the generic answer of "in memory", the .find() calls live where ever you return it to. Generally speaking, this is meant to be set on a 'content' property of a controller.
2) I more or less answered this, but generally speaking you take the name of your route, and base it off that. So for a route TileRoute, you have:
TileController = Ember.ObjectController.extend
Tile = DS.Model.extend
TileView = Ember.View.extend
tile.handlebars
I generally store all my handlebars files in a templates/ folder. If you nest them deeper, just specify the path in your view object:
App.TileView = Ember.View.extend({
templateName: "tiles/show"
});
3) This really depends on your app. Generally speaking its better for the id to be either obtained from the URL, or constructed locally in a function. Since you are encoding a hash, i imagine you're doing this in a function, and then calling find. I do something a bit similar for an Array controller.
I don't know at what point you are generating a hash, so let's say it's onload. You should be able to generate the hash just in the setupController function.
App.TileRoute = Ember.Route.extend({
generateHashBasedOnCoords: function() {
// ...
},
setupController: function(controller) {
var MYHASH = this.generateHashBasedOnCoords();
controller.set('content', App.Tile.find(MYHASH));
}
});
I hope that helps.

I believe that you can make use of the data binding in ember and basically have an array controller for tiles and set the content initially to an empty array. Then we you get back your response do a App.find() and set the content of the tiles controller with the data that is returned. This should update the view through the data binding. (Very high level response)
The data itself is stored in a store that is setup with ember data. You access it with the same method you are using the model methods App.Tile.find() ect. It checks to see if the data that is needed is in the store if so it returns the data otherwise it makes a call to the api to get the data.

Using Visualsearch.js in a backbone.js framework

First of all, thanks to the guys of DocumentCloud for releasing those two super-useful tools.
Here goes my question(s):
I'm trying to use visulasearch.js in a backbone.js app.
In my app I have a basic index.html and a myapp.js javascript file wich contains the main application done with backbone.js
I use CouchDB as data storage, and I successfully can retrieve in a restful way all the data to be put in the collection.
I must retrieve the search query given by visualsearch.js and use it to filter a collection.
I surely need a view for the searchbox, to trigger an event when enter is hit, but..
Should I initialze the searchbox externally to myapp.js, within an additional js file or my index.html page (as suggested in the visualsearch mini.tutorial)?
Or I should initialize it within the searchbox view (myapp.js)? This latter solution seems to be too tricky (it was what I was trying to do, but even when I succeed, it's too complicated and I lost the simplicity of bacbone mvc).
Let's say I succeed in retrieving the search string as a JSON object like {name:'Fat DAvid', address:'24, slim st', phone:'0098876534287'}. Once done that, which function can I use to retrieve, in the collection, only the models whose fields match with the given string. I understand that I should do a map or a filter, but those function seems to serve natively for slightly different tasks.
a. is it really the best way to filter results? It charges the client (which must filter the results), while making a new query (a view or a filter) to CouchDB would be quite simple and, considered the small amount of data and the low access rate to the site, not so expensive. However, making all the filtering action client-side, it's much simpler than making new view(or list or filters) in CouchDB and linking it the the backbone.js view

You can initialize your VisualSearch.js search box right in your myapp.js. Just make sure you keep a reference to it so you can then extract out the facets and values later.
For example:
var visualSearch = VS.init({...})
// Returns the unstructured search query
visualSearch.searchBox.value()
// "country: "South Africa" account: 5-samuel title: "Pentagon Papers""
// Returns an array of Facet model instances
visualSearch.searchQuery.facets()
// [FacetModel<country:"South Africa">,
// FacetModel<account:5-samuel>,
// FacetModel<title:"Pentagon Papers">]
If you have these models in a Backbone collection, you can easily perform a filter:
var facets = visualSearch.searchQuery.models;
_.each(facets, function(facet) {
switch (facet.get('category')) {
case 'country':
CountriesCollection.select(function(country) {
return country.get('name') == facet.get('value');
});
break;
// etc...
}
});