We Keep Coding

Where to store remote data in Svelte for responsive data vizualisations with Svelte and D3 (best practices)

Disclaimer: I am really not well experienced with neither svelte, nor D3, nor general JavaScript patterns. But I really like it, so I really want to learn it and already invested quite some time. Still, this feels like a super basic question that annoys me a lot. I hope it is not too confusing and someone might has an idea.
It is basically about how to setup a simple graph (let it be a bar chart) in an efficient, reproducible and "best-practice" way. I guess my main concern is on how to pass around the data and use it for different tasks. E.g. I think it might be a good idea to separate out the construction of the scales (using d3) in a separate component. However, this component needs access to the data (and probably also access to the, in the best case resposive width, of the chart-container).
However, also the bars, which are in another component, need access to the data in order to know how do draw the rectangles.
A general misunderstanding (i guess that is the right word) I have with JavaScript is that I do not understand how to fetch data asynchronously (using e.g. the browsers fetchor D3's csvmethod). I simply can not fetch the data and then pass it as prop to another component. Because what I would be passing would be a promise...
So I have this very basic REPL that kind of shows a bit this know I have in my head: https://svelte.dev/repl/398f4c21b7a9409a9811fd8e38703a36?version=3.44.1
It looks like this. In the App.html I fetch the data that I want to use for multiple purposes. However I cannot "get it out" of that component.
<script>
import Chart from "./Chart.svelte"
const url = "https://api.github.com/search/repositories?q=stars:>100000";
async function getData(){
let response = await fetch(url)
let data = await response.json()
console.log(data)
}
//async function getDataLocal(){
// let data = await d3.csv(<path_to_data>)
// return await data
// }
let data = await getData()
</script>
<Chart {data}>Do Something with the data. Make the chart, build the scales, ....</Chart>
So the main questions are:
Are there any ressources on how to learn building sustainable graphics with remote data, svelte and a bit of D3. I already watched many many youtube videos and I guess I will rewatch the one from Matthias Stahl;)
Is it a good idea to use stores in such a case to store the data
And a little more specific: As the data is (probably) fixed, however the dimension arent't: What is a good way/place to let the app know to recalculate the scales etc.

There are 3 separate concerns here:
fetching, storing and retrieving data (aka the data source layer)
manipulating/transforming data (aka the business logic layer)
displaying data (aka the presentation layer)
I will leave the last part aside as it solely concerns D3 (if that is your visualization library of choice) and there are plenty of resources available online on this topic, and I will instead focus on what seems to be the heart of your question, i.e. how to fetch data in Svelte, where to store it, how to pass it around to components, and how to manipulate the data.
1. Asynchronous queries in Svelte
Your first inquiry is about how to deal with asynchronous requests. You cannot use await statements at the root level of the <script> section of a Svelte file, meaning the following reactive statement would generate an error:
// will fail
$: data = await getData(url)
However, you can call an asynchronous function that will handle the assignment. Reactivity will still work and your component will re-render when the url is changed and the new data retrieved:
// will work
$: updateData(url)
async function updateData(url) {
data = await getData(url)
}
Here is a working example based on the REPL in your question
2. Using stores
As you could see from the above example, you had to pass the data to your <Header> and <Chart> components for it to be used in either:
<Header {data}>GitHub Lookup</Header>
<Chart {data}/>
But what if you want to use your Chart somewhere else in your application? What if you have another component that wants to make use of the same data?
Obviously you do not want to fetch the same data over & over (unless the request itself has changed). You also want to avoid passing the data around as a prop everywhere in your app. You will want to make the data available only to these components that will use it.
This is where stores come in handy. Stores can be subscribed to by any component. A writable store will allow its contents to be updated, while a readable store will be -as the name implies- read-only.
A store need not be complex. The following is a very basic writable store:
import { writable } from 'svelte/store'
export const githubStore = writable(null) // initialized with a null value
All you have to do then is interact with your store.
Updating the store in your App component:
import { githubStore as data } from './githubStore.js' // import the store we defined above under the name 'data'
.
.
.
async function updateData(url) {
$data = await getData(url) // using the $ shorthand to access the store (assigning a new value will update the store content)
}
Using (i.e. subscribing to) the store in your components:
import { githubStore as data } from './githubStore.js' // import the store we defined above under the name 'data'
.
.
.
// using the $ shorthand to access the store
{#each $data.items as item (item.id)}
<li><a href={item.html_url}>{item.full_name}</a> [{item.stargazers_count}⭐]</li>
{/each}
Read here for details on using the $ reactive syntax with stores
Now that your child components are subscribing to the store where you stored your data, you do not need to pass that data as a prop any more:
<Header>GitHub Lookup</Header>
<Chart />
Here is an updated version of the REPL above, using stores
3. Further considerations
When you want to start manipulating or transforming data that has been put into a store, derived stores come in handy. When the data in your original store is updated, the derived store will automatically update itself based on the changes to the original.
You can also build on the provided readable/writable stores by adding your own functionality and custom methods. These are slightly more advanced topics but would come in handy where data manipulation is concerned.
Finally, D3 will provide its own data manipulation methods, so it will be up to you to decide how much manipulation you handle directly in Svelte, and how much you delegate to D3. I would probably leave everything connected to visualization (scaling, zooming, etc.) on the D3 side, and have the actual pre-visualization manipulation of data (i.e. the business logic) on the Svelte side (or better yet, directly on the back-end if you have access to that!).

how to scan all decorators value at runtime in nestjs

I have to collect all decorator value that appears in different place in my app as string and then saving them to database at runtime, i don't have to add them twice (in database and in code),
i have tried to do it but i could not figure out i use
Reflector api from nestjs as following
this.reflector.getAll<string>('access', context.getHandler())
but i could not get context.getHandler() during run time
async function bootstrap() {
const app = await NestFactory.create(AppModule);
app.useGlobalPipes(new ValidationPipe());
// Here is where i want to save
await app.listen(3000);
}
bootstrap();
here is my decorator
#HashPermission('access_value')
Please assist

For something like this, you'll either need to make use of something like Nest's undocumented DiscoveryService or a package like #golevelup/nestjs-discovery which is a friendly wrapper around Nest's package. You can then make use of methods like this.discoveryService.methodsAndControllerMethodsWithMetaAtKey to get the classes and methods that have that metadata, then you can use the reflector class on each method to get the metadata value.

share firestore collection paths across admin and web

I'd like to make re-usable functions that get the Firestore Document/Collection reference across web and admin (node.js).
for example:
getUserDocumentReference(company: string, user: string) {
return firebase.collection("companies")
.doc(company)
.collection("users")
.doc(user);
}
This will reduce errors and coordinate changes across both environments.
Problem: Admin imports firestore from firebase-admin, and web imports from firebase.
I've tried making some class/function where I pass in my firestore reference, but it becomes a pain where I have to declare the return types:
const ref = (
getUserDocumentReference("a", "1") as
firebase.firestore.DocumentReference
)
.withConverter(converter)
Is there a smarter/cleaner way to do this without re-inventing the wheel (i.e. somehow passing an array or re-creating paths in a complex way)?
my current approach:
class FirestoreReferences {
constructor(firestore: firebase.firestore.Firestore
| admin.firestore.Firestore) {
this.firestore = firestore;
}
getUserDocumentReference(company: string, user: string): FirebaseFirestore.DocumentReference | firebase.firestore.DocumentReference {
return this.firestore.collection(...).doc(...);
}
}

Just found out about Typesaurus which provides generic types to share across web/admin!

The simplest answer: DO NOT use the .doc() or the .doc().ref. Use the .doc.ref.path - which is a string with the FULL PATH to the document. Save/share it as let refPath = whatever.doc().ref.path and re-build it as .doc(refPath) is either environment.
I DO NOT actually RECOMMEND this - it exposes your internal structure - but it isn't inherently insecure (your security rules better be taking care of that).
btw, I'm building an entire wrapper npm package (#leaddreamer/firebase-wrapper) for this specific purpose.

You should not do this. The Admin SDK is meant for server-side usage because it has full control over your entire project. If a user gets access to this, they have control over your app. Keep firebase and firebase-admin seperate.

Mocking jest network - how do I not DRY my formatter?

I have a network module to ping a legacy database with data in multiple formats, and I want to standardize it here in the network module before passing it into the application so my application can expect a certain format of data (don't want the old, poor formatting polluting my business logic). I'm struggling with how to pass mock data through as this network module, specifically as it relates to the formatter. Here's what I mean:
// User API Network Module
// UserAPI.ts
export const getUser = (uid: String, callback: (GetUserResponse) => void): void => {
// Do network call here and format the data into a typescript type
// matching the GetUserResponse structure by business logic expects
callback(formattedData)
}
In my test file, I can mock this call easily with:
import { getUser } from "./UserAPI"
jest.mock("./UserAPI", () => ({
getUser: (uid: String, callback: (GetUserResponse) => void) => {
const mockedUserData = require("./mockUser.json")
// But how do I format it here?
return formattedMockedUserData
},
}))
I can create a formatter function in my UserAPI.ts file, export it, and run it in the jest mock, but I'm wondering if that's a best practice because it technically leaks the UserAPI implementation details. And I point that out only because no other file cares about how UserAPI formats things except UserAPI. If I have to leak it for testing purposes, I'll do that. But is there a better way to mock the network call and run it through a formatter without exposing additional implementation details?
And please be gentle on my typescript - I come from both a JS and strongly typed background, but this is my first venture into using typescript :)

Even though it's not used multiple places extract it - following Single Responsibility Principle - into its own construct. You test all formatting logic in Formmater Test not in User API Test. Additionally you can test the integration of Formatter with User API in an Integration Test.

How set graphql-js module in debug mode

i'm new in graphql and setting up server graphql api use [https://github.com/graphql/graphql-js] graphql-js.
I followed their document and everything work, expect 1 problem:
when make query and got error from deep module, graphql-js try catch error and return result with errors: [...] property, but in terminal console say nothing. I would like force grahql-js return actual Error object so i can debug issue when looking to stack.
this is code in api
query: (ctx) -> (
await graphql(
schema
query
context
graphContext
variables
)
)
I search a lot of topic but no ones say about set this package in debug mode, also no see any parameter allow set debug. Anyone know about this?
Thanks !

Normally, you would use a library like express-graphql or apollo-server (or apollo-server-express, apollo-server-hapi, etc.) to provide an endpoint for your GraphQL API. Unless your use case prohibits you from doing so, I would strongly encourage you to utilize one of these libraries since they provide a number of additional features that are helpful in both development and production.
You can add logging to your endpoint when using one of these libraries simply by providing a custom formatError function in your configuration. See each library's documentation here and here. As a bonus, if you use Apollo, then the full stack trace will be exposed inside of your errors array anyway, but only in development.
If you're calling the graphql function directly in your code somewhere, then you can just log the errors yourself. The call to graphql returns a Promise that resolves to a ExecutionResult object, which includes both your data and errors. So you can do something like:
const result = await graphql(/* your args */)
if (result.errors) {
result.errors.forEach((error) => {
// log the error
})
}

You have to instantiate the graph object with the debug option enabled, like this
var graph = graphql("/api", { debug: true })