I'm looking for a way to fetch data from multiple collections in a subscription.
I have two collections: inspection_processes and inspections. Each inspection belongs to exact one inspection_process. Each inspection contains data (foreign keys) from another collection called devices.
My goal is to get a list of all inspection_processes and fetch their respective inspections and the devices specified in each inspection.
Here's the data from Firestore (displayed JSON-like for better readability:
inspection_processes = [{
id: 'AAA',
process_name: 'P-001'
}]
inspections = [{
inspection_id: 'BBB',
inspection_process_id: 'AAA',
device_id: 'CCC'
}]
devices = [{
device_id: 'DDD',
device_name: 'Computer'
}]
My current (simplified) approach:
fetchInspectionProcesses() {
let inspecion_processes_data = [];
// Get inspection processes
this.getInspectionProcesses().subscribe(inspecion_processes => {
for (let inspection_process of inspecion_processes) {
// Get inspections of inspection process
this.getInspectionsByProcessId(inspection_process['id']).subscribe(inspections => {
for (let inspection of inspections) {
// Get device info
this.getDeviceById(inspection['device_id']).subscribe(device => {
// ... push data to inspecion_processes_data array
})
}
})
}
})
}
I know that this approach isn't recommended since it leads to different problems (duplicate output in the UI, memory leaks, ...).
I did my best trying to solve it by using RxJS (e.g. by piping) but I didn't manage to use the right operators. The problem is not limited to having two or three collections to combine, I'm looking for a general approach on how to do this.
Working with duplicate data (e.g. putting the device's name in the inspections document isn't an option since my example is just simplified, there are many nestings I have to deal with.
EDIT: Screenshot of the Firebase Console
Can anyone help, please? Thank you in advance!
What you're talking about is basically a join in firestore. You should really duplicate your data so you don't have to do this. I wrote some functions to do this in my package
https://github.com/jdgamble555/adv-firestore-functions#Aggregation
But you could also do SQL like joins (but you will incur more reads):
https://fireship.io/lessons/firestore-joins-similar-to-sql/
J
This is how you might remove your nested subscriptions
fetchInspectionProcesses() {
let inspecion_processes_data = [];
this.getInspectionProcesses().pipe(
map(inspecion_processes => inspecion_processes.map(
inspection_process => this.getInspectionsByProcessId(inspection_process['id'])
)),
mergeMap(processes => merge(...processes)),
map(inspections => inspections.map(
inspection => this.getDeviceById(inspection['device_id'])
)),
mergeMap(devices => merge(...devices)),
toArray()
).subscribe(deviceArray => {
// ... This is your inspecion_processes_data array
});
}
Or, a little bit more compactly by combining map and mergeMap together:
fetchInspectionProcesses() {
let inspecion_processes_data = [];
this.getInspectionProcesses().pipe(
mergeMap(inspecion_processes => merge(...inspecion_processes.map(
inspection_process => this.getInspectionsByProcessId(inspection_process['id'])
))),
mergeMap(inspections => merge(...inspections.map(
inspection => this.getDeviceById(inspection['device_id'])
))),
toArray()
).subscribe(deviceArray => {
// ... Do something with deviceArray
});
}
Related
I am building an order form that limits how many items you can order based on the stock of the item. I have a menu collection which has items
// menu
{ id: "lasagna", name: "Lasagna", price: 10, stock: 15 }
{ id: "carrot-soup", name: "Carrot Soup", price: 10, stock: 15 }
{ id: "chicken-pot-pie", name: "Chicken Pot Pie", price: 10, stock: 15 }
And an orders collection
// orders
{ id: <auto>, name: "Sarah", cart: {lasagna: 1, carrot-soup: 3}, ... }
{ id: <auto>, name: "Wendy", cart: {chicken-pot-pie: 2, carrot-soup: 1}, ... }
{ id: <auto>, name: "Linda", cart: {lasagna: 3}, ... }
4 carrot-soup has been ordered so the stock should be updated
// updated stock
{ id: "carrot-soup", name: "Carrot Soup", stock: 11 }
Orders are inserted from my Form component
function Form(props) {
// ...
// send order to firestore
const onSubmit = async _event => {
try {
const order = { cart, name, email, phone, sms }
dispatch({ action: "order-add" })
const id = await addDocument(store, "orders", order)
dispatch({ action: "order-add-success", payload: { ...order, id } })
}
catch (err) {
dispatch({ action: "order-add-error", payload: err })
}
}
return <form>...</form>
}
This is my database addDocument function
import { addDoc, collection, serverTimeStamp } from "firebase/firestore"
async function addDocument(store, coll, data) {
const docRef = await addDoc(collection(store, coll), { ...data, timestamp: serverTimestamp() })
return docRef.id
}
How should I decrement the stock field in my menu collection?
Ideally the client should have only read access to menu but to update the stock the client would need write access.
Another possibility is to have the client query the orders, sum the items, and subtract them from the read-only menu. But giving the client read access to other people's orders seems wrong too.
I am new to firestore and don't see a good way to design this.
You should deffinitely use a cloud function to update the stock. Create a function onCreate and onDelete functions trigger. If users can change data you would also need to onWrite function trigger.
Depending on the amount of data you have you woould need to create a custom queue system to update the stock. Belive me! It took me almost 2 years to figure out to solve this. I have even spoken with the Firebase engeeners at the last Firebase Summit in Madrid.
Usualy you would use a transaction to update the state. I would recommend you to do so if you don't have to much data to store.
In my case the amount of data was so large that those transactions would randomly fail so the stock wasn't correct at all. You can see my StackOverflow answer here. The first time I tought I had an answer. You know it took me years to solve this because I asked the same question on a Firebase Summit in Amsterdam. I asked one of the Engeeners who worked on the Realtime Database before they went to Google.
There is a solution to store the stock in chunks but even that would cause random errors with our data. Each time we improved our solution the random errors reduced but still remained.
The solution we are still using is to have a custom queue and work each change one by one. The downside of this is that it takes some time to calculate a lot of data changes but it is 100% acurate.
Just in case we still have a "recalculator" who recalculates one day again and checks if everything worked as it should.
Sorry for the long aswer. For me it looks like you are building a similar system like we have. If you plan to create a warehouse management system like we did I would rather point you to the right direction.
In the end it depends on the amount of data you have and how often or fast you change it.
Here is a solution based on Tarik Huber's advice.
First I include functions and admin
const functions = require("firebase-functions")
const admin = require("firebase-admin")
admin.initializeApp()
Then I create increment and decrement helpers
const menuRef = admin.firestore().collection("menu")
const increment = ([ id, n ]) =>
menuRef.doc(id).update({
stock: admin.firestore.FieldValue.increment(n)
})
const decrement = ([ id, n ]) =>
increment([ id, n * -1 ])
Here is the onCreate and onDelete hooks
exports.updateStockOnCreate =
functions
.firestore
.document("orders/{orderid}")
.onCreate(snap => Promise.all(Object.entries(snap.get("cart") ?? {}).map(decrement)))
exports.updateStockOnDelete =
functions
.firestore
.document("orders/{orderid}")
.onDelete(snap => Promise.all(Object.entries(snap.get("cart") ?? {}).map(increment)))
To handle onUpdate I compare the cart before and after using a diff helper
exports.updateStockOnUpdate =
functions
.firestore
.document("orders/{orderid}")
.onUpdate(snap => Promise.all(diff(snap.before.get("cart"), snap.after.get("cart")).map(increment)))
Here is the diff helper
function diff (before = {}, after = {}) {
const changes = []
const keys = new Set(Object.keys(before).concat(Object.keys(after)))
for (const k of keys) {
const delta = (before[k] ?? 0) - (after[k] ?? 0)
if (delta !== 0)
changes.push([k, delta])
}
return changes
}
The title may be miss leading but I'm not really sure how do I ask this question correctly. Here is the problem: I'd like to query my own API(not created yet so I made placeholder data) for global settings which might change in the future and I will only need to rebuild the website instead of editing it manually, I want to create source node called CmsSettings and pass it to GraphQL (structure similar to site.siteMetadata) but I don't know how can I achieve that. What I achieved so far is to create a source node called allCmsSettings which has my data as an object in nodes array.
exports.sourceNodes = ({ actions, createNodeId, createContentDigest }) => {
const { createNode } = actions;
const myData = {
key: 123,
app_title: `The foo field of my node`,
...
}
const nodeContent = JSON.stringify(myData);
const nodeMeta = {
id: createNodeId(`my-data${ myData.key }`),
parent: null,
children: [],
internal: {
type: `CmsSettings`,
mediaType: `text/html`,
content: nodeContent,
contentDigest: createContentDigest(myData)
}
}
const node = Object.assign({}, myData, nodeMeta);
createNode(node);
}
Here is the query used to get the data of the source node
allCmsSettings {
edges {
node {
id
app_title
...
}
}
}
Creating a query results in an array of results(which I know is the result of creating source nodes) but I'd like to create that source so that I could query it like this and:
CmsSettings {
app_title
app_keywords
app_descriptions
app_logo_path
brand_name
...
}
You get the point. I was browsing the gatsby node API but I can't find how to achieve this.
Thank you for your help
Nevermind, the answer is pretty simple, if you are new to gatsby just like me the sourceNodes export creates 2 graphql fields for you with all prefix and camel case source node. The thing that I wanted to make is already there and is queryable with
cmsSettings {
app_title
app_keywords
app_descriptions
app_logo_path
brand_name
...
}
Notice the lowercase letter even though it was declared as CmsSettings. It seems that gatsby really does some magic under the hood.
When we execute queries to our PostgreSQL Database and receive responses, we then pass these responses to our clientside to display/work with.
Example:
const response = [
{
first_name: 'Bob',
last_name: 'English',
title: 'The Dude',
},
{
first_name: 'Harry',
last_name: 'Smith',
title: 'Unknown',
},
];
Our app then has to map over this and rename the keys in a rather inefficient and verbose manner. Is there a better way? Perhaps using pg-promise?
Worth noting we also have to convert back when we send the data as an UPDATE to the DB.
It's worth noting we are not looking to use a ORM like Objection or a query builder like Knex.
Event receive in the API offers a usable example of how this can be done:
// Example below shows the fastest way to camelize all column names.
// NOTE: The example does not do processing for nested JSON objects.
const initOptions = {
// pg-promise initialization options...
receive(e) {
camelizeColumns(e.data);
}
};
function camelizeColumns(data) {
const tmp = data[0];
for (const prop in tmp) {
const camel = pgp.utils.camelize(prop);
if (!(camel in tmp)) {
for (let i = 0; i < data.length; i++) {
const d = data[i];
d[camel] = d[prop];
delete d[prop];
}
}
}
}
It also has been discussed in various issues on the project in the past, and documented by other developers, like in this article, which is a good way to get started with it. UPDATE: That article is obsolete for pg-promise v11.
It is a universal approach that works for all types of queries, including streams.
UPDATE
The example above has been updated to use pg-promise v11 or later.
What I am trying to do
I am creating a social media app with react native and firebase. I am trying to call a function, and have that function return a list of posts from off of my server.
Problem
Using the return method on a firebase query gives me a hard to use object array:
Array [
Object {
"-L2mDBZ6gqY6ANJD6rg1": Object {
//...
},
},
]
I don't like how there is an object inside of an object, and the whole thing is very hard to work with. I created a list inside my app and named it items, and when pushing all of the values to that, I got a much easier to work with object:
Array [
Object {
//...
"key": "-L2mDBZ6gqY6ANJD6rg1",
},
]
This object is also a lot nicer to use because the key is not the name of the object, but inside of it.
I would just return the array I made, but that returns as undefined.
My question
In a function, how can I return an array I created using a firebase query? (to get the objects of an array)
My Code
runQ(group){
var items = [];
//I am returning the entire firebase query...
return firebase.database().ref('posts/'+group).orderByKey().once ('value', (snap) => {
snap.forEach ( (child) => {
items.push({
//post contents
});
});
console.log(items)
//... but all I want to return is the items array. This returns undefined though.
})
}
Please let me know if I'm getting your question correctly. So, the posts table in database looks like this right now:
And you want to return these posts in this manner:
[
{
"key": "-L1ELDwqJqm17iBI4UZu",
"message": "post 1"
},
{
"key": "-L1ELOuuf9hOdydnI3HU",
"message": "post 2"
},
{
"key": "-L1ELqFi7X9lm6ssOd5d",
"message": "post 3"
},
{
"key": "-L1EMH-Co64-RAQ1-AvU",
"message": "post 4"
}
...
]
Is this correct? If so, here's what you're suppose to do:
var items = [];
firebase.database().ref('posts').orderByKey().once('value', (snapshot) => {
snapshot.forEach((child) => {
// 'key' might not be a part of the post, if you do want to
// include the key as well, then use this code instead
//
// const post = child.val();
// const key = child.key;
// items.push({ ...post, key });
//
// Otherwise, the following line is enough
items.push(child.val());
});
// Then, do something with the 'items' array here
})
.catch(() => { });
Off the topics here: I see that you're using firebase.database().... to fetch posts from the database, are you using cloud functions or you're fetching those posts in your App, using users' devices to do so? If it's the latter, you probably would rather use cloud functions and pagination to fetch posts, mainly because of 2 reasons:
There might be too many posts to fetch at one time
This causes security issues, because you're allowing every device to connect to your database (you'd have to come up with real good security rules to keep your database safe)
I've been working with Firebase for a little while, and like their suggestion to keep data denormalized. My only problem is figuring out the best way to query across multiple collections. For example, I have an Identity object which holds info on my users:
identities: {
$identity: {
name: string,
studio: $studioID
}}
That corresponds to a Studio object:
studios: {
$studio: {
name: string,
owner: $identityID,
location: $locationID
}}
This object references the owner, and also their location. The location object references Classes, which references Students.... on and on. Right now, in order to fetch a referenced object, I'm doing something like this:
Auth.loginUser(email, password, (success) => {
const identityRef = firebase.database().ref('/identity');
identityRef.child(success.uid).on("value", function(identitySnapshot) {
const identity = identitySnapshot.val();
const studioRef = firebase.database().ref('/studios');
dispatch({type: 'UPDATE_IDENTITY', identity})
studioRef.child(identity.studioID).on("value", function(studioSnapshot) {
const studio = studioSnapshot.val();
dispatch({type: 'UPDATE_STUDIO', studio});
})}); });
I would continue nesting my calls for Location, Classes, Students, etc. Is there a better way to do this?
Consider the following structures:
identities: {
$identityKey: {
name: string
}
}
identities_studios: {
$identityKey: {
$studioKey: {
name: string
}
}
}
identities_studios_locations: {
$identityKey: {
$studioKey: {
$locationKey: {
lat: string,
lng: string
}
}
}
}
The first, identities, only stores info about the identities as usual.
The second, identities_studios, only stores info about the studios, but the studios are grouped by $identityKey.
The third, identities_studios_locations, only stores info about the locations, but they are grouped firstly by $studioKey and secondly by $identityKey.
Now you can do this:
const db = firebase.database()
Auth.loginUser(email, password, success => {
db.ref(`/identities/${success.uid}`).on("value", snap => { ... }
db.ref(`/identities_studios/${success.uid}`).on("value", snap => { ... }
db.ref(`/identities_studios_locations/${success.uid}`).on("value", snap => { ... }
}
Instead of making multiple requests one after the other, we get them to run simultaneously.
If you want, after getting all this info from the database, you can transform the data structure to whatever you want: one array for identities, another for studios, another for locations, etc.; or a single array of identities with nested studios which in turn have nested locations etc.