I've been asked to look at flatting API responses on a Node.js project I'm working on but I'm not entirely sure why it needs to be flattened. No responses are nested beyond 2 levels so to me it's not too bad. Can anybody tell me why an API response would be flattened? I'd be keen to know the pros and cons aswell an hear any suggestions on how to do it? I'm currently looking at npm package flat.
Here's an example response:
{
"users": [
{
"id": 1,
"name": "John Doe",
"email": "john#doe.com",
"suppliers": [
{
"id": 1,
"name": "Supplier1",
}
]
},
{
"id": 2,
"name": "Jane Doe",
"email": "jane#doe.com",
"suppliers": [
{
"id": 1,
"name": "Supplier1",
},{
"id": 2,
"name": "Supplier2",
}
]
}
]
}
Can anybody tell me why an API response would be flattened? I'd be
keen to know the pros and cons aswell an hear any suggestions on how
to do it?
Pros:
An API needs to be flattened if the consumer of that API requires it to be flattened. For example, if the purpose of the API is to provide data that gets loaded into a table view of some kind, it would be much more convenient to the consumer if the API returned the data flattened to match the shape of that table.
Cons:
Flattening hierarchical data often times requires either limiting the number of child elements that can be returned or creating more rows that makes sense.
Consider your data flattened in these two approaches:
Flattened approach #1
{
"users": [
{
"id": 1,
"name": "John Doe",
"email": "john#doe.com",
"suppliers_1_id": 1,
"suppliers_1_name": "Supplier1",
"suppliers_2_id": null,
"suppliers_2_name": null
}, {
"id": 2,
"name": "Jane Doe",
"email": "jane#doe.com",
"suppliers_1_id": 1,
"suppliers_1_name": "Supplier1",
"suppliers_2_id": 2,
"suppliers_2_name": "Supplier2"
}
]
}
In this example, it has to be decided ahead of time what the maximum number of suppliers that can be returned. I really dislike this kind of design for storing data, but often it is the requested way for data to be displayed in a table. If for example there are 3 suppliers for a user, you would have no way to return that data without adding more columns. It very quickly becomes unmanageable unless you have a very small and finite maximum number of child rows.
Flattened approach #2
{
"users": [
{
"id": 1,
"name": "John Doe",
"email": "john#doe.com",
"supplier_id": 1,
"supplier_name": "Supplier1"
}, {
"id": 2,
"name": "Jane Doe",
"email": "jane#doe.com",
"supplier_id": 1,
"supplier_name": "Supplier1"
}, {
"id": 2,
"name": "Jane Doe",
"email": "jane#doe.com",
"supplier_id": 2,
"supplier_name": "Supplier2"
}
]
}
This approach simply repeats each user for the number of suppliers. This format is convenient for making it possible to convert the flattened data back to hierarchical data. It is a common approach when retrieving hierarchical data from a relational database in a single rowset. A client application can easily reassemble the original hierarchy by grouping by user id. The negative to this approach is it returns more than one row per top-level object, which can cause larger data payloads. If you have a single, internal consumer of the API, this approach may work. If it is for a public API, you'll have to spend more time documenting and supporting the API, because it may not make sense to your API consumers.
I'm not entirely sure why it needs to be flattened.
Whoever is asking you to flatten it should specify that actual shape they are looking for, and that may provide clarity. I've described two possible ways to flatten your data, but there are plenty more possibilities.
Related
I have a JSON file that I used to create my database in MongoDB, which I am using to build a website. Here is a snippet, before I explain what I'm trying to do.
[
{
"name": "Field 1",
"description": "",
"completed": false,
"category": "Field",
"resources": [],
"items": [
{
"name": "Field 2",
"description": "",
"completed": false,
"category": "Field",
"resources": [],
"items": [
{
"name": "Field 3",
"description": "",
"completed": false,
"category": "Field",
"resources": [],
"items": [
{
"name": "Topic 1",
"description": "",
"completed": false,
"category": "Topic",
"resources": [],
"items": []
},
{
"name": "Topic 2",
"description": "",
"completed": false,
"category": "Topic",
"resources": [],
"items": []
}
My data (and the way I'd like to use it) is heavily dependent on the parent and children of any individual object. My database has 8 documents, but there's over many embedded objects in total, all embedded at different depths in the 'items' array of each object.
I just created my first draft of the site, which loads the original 8, and upon clicking any of them, it then lists the immediate children in the 'items' array, and it continues doing that until the end of that specific path. To do this, I rely on the index of each object in its array, and keep track of a path
But I would also like my users to be able to navigate directly to the page for a certain object without starting at the top and navigating through. If I wanted to directly access the object for Field 3 in the above example, what's the best way to do this with a single function or piece of code that could work with any of them?
The JSON file that I've used to create my database is 22,000+ lines long, and I'd love to not have to go back and change it more than is absolutely necessary. I was thinking I could add an ID somehow, and if that's unique, I could use that. The names of some objects will be the same, depending on where they are in the data.
EDIT: Bonus question - Would this sort of data be best stored in a relational database? I thought non-relational would be best because of the nested functionality, but I suppose I could make it work with either.
I would like to get the live results from an Api endpoint preferably using javascript, nodejs. I am looking for more like a technique or idea how to handle this.
Api response looks as following:
[
{
"id": 6,
"team1": {
"name": "Home Team",
"goals": 2
},
"team2": {
"name": "Away Team",
"goals": 1
},
"starting": "2018-09-14T15:30:00Z"
}
]
Probably one way I can think of doing is making a new request every second and get up to date. So each time a team scores this api changes so I get the latest score goal results.
What is the best way I can handle this without overwhelming the resources of the server or probably not get banned from the API Endpoint if I am doing too many requests.
i come from a MySQL workflow and now i am trying to do something in Firebase, and i have some dilemmas with structuring my data, because i don't know how would i query it.
fallowing the example below where i have some users and some comments, what is a good way to find
how many post likes a user had
how many post comments a user had
what were the posts that a user liked
...
i was thinking on adding that information to a user, like:
"user:2": {
"name": "Doe",
"email": "doe#bob.com",
"post_likes": {
"post:1": 1,
"post:2": 1
},
"post_comments": {
"post:1": 15,
"post:2": 5
}
}
but this seems redundant and duplicates data..
i need to find a way to search in posts everything that has to do with user:1, maybe i need to create another object that connects the users and posts ??
any ideas?
{
"array": {
"users": {
"user:1": {
"name": "John",
"email": "john#bob.com"
},
"user:2": {
"name": "Doe",
"email": "doe#bob.com"
}
},
"posts": {
"post:1": {
"name": "First Post",
"content": "some post content",
"comments": {}
},
"post:2": {
"name": "Second Post",
"content": "some other post content",
"likes": 2,
"comments": {
"comment:1": {
"uid": "user:1",
"email": "some comment"
},
"comment:2": {
"uid": "user:2",
"email": "some other comment"
}
}
}
}
}
}
How to structure data is not a simple question to answer and is highly dependent on how the data will be read. The general rule is to work hard on writes to make reads easy. Firebase offers two guides on understanding data and on structuring data.
They won't fit nicely into this text box, so you'll find better answers there. Here's a birds-eye view.
It's JSON data.
Flatten your data. Don't nest just because you can. Treat each logical data component like you would in SQL and keep it in its own path.
Avoid arrays in distributed systems. Sequential numeric ids are error prone.
Utilize indices to create sub-lists of master data rather than trying to nest lists in lists in lists.
I'm still trying to write a function in JavaScript where the user can type in an artist, and it will return a link to that artist's SoundCloud page.
For example,
/artist beyonce --> https://soundcloud.com/beyoncemusic
But the SoundCloud URLS don't all act the same. For example,
/artist dave matthews band --> https://soundcloud.com/dave-matthews-band.
For this reason, I can't simply just output scLink/artistName because they all have different URLs. I'm using Node.js, so I looked through a lot of npm packages, but couldn't figure out how to use any for this purpose. Perhaps Soundclouder will work somehow (though I couldn't figure it out myself). Does anyone know how I could write a command like this?
You are using the SoundCloud API, right?
A simple HTTP request to the right API should return the data you want. For example:
http://api.soundcloud.com/users.json?q=beyonce
[
{
"id": 4293843,
"kind": "user",
"permalink": "beyoncemusic",
"username": "Beyoncé",
"uri": "http://api.soundcloud.com/users/4293843",
"permalink_url": "http://soundcloud.com/beyoncemusic",
"avatar_url": "http://i1.sndcdn.com/avatars-000036935308-a2acxy-large.jpg?435a760",
"country": "United States",
"full_name": "Beyoncé",
"description": "",
"city": "New York",
"discogs_name": null,
"myspace_name": "beyonce",
"website": "http://www.beyonceonline.com",
"website_title": "",
"online": false,
"track_count": 33,
"playlist_count": 2,
"plan": "Pro Plus",
"public_favorites_count": 0,
"followers_count": 478783,
"followings_count": 0,
"subscriptions": [
{
"product": {
"id": "creator-pro-unlimited",
"name": "Pro Unlimited"
}
}
]
},
...
]
...so you could just do results[0].permalink_url.
You can use the request module to make the HTTP request manually, or use soundclouder to handle SoundCloud API's authentication details.
Most of the above does not apply if you want to make the actual requests from a browser. (The question is tagged node.js, but it sounds like you want to do this from a web page.)
If you're doing this from a webpage, use the SoundCloud JS SDK. The data you get back will look like the example above.
I don't think you'd be able to get an exact match reliably. Your best bet would be to search for users with the string you are looking for - example: "beyonce" and then to show the results and let them pick the correct link. You may be able to filter out likely results with follower count (high follower count) or something after you've pulled the initial list from soundcloud.
Search code:
users = SC.get('/users', { q: 'beyonce' });
Then iterate over users and display the permalink url. Hope this helps.
I am tackling frontend development (AngularJS) and rather than pull data from the backend (which isn't complete but renders everything to JSON), I am looking to just use hardcoded JSON.
However, I am new to this and can't seem to find anything about complex JSON structure. In a basic sense, my web app has users and the content they create. So, in my mind, there will be two databases, but I'm not sure if I'm approaching it correctly.
Users - username, location, created content, comments, etc.
"user": [
{
"userID": "12",
"userUserName": "My Username",
"userRealName": "My Real Name",
"mainInterests": [
{
"interest": "Guitar"
},
{
"interest": "Web Design"
},
{
"interest": "Hiking"
}
],
"userAbout": "All about me...",
"userComments": [
{
"comment": "this is a comment", "contentID" : "12"
},
{
"comment": "this is another comment", "contentID" : "123"
}
],
}
]
Created Content - title, description, username, comments, etc.
"mainItem": [
{
"mainID": "1",
"mainTitle": "Guitar Lessons",
"mainCreatorUserName": "My Username",
"mainCreatorRealName": "My Real Name",
"mainTags": [
{
"tag": "Intermediate"
},
{
"tag": "Acoustic"
},
{
"tag": "Guitar"
}
],
"mainAbout": "Learn guitar!",
"mainSeries": [
{
"videoFile": "file.mov",
"thumbnail": "url.jpg",
"time": "9:37",
"seriesNumber": "1",
"title": "Learn Scales"
},
{
"videoFile": "file.mov",
"thumbnail": "url.jpg",
"time": "8:12",
"seriesNumber": "2",
"title": "Learn Chords"
}
],
"userComments": [
{
"comment": "this is a comment", "userID" : "12"
},
{
"comment": "this is another comment", "userID" : "123"
}
]
}
]
And there is more complexity than that, but I just would like to understand if I'm approaching this right. Maybe I'm even approaching this entirely incorrectly (for instance, CRUD vs. REST? Does it matter here? As I understand it, REST implies that each of the objects above are resources with their own unique URI? So would JSON rendered be impacted?). I really am not sure. But ultimately, I need to use the JSON structure properly pull data into my frontend. Assumably, whatever said structure is will be mirrored and rendered in the backend.
Edit* thank you guys for the replies. I think part of my question, where I clarify "complex", is missing. So I'd like to explain. I guess more than the JSON itself, I really mean the structure of the data. For instance, in my example, I am structuring my data to all be beneath two unique objects (user and content). Is this correct? Or should I think about my data more diverse? For instance, technically I could have a comments database (where each comment is the main object). Or is that still implied in my dataset? Perhaps my question isn't even about JSON as much as it is the data structure which will happen to get rendered in JSON. Hopefully this clarifies what I mean by complex?
Any and all help is appreciated.
I'm not sure why you're making what seems to be objects into single-item arrays (as evidenced by the opening square brackets). Other than that, it looks fine to me. Generally speaking single items (like "User") are structured as an object and multiples are arrays of objects.
As for the Angular stuff, if you want to pull direct from a JSON file as a test, take a look here:
var services = angular.module('app.services', [])
services.factory('User', function($http) {
var User = function(data) {
return data;
}
User.findOne = function(id) {
return $http.get('/test_user.json').then(function(response) {
return new User(response.data);
});
};
return User;
});
I also recomment looking into Deployed for doing development without access to live data services.