We Keep Coding

Making really simple app including front & backend skills (js, node.js, psql, react...)

I'm trying to make a simple todo app in order to understand how frontend and backend are connected. I read some of the websites showing a tutorial for using and connecting rest API, express server, and database, but still, I was not able to get the fake data from a database. Anyway, I wanted to check if my understanding of how they are connected and talk to each other is correct or not. So could you give some advice please?
First of all, I'm planning to use either Javascript & HTML or React for frontend, Express for server, and Postgres for the database. My plan is a user can add & delete his or her task. I have already created a server in my index.js file and created a database using psql command. Now if I type "" it takes me to the page saying "Hello" (I made this endpoint), and I'm failing to seed my data to the database. Here are my questions↓
After I was able to seed my fake data into the database, how should I get the data from the database and send to the frontend? I think in my index.js file, create a new endpoint something like "app.get("/api/todo", (res, req) => ..." and inside of the callback function, I should write something like "select * from [table name]". Also, form the front end, I should probably access certain endpoints using fetch. Is this correct?
Also, how can I store data which is sent from the frontend? For example, if I type my new todo to <input> field and click the add <button>, what is the sequence of events looks like? Adding event listener to button and connect to the server, then create post method in the server and insert data, kind of (?) <= sorry this part it's super unclear for me.
Displaying task on the frontend is also unclear for me. If I use an object like {task: clean up my room, finished: false (or 0 ?)} in the front end, it makes sense but, when I start using the database, I'm confused about how to display items that are not completed yet. In order to display each task, I won't use GET method to get the data from the database, right?
Also, do I need to use knex to solve this type of problem? (or better to have knex and why?)
I think my problem is I kind of know what frontend, server, database for, but not clear how they are connected with each other...
I also drew some diagrams as well, so I hope it helps you to understand my vague questions...

how should I get the data from the database and send to the frontend?
I think in my index.js file, create a new endpoint something like
"app.get("/api/todo", (res, req) => ..." and inside of the callback
function, I should write something like "select * from [table name]".
Typically you use a controller -> service -> repository pattern:
The controller is a thin layer, it's basically the callback method you refer to. It just takes parameters from the request, and forwards the request to the service in the form of a method call (i.e. expose some methods on the service and call those methods). It takes the response from the service layer and returns it to the client. If the service layer throws custom exceptions, you also handle them here, and send an appropriate response to the client (error status code, custom message).
The service takes the request and forwards it to the repository. In this layer, you can perform any custom business logic (by delegating to other isolated services). Also, this layers will take care of throwing custom exceptions, e.g. when an item was not found in the database (throw new NotFoundException)
The repository layer connects to the database. This is where you put the custom db logic (queries like you mention), eg when using a library like https://node-postgres.com/. You don't put any other logic here, the repo is just a connector to the db.
Also, form the front end, I should probably access certain endpoints
using fetch. Is this correct?
Yes.
Also, how can I store data which is sent from the frontend? For
example, if I type my new todo to field and click the add , what is
the sequence of events looks like? Adding event listener to button and
connect to the server, then create post method in the server and
insert data, kind of (?) <= sorry this part it's super unclear for me.
You have a few options:
Form submit
Ajax request, serialize the data in the form manually and send a POST request through ajax. Since you're considering a client library like React, I suggest using this approach.
Displaying task on the frontend is also unclear for me. If I use an
object like {task: clean up my room, finished: false (or 0 ?)} in the
front end, it makes sense but, when I start using the database, I'm
confused about how to display items that are not completed yet. In
order to display each task, I won't use GET method to get the data
from the database, right?
If you want to use REST, it typically implies that you're not using backend MVC / server rendering. As you mentioned React, you're opting for keeping client state and syncing with the server over REST.
What it means is that you keep all state in the frontend (in memory / localstorage) and just sync with the server. Typically what is applied is what is referred to as optimistic rendering; i.e. you just manage state in the frontend as if the server didn't exist; yet when the server fails (you see this in the ajax response), you can show an error in the UI, and rollback state.
Alternatively you can use spinners that wait until the server sync is complete. It makes for less interesting user perceived performance, but is just as valid technical wise.
Also, do I need to use knex to solve this type of problem? (or better
to have knex and why?) I think my problem is I kind of know what
frontend, server, database for, but not clear how they are connected
with each other...
Doesn't really matter what you use. Personally I would go with the stack:
Node Express (REST), but could be Koa, Restify...
React / Redux client side
For the backend repo layer you can use Knex if you want to, I have used node-postgres which worked well for me.
Additional info:
I would encourage you to take a look at the following, if you're doubtful how to write the REST endpoints: https://www.youtube.com/watch?v=PgrP6r-cFUQ

After I was able to seed my fake data into the database, how should I get the data from the database and send to the frontend? I think in my index.js file, create a new endpoint something like "app.get("/api/todo", (res, req) => ..." and inside of the callback function, I should write something like "select * from [table name]". Also, form the front end, I should probably access certain endpoints using fetch. Is this correct?
You are right here, you need to create an endpoint in your server, which will be responsible for getting data from Database. This same endpoint has to be consumed by your Frontend application, in case you are planning to use ReactJS. As soon as your app loads, you need to get the current userID and make a fetch call to the above-created endpoint and fetch the list of todos/any data for that matter pertaining to the concerned user.
Also, how can I store data which is sent from the frontend? For example, if I type my new todo to field and click the add , what is the sequence of events looks like? Adding event listener to button and connect to the server, then create post method in the server and insert data, kind of (?) <= sorry this part it's super unclear for me.
Okay, so far, you have connected your frontend to your backend, started the application, user is present and you have fetched the list of todos, if any available for that particular user.
Now coming to adding new todo the most minimal flow would look something like this,
User types the data in a form and submits the form
There is a form submit handler which will take the form data
Check for validation for the form data
Call the POST endpoint with payload as the form data
This Post endpoint will be responsible for saving the form data to DB
If an existing todo is being modified, then this should be handled using a PATCH request (Updating the state, if task is completed or not)
The next and possibly the last thing would be to delete the task, you can have a DELETE endpoint to remove the todo item from the list of todos
Displaying task on the frontend is also unclear for me. If I use an object like {task: clean up my room, finished: false (or 0 ?)} in the front end, it makes sense but, when I start using the database, I'm confused about how to display items that are not completed yet. In order to display each task, I won't use GET method to get the data from the database, right?
Okay, so as soon as you load the frontend for the first time, you will make a GET call to the server and fetch the list of TODOS. Store this somewhere in the application, probably redux store or just the application local state.
Going by what you have suggested already,
{task: 'some task name', finished: false, id: '123'}
Now anytime there has to be any kind of interaction with any of the TODO item, either PATCH or DELETE, you would use the id for each TODO and call the respective endpoint.
Also, do I need to use knex to solve this type of problem? (or better to have knex and why?) I think my problem is I kind of know what frontend, server, database for, but not clear how they are connected with each other...
In a nutshell or in the most minimal sense, think of Frontend as the presentation layer and backend and DB as the application layer.
the overall game is of sending some kind of request and receiving some response for those sent requests. Frontend is what enables any end-user to create these so-called requests, the backend (server & database) is where these requests are processed and response is sent back to the presentational layer for the end user to be notified.
These explanations are very minimal to make sure you get the gist of it. Since this question almost revolves around the entire scope of web development. I would suggest you read a few articles about both these layers and how they connect with each other.
You should also spend some time understanding what is RESTful API. That should be a great help.

Get an http call from express to angular when event happens

i'm building an angular app that will make about a thousand people to connect simultaneously to book a ticket. I want only "XYZ" of them to access simultaneously at the registration Angular component. The other ones will see a "waiting room" component until it's their turn.
I set up the whole thing like this:
User enters the page.
I make an http call to expressjs server
The server checks if the "connections" collection constains less than XYZ docs
If true, it unlocks the user registation component and with an http post req, it creates a new doc in the db. if false it leaves it hidden and shows up the waitingroom component
When user leaves the page, his doc in "connections" collection gets destroyed with an http delete call.
Fully working.
The problem is now that i want to create a kind of "priority" system, because, going like that, if you just refresh you may be lucky and get access, even if you are soon arrived and there is who is waiting since 1990's. So i introduced a "priority" system. When the user makes the first http call, if user is not allowed, the server creates a timestamp and pushes it into an array.
const timestamps = []
.
.
.
// this below is in http get req
Connessione.countDocuments({},(err,count)=>{
if(count<=nmax){
console.log("Ok")
res.status(200).json({allowed: true})
}
else{
const timestamp = req.params.timestamp;
timestamps.push(timestamp);
console.log("Semo troppi")
res.status(401).json({allowed: false})
}
});
The idea is to listen to db changes, and when there is just XYZ-1 in the db. Make a call to the first timestamp's angular frontend to say him: "Hey there, if you want we're done. You can go" and unlock him the access to registration component.
The problem is that i can't make continuous http requests every second from angular until there's a free place...
Is there any method to send a request at the server, and when server says OK, calls angular and says "Hey dude. You can go!"?
Hope you understood my question. If not ask me in the comments.
Thanks in advance

Even i had trouble with sockets in the beginning so i'll try to explain the concept in a simple way, Whenever you write an API or Endpoint you have a one way connection i.e. you send request to server and it return back some response as shown below.
Event 1:
(Client) -> Request -> (Server)
Event 2:
(Client) <- Response <- (Server)
For API's, without request you cannot get response.
To overcome this issue as of now i can think of two possible ways.
Using Sockets, With sockets you can create a two way connection. Something like this
(Server) <-> data <-> (Client)
It means you can pass data both ways, Client to server and Server to client. So whenever an event occurs(some data is added or updated in database) one can emit or broadcast it to the client and the client can listen to the socket and receive it.
In your case as it's a two connection you can emit the data from angular and
I've attached few links at the bottom. please have a look.
Using XML/AJAX Request, This is not a preferable method, using setInterval you can call the server in every 5 seconds or so and do the operation needed.
setInterval(ajaxCall, 5000); //5000 MS == 5 seconds
function ajaxCall() {
//do your AJAX stuff here
}
Links:
https://socket.io/docs/
https://alligator.io/angular/socket-io/

how to await subscriptions established?

I have the following js code:
stompClient.subscribe('/topic/clients', function (calResult) {
updateClientsTable(JSON.parse(calResult.body));
});
$.get("/clients", null);
and following server code(last line invokes it):
#GetMapping(value = {"/clients"})
#ResponseBody
public void loadClients() {
brokerMessagingTemplate.convertAndSend("/topic/clients", clientService.getClientList());
}
Sometime front-end misses result of $.get("/clients", null);
As I understand problem: at the moment of result getting on front end, subscriptions is not happens.
if to put $.get("/clients", null); below in the code - all works fine.
Can you explain how to await subscriptions established?

I think it would make more sense to not mix REST requests with this messaging pattern.
Have you considered sending the "updateClients" command through SockJS into an "/apps/updateClients" channel which replies to the "/topic/clients" channel?

As #light_303 already mentioned, mixing HTTP requests with notification mechanism isn't good. You can register moment, when client connects (GET request on /clients), but you can't register when he disconnects.
You should think in one of the next ways. When user subscribes to /topic/clients:
You individually send him response with all client list and then push updates only.
You individually send him current server time or some kind of ID and then push updates only. User uses given time/ID in GET request to /clients and receives full client list on that moment. This option can be good in situation, when you have incremental updates (i. e. adding new elements to list) and otherwise not so good.
Check this question: Sending message to specific user on Spring Websocket.
This is actually ridiculous, how Spring can complicate things. I recommend you to look on another frameworks for real-time web communication, such as Vert.x or Netty and on Go programming language. Use WebSockets or SockJS instead of STOMP. All that technologies can give you more flexible and performant solution in obvious way. Also, check Centrifugo project, maybe it's relevant to your task.

You can use #SubscribeMapping annotation from spring-messaging.
If you have spring-messaging configured as described here and here, the server-side code could look like following:
#Controller
public class MessagingController {
#SubscribeMapping("/clients")
public List<Client> loadClients() {
return clientService.getClientList();
}
}
This way you don't have to call $.get("/clients", null); because JS message handler receives result of loadClients() call right after subscription happens. JS code would look like:
stompClient.subscribe('/topic/clients', function (calResult) {
updateClientsTable(JSON.parse(calResult.body));
});

Meteor: How to prevent client from accessing methods

All meteor methods can be called same way from client and server side.
Let's say user knows or can predict all the method names on server, then he is able to call them and use it's result however he want.
example:
A method which performs cross domain http request and return response can be used to overload server by calling huge amounts of data Meteor.call(httpLoad, "google.com");, or a method which load data from mongo can be used to access database documents if the client know document _id Meteor.call(getUserData, "_jh9d3nd9sn3js");.
So, how to avoid this situations, may be there is a better way to store server-only functions than in Meteor.methods({...})?

Meteor methods are designed to be accessed from the client, if you don't want this, you just need to define a normal javascript function on the server. A really basic example would be:
server/server.js:
someFunction = function(params) {
console.log('hello');
}
As long as it's in the server folder, the function won't be accessible from the client.
For coffeescript users, each file is technically a separate scope, so you would have to define a global variable with #, e.g.
#someFunction = (params) ->
console.log 'hello'
or if you want to scope the function to a package:
share.someFunction = (params) ->
console.log 'hello'
If you have methods that need to be accessible from the client but only for say admin users, you need to add those checks at the start of the meteor method definition:
Meteor.methods({
'someMethod': function(params) {
var user = Meteor.user();
if (user && (user.isAdmin === true)) {
// Do something
} else {
throw new Meteor.Error(403, 'Forbidden');
}
}
});
I'm not going to vouch for the security of this example - it's just that, an example - but hopefully it gives you some idea of how you would secure your methods.
EDIT: Noticed the other answers mention using a if (Meteor.isServer) { ... } conditional. Note that if you are doing this inside methods which are also accessible on the client, the user will be still be able to see your server code, even if they can't run it. This may or may not be a security problem for you - basically be careful if you're hardcoding any 3rd-party API credentials or any kind of sensitive data in methods whose code can be accessed from the client. If you don't need the method on the client, it would be better to just use normal JS functions. If you're wrapping the whole Meteor.methods call with a isServer conditional, the code will be on the server only, but can still be called from the client.

as rightly stated in other answers, your methods will always be accessible from the client (per design). yet, there is a simple workaround to check if the call originates from the client or from the server. if you do a
if ( this.connection == null )
this will return true if the method was called from server. like that you can restrict the method body execution to 'secure' calls.

I think this page explains it: http://meteortips.com/first-meteor-tutorial/methods/
I'm quoting:
"The safer approach is to move these functions to the isServer conditional, which means:
Database code will execute within the trusted environment of the server.
Users won’t be able to use these functions from inside the Console, since users don’t have direct access to the server.
Inside the isServer conditional, write the following:
Meteor.methods({
// methods go here
});
This is the block of code we’ll use to create our methods."
and so on. I hope this helps.

With proper app design, you shouldn't care whether a request was through the web UI or via something typed in a console window.
Basically, don't put generic, abuse worthy functions in Meteor.methods, implement reasonable access controls, and rate-limit and/or log anything that could be a problem.
Any server-side function defined in Meteor.methods will have access to the current user id through this.userid. This userid is supplied by Meteor, not a client API parameter.
Since that Meteor Method server-side code knows the login status and userid, it can then do all the checking and rate limiting you want before deciding to do that thing that the user asked it to do.
How do you rate limit? I've not looked for a module for this lately. In basic Meteor you would add a Mongo collection for user actions accessible server-side only. Insert timestamped, userid specific data on every request that arrives via a Meteor method. Before fulfilling a request in the server method code, do a Mongo find for how many such actions occurred from this userid in a relevant period. This is a little work and will generates some overhead, but the alternative of rate-limiting via a server-wide underscore-style debounce leaves a function open for both abuse and denial-of-service by an attacker.

How to avoid too many ajax calls and cache json data on the client side

I have a calendar application and it loads all of the event data using ajax and json results. the issue is that i have different view and right now i have to re call the server when i change views.
Is there any recommendation for ways i can cache this data on the client side and check if i have loaded these events already before firing off more ajax calls.
What is the best practice for this ?

Like hvgotcodes said, an MVC framework would help; try backbone.js (http://documentcloud.github.com/backbone/), for instance.
Alternatively, you might want to consider using jStorage (http://www.jstorage.info/). Every time you need to make an AJAX call, check first if it's in your storage object, then run the AJAX call if it isn't. On the other end, whenever you finish an AJAX call, store the results in the storage object. Make sure you have some kind of index (a CalendarEvent id) to reference when looking it up in the data store. Might want to add some kind of "expire time" to the data in your storage, too ... a timestamp after the AJAX call, and re-request up front if it's out of date.

It's called MVC.
You need to construct a data model for you application, write some sort of Record objects, and then you can determine their status. So your application would have some sort of CalendarEvent model, and when you load data from the server, you would instantiate instances.
So when changing views, you would first check to see if you had the model object for that view, and if you did, you wouldn't need to load it from the server (unless you want to check for changes).
Your scheme doesn't need to be that complicated. If you load events by Id, you can do something like
window.App = {};
window.App.Models = {};
when you load a record you could put
window.App.Models[id] = InstanceOfYourRecord
and that way its pretty fast to look for records. Or just use a framework (like Sproutcore) that has a robust data layer.

I had similar issues on a recent project.
Conceptually, I have the "real" data model (DM) kept on the server, persisted to a database.
To make life sane, the client keeps its own local data model. Outside of the client DM, all the client code thinks it's pulling results locally.
When reading data (GET) from the client DM it:
checks the cache for existing results
invokes appropriate AJAX queries when cached data is not available, then caches the results.
When changing data (POST) via the client DM it:
invalidates the cache as appropriate
invokes appropriate AJAX queries
emits custom jQuery event indicating client DM changed
Note that this client DM also:
centralizes AJAX error handling
tracks AJAX calls still in-flight. (Lets us warn users when leaving pages with unsaved changes).
allows a drop-in, dummy replacement for unit testing, where all the calls hit local data and are completely synchronous.
Implementation notes:
I coded this as a JavaScript class called DataModel. As the design becomes more complex, it makes sense to further break-down the responsibilities in to separate objects.
jQuery's custom events let you easily implement the observer pattern. Client components update themselves from the client DM whenever it indicates data has changed.
JSON in your remote API helps simplify the code. My client DM stores the JSON results directly in its cache.
The client dm function arguments include call-backs so everything can naturally be passed along via AJAX when needed: function listAll( contactId, cb ) { ... }
My project only allowed single user logins. If outside parties can change the server datamodel, some sort of has-data-changed probe should be fired regularly to ensure the client cache is still valid.
For my app, multiple client components would request the same data when receiving a client DM changed event. This resulted in multiple AJAX calls with the same info. I fixed this problem with a getJsonOnce() helper, which manages a queue of client component call-backs awaiting the same result.
Example function in my implementation:
listAll:
function( contactId, cb ) {
// pull from cache
if ( contactId in this.notesCache ) {
cb( this.notesCache[contactId] );
return;
}
// init queue if needed
this.listAllQueue[contactId] = this.listAllQueue[contactId] || [];
// pull from server
var self = this;
dataModelHelpers.getJsonOnce(
'/teafile/api/notes.php',
{'req': 'listAll', 'contact': contactId},
function(resp) { self.notesCache[contactId] = resp; },
this.listAllQueue[contactId],
cb
);
}
The getJsonOnce() helper makes sure that if multiple client components request the exact same (uncached) data, that we only send out a single AJAX request and inform everyone once it comes in.
The notesCache is just a simple javascript object:
this.notesCache = {};