We Keep Coding

Get ajax call progress status in front end

I have a web page which allows users to upload and process specific files. After an user uploads some files, after clicking the 'Process' button an ajax call is being sent to a backend service. In the beforeSend function there is an overlay applied to the screen and a spinner is displayed. When the success function is triggered, then the overlay is removed and a toast notification is being shown like 'Files were processed!'
My goal is to somehow show a progress status for each file based on specific checkpoints in the backend service.
Let's say that the backend service when called does following tasks: parse file, map to specific format, send data to database A.... and in the end it sends back http status 200 and a JSON like
{
"status":"Success",
"message": "File X was processed"
}
Now what I want is that instead of just getting an overlay and disabling the whole page until the success event is triggered, to have a progress bar which is updated for each file based on the exact step where the backend has reached.
For instance, for file A, I would like to see below transitions: 5 % Parsing file, 10 % Mapping file...90% sending data to database, 100% processed.
Is this somehow achievable?

There are few points that you need to look into.
Usually in production code, we need to have timeouts. If you are making an ajax call to the backend API, there will be a timeout associated with that api call. Suppose if the timeout is more than 2 mins, then it will send you a 504 Gateway timeout error.
To overcome this and to implement the functionality which you want, you can have any DB(lets consider SQL server). In your SQL server, make a table:
Process_Table
With schema:
Process_id( Will store the process id/name )
Percentage( Will store the percentage )
At_step ( Parsing, Mapping, Sending to DB etc)
Using Javascript(Or any framework of your choice), use setInterval(), to make check_process() api calls. For check_proceess, you can pass in the process_id, and check against the db. For interval, you can set it to 5 seconds. So that every 5 seconds the call is made.
You can read the response of those API calls and do your processing.

An HTTP request consists of a request and a response. There's no direct way to get status updates beyond the onprogress event which would let you see how much data has been transferred. This is useful for determining how much of the data has been sent to the server, but not so much for how far the server has got with working with that data.
You could store progress in a database and poll a webservice to read the most recent status.
You could also have the server push updates to the client using Websockets for bi-directional communication.
A rough outline for such a system might look like:
Open a Websocket
Send files with Ajax
Get server generated ID back in HTTP response
Pay attention to messages coming over the Websocket that mention that ID
You could also look at doing the whole thing over Websockets (i.e. upload the files that way too). A quick Google search turns up this library for uploading files to a Websocket service hosted on Node.js.

Express.js how to make html render on client side with response.render?

I have basic express js application with following route:
router.get('/', function(req, res){
res.render('login');
});
It works fine - after logging into main page on my localhost, html from login.pug is nicely rendered on client side. However when my app runs, sometimes I want to render another pug file, when there already is html rendered on client side:
app.get('/dashboard', function(req, res){
res.render('dashboard', {user: req.query.login});
});
But when get request is send on'dashboard'path, nothing happens. As I understand, this happens because res.render just parses pug file into HTML and sends it to client as plain string (which I can inspect in browser developers tool, when I check AJAX response I see exactly rendered HTML).
Now my question is: is there a way to render HTML from res.render in client automatically? So on server side I just write res.render('template') and on client side page is re-rendered without handling the response?
I know I can clear whole DOM and append received string into document.body, I know also that I can make a POST form request and then page will be re-rendered automatically, but I want to avoid both solutions (don't ask why).
Thank you for help in advance.

When your Javascript sends an Ajax request, the response from that Ajax request is just returned back to your Javascript. That's it. The browser does not show anything. If you want to do something with that response, then it is the responsibility of your Javascript to do something with it (insert it in the page to show it, etc...).
If what you really want is that you want the browser to actually go to the /dashboard URL and then show that page, then your Javascript can just do:
window.location = '/dashboard';
This will tell the browser to fetch the contents of that URL, it will make a request to your server, your server will return the rendered HTML and the browser will display it in the page and show /dashboard as the URL in the browser bar. That should do everything you want.
So, it's totally up to your Javascript. Pick the right tool for the job. Either fetch data for your Javascript with an Ajax call and process the result in your Javascript or instruct the browser to load a new page. One or the other.
But when get request is send on'dashboard'path, nothing happens. As I understand, this happens because res.render just parses pug file into HTML and sends it to client as plain string (which I can inspect in browser developers tool, when I check AJAX response I see exactly rendered HTML).
Yes, that what Ajax requests do. They fetch content from the server and return the data back to your Javascript (and only to your Javascript).
is there a way to render HTML from res.render in client automatically?
Yes, use window.location = '/dashboard'; from your Javascript.
So on server side I just write res.render('template') and on client side page is re-rendered without handling the response?
Not from an ajax call. Ajax calls never automatically display content. Never. Ajax calls are programmatic requests that return data to your script. That's what they are. But, yes from Javascript you can cause content to be automatically displayed by setting window.location to a new URL.

load remote content that need authentication?

This is the scenario :
-User is logged in the remote website (via iframe)
-With that cookie saved the user is able to make a get request and obtain a table
-This table is loaded in a second iframe
Problem : I would like to apply styles to that received information.
The output of the request is a simple html file with a table , I cannot access that file from a server (php for example) , because I have no control over the remote site , and in order to make that request user need to be logged (and as said what I do is load an iframe with the get request response..) .
The only thing that comes to my mind is to parse that response , but without a server-side tool , I do not think this is approachable. Any idea?

Send request to download file, catch response in Javascript

In a Rails2 Webapp, I am creating in javascript a form to forge a request to download a file. In the controller, I render the response using send_file so the file is downloaded.
There is a rather long process going on looking up the file to download and meanwhile I have the client waiting for the file to start being downloaded. For this reason, as soon as the client clicks the download button I display a small massage saying "Requesting file...".
Problem:
How can I (on javascript side) know when the response is rendered (aka the file starts to download) so I can hide the message I am displaying? I use ajax everywhere and I throw the showMessage method in the "before" part of the call and the hideMessage in the "complete" part of the jquery ajax call. But since I cannot request a file to be downloaded with an ajax call I have to do it trough a form but then I don't know how to realize when I get the repsonse back (or the file starts to download)

The basic idea is to send a cookie along with the file. Some javascript code checks the cookie periodically. When javascript gets the cookie, you know the downloading begins.
You can find sample javascript code here (server code in Java) and here (server code in ASP.NET).
In Rails, the cookie could be sent like this:
cookies['download_token'] = params[:download_token]
send_data ...

Force browser download a large file

I'm using a POST request with body to dynamically create a specified file on the server side and streaming it to the client as a response. The problem on the client side is that the file is being downloaded into the response object (browser's cache memory I suppose, as a temporary file) instead of the PC's hard drive.
How can I pass the byte stream directly for the browser to handle and direct into the /Downloads folder or wherever the user has set the downloads go?
The file size is really large and there's not enough space on the gadget's hard drive to create it there and give a link for GET request. So I need to create it on the fly and stream to the client as a POST response.
Right now I've written my code in Angular, but the language doesn't really matter. I'd like to understand how the browser handles the situation.
When I execute the HTTP request, I bet the browser does something with it before it's passed to the socket. And I'm sure the browser does something with the received request too before passing it to the piece of code which performed the request.
I mean can you only by setting the response headers make the browser download the file into /Downloads? What happens then to the waiting HTTP request if the packets are "stolen" by the browser? Or when the response headers arrive to the response object, can you somehow notify the browser that the body of the response is a file byte stream we wish to download into /Downloads?
And I have the possibility to modify my server code too.
EDIT More detailed explanation:
My server is sending the 5GB file to the client. Instead of just receiving that file in the temporary memory of the browser I would like to save the file directly on hard drive.
But naturally the browser prevents the direct access to the users computer so I can't write it there. So how do I pass the byte stream for the browser to handle and save into a file?

Ok, I have solved the problem. So if you send a file request yourself to the server from your javascript code the browser will just pass the received response directly to your response object, but you can't store it on the hard drive without some external (browser supported) tools.
What I mean is that if you perform any kind of $http.post() or .get() from your scripts the response will be passed to the $http.post().then(function(response){...}) directly and it's passed only after the whole response is received. Meaning that if you're expecting a file with the size of 5GB it will fail for being too large to be received inside the response object.
Instead what you have to do is to trigger the download in another way. What I did in my code is that I built a hidden
<form method="POST" action="/set.uri.here/" id="download">...</form>
and when the user clicks the button to download the file, I run a function that builds the body of the form and then submits it. Which looks like
var form = document.getElementById("download");
// magic here... create the body for the form and so on
form.submit();
This way you can pass the request to be done by your browser so that then the response will be also handled by the browser. This is the crucial difference between doing the request yourself or by the browser. If you do the request yourself then the browser won't download the object but you will receive it yourself to do something with it.
If you wish the browser to download the object, then, make the browser do the request for the object as well!