I'm trying to scrape a site that uses lots of ajax effects to show data in a table.
There is some data returned via JSON when you interact with the site.
I know the URL and how to construct it but the server returns a HTTP 410 status if I try and re-request this JSON (I guess the server is expiring the data).
I have one chance to capture the data and I'm looking for a jQuery function, something like onJSONResourceReceived would be nice so that I can catch the response and store it in a variable.
Either a callback or a way to cache the data in a variable would be great.
Or if there is already a variable that stores all JSON resource already received in memory, that is even better.
All the functions I've looked at are for situations where you know or can re-request the URL.
This question is similar but for CasperJS:
How to get the response after a POST request in CasperJS
Look at the $.ajaxSuccess
Attach a function to be executed whenever an Ajax request completes
successfully.
$(document).ajaxSuccess(function( event, request, settings ) {
});
Related
I have a web page which allows users to upload and process specific files. After an user uploads some files, after clicking the 'Process' button an ajax call is being sent to a backend service. In the beforeSend function there is an overlay applied to the screen and a spinner is displayed. When the success function is triggered, then the overlay is removed and a toast notification is being shown like 'Files were processed!'
My goal is to somehow show a progress status for each file based on specific checkpoints in the backend service.
Let's say that the backend service when called does following tasks: parse file, map to specific format, send data to database A.... and in the end it sends back http status 200 and a JSON like
{
"status":"Success",
"message": "File X was processed"
}
Now what I want is that instead of just getting an overlay and disabling the whole page until the success event is triggered, to have a progress bar which is updated for each file based on the exact step where the backend has reached.
For instance, for file A, I would like to see below transitions: 5 % Parsing file, 10 % Mapping file...90% sending data to database, 100% processed.
Is this somehow achievable?
There are few points that you need to look into.
Usually in production code, we need to have timeouts. If you are making an ajax call to the backend API, there will be a timeout associated with that api call. Suppose if the timeout is more than 2 mins, then it will send you a 504 Gateway timeout error.
To overcome this and to implement the functionality which you want, you can have any DB(lets consider SQL server). In your SQL server, make a table:
Process_Table
With schema:
Process_id( Will store the process id/name )
Percentage( Will store the percentage )
At_step ( Parsing, Mapping, Sending to DB etc)
Using Javascript(Or any framework of your choice), use setInterval(), to make check_process() api calls. For check_proceess, you can pass in the process_id, and check against the db. For interval, you can set it to 5 seconds. So that every 5 seconds the call is made.
You can read the response of those API calls and do your processing.
An HTTP request consists of a request and a response. There's no direct way to get status updates beyond the onprogress event which would let you see how much data has been transferred. This is useful for determining how much of the data has been sent to the server, but not so much for how far the server has got with working with that data.
You could store progress in a database and poll a webservice to read the most recent status.
You could also have the server push updates to the client using Websockets for bi-directional communication.
A rough outline for such a system might look like:
Open a Websocket
Send files with Ajax
Get server generated ID back in HTTP response
Pay attention to messages coming over the Websocket that mention that ID
You could also look at doing the whole thing over Websockets (i.e. upload the files that way too). A quick Google search turns up this library for uploading files to a Websocket service hosted on Node.js.
I just get started in Ajax and httpRequest. While I was playing around with the code, I noticed that $.get works fine but $.post doesn't work. Here's my code:
$(document).ready(function() {
$.post('hello.txt', function(data) {
alert(data);
}).fail(function() {
alert('fail');
});
});
It always gives me a fail, and I cannot figure it out.
Thanks
Barmar is correct in the comments, but for an answer, let's go over what it is these functions are doing.
When you're using the jQuery AJAX methods, they are performing HTTP requests to the resource you're providing in the url parameter for the function. As long as the value is something sitting on your server (an endpoint) the function will hit it.
$.get() performs an HTTP GET action which is how we'd fetch data over HTTP. In your example, you specify hello.txt as the url, which as long as that is a file sitting on your server, the application will make a GET request to that resource. If it is found, the contents of that resource are returned. This can be done with a text file, a JSON payload, HTML web pages, etc. As long as the resource has returnable content, it will return that content.
$.post(), on the other hand, performs an HTTP POST action which sends data up to a resource to be processed. A POST action is not intended to fetch a resource's data, but to push data into it. Canonically, you would use a POST to create something with the data you push to the resource (as opposed to PUT for modifying and DELETE for removal, but that's beyond this answer).
So, the GET works because the action is intended to fetch data and the resource you provided has data to return. The POST fails because it is intended to give data to the resource to process, which your text file is not equipped to handle.
Hope this sheds a bit of light on the problem.
I have a scenario and cant find reasonable answer. So posting the question directly !
Scenario -
I make a ajax request and send a file to be uploaded to the server.
On server(tomcat), I download the file and start processing it.
I run the download-file method on main thread and process-file method on new thread since it takes too long for processing and I dont want user to wait so long.
My Issue :
The Ajax response will catch the response in call back method for download-file method.
My process-file method also return the status response which I require to display on front-end when the processing is complete.
But I cannot because Runnable Interface has return type void for its run method.
Question -
How can I get result from process-file method and send in as response to front-end
If I am able to acheive above, how can I catch the response on front end javascript since main thread response will be caught in ajax success method already.
First: You can get result from process-file method by using Callable along with ExecutorService. Here is an example. And return to the front end is explaining below.
Second: As you do not want to wait user for process-file method so that you already return from main thread and you can not get second return by one single request. The following process make you to understand for better and easier solution for this situation.
Assign a id (e.g. UUID) for the process-file method. Pass it to the process-file method and return the id from main thread to front-end if successfully downloaded.
Then after processing your file, store in DB/wherever you want or put result in a Map<id, result> where id will be the key.
In front end you get the id for the process-file method. You can do call ajax request for certain interval to get the result of this id to a different endpoint (e.g. /file/status/{id}).
In that endpoint for retrieving result, you can get result from Map by id which is key of the Map or if you store in DB then you can easily get by the id and return it to front end.
In case you can not get result in Map/DB (in case of the file-processing not finished yet) you can send exception to frontend so that you can send request again after next interval.
I have an AJAX call that is running a long PHP script where it has 20 different results, I would like to show when each step in the script is done.
Like so 1/20 done, 2/20 done, 3/20 done.
Latest 29-12-2015 03:17.
I tried to create the JSON file as so (jsonFileName_uniqueTimeStampHere.json) by PHP, but the time taken to create the file with PHP, result in a 404 file not found error!
Because when the AJAX call is running it comes to the progress call before the file has been created, I know I can't create the file with JavaScript but is there anyway to create.
The file before the success callback from jQuery AJAX?
What would be the best way to show progress information while AJAX call is running.
The way I have it now, I have a JSON file saved on the server that gets updated with the latest state that has completed, but if multiple users is running the same script the JSON file with the state gets overwritten.
Should I save the state of each progress in DB and then receive it with multiple calls to a PHP method that get state that has been completed?
Should I keep using the current method I use and add a userID to the JSON file so it is unique on each call to the file?
How should I go about doing it the same way as Seositecheckup?
What is the best way to make a progress with AJAX and PHP?
Please tell me if you need any more information.
I have looked around and don't feel like the info or half of info, there is to find online has been enough to do this myself.
I would like to use jQuery AJAX and not XMLHttpRequest, I'm looking for something similar to seositecheckup.com, when you scan a page you can see the state update on each completed function in the console and is done with different AJAX calls. How is that possible?
Should I forget about using jQuery and keep focus on plain JavaScript instead?
Right now I have a setup with jQuery that works the problem is, that I use a JSON file to get the result from and it gets overwritten when multiple users request the same script, is it possible to store the state in db instead and receive it from there with some unique identifier?
In the future I would like to make it possible to put the script into a queue that could be run and when the script ends it should send an e-mail to the user.
The HTTP way of handling requests that may take a long time is for requests to return a 202 and the body of the response should contain the URL where the user can query for the result.
#Request
POST /some/entitities
...
#Response
HTTP/1.0 202 Accepted
/jobs/{jobId}
The user can then poll /jobs/{jobId} which can return a number to represent progress. Do you have to use this? No, but if you do, others developers can immediately recognize what is going on.
Even if you don't use the approach I recommend, you will also have to keep track of job progress in your database and have a separate AJAX call to find out the current progress.
I have an application in which most requests are submitted via AJAX, though some are submitted via "regular" HTTP requests. If a request is submitted and the user's session has timed out, the following JSON is returned:
{"authentication":"required"}
The JavaScript function which submits all AJAX requests handles this response by showing a popup message and redirecting the user back to the login page.
However, when a non-AJAX request receives this response the JSON is simply shown in the browser because the response is processed directly by the browser (i.e. the aforementioned JavaScript function is bypassed). Obviously this is not ideal and I would like the non-AJAX requests that receive this response to behave the same as the AJAX requests. In order to achieve this, I can think of 2 options:
Go through the application and convert all the requests to AJAX requests. This would work, but could also take a long time!
The JSON shown above is generated by a very simple JSP. I'm wondering if it might be possible to add a JavaScript event handler to this JSP which is run just before the content is displayed in the browser - I'm assuming this would never be called for AJAX requests? This handler could call the other JavaScript code that displays the popup and performs the redirection.
If anyone knows how exactly I can implement the handler I've outlined in (2), or has any other potential solutions, I'd be very grateful if they'd pass them on.
Cheers,
Don
3) Change your AJAX code to add a variable to the GET or POST: outputJson=1
You cannot add a handler to the JSP that way. Anything you add to it will make it a non-JSON producing page.
There are two options that I can see:
Add a parameter to the page by appending a URL parameter to the screen that modifies the output.
URL: http://domain/page.jsp?ajaxRequest=true
would output json only
URL: http://domain/page.jsp
would display a jsp page that could forward to another page.
OR
change the response to have the forwarding code in the JSP that will get executed by the web browser if it is hit directly. Then have your calling AJAX to strip the forwarding code out, and then process what is left.
4) Read up on the 'Accept' request HTTP header.
Then, on the server side tailor the output:
e.g.
if(Accept contains application/json...) { // client asking for json, likely to be XHR
return {"foo":"bar"}
} else { // other
return "Location: /login-please";
}
Start with a smarter error message, like this:
{"error":"authentication required"}
Wrap the JSON output in a callback:
errorHandler({"error":"authentication required"});
Have a handler waiting in your script:
function errorHandler(r) {
alert(r.error);
}
And don't forget to send it down as text/javascript and not application/x-json.