Get an element from another page - javascript

How to get an element (opening and closing times) from this site https://www.costco.com/warehouse-locations/brooklyn-NY-318.html using javascript and display it on another page?
element = document.getElementsByClassName('core')[0].firstElementChild.childNodes[0].data
Thanks

Unfortunately, if you plan to do this with purely browser-based Javascript, CORS will likely stop you from requesting the page and pulling the data there. You can try looking for an API that Costo provides, or some other online tool that can get you that information that supports cross-origin requests.
The other solution is to create your own server script. Having a back-end like NodeJS pull the page with packages like axios and passing the resulting string over to jsdom. This way, you can run the same element script you wrote to get the data you were looking for. I can elaborate but I'm just not sure if Google Sites has this capability.

Related

How can i get DOM of specific page like "google.com"?

for this problem I need to say I use fetch method but some web, like google and ..
does not allow it.
I want to get DOM of a page for tracking it, for example : I wanna check if an element change let me know.
How can I get an specific web page DOM without exception, like: google in using fetch method?
If you have another solutions, may you tell me (With Java script)?
TLDR; You don't
You can insert a page via <iframe>, but it won't (for security reasons) let you access any of its data, thus nor the DOM.
You can't fetch the HTML of google.com either, it is because Google doesn't want to give you the HTML... You can, however, create a proxy server, because a nodeJS server can fetches the HTML of google.com. But then it will be simple HTML yet to be parsed...

Sending safely JSON to server via an HTML form page

I have an html page that has a form and is communicating to a server via JSON requests, when loading (e.g. in order to remember user's previous answers) and when submit button is pressed (e.g. in order to save his new answers). Every user has a specific user_id that is located as an attribute in the url of the website. I have only HTML pages, CSS and Javascript that makes some simple functions, as well as received and sends the requests.
The problem is that the server needs an api-key for the request to happen, that I need to be kept hidden, but instead is easily discovered when the user sees the source code. I want this specific line to be hidden, but I guess this is not possible without any backend in the game.
I would like to know the easiest and fastest way to get from this state (just frontend, where every piece of information in the source code is totally insecure) to another where the api-key (at least) is not on the open.
I guess the solution is to use a server for that part but I need suggestion on the easiest transition from my code to another. I tried to use GWT, as I am a bit more familiar with JAVA backend application (but not with GWT), but seems like a big trouble, where I need to change my HTML, my Javascript and also the CSS that I have may not be useful, as well as I face a lot of problems when trying to read my parameters.
I know that it is not the best way but I do not have a lot of time to make it work, so sorry if it seems lazy (I am new to this type of programming), but I haven't found anything helpful and I cannot study for 2 weeks in order to be able to begin implementing it.
Is node.js (which is Javascript and I already have implemented the request sending/receiving in this language) easier than GWT for that matter? Will my sensitive data be secure in that way? I would be grateful if there was a similar sample, that I could start using for my implementation, as I haven't find anything that is specifically helpful for my situation.
Thanks in advance!
NodeJs is not javascript, NodeJs is specific javascript "interpreter" whose is purpose is mainly to be executed server-side. If you have an HTML page, it is likely to be loaded in a web browser (client-side), so not in a NodeJs environnement.
Everything that is hard-coded in the javascript of you web page is visible from the client, there is no way around that. So yes, you need some server-ish thing somewhere.
If you are not to manage a server by yourself or via PaaS, you can go for a serverless architecture. For instance, If you consider AWS which I know the most, you can easilly add some user management to your web page using Aws Cognito User Pool. Once a user is connected and have the good permission, he can be granted access to some other resources via a JWT token that you send along with your request.

Simplest way to display API results

Im kind of new to this and looking to expand pulling API results and displaying them on page, whether it's from a blog resource or content generation.
For example, I want to pull from VirusTotal's API to display returned content. What is the best way to capture that in an input tag and display it in a DIV. And what if it were an option to pull from different API's based on drop down selection?
An example of the API to pull content would be here https://developers.virustotal.com/reference#api-responses under the /file/report section.
To call the data from the API, you need to send a request. However, there is a problem with CORS. Basically, you can't call the website from inside your web browser from a page on your local machine, because your browser blocks the request. The web browser will only allow calls to and from the same server, except for a few exceptions.
There's two ways to approach this.
The simplest one is to make a program that calls the API and outputs an HTML file. You can then open that HTML file to read the contents. If you want to update the info, you would need to run that program once again manually. You could easily do this building off the python they provided.
The other, little bit more complex way, is where you host a server on your PC. When you go to the webpage on that server, it sends a request to the website, and then provides the latest information. There's tons of frameworks and ways to do this. For an absolute beginner on this subject, ExpressJS is a good start. You can make a hello world program, and once you do that you can figure out how to call the API whenever a page is loaded, and display the results.

Is it possible to scrape a website with a login using purely javascript - on client side

I've managed to scrape websites that require no login using js only and a little help from websites that allow me to pass the CORS issues(like allorigins), but I just couldn't manage to get pass through the login problem.
I've seen many posts discussing of doing it using node.js and python beautifulsoup, but none on how to do it with javascript.
So how do I go about it?
Is it even possible doing it purely on client-side?
I'm willing to do all the learning and searching needed, but I need some direction in this vast subject.
Assuming you meant using in-browser JavaScript, how did you get around CORS? And if you did, then once the page refreshed after a successful login you code would stop running anyway unless you were a browser extension.
If you mean on your computer, then Node is what you're looking for, but unless you use a project like Headless Chrome then you'll run into the issue of saving the cookies between requests which is what keeps track of your session and actually keeps you logged in.
Login requires a direct interaction with your browser, like saving a cookie, returning a security token etc.
If you use JavaScript from a html page, it would theoretically require to visit the other page, at least inside a iFrame. There is a limit of how much you can do with javascript inside a iFrame.
With other words you try to imitate something like Selenium. Give it a try. It works with Java. You can control you browser, telling what to do, like a real user, and fetch the results, make even screenshots.

What's the best fallback if Google Ajax API is not available (since Google does not permit to install it locally)?

In my site I have:
...
<script type="text/javascript" src="https://www.google.com/jsapi"></script>
...
The script above is the Google script to load up other resources dynamically.
(eg Google charts API)
This works 99.99% of the time.
However, I just got a client that for some reasons got his company restricting access to google.com.
As a consequence of this my website simply threw a JavaScript error.
Now I know how to handle that, and I can check if window.Google exists.
but my question is
"what's the standard way to deal with this? "
In other words if you embed 3rd party JavaScript how best do you deal with their JS not available?
NOTE: VERY IMPORTANT
You can not host the chart code locally or on an intranet.
SEE FAQ from Google: https://developers.google.com/chart/interactive/faq#localdownload
Can I download and host the chart code locally, or on an intranet?
Sorry; our terms of service do not allow you to download and save or
host the Google.load or Google.visualization code.
There is no real alternative. Due to Google's terms of service you cannot use Google API without access to google.com.
Check the connection to Google and iform user that function is not available
Develop your own or use non-google api. Still you can use Google if available
The solution is that your client's company review their content filtering policies. Google are quite clear in their previous answer concerning offline access:
…your computer must have live access to http://www.google.com/jsapi in order to use charts.
You are using a third-party solution according to their terms and conditions, which naturally imposes limits on how that solution may be used by your clients. You need to stand firm or find a more liberally-licensed solution. (At any rate, you are more likely to succeed at convincing your client's IT department than petitioning Google to change their TOS.)
For the more general case of third party JS APIs that may not load but for which you are allowed to keep a local copy on your server, see this question.
You can try it like this:
Instead of using the direct link to the Google libraries you want to use, use a link which points to your server:
<script type="text/javascript" src="https://www.myserver.com/jsapi"></script>
When your server gets an incoming request to this URL, your server now makes a request to Google to get the API and sends the response to the client.
That means you do not install the API anywhere locally or on a server and always get the most actual version directly from Google. People also do not need access to Google (as in the company you mentioned) and therefore can use your service.
Use Firebug or the Chrome Dev Tools to inspect your HTML source once the charts scripts are loaded. Access the scripts in your browser and save them locally, then serve them from your own server. This isn't recommended, of course, but if you don't have any other choice...
For example, checking the code of one of the pages I use it on, the core script for the Google Charts library is located at:
https://www.google.com/uds/api/visualization/1.0/3d781368978b51b3ca00a01566dccf40/format+en,default,corechart.I.js
Use the javascript window.onload to check whether the api has loaded or not, if no then load it from your server.
You already know how to check whether or not your library has been loaded (checking the object), if it fails, than what you can do within giving constrains:
Keep checking the object with timer and trying to download library, displaying message for a user
In case first one fails, you have two ways again:
Stopping your application and displaying an error: "Application error... try later"
Or downloading different library as a fallback
Are you progressively enhancing or gracefully degrading the page? If so, what do you display to users without JavaScript for this chart? A table? A list? This is what you should leave in the page and only start changing it once google's JS is available. Either that, or find an alternative library like raphaeljs that lets you keep all your code within your project.
IF (BIG IF) you are not worried about the interactivity the Google Charts and want to display them to the user just to see - maybe add your own javascript to it but not depend on the Google Javascript at all, this can turn the google charts into a image that you can display to the user.
Also this requires access to install a command line tool on the server.
http://code.google.com/p/wkhtmltopdf/ is a command line tool that will generate an image from an html page. If you build a simple page that only shows the chart you want and point the wkhtmltoimage tool at the local html file it will load the Google Charts javascript and generate the chart then generate an image out of the results.
YES I understand this is VERY kludgy and is adding a big tool for a small problem but with the browser restriction and the Google Terms of Service this will solve most of the problem.
You can try going straight to google and if it fails (if google is restricted) you can bounce the request off of your server which forwards the request using CURL to google. If that doesn't work then Google is most likely down. This should cover the issue that you described in your question, but there isn't really a fix for if google itself actually goes down. It should, however, give your application access around domain restrictions because the request will be routed to your server rather than straight to google. I use this architecture for all requests so that I don't have ajax requests routed to random servers. It allows me to control what interacts with my front end using my backend. There are other benefits to this, especially if you are using something like AngularJS with NodeJS because you can decouple a lot of your third party libraries. This however, is beyond the scope of your question!
Basically, it works like this (pseudo code):
If(!Browser->Google->Browser){
return Browser->MyServer->Google->MyServer->Browser;
}
An answer has been accepted already, but still I would like to leave an additional aspect elaborating on the comment I made above ....
It has been accepted that the Google Server is the only place from where the API can be loaded. We don't know whether the client's IT manager will re-think their content policy, they might have good reasons for that.
Given a non-100% availability of all the components along the path between a user browser and the Google API, sooner or later a user will end up in an error situation; statistically this is unavoidable.
What is not acceptable (and avoidable) for a user is to receive an "unspecific" JS error making him/her believe there's a bug on the page. So my solution would be to trap the failure loading the Google API and display a message "Third party components temporarily unavailable - Please try later".
This will demonstrate to the user that
we know what's going on
there's nothing we can do about it now
but it's not totally unexpected and still somehow under control

Categories