When a webpage like https://poloniex.com/exchange#btc_eth is opened in the browser, we see that the browser constantly shows updated buy and sell orders. Also, in the Elements section in the chrome console, these updates are visible in the HTML tables.
Is there a way I can use a nodejs script run on my pc (so not in the browser console) to get these live html table updates from that website, without having to do a GET request every time?
If the chrome browser is able to do it, nodejs / jQuery / ajax should be able to do it as well. I tried the XMLHttpRequest nmp module but no luck yet.
It's possible they are using token authentication which means you wouldn't be able to get all the connection info you need just from their client-side code. Have you downloaded it and looked at it yet?
If you find it's not possible to call their services, there are other free products designed for webscraping. AutoHotKey is one that can open a web page and traverse its DOM. I believe it has the ability to run in the background, but don't quote me.
Related
This is my first time delving into an HTML editor with goal of learning it. I opened the website option-price.com and right clicked in chrome to get the source.
I am not able to figure where the real calculation happens when I hit the button Calculate.
Mostly such content is embedded.like most of the web pages don't allow you to see the source code.
But
Firebug is a discontinued free and open-source web browser extension for Mozilla Firefox that facilitated the live debugging, editing, and monitoring of any website's CSS, HTML, DOM, XHR, and JavaScript.
You need to open the console (option + cmd + J on Mac and Chromium), and not the source.
Then you might want to look at the Sources tab, and try and find the Javascript (file or from the HTML (here index.php) responsible for what you want to debug. However it seems for this very site that the Calculate does a server call to do the calculation.
Such calculations would not typically be on the front-end - they would likely be happening on whatever back-end they are using (ex. PHP, Django, etc.) by making a server call, which is definitely not displayed in the source.
I have a client-side Golang application running on my machine. I also have a browser open, and in that browser there might be a tab running my web application (which is completely separate from the Golang app).
From the Golang app, I would like to programmatically refresh the browser tab (and maybe if possible, bring it to front, but that's less important).
I researched quite a lot already, and I concluded this is not possible just by communicating to the browser, there is no standard (especially cross-platform and cross-browser) interface with which we can trigger the refresh of a specific tab of a browser.
So I suppose I'll need to have some custom JS code running on the website with which my Golang application can communicate and trigger the refresh of the tab.
What's the easiest way to do this?
(I was looking at livereload.js and lrserver, but these all start with the premise that there is a folder of content we'd like to watch and automatically reload on any change. But I don't want that, I just programmatically want to trigger the refresh. Also, this Golang app is not hosting the website, it's just a separate client-side application.)
As suggested by some comments, there seems to be no API through which we could connect to a browser from Golang, query the list of tabs, and refresh a particular page (at least not in a cross-browser and cross-platform way).
One possible approach to do this is to host a small WebSocket endpoint in Golang, and connect to it from the site we want to refresh. Then send a message through the WS connection every time we want to reload the site, and in JavaScript call location.reload() when we receive the message.
I described all the details in a blog post, and uploaded a complete working example to GitHub.
This past week, there was the launch of a new tool called #Homescreen that allows people to share a screenshot of the apps that they have on their iPhone home screen. For example: https://homescreen.is/iamfinnym
I'd like to build a scraper that extracts the names of all the apps given a user's page (in addition to their location on the screen). How would I do this? I know how to build a normal HTML scraper, but it looks like the apps are generated onto the page via some kind of React.js javascript call, and I'm not sure how to go about figuring this out. (I can write basic Javascript, but have never used React.js before and I don't know how to get started.)
This is how you can get the data through Chrome's dev console:
If you open the Chrome dev console (Ctrl / Cmd+Shift+C), and head to the Network, you will find this:
If you click on it, you will see that the API is set up so that when you make a GET request to https://homescreen.is/api/user/{USERNAME}, you will get the data of their apps as responseData.apps. Click on Preview to get a preview of the data sent by the API:
Now you can use any language's http library to make GET requests to the API.
I'm sorry if this is a newbie question but I don't really know what to search for either. How do you keep content from a previous page when navigating through a web site? For example, the right side Activity/Chat bar on facebook. It doesn't appear to refresh when going to different profiles; it's not an iframe and doesn't appear to be ajax (I could be wrong).
Thanks,
I believe what you're seeing in Facebook is not actual "page loads", but clever use of AJAX or AHAH.
So ... imagine you've got a web page. It contains links. Each of those links has a "hook" -- a chunk of JavaScript that gets executed when the link gets clicked.
If your browser doesn't support JavaScript, the link works as it normally would on an old-fashioned page, and loads another page.
But if JavaScript is turned on, then instead of navigating to an HREF, the code run by the hook causes a request to be placed to a different URL that spits out just the HTML that should be used to replace a DIV that's already showing somewhere on the page.
There's still a real link in the HTML just in case JS doesn't work, so the HTML you're seeing looks as it should. Try disabling JavaScript in your browser and see how Facebook works.
Live updates like this are all over the place in Web 2.0 applications, from Facebook to Google Docs to Workflowy to Basecamp, etc. The "better" tools provide the underlying HTML links where possible so that users without JavaScript can still get full use of the applications. (This is called Progressive Enhancement or Graceful degradation, depending on your perspective.) Of course, nobody would expect Google Docs to work without JavaScript.
In the case of a chat like Facebook, you must save the entire conversation on the server side (for example in a database). Then, when the user changes the page, you can restore the state of the conversation on the server side (with PHP) or by querying your server like you do for the chat (Javascript + AJAX).
This isn't done in Javascript. It needs to be done using your back-end scripting language.
In PHP, for example, you use Sessions. The variables set by server-side scripts can be maintained on the server and tied together (between multiple requests/hits) using a cookie.
One really helpful trick is to run HTTPFox in Firefox so you can actually monitor what's happening as you browse from one page to the next. You can check out the POST/Cookies/Response tabs and watch for which web methods are being called by the AJAX-like behaviors on the page. In doing this you can generally deduce how data is flowing to and from the pages, even though you don't have access to the server side code per se.
As for the answer to your specific question, there are too many approaches to list (cookies, server side persistence such as session or database writes, a simple form POST, VIEWSTATE in .net, etc..)
You can open your last closed web-page by pressing ctrl+shift+T . Now you can save content as you like. Example: if i closed a web-page related by document sharing and now i am on travel web page. Then i press ctrl+shift+T. Now automatic my last web-page will open. This function works on Mozilla, e explorer, opera and more. Hope this answer is helpful to you.
Im running a javascript trading application in my browser, provided by my bank. The application gives me realtime data on stock quotes etc. Is there anyway i can get the browser to make the data available outside the browser, like writing the info to a file every ten seconds? Im using firefox.
No, the Javascript is running in a sandbox. You can write it into a html5 database build in with your browser.
May be, the application is using a SOAP-Service, which you can use directly.
there is a way save the web pages. hope it will help you.
The add on is available to save web pages in mozilla.
see https://addons.mozilla.org/en-US/firefox/addon/auto-save-document/
There is a straight-forward approach, that requires a server (even if it's a very dumb one): You could post the data to the server and have it respond with a file containing the posted data (no more, no less) that the user is prompted to save.