Trying to fiddle around with site scraping. The example I'm using is this site:
http://www.oddsshark.com/nhl/computer-picks
With the goal being to use data to help with hockey pool etc etc
Now, when I do a simple fetch like:
var data = UrlFetchApp.fetch('http://www.oddsshark.com/nhl/computer-picks');
Logger.log(data.getBlob().getDataAsString());
None of the data in the body shows up. I assume its because the data is being retrieved from somewhere via javascript, but thats kinda the end of my rope in terms of understanding web development. I guess if that is the case, two parts,
How can I see where in that HTML that I'm grabbing the data is being fetched
and
How do I fetch it?
Thanks!
-Josh
This actually works. Since you are using Logger.log, you have to view logs (View -> Logs) in the script editor to see the fetched response, which I pasted here. Check it out.
Related
I am developing my first website. At this time i am generating a new html design that would be a ticket.
From my main page, i will load this html when the user clicks the "See ticket" button. This html has a table which is filled on document.ready with javascript. The data used is a JSON created in the main page.
I coded a working solution using localStorage. The problem is that the next step is to convert that HTML website to PDF and the software i am using does not work properly with localStorage, so i need to pass the JSON from main page to the ticket page. I can't neither use URL encoding cause string could be sometimes longer than 2000 characters and it is not productive.
So i thought that maybe i could do and $.get call from the ticket.html to index.html and get the needed JSON. Is this approach correct, or is there any better solution?
Regards
As suggested earlier comments, you need to use serverside code to accept post params and you need to do a ajax post to send the data. This is very good approach. I have one more idea for implementing this.
Let say you open ticket.html in a window.open. And have a JS function ( say GetValue) in index.html, that returns JSON . So you need to get JSON in ticket.html.
You need to define a JS function in ticket.html , using windown.opener.GetValue() , you can get JSON value.
Hope, i am in same direction, which you need. If not, please clarify.
Other way, would be use iFrame and use message communication to pass large data between them, you are interested in this, please read this - https://developer.mozilla.org/en-US/docs/Web/API/Window.postMessage
I have a website that mirror the hacked websites
From the day of starting everything is cool till it's being bigger
I have 2,197 records in MySQL DB
Every time I want to load this page http://red-zone.org/archive/ 2,197 have to be shown.
The cause that i'm using JavaScript for pagination and the JavaScript will work just after loading all the records
I don't like to use PHP for pagination
Does there's any other solution ?
I think the best way to use paging with big data is using Server-side processing with ajax/java script.
If you want a cool example you can look into datatables:
https://datatables.net/examples/data_sources/server_side.html
Of you can make your own javascript, but i see your already using datatables try the server side processing it will resolve your issue.
Ok, I've been wondering what I'm doing wrong with a Knockout web app I'm trying to put together.
Here is the code excluding the AJAX service with my MVC PHP interface.
my.js = namespace declaration that I use for my app
data.js = contains static data which works to load the data into the view (leftPanel.php)
appViewModel.js = call the functions that loads the data from PHP server and declare the view model.
So, basically, here is the issue:
I've an AJAX Service that works and get the data as show in accounts_JSON.txt and currencies_JSON.txt. The JSON is well formatted and in the text files, I've only added returns to make it easy to read.
In appViewModel.js, the "getCurrencies" and "getAccounts" methods are called and work properly: iterate through the JSON data and put it in the specified array.
Then, at the end of appViewModel.js, I log in the Chrome console each step because I can't find out why "my.app.data.currencies" and "my.app.data.accounts" are logged as empty when they are not (screenshot_chrome_console.png).
Those arrays are populated before the ko.applyBiddings so if I were to use "my.app.data.currencies" and "my.app.data.accounts" to populate the view, it should work but it doesn't. Only the static data "my.app.leftPanel.currencies.list" and "my.app.leftPanel.accounts.list" works.
What am I missing? I really can't see!
Thanks a lot for your help :)
Well, I've solved my issue :) Thanks to this post (from Irakli Nadareishvili).
My problem was definitely the loading method of the data retrieved via AJAX. Using Underscore to load the currencies and the accounts first and when, and only when all is loaded, filteredAccounts is loaded since I need the full list of accounts.
Here is the final and cleaned-up code in case anyone is interested or stumble on the same problem.
Happy coding to all!
I've searched around the internet for a couple of hours and could not find anything related to what I'm trying to do. I wrote a HTML document that collects data from a user and stores it in a javascript array. This array is then joined together and stored as a string in a document which is hidden. Originally, I was going to transfer this string to a program I wrote in C#, but now I am using LabVIEW.
In C#, I used two simple lines of code to do what I wanted:
System.Windows.Forms.HtmlElement hidden = webBrowser1.Document.GetElementById("hiddenfield1");
List<latlng> data = formharvest.extract(hidden.GetAttribute("value"));
But now I cannot find a way to access the data that is in this hidden document. I'm using the IWebBrowser2 block to embed my HTML code in my VI. Any help would be greatly appreciated. Thank you for your time!
A solution would be to start a Web server in your LabVIEW program, and serve your HTML form from it. I suppose it wouldn't be too hard to retrieve form data then, but I haven't done such thing myself.
Here's an interesting discussion on this with sample code.
I'm not sure I really understand what you are doing above: in the C#, it looks like you are embedding an HTML-rendering engine in a Windows form (i.e. window).
You can embed .NET in your labview code and therefore you should be able to embed the same HTML rendering engine in a LabView VI, but you might consider changing your approach, as CharlesB suggests, to something more traditional where a server serves some HTML to a web browser, which then sends some data back to the server via HTTP GET or POST.
if im loading data for the markers from a database do i write the output queried from the db into a javascript file or is there a cleaner way of doing it?
thanks
Yeah, writing to a file is a good way to do it. Just write the data as JSON. Your file would look like:
var map = {waypoints:[...]};
And then you can do:
for(var i=o; i<map.waypoints.length; ++i) {
addWaypoint(map.waypoints[i]);
}
I actually do some static caching of nodes using this method: http://www.trailbehind.com/site_media/javascript/gen/national-parks.js
We use that set of National Parks a lot, so we cache it. But we also have urls where you can fetch JSON for a node on the fly, such as: http://www.trailbehind.com/map/node/7538973/632/735/
This URL gets the map for node 7538973, and specifies the dimensions of their map in pixels as well.
The needed Javascript can of course be wrapped in whatever language you prefer to use, see e.g. pymaps for a Python example. While pymaps is actualally inserting the JS code into an HTML template, if you're writing a web app you can perfectly well choose to serve that JS code on the fly at an appropriate URL and use that URL in a <script> tag in your pages.
Depending on the size of your application, you may want to consider printing out plain javascript.
I have a map that uses server-side clustering, so markers update frequently. I found that parsing JSON markers slowed the app significantly, and simply wasn't necessary.
If speed is an issue, I'd suggesting removing all of the unnecessary layers possible (JSON, AJAX, etc.). If it's not, you'll be just fine with JSON, which is cleaner.
I agree with Andrew's answer (+1).
I guess the only point I would add is that rather than including some server side generated JavaScript, you could use an AJAX request to grab that data. Something like:
var request = new Request.JSON (url: 'get_some_json.php',
onSuccess: function(data) {
// do stuff with the data
}).get ();
(This is a Mootools AJAX thing, but you could use any kind of AJAX request object).
Edit: ChrisB makes a good point about the performance of parsing JSON responses and re-reading my answer I certainly didn't make myself clear. I think AJAX requests are suitable for re-requesting data based on parameters generated by user interaction. I guess an example use case might be, a user filtering the data displayed on the map. You might grab the filtered data via an AJAX/SJON request rather than re-loading the page.