We Keep Coding

Node.js: requesting a page and allowing the page to build before scraping

I've seen some answers to this that refer the askee to other libraries (like phantom.js), but I'm here wondering if it is at all possible to do this in just node.js?
Considering my code below. It requests a webpage using request, then using cheerio it explores the dom to scrape the page for data. It works flawlessly and if everything had gone as planned, I believe it would have outputted a file as i imagined in my head.
The problem is that the page I am requesting in order to scrape, build the table im looking at asynchronously using either ajax or jsonp, i'm not entirely sure how .jsp pages work.
So here I am trying to find a way to "wait" for this data to load before I scrape the data for my new file.
var cheerio = require('cheerio'),
request = require('request'),
fs = require('fs');
// Go to the page in question
request({
method: 'GET',
url: 'http://www1.chineseshipping.com.cn/en/indices/cbcfinew.jsp'
}, function(err, response, body) {
if (err) return console.error(err);
// Tell Cherrio to load the HTML
$ = cheerio.load(body);
// Create an empty object to write to the file later
var toSort = {}
// Itterate over DOM and fill the toSort object
$('#emb table td.list_right').each(function() {
var row = $(this).parent();
toSort[$(this).text()] = {
[$("#lastdate").text()]: $(row).find(".idx1").html(),
[$("#currdate").text()]: $(row).find(".idx2").html()
}
});
//Write/overwrite a new file
var stream = fs.createWriteStream("/tmp/shipping.txt");
var toWrite = "";
stream.once('open', function(fd) {
toWrite += "{\r\n"
for(i in toSort){
toWrite += "\t" + i + ": { \r\n";
for(j in toSort[i]){
toWrite += "\t\t" + j + ":" + toSort[i][j] + ",\r\n";
}
toWrite += "\t" + "}, \r\n";
}
toWrite += "}"
stream.write(toWrite)
stream.end();
});
});
The expected result is a text file with information formatted like a JSON object.
It should look something like different instances of this
"QINHUANGDAO - GUANGZHOU (50,000-60,000DWT)": {
 "2016-09-29": 26.7,
"2016-09-30": 26.8,
},
But since the name is the only thing that doesn't load async, (the dates and values are async) I get a messed up object.
I tried Actually just setting a setTimeout in various places in the code. The script will only be touched by developers that can afford to run the script several times if it fails a few times. So while not ideal, even a setTimeout (up to maybe 5 seconds) would be good enough.
It turns out the settimeouts don't work. I suspect that once I request the page, I'm stuck with the snapshot of the page "as is" when I receive it, and I'm in fact not looking at a live thing I can wait for to load its dynamic content.
I've wondered investigating how to intercept the packages as they come, but I don't understand HTTP well enough to know where to start.

The setTimeout will not make any difference even if you increase it to an hour. The problem here is that you are making a request against this url:
http://www1.chineseshipping.com.cn/en/indices/cbcfinew.jsp
and their server returns back the html and in this html there are the js and css imports. This is the end of your case, you just have the html and that's it. Instead the browser knows how to use and to parse the html document, so it is able to understand the javascript scripts and to execute/run them and this is exactly your problem. Your program is not able to understand that has something to do with the HTML contents. You need to find or to write a scraper that is able to run javascript. I just found this similar issue on stackoverflow:
Web-scraping JavaScript page with Python
The guy there suggests https://github.com/niklasb/dryscrape and it seems that this tool is able to run javascript. It is written in python though.

You are trying to scrape the original page that doesn't include the data you need.
When the page is loaded, browser evaluates JS code it includes, and this code knows where and how to get the data.
The first option is to evaluate the same code, like PhantomJS do.
The other (and you seem to be interested in it) is to investigate the page's network activity and to understand what additional requests you should perform to get the data you need.
In your case, these are:
http://index.chineseshipping.com.cn/servlet/cbfiDailyGetContrast?SpecifiedDate=&jc=jsonp1475577615267&_=1475577619626
and
http://index.chineseshipping.com.cn/servlet/allGetCurrentComposites?date=Tue%20Oct%2004%202016%2013:40:20%20GMT+0300%20(MSK)&jc=jsonp1475577615268&_=1475577620325
In both requests:
_ is a decache parameter to prevent caching.
jc is a name of a JS wrapper function which should be invoked with the result (https://en.wikipedia.org/wiki/JSONP)
So, scrapping the table template at http://www1.chineseshipping.com.cn/en/indices/cbcfinew.jsp and performing two additional requests you will be able to combine them into the same data structure you see in the browser.

Loading TXT files before displaying them with ajax [duplicate]

This question already has answers here:
Sequencing ajax requests
(10 answers)
Closed 7 years ago.
I'm working on a small text game in js, and the easiest way I found to have save text is to use text files. I can order them in different folders, they really light and they're easily identifiable and editable in case I need to make changes.
I'm loading them using ajax
$.ajax({
url: 'text/example.txt',
success: function(text){
document.getElementById("main").innerHTML = document.getElementById("main").innerHTML + text;
}
});
As it was suggested to me in another thread.
And honestly, so far it's been working pretty well, in single-cases scenarios. When only one TXT file needs to be displayed there are literally no problems. But, unfortunately in cases where a lot of different files need to be displayed in a correct order (let's say around 10 different files), the text gets messed up and loads out of order. I'm going to suppose this is because it just can't fetch the txt file fast enough.
So at this point I'm really not too sure what to do.
Is there a way to get my script to wait before printing the next piece of text before displaying one that still hasn't loaded?
Maybe a way to load all the txt files when the site is accessed?
My knowledge is pretty limited so I'm really not sure how this could be fixed.
Tried searching google and stackoverflow, none of the threads I found are helping me, perhaps because I'm really not an expert.

You can achieve with callback, the following way will call ajax one by one after they finish:
//setup an array of AJAX url
var ajaxes = [{ url : 'text/example.txt'}, { url : 'text/example1.txt'}],
current = 0;
//declare your function to run AJAX requests
function do_ajax() {
//check to make sure there are more requests to make
if (current < ajaxes.length) {
//make the AJAX request with the given data from the `ajaxes` array of objects
$.ajax({
url : ajaxes[current].url,
success : function (text) {
document.getElementById("main").innerHTML = document.getElementById("main").innerHTML + text;
//increment the `current` counter and recursively call this function again
current++;
do_ajax();
}
});
}
}
//run the AJAX function for the first time when you want
do_ajax();

Ajax call via POST not working

I have a lil problem with an AJAX request.
We have a lil PHP, JavaScript application (Website). The application is running fine on all desktop browsers + on our old MDE's (some Windows CE6 MDE). Now on our new Motorola MC9200 (Windows Embedded Compact 7 formerly CE7) it's not working anymore.
The problem is some small JavaScript function. It disables the buttons/input fields, starts a Ajax.Request (prototype 1.72 but I tested jQuery 1.11.1 too), does something on the database and when everything went right it is refreshing the site via window.location. This function isn't working always on the new devices. Sometimes it does, sometimes not.
simplified code:
function loadSite(siteName) {
disableForm();
var parameters = {
/* SOME PARAMETERS */
};
new Ajax.Request('ajax/ajax_db_execute.php', {
method: 'post',
parameters: parameters,
onSuccess: callbackFunc
});
}
function callbackFunc(transport) {
response = transport.responseText.evalJSON(true);
if(response.retcode === 0) {
window.location = "start.php?id=<?php echo $id; ?>";
} else {
show_error_box(response.errortext);
enableForm();
}
}
I tried to output the response in the callbackFunc but that function wasn't even called. Next thing I tried was to put some alert at the end of the loadSite function, it was fired everytime. I already checked the parameters and they look fine too.
After that I put some simple fwrite in the php file. It looks like that file isn't even called sometimes. So the question is why?
By changing the method to 'get' I couldn't reproduce the problem and everything is working fine. Problem about that is that I don't want to use get + some parameters might be too long for get to handle.
The parameters in that example were just some simple integers and strings. Does anyone have an idea what might cause the problem and some workaround?

It seems that your post request response is not synchronized. So please use setTimeout function in your post callback like that
setTimeout(function(transport) {
response = transport.responseText.evalJSON(true);
if(response.retcode === 0) {
window.location = "start.php?id=<?php echo $id; ?>";
}
else
{
show_error_box(response.errortext);
enableForm();
}
}, 3000);

Create HTML table from text file using Javascript?

I don't even know if my project is possible. After looking around for a few hours and reading up on other Stack Overflow questions, my hopes are slowly diminishing, but it will not stop me from asking!
My Project: To create a simple HTML table categorizing our Sales Team phone activity for my superior. Currently I need something to pull data values from a file and use those values inside the table.
My Problem: Can Javascript even do this? I know it reads cookies on the client side computer, but can it read a file in the same directory as the webpage? (If the webpage is on the company server?)
My Progress: I will update as I find more information.
Update: Many of you are curious about how the file is stored. It is a static webpage (table.html) on our fileserver. The text file (data.txt) will be in the same directory.

I've recently completed a project where i had almost the exact conditions as yourself (the only difference is that users exclusively use IE).
I ended up using JQuery's $.ajax() function, and pulled the data from an XML file.
This solution does require the use of either Microsoft Access or Excel. I used as early as the 2003 version, but later releases work just fine.
My data is held in a table on Access (on Excel i used a list). Once you've created your table in Access; it's honestly as simple as hitting 'Export', saving as XML and then playing around with your 'ajax()' function (http://api.jquery.com/jQuery.ajax/) to manipulate the data which you want to be output, and then CSS/HTML for the layout of your page.
I'd recommend Access as there's less hastle in getting it to export XML in the right manner, though Excel does it just fine with a little more tinkering.
Here's the steps with ms-access:
Create table in access & export as XML
The XML generated will look like:
<?xml version="1.0" encoding="UTF-8"?>
<dataroot xmlns:od="urn:schemas-microsoft-com:officedata" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="Calls.xsd" generated="2013-08-12T19:35:13">
<Calls>
<CallID>1</CallID>
<Advisor>Jenna</Advisor>
<AHT>125</AHT>
<Wrap>13</Wrap>
<Idle>6</Idle>
</Calls>
<Calls>
<CallID>3</CallID>
<Advisor>Edward</Advisor>
<AHT>90</AHT>
<Wrap>2</Wrap>
<Idle>4</Idle>
</Calls>
<Calls>
<CallID>2</CallID>
<Advisor>Matt</Advisor>
<AHT>246</AHT>
<Wrap>11</Wrap>
<Idle>5</Idle>
</Calls>
Example HTML
<table id="doclib">
<tr><th>Name</th><th>AHT</th><th>Wrap</th><th>Idle</th></tr>
</table>
jQuery:
$(document).ready(function(){
$.ajax({
type: "GET",
url: "Calls.xml",
dataType: "xml",
success: function(xml) {
$(xml).find('Calls').each(function(){
var advisor = $(this).find('Advisor').text(),
aht = $(this).find('AHT').text(),
wrap = $(this).find('Wrap').text(),
idle = $(this).find('Idle').text(),
td = "<td>",
tdc = "</td>";
$('#doclib').append("<tr>" +
td + advisor + tdc + td + aht + tdc + td + wrap + tdc + td + idle + tdc + "</tr>")
});
}
});
});

JavaScript cannot automatically read files due to security reasons.
You have two options:
If you can rely on IE being used, you could use some fancy ActiveX stuff.
Use a backend which either constantly pushs data to the JS client or provides the data on pull requests.

This could work if you had a server like build with Node.js, PHP, ...etc.
JavaScript can read files with the Ajax protocol, but this mean that you need a server.
Otherwise your requests will go through the file:// protocol which doesn't support Ajax.

You can try looking into FileReader:
https://developer.mozilla.org/en-US/docs/Web/API/FileReader
The FileReader object lets web applications asynchronously read the contents of files (or raw data buffers) stored on the user's computer
I've never personally gotten it to work properly, but it's supposed to be able to allow this sort of thing.

Try with XMLHttpRequest or ActiveXObject in IE 5 or IE 6.
Here you can find an explanation:
http://www.w3schools.com/xml/xml_http.asp
Or try this example:
http://www.w3schools.com/ajax/tryit.asp?filename=tryajax_first

It sounds like you just want to get the contents of a static file from your server; is that right? If that's what you need to do, you're in luck. That's very easy.
load('textTable.txt', function(err, text) {
buildTable(text);
});
function load(url, callback) {
var xhr = new XMLHttpRequest();
xhr.onreadystatechange = function() {
if (xhr.readyState < 4) return;
if (xhr.status !== 200) {
return callback('HTTP Status ' + xhr.status);
}
if (xhr.readyState === 4) {
callback(null, xhr.responseText);
}
};
xhr.open('GET', url, true);
xhr.send('');
}
If you go with qwest, it'll look something like this:
qwest.get('textTable.txt').success(function(text) {
buildTable(text);
});
With jQuery:
jQuery.get('textTable.txt', function(text) {
buildTable(text);
});

jquery load() equivalent for offline use

I am looking for an equivalent to jquery's load() method that will work offline. I know from jquery's documentation that it only works on a server. I have some files from which I need to call the html found inside a particular <div> in those files. I simply want to take the entire site and put it on a computer without an internet connection, and have that portion of the site (the load() portion) function just as if it was connected to the internet. Thanks.
Edit: BTW, it doesn't have to be js; it can be any language that will work.
Edit2:
My sample code (just in case there are syntax errors I am missing; this is for the files in the same directory):
function clickMe() {
var book = document.getElementById("book").value;
var chapter = document.getElementById("chapter").value;
var myFile = "'" + book + chapter + ".html'";
$('#text').load(myFile + '#source')
}

You can't achieve load() over the file protocol, no other ajax request is going to work for html files. I have tried even with the crossDomain and isLocale option on without anything success, even if precising the protocol.
The problem is that even if jQuery is trying the browser will stop the request for security issues (well most browsers as the snippet below works in FF) as it allows you to load locale file so you could get access to a lot of things.
The one thing you could load locally is javascript files, but that probably means changing a lot of the application/website architecture.
Only works in FF
$.ajax({
url: 'test.html',
type: 'GET',
dataType: 'text',
isLocale: true,
success: function(data) {
document.body.innerHTML = data;
}
});
What FF does well is that it detect that the file requesting local files is on the file protocol too when other don't. I am not sure if it has restriction over the type of files you can request.

You can still use the JQuery load function in this context:
You would could add an OfflineContent div on your page:
<div id="OfflineContent">
</div>
And then click a button which calls:
$('#OfflineContent').load('OfflinePage.html #contentToLoad');
Button code:
$("#btnLoadContent").click(function() {
$('#OfflineContent').load('OfflinePage.html #contentToLoad');
});
In the OfflinePage.html you could have to have another section called contentToLoad which would display on the initial page.