I have a firefox extension that collaborates with a web page, and occasionally needs to inject data into it which the page formats and displays.
The way I do it now is :-
var element = doc.createElement("MyData");
doc.documentElement.appendChild(element);
for(....)
{
var x = ....
var y = ....
var z = ....
var row = doc.createElement("row");
row.setAttribute("x", x);
row.setAttribute("y", y);
row.setAttribute("z", z);
element.appendChild(row);
}
This gets really slow for 1000s of items, and some more time is spent by the page parsing the data and displaying it in HTML elements.
Is there a better way?
Would it make sense to dump the entire data as a single string for example?
Thanks in advance
In my experience with regular site scripting, large HTML insertions appear to be faster if you inject raw HTML with the .innerHTML property. Perhaps that's true for extensions as well.
Related
I've seen some answers to this that refer the askee to other libraries (like phantom.js), but I'm here wondering if it is at all possible to do this in just node.js?
Considering my code below. It requests a webpage using request, then using cheerio it explores the dom to scrape the page for data. It works flawlessly and if everything had gone as planned, I believe it would have outputted a file as i imagined in my head.
The problem is that the page I am requesting in order to scrape, build the table im looking at asynchronously using either ajax or jsonp, i'm not entirely sure how .jsp pages work.
So here I am trying to find a way to "wait" for this data to load before I scrape the data for my new file.
var cheerio = require('cheerio'),
request = require('request'),
fs = require('fs');
// Go to the page in question
request({
method: 'GET',
url: 'http://www1.chineseshipping.com.cn/en/indices/cbcfinew.jsp'
}, function(err, response, body) {
if (err) return console.error(err);
// Tell Cherrio to load the HTML
$ = cheerio.load(body);
// Create an empty object to write to the file later
var toSort = {}
// Itterate over DOM and fill the toSort object
$('#emb table td.list_right').each(function() {
var row = $(this).parent();
toSort[$(this).text()] = {
[$("#lastdate").text()]: $(row).find(".idx1").html(),
[$("#currdate").text()]: $(row).find(".idx2").html()
}
});
//Write/overwrite a new file
var stream = fs.createWriteStream("/tmp/shipping.txt");
var toWrite = "";
stream.once('open', function(fd) {
toWrite += "{\r\n"
for(i in toSort){
toWrite += "\t" + i + ": { \r\n";
for(j in toSort[i]){
toWrite += "\t\t" + j + ":" + toSort[i][j] + ",\r\n";
}
toWrite += "\t" + "}, \r\n";
}
toWrite += "}"
stream.write(toWrite)
stream.end();
});
});
The expected result is a text file with information formatted like a JSON object.
It should look something like different instances of this
"QINHUANGDAO - GUANGZHOU (50,000-60,000DWT)": {
"2016-09-29": 26.7,
"2016-09-30": 26.8,
},
But since the name is the only thing that doesn't load async, (the dates and values are async) I get a messed up object.
I tried Actually just setting a setTimeout in various places in the code. The script will only be touched by developers that can afford to run the script several times if it fails a few times. So while not ideal, even a setTimeout (up to maybe 5 seconds) would be good enough.
It turns out the settimeouts don't work. I suspect that once I request the page, I'm stuck with the snapshot of the page "as is" when I receive it, and I'm in fact not looking at a live thing I can wait for to load its dynamic content.
I've wondered investigating how to intercept the packages as they come, but I don't understand HTTP well enough to know where to start.
The setTimeout will not make any difference even if you increase it to an hour. The problem here is that you are making a request against this url:
http://www1.chineseshipping.com.cn/en/indices/cbcfinew.jsp
and their server returns back the html and in this html there are the js and css imports. This is the end of your case, you just have the html and that's it. Instead the browser knows how to use and to parse the html document, so it is able to understand the javascript scripts and to execute/run them and this is exactly your problem. Your program is not able to understand that has something to do with the HTML contents. You need to find or to write a scraper that is able to run javascript. I just found this similar issue on stackoverflow:
Web-scraping JavaScript page with Python
The guy there suggests https://github.com/niklasb/dryscrape and it seems that this tool is able to run javascript. It is written in python though.
You are trying to scrape the original page that doesn't include the data you need.
When the page is loaded, browser evaluates JS code it includes, and this code knows where and how to get the data.
The first option is to evaluate the same code, like PhantomJS do.
The other (and you seem to be interested in it) is to investigate the page's network activity and to understand what additional requests you should perform to get the data you need.
In your case, these are:
http://index.chineseshipping.com.cn/servlet/cbfiDailyGetContrast?SpecifiedDate=&jc=jsonp1475577615267&_=1475577619626
and
http://index.chineseshipping.com.cn/servlet/allGetCurrentComposites?date=Tue%20Oct%2004%202016%2013:40:20%20GMT+0300%20(MSK)&jc=jsonp1475577615268&_=1475577620325
In both requests:
_ is a decache parameter to prevent caching.
jc is a name of a JS wrapper function which should be invoked with the result (https://en.wikipedia.org/wiki/JSONP)
So, scrapping the table template at http://www1.chineseshipping.com.cn/en/indices/cbcfinew.jsp and performing two additional requests you will be able to combine them into the same data structure you see in the browser.
I'm looking for a way, how set data from var title, to other data ( var t ), and see result in page source.
Now i have:
// example: home
var title = document.title;
var t = title;
Now, in souce view, var title = document.title and var t also.
How can i parse this code, and see document.title returned value ?
Thanks for explain.
With this code:
var title = document.title
i Want to see in source code of browser:
var title = 'title of page being viewed' instead of var title = document.title
This isn't how JavaScript works.
It is a dynamic scripting language that runs in the web browser and affects that visitors page only.
When you choose View Source you are looking at the underlying code before it is executed.
If you want to pull information from another source then you need to look into either:
Screen scraping
Web services
Screen scraping is the process of downloading a webpage and then looking inside of it to pull information out. This can be messy if the website changes the layout of the page or the content is badly formatted.
A better way to provide information to 3rd party sources is to create a web service. This gives you an API which you can query and then you can return data in a computer readable format.
You mean that display the value of variable on your browser?
Just use this code:
console.dir(t);
And you can see the result on console panel.
I would like to save the results calculated on html page in a textfile using javascript.
<script type="text/javascript">
window.onload = function () {
var sw : StreamWriter = new StreamWriter("HTML_Results.txt");
sr.Write('xyz");
*** calculations ******
sr.Write (result);
}
</script>
by doing this, my WP8 App is misbehaving and not displaying images as usual. This app is an Image Fader (calculates FPS).
Also tried:
StreamWriter sr;
try {
sr = new StreamWriter("\HTML5\HTMLResults.txt");
sr.Write("xyz");
File.SetAttributes("HTML5\HTMLResults.txt", FileAttributes.Hidden);
} catch(IOException ex) {
console.write ("error writing"); //handling IO
}
The aim is to:
Extract calculated values of several html pages(after getting loaded
one by one) in a single text file.
A Resultant HTML that reads this
text file and displays results in a tabular format.
Is there a better way to this job or the above can be rectified and used? Appreciate help.
Perhaps I've misunderstood your code but it looks like you're trying to write Java within JavaScript scripting tags. You cannot write Java in an HTML document. As far as I know, client-side JavaScript (which given your <script> tags is I guess what you're trying to write) can't perform the kind of file I/O operations you seem to want here.
You need to use Node JS to use JavaScript for something like that and then you're talking server-side. The closest you can get on client-side is using the new localStorage feature in HTML5 (not supported by all browsers).
Found solution from Microsoft Blog... see below
OK, to start I don't like the word random but I cannot find any correlation in test cases for this problem so I am going to use random to describe parts of this problem.
The setup: I have a list where i have crated a customized UI for the EditForm.aspx and NewForm.aspx. I use the same JS file and JavaScript between the two of them. I have added in a google map to help illustrate the location selection. I have added extra code to the "OK" button for some dynamic validation. I have done a lot of dynamic menu things as well. All users use IE 9 and the site is on a MOSS 2007 server.
The problem: Only on the EditForm.aspx, clicking OK "Randomly" results in an immediate white screen. The form is not saved and when viewing the source code of the white screen i find a blank html page.
What I have tried to find this problem:
- I tried to narrow down the user and computer this happens on and found that it happens for everyone on every computer(once again "Randomly").
- I tried disabling the code that is pre-pended to the "OK" button
- I tried following the code with the IE9's external script debugged and found no errors
I can provide the code but it is a bit long and I really do not know where to begin. So i can provide it if needed.
Thanks for the help ahead of time.
Edit:
This is the code re-wiring my OK button(i reset the value to "Save" earlier)
var okBtns = $('input[value="Save"]')
$.each(okBtns, function(index,value){
okFunction=$(value).attr('onclick');
$(value).attr('onclick','return false;')
$(value).bind('click', function(){
if ($('#'+StatusBox).val()=='Draft') {$('#'+StatusBox).val('New Request')}
var err = clickOKbutton();
if(err==0) {okFunction()};
});
});
This is the clickOKbutton function witch is th code prepended to the orgianal sharepoint operations:
function clickOKbutton()
{
//all of the imput validation i could ever wish for!!!!
var NoteVal = ''
var NameAry = $('#'+PersonnelBox).parent().children(":first").children("SPAN").children("SPAN");
$.each(NameAry, function(index,value){
var $n=$(value).html();
if(NoteVal.length==0) {NoteVal=$n} else {NoteVal=NoteVal+';'+$n};
});
//$('#'+AddNotes).val(NoteVal);
var plh = $('#'+PersonnelBox).parent().html()
userNameTx = $('#zz8_Menu').text();
userNameTx = userNameTx.replace('Welcome ','');
$.each(OICUsers, function(i,v){
if(plh.indexOf(v) > -1 && st=='New Request'){
$('#'+StatusBox).val('OIC Bypassed')
$('#'+CommentsBox).val('OIC is travling on this TDY/TAD and cannot approve. So this request is bypassing the "OIC Approval" step')
}
});
/*userNameTx = $('#zz8_Menu').text();
userNameTx = userNameTx.replace('Welcome ','')
$('#' + ModBox).closest('TR').show();*/
var message=''
message = detectFieldChanges(AllFieldsArray,AllOrgValArray,"Draft,New Request,Modified")
if(message.length>0){
$('#'+ModBox).val(message);
AutoResizeTextarea(ModBox);
}
message = detectFieldChanges(ValFieldsArray,OrgValuesArray,"Draft,New Request,Modified,OIC Approved,OIC Bypassed,Pending RFI,Ready for COS")
userNameTx = $('#zz8_Menu').text();
userNameTx = userNameTx.replace('Welcome ','');
if(message.length>0&&$.inArray(userNameTx,COSUsers)==-1){
$('#'+StatusBox).val('Modified').change;
$('#'+StatusLongBox).val('Modified').change;
}
//Subject box
var pb = NoteVal;
var ep = $('#'+ExtPersonnel).val();
var ab = $('#'+AddressBox).val();
var sd = $('#'+sDateBox).val();
var ed = $('#'+eDateBox).val();
var st = $('#'+StatusBox).val();
var p = pb+';'+ep;
var p = p.replace(/mossaspnetmembershipprovider:/g,'');
var p = p.slice(0,-1);
var ad = ab+' '+sd+' to '+ed;
var s = 'eTDY | '+st+' - '+p+' - '+ad;
if(s.length>255){
var l = s.length-255;
p = p.substring(0,p.length-l);
s = 'eTDY | '+p+' - '+ad;
}
$('#'+Subject).val(s);
//check Lat/Lng value
if($('#'+LatBox).val()=='' || $('#'+LngBox).val()==''){
//alert("Cannot continue unless the Lat Lng has a vallid coordinate");
if($('#LatLngError').length==0){
errorHTML='<br><span class="ms-error" id="LatLngError">You must specify a value for Lat and Lng</span>'
$('#'+AddressBox).closest('TD').append(errorHTML)
}
return -1
}
return 0
};
It is messy but hopefully you can make sense of it.
Edit 2:
I think I have tracked the randomness down... I completely turned off all custom code and still have the problem. I then tried comparing a working record with a non working record. Everything looked normal until i got to the field with a multiple people picker. If i have more than 2 people in that field it will save normal but when i go to make a modification on that record with more than 2 people in the people picker field is causes this problem. I am going to do some more research and will post my results.
Edit 3:
http://blogs.msdn.com/b/jorman/archive/2009/12/22/mystery-of-the-sharepoint-white-screens.aspx
This problem all boils down to IIS configuration and the Impersonation Level. Apparently our server admins decided to change it without telling anyone.
Usually, when you get [seemingly random] behavior from a web page (especially in MOSS), it means that you have ambiguous events defined on the page. Usually, I get this when I add some kind of JScript to a button or form on_submit.
Without seeing your code, I can't really narrow it down further than that. I recommend: look for JavaScript events on your HTML form or on your button click events or look for anchor [a] tags that point to nowhere (href=#) but have javascript. Then decide to do it (strictly) the HTML way (forms, submit buttons) or the javascript way, (no forms, no asp:button) and un-wire the other.
This problem all boils down to IIS configuration and the Impersonation Level. Apparently our server admins decided to change it without telling anyone.
http://blogs.msdn.com/b/jorman/archive/2009/12/22/mystery-of-the-sharepoint-white-screens.aspx
I am looking for a simple way to take a screenshot of an iFrame in my ASP page. I just couldn't achieve it with C# and I lack of knowledge of Javascript! Does anyone out there know the simple and best way to achieve this?
What I am trying to do is, I am building a website that students can log in to e-government website in my country and prove if they are continuing student with a single click so that they can get discount from our service.
Edit: The puzzle should be solved in local.
this piece of code worked for me. I hope it does the same to the others.
private void saveURLToImage(string url)
{
if (!string.IsNullOrEmpty(url))
{
string content = "";
System.Net.WebRequest webRequest = WebRequest.Create(url);
System.Net.WebResponse webResponse = webRequest.GetResponse();
System.IO.StreamReader sr = new StreamReader(webResponse.GetResponseStream(), System.Text.Encoding.GetEncoding("UTF-8"));
content = sr.ReadToEnd();
//save to file
byte[] b = Convert.FromBase64String(content);
System.IO.MemoryStream ms = new System.IO.MemoryStream(b);
System.Drawing.Image img = System.Drawing.Image.FromStream(ms);
img.Save(#"c:\pic.jpg", System.Drawing.Imaging.ImageFormat.Jpeg);
img.Dispose();
ms.Close();
}
}
Unless I'm misunderstanding you, this is impossible.
You cannot instruct the user's browser to take a screenshot (this would be a security risk … and has few uses cases anyway).
You cannot load the page you want a screenshot of yourself (with server side code) because you don't have the credentials needed to access it.
server side
Take a screenshot of a webpage with JavaScript?
javascript
http://html2canvas.hertzen.com/