When I click on a certain link the server writes if I have done it by stating either true or false in a text file...
Just like this:
{"needs_click":false,"has_clicked":false,"sites":[{"id":2134,"name":"testing","has_clicked":false,"click_url":"http://testing.com?siteid=5433"}]}
Now what I have done is uploaded a text file to my FTP already and changed all the false results into true like this:
{"needs_click":true,"has_clicked":true,"sites":[{"id":2134,"name":"testing","has_clicked":true,"click_url":"http://testing.com?siteid=5433"}]}
Now in one of the JavaScript files attached to the source code there is the variable at the top directing to the text file I just need to know how to make my browser read the spoofed link I created instead of the original.
I need to change the variables:
var API_VERSION = 0.1;
var API_URL = "http://api.testwebsite.com/" + API_VERSION + "/";
To:
var API_VERSION = 0.1;
var API_URL = "http://api.testwebsite2.com/" + API_VERSION + "/";
So it reads the other text file stating all the true options instead of the original... any suggestions?
What if you used a personal proxy to shape the query? Something like Charles proxy could work. In any case you will need to get in between the browser and server to do this sort of thing.
Alternatively you could try to rewrite entire JavaScript (ie. remove the original from source before loading and inject your own with the url fixed, you might be able to get away with a replace on the original). I don't know how feasible this is, though, as I don't know internals of Greasemonkey well enough. But this might be something to explore.
Was impossible, at the time I was very new to programming and limitations on what I could and could not do.
Related
I've seen some answers to this that refer the askee to other libraries (like phantom.js), but I'm here wondering if it is at all possible to do this in just node.js?
Considering my code below. It requests a webpage using request, then using cheerio it explores the dom to scrape the page for data. It works flawlessly and if everything had gone as planned, I believe it would have outputted a file as i imagined in my head.
The problem is that the page I am requesting in order to scrape, build the table im looking at asynchronously using either ajax or jsonp, i'm not entirely sure how .jsp pages work.
So here I am trying to find a way to "wait" for this data to load before I scrape the data for my new file.
var cheerio = require('cheerio'),
request = require('request'),
fs = require('fs');
// Go to the page in question
request({
method: 'GET',
url: 'http://www1.chineseshipping.com.cn/en/indices/cbcfinew.jsp'
}, function(err, response, body) {
if (err) return console.error(err);
// Tell Cherrio to load the HTML
$ = cheerio.load(body);
// Create an empty object to write to the file later
var toSort = {}
// Itterate over DOM and fill the toSort object
$('#emb table td.list_right').each(function() {
var row = $(this).parent();
toSort[$(this).text()] = {
[$("#lastdate").text()]: $(row).find(".idx1").html(),
[$("#currdate").text()]: $(row).find(".idx2").html()
}
});
//Write/overwrite a new file
var stream = fs.createWriteStream("/tmp/shipping.txt");
var toWrite = "";
stream.once('open', function(fd) {
toWrite += "{\r\n"
for(i in toSort){
toWrite += "\t" + i + ": { \r\n";
for(j in toSort[i]){
toWrite += "\t\t" + j + ":" + toSort[i][j] + ",\r\n";
}
toWrite += "\t" + "}, \r\n";
}
toWrite += "}"
stream.write(toWrite)
stream.end();
});
});
The expected result is a text file with information formatted like a JSON object.
It should look something like different instances of this
"QINHUANGDAO - GUANGZHOU (50,000-60,000DWT)": {
"2016-09-29": 26.7,
"2016-09-30": 26.8,
},
But since the name is the only thing that doesn't load async, (the dates and values are async) I get a messed up object.
I tried Actually just setting a setTimeout in various places in the code. The script will only be touched by developers that can afford to run the script several times if it fails a few times. So while not ideal, even a setTimeout (up to maybe 5 seconds) would be good enough.
It turns out the settimeouts don't work. I suspect that once I request the page, I'm stuck with the snapshot of the page "as is" when I receive it, and I'm in fact not looking at a live thing I can wait for to load its dynamic content.
I've wondered investigating how to intercept the packages as they come, but I don't understand HTTP well enough to know where to start.
The setTimeout will not make any difference even if you increase it to an hour. The problem here is that you are making a request against this url:
http://www1.chineseshipping.com.cn/en/indices/cbcfinew.jsp
and their server returns back the html and in this html there are the js and css imports. This is the end of your case, you just have the html and that's it. Instead the browser knows how to use and to parse the html document, so it is able to understand the javascript scripts and to execute/run them and this is exactly your problem. Your program is not able to understand that has something to do with the HTML contents. You need to find or to write a scraper that is able to run javascript. I just found this similar issue on stackoverflow:
Web-scraping JavaScript page with Python
The guy there suggests https://github.com/niklasb/dryscrape and it seems that this tool is able to run javascript. It is written in python though.
You are trying to scrape the original page that doesn't include the data you need.
When the page is loaded, browser evaluates JS code it includes, and this code knows where and how to get the data.
The first option is to evaluate the same code, like PhantomJS do.
The other (and you seem to be interested in it) is to investigate the page's network activity and to understand what additional requests you should perform to get the data you need.
In your case, these are:
http://index.chineseshipping.com.cn/servlet/cbfiDailyGetContrast?SpecifiedDate=&jc=jsonp1475577615267&_=1475577619626
and
http://index.chineseshipping.com.cn/servlet/allGetCurrentComposites?date=Tue%20Oct%2004%202016%2013:40:20%20GMT+0300%20(MSK)&jc=jsonp1475577615268&_=1475577620325
In both requests:
_ is a decache parameter to prevent caching.
jc is a name of a JS wrapper function which should be invoked with the result (https://en.wikipedia.org/wiki/JSONP)
So, scrapping the table template at http://www1.chineseshipping.com.cn/en/indices/cbcfinew.jsp and performing two additional requests you will be able to combine them into the same data structure you see in the browser.
I'm using Electron for a project of mine. I need to pass an URL between windows, which I'm doing by using the URL the following way:
function openWindow(url) {
url=encodeURIComponent(url);
const remote = require('electron').remote;
const BrowserWindow = remote.BrowserWindow;
var win = new BrowserWindow({ width: 800, height: 600 });
win.loadURL('file://' + __dirname + '/otherwindow.html?url=' + url);
}
On the receiving end (in otherwindow.html) I get the parameter this way:
var urlParam = function(name, w){
w = w || window;
var rx = new RegExp('[\?]'+name+'=([^\#]+)'),
val = w.location.search.match(rx);
return !val ? '':val[1];
}
I call this function as:
var decoded=decodeURIComponent(urlParam('url'));
And this all works fine. It's kind of ugly right now, but it works. Or so it seems. If I print out the decoded URL to the console it displays correctly. It seems to be fine, if I open up an Electron window with it, it displays the destination with no problem what so ever.
Here's the catch. I'm using wcjs-player for this project. In the destination page (otherwindow.html) is a wcjs-player instance, which would take the decoded URL and play the media located there with .addPlaylist(); or .vlc.play();.
It works fine if I put the destination URL in a variable in the same page then l put it as a parameter to these 2 functions, even works if I use encodeURIComponent(); and then decodeURIComponent(); on said variable, so encoding the URL is not the problem. I even tried with a base64 encoding to pass between the pages, no success.
So judging from all this, I reckon that the problem is not the encoding itself, but the passing between pages. My (probably wrong) theory is that the URL might get somehow very slightly altered (losing/gaining some information, special characters maybe?), which wcjs-player is not prepared to handle, but Chromium is (since there's no problem with the Electron window using the result URL).
I have no idea to fix this, I've tried all my ideas. Did a fair bit of searching, but didn't really find anything useful. I can solve it another way, but that would involve opening and processing the same file twice - in both windows - which I'm trying to avoid.
I've pretty new to Javascript overall, so please excuse me if I'm missing something trivial. Any help is appreciated!
I am building an app with Phonegap and jQuerymobile. The app roughly works like this:
1) The app downloads a ZIP file from a public server and then unzips them to a local folder. I got the local folder path from fileSystem.root.toNativeURL() (in OS, it's something like this: file://var/mobile/Container/Data/Application/xxxx/Documents/)
2) App redirects to HTML that was unzipped in local folder (ex: file://var/mobile/Container/Data/Application/xxxx/Documents/index.html)
I am now facing issues b/c inside the index.html file, all the links are absolute path (ex: Link). This breaks all the links since (I assume) they are all now pointing to file://content/index2.html instead of file://var/mobile/Container/Data/Application/xxxx/Documents/content/index2.html.
My question is, how should I handle the links? I am thinking i should just rewrite all the links to force prepend the local folder URL in front of it. Is there a better way?
And if rewriting links is the way to go, how can I do this with jQuerymobile? I did this in jQuery which seems to work http://jsfiddle.net/jg4ouqc5/ but this code doesn't work in my app (jQueryMobile)
When you are loading index.html, you are getting file://some_path/..../index.html as your base URL. Any links which will be encountered now own-wards can be resolved in relation to the base URL.
You would know your scenario better. There could be multiple ways in which this can be fixed.
Have a contract with the CMS/Code generator. Links should always be generated either Relative to the base URL or Absolute. The links you are getting in the page are wrong - Link it ideally should be Link or fully qualified like https://www.google.com.
If you want to change the URL then you can use native code to change it after unzipping the content. It will be really straight forward.
If you want to change the URL in browser then you will have to persist the base url and then take care of couple of things:
a. absolute urls - In your case you can just check the window.location.protocol, if it starts with 'http' and then skip it.
b. sub-directories
Here is a small I have written:
Note: I have not tried this code and you might have to change it according to your need.
$(document).ready(function(){
var base_file_name = window.location.pathname.substring(window.location.pathname.lastIndexOf('/') + 1);
//In index.html (persist this value in native)
var baseUrl = window.location.href.replace(base_file_name, "");
$("a").each(function () {
this.href = baseUrl + this.pathname;
$(this).click(function (e) {
e.preventDefault();
alert(this.pathname);
window.location.href = this.href;
});
});
});
The example you linked should work, make sure you have the <base> set correctly and that you are using the correct string to replace.
Yeah, your going to have to normalize all URL's when your page loads. I can't test with phonegap right now, but your basePath will need to be one of the following:
The file path as you described in your answer (not likely)
window.location.origin (optionally including window.location.pathname)
CODE:
// mini dom ready - https://github.com/DesignByOnyx/mini-domready
(function(e,t,n){var r="attachEvent",i="addEventListener",s="DOMContentLoaded";if(!t[i])i=t[r]?(s="onreadystatechange")&&r:"";e[n]=function(r){/in/.test(t.readyState)?!i?setTimeout(function(){e[n](r)},9):t[i](s,r,false):r()}})
(window,document,"domReady");
domReady(function () {
var anchors = document.getElementsByTagName['a'],
basePath = /* get your base path here, without a trailing slash */;
Array.prototype.forEach.call(anchors, function( anchor ){
anchor.setAttribute('href', basePath + anchor.getAttribute('href'));
});
});
Remove the forward slash from the beginning of your links.
href="content/index2.html">
I created a bilingual website using two databases:
www.martyregan.com/
www.martyregan.com/jp/
You can choose the language of the website using the 'Website Language' flags, but currently the links only bring you to the homepage. The paths/URLs on both sites are exactly the same, other than /jp/ directory on the Japanese site.
I'm looking for a way to alter the hyperlinks to go to the parallel page, based on the URL of the page the visitor is currently on. I figure it'd be quite simple being that the paths are identical, but not really sure where to start with my little knowledge of jquery.
This assumes your language is accessible as seen below.
"http://" is removed from the URL for convenience.
$(function(){
var lang = 'jp';
$('a').attr('href', function(x, url){
var split = url.replace(/(http:)?(\/\/)?/, '').split('/');
return split.shift() + '/' + lang + '/' + split.join('/');
});
});
You may be able to get away with using the <base> tag depending up on the browsers you need to support. You can just set that to whatever the base is by acquiring it from the server.
If you want to use jQuery to do it, it should be fairly simple if all of your hrefs are absolute:
$("a").attr('href', function (_, href) {
return $("base").attr('href') + href;
});
This assumes that you are using <base>. If you're not you can get the path from window.location.path, or even some other element on the page.
In case you are confused, the JavaScript is only required if <base> is not enough to work on its own
Question:
IE and Firefox / Safari seem to deal differently with BASE HREF and Javascript window.location type requests. First, is this an accurate description of the problem? What's going on? And what's the best cross-browser solution to deal with this situation?
Context:
I have a small PHP flat file sitelet (it's actually a usability testing prototype).
I dynamically generate the BASE tag's HREF value in PHP, i.e. if it's running on our company's server, it's:
$basehref = 'http://www.example.com/alpha/bravo/UsabilityTest/';
and on my local dev machine, it's:
$basehref = 'http://ellen.local/delta/echo/foxtrot/UsabilityTest/';
For one of the tasks, I collect some user input, do some transformations on it in Javascript, and send to the server using code like this:
function allDone() {
// elided code for simplicity of stackoverflow question
var URI = "ProcessUserInput.php?";
URI = URI + "alphakeys=" + encodeURI( keys.join(",") );
URI = URI + "&sortedvalues=" + encodeURI( values.join(",") );
window.location = URI;
}
Both the javascript file (containing function allDone()) and the processing PHP script (ProcessUserInput.php) live in a subdirectory of UsabilityTest. In other words, their actual URL is
http://www.example.com/alpha/bravo/UsabilityTest/foxtrot/ProcessUserInput.php
aka
$basehref . '/foxtrot/ProcessUserInput.php'
The Problem
IE's JavaScript basically seems to ignore the BASE HREF. The javascript and the PHP processor live in the same directory, so the call to ProcessUserInput.php works out fine. The input gets processed and everything works fine.
But when I test on Firefox, the JavaScript does appear to use the BASE HREF, because the script's output gets sent to
$basehref . '/ProcessUserInput.php'
This breaks, because ProcessUserInput.php is in a subdirectory of basehref. However, if I add the subdirectory name to the javascript, it no longer works in IE.
Solutions?
I can think of a few ways to solve this:
In Javascript, read the HREF property of the BASE tag and manually prepend to var URI in the javascript, calling a fully-resolved absolute URL
Process the .js file with PHP and insert the $basehref variable into the script
Move the files around
Something else?
I'm sure there must be other ways to solve this too. What's the best way to deal with BASE HREF in JavaScript when IE and Firefox apply it differently in JavaScript?
Using the assign method of window.location seems like the most straightforward answer.
Instead of
window.location = URI;
I'm using this:
window.location.assign( URI );
which is doing the right thing in both IE and Firefox.
IE and Firefox / Safari seem to deal differently with BASE HREF and Javascript window.location type requests.
Yes, this is a long-standing difference going back to the early days of Netscape-vs-IE.
IE enforces base-href only at the point a document element is interacted-with. So, you can createElement('a'), set a relative href and click() it*, but the base-href will be ignored; appendChild it to the document containing the base-href, and it'll work.
On the other browsers the base-href is taken as global per-window and always applied. Which is right? It seems to be unspecified. The original JavaScript docs say only that location.hash (and hence, location applied as a string):
represents a complete URL
So setting it to a relative URL would seem to be an undefined operation.
(*: link.click() is a non-standard method supported by IE and Opera)
read the HREF property of the BASE tag and manually prepend
Probably what I'd do, yeah, if you're dead set on using <base>.
I believe you want to modify window.location.pathname, not window.location. window.location is a Location object, that has multiple variables. As a result, the effects of changing it is not well defined. However, window.location.pathname is defined as the path relative to the host, which is what you want.
If you want to read up more on the many variables you can change in window.location, I'd check here. According to Mozilla's documentation, changing any variable in window.location should reload the page with a new URL corresponding to those changes.
I had the same problem today, after some researching, couldn´t findn any way to override this issue in IE9, what is a requiremente for my project, so, i did the following approach (jquery based, but it´s really easy to make it in simple javascript).
href = function(url){
if ($("base").length > 0 ){
location.href= $("base").attr("href")+url;
}else{
location.href = url;
}
}
And then, change
location.href= 'emp/start'
to
href('emp/start');
just add $('base').attr('href') before the link. (using jquery) or
document.getElementBytagname('base').href
You can always use Vanilla JS :)
var href = document.getElementBytagname('base')[0].href
I hope this helps.