We Keep Coding

How to get html source of page, after all it scripts was called?

I'am trying to parse site. The site (i suppose) using scripts and data bases to load data from (dynamically?). And this is my problem... I am trying to grab data through C# (unfortunately i don't have access to code right now) or JS. And it seems like either C# and JS, get only template of the site, but don't wait until all scripts executed. So this is my question, is there any way to get ALL html source? Maybe call scripts somehow. Or make a request, wait for 10 seconds, and then write source html data into variable?
Here is my JS code.
function request(link)
{
var xhr = new XMLHttpRequest();
xhr.open('GET', link, true);
xhr.onreadystatechange = function() .
{console.log(xhr.readyState);};
xhr.send();
let data = xhr.responseText;
var tempDiv = document.createElement('div');
tempDiv.innerHTML = data.replace(/<script(.|\s)*?\/script>/g,
'');
return tempDiv;
}
function loadFile(url, timeout, callback)
{
var args = Array.prototype.slice.call(arguments, 3);
var xhr = new XMLHttpRequest();
xhr.ontimeout = function () {
console.error("The request for " + url + " timed out.");
};
xhr.onload = function() {
if (xhr.readyState === 4) {
if (xhr.status === 200) {
callback.apply(xhr, args);
} else {
console.error(xhr.statusText);
}
}
};
xhr.open("GET", url, true);
xhr.timeout = timeout;
xhr.send(null);
let data = xhr.responseText;
return data;
}
function showMessage (message) {
console.log(message + this.responseText);
}
function include(scriptUrl)
{
var xmlhttp = new XMLHttpRequest();
xmlhttp.open("GET", scriptUrl);
xmlhttp.onreadystatechange = function()
{
if ((xmlhttp.status == 200) && (xmlhttp.readyState == 4))
{
eval(xmlhttp.responseText);
}
};
xmlhttp.send();
let data = JSON.parse(xmlhttp.responseText);
var tempDiv = document.createElement('div');
tempDiv.innerHTML = data.replace(/<script(.|\s)*?\/script>/g,
'');
return tempDiv;
}
All this functions do not work as i want.

This isn't really practical - you're trying to load an HTML page, all associated scripts, then run them on the HTML page as if they were in a proper browser environment, but within your current browser session.
This sort of thing is feasible with the jsdom library if you were running on the server-side (NodeJS), because it simulates browser behaviour: https://github.com/jsdom/jsdom. So you could do
JSDOM.fromURL("https://example.com/", { runScripts: "dangerously" }).then(dom => {
console.log(dom.serialize()); //turn the page back into HTML
});
...to get the whole thing.

In javascript XMLHttpRequest, changing POST request to GET request

My javascript library is sending payload data from client browser to web server.
And, intermittently, I find a weird situation in Web server logs.
When sending data from the client through my library, using the POST method of XHR and the GET method is sent intermittently with the same target URL.
My library code is as follows...
const xhr = new XMLHttpRequest()
xhr.onreadystatechange = function (event) {
if (xhr.readyState === 4 /** responseText is not available yet */) {
const statusCode = xhr.status
const responseText = xhr.responseText
if (responseValidator(statusCode, responseText)) {
if (successCallback) successCallback(xhr)
} else {
const error = new Error(`${ErrorType.INVALID_RESPONSE}: ${statusCode}`)
Logger.error(error)
handleServerError(payloads)
if (failureCallback) failureCallback(error)
}
}
}
xhr.open('POST', url, true /** async */)
xhr.setRequestHeader('Content-Type', 'text/plain; charset=utf8')
xhr.withCredentials = true
xhr.timeout = timeoutMillis
xhr.ontimeout = function (event) {
const error = new Error(`${ErrorType.REQUEST_TIMEOUT} (${timeoutMillis} millis)`)
Logger.error(error)
if (failureCallback) failureCallback(error)
}
xhr.onerror = function (e) {
handleServerError(payloads)
}
xhr.send(JSON.stringify(payloads))
Why is this happening?

Equivalent to $.load without jQuery

I want to load some Jade content into a certain div on button click. I have found how to do this with jquery, there are several posts on it, and essentially what I want to do is
$('#div').load('/somePage');
However, I am unable to use jQuery in my project. Is there an equivalent function in vanilla javascript?

I think you can do this with the following;
var request = new XMLHttpRequest();
request.open('GET', '/somepage', true);
request.onload = function() {
if (request.status >= 200 && request.status < 400) {
var resp = request.responseText;
document.querySelector('#div').innerHTML = resp;
}
};
request.send();
By the way, you can do this with fetch API too.
fetch('/somepage')
.then(function(response) {
return response.text();
})
.then(function(body) {
document.querySelector('#div').innerHTML = body;
});
By the way, you can read this blog post for learning something about fetch API.

While I was trying to solve the same problem, I made this which is based on Ali BARIN's answer, and seems to work great but is a bit more explicit version, adding init information, and has some logic to use document.getElementById instead of querySelector.
/*
* Replicates the functionality of jQuery's `load` function,
* used to load some HTML from another file into the current one.
*
* Based on this Stack Overflow answer:
* https://stackoverflow.com/a/38132775/3626537
* And `fetch` documentation:
* https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/fetch
*
* #param {string} parentElementId - The ID of the DOM element to load into
* #param {string} htmlFilePath - The path of the HTML file to load
*/
const loadHtml = function(parentElementId, filePath) {
const init = {
method : "GET",
headers : { "Content-Type" : "text/html" },
mode : "cors",
cache : "default"
};
const req = new Request(filePath, init);
fetch(req)
.then(function(response) {
return response.text();
})
.then(function(body) {
// Replace `#` char in case the function gets called `querySelector` or jQuery style
if (parentElementId.startsWith("#")) {
parentElementId.replace("#", "");
}
document.getElementById(parentElementId).innerHTML = body;
});
};

You can do it like that, but there is something you'll have to pay attention to it.
const getData = (url) => {
return new Promise((resolve, reject) => {
let request = new XMLHttpRequest();
request.onload = function () {
if (this.readyState === 4 && this.status === 200) {
resolve(this.responseText);
} else {
reject(this.responseText);
}
};
request.open("get", url, true);
request.send();
});
};
getData("Your URL")
.then((resolve) => {
console.log(resolve);
})
.catch((reject) => {
console.error(reject);
});
What I want you to pay attention to is if you put URL to a page it will return it from <html> to </html> as string, I guess there is no way to get just apart from it like method .load() in jQuery.

Adjacently Dependent AJAX (improved)

This question was posted a couple of days ago, but since I'm a nub it was filled with spaghetti code and that sort of thing (please pardon the form handling as well) That aside, I've added some notes and given some context, but the problem still lies in the second AJAX call.
This is the error that Chrome is throwing "Cross origin requests are only supported for protocol schemes: http, data, chrome, chrome-extension, https, chrome-extension-resource."
I have hidden the URL because it contains an API key that I would rather not share.
Any and all criticisms are warmly welcomed
/*
This module will take a user's name, return an ID
then search more stats in the api with the ID.
*/
var search = document.getElementById('search');
search.addEventListener('click', function(){
var demo = document.getElementById('demo');
var player_name = document.getElementById('player_name').value;
var player_id;
// Interpolated API URLs
var name_url = 'URL.pre'+player_name+'URL.end';
var stats_url; //nested in the second ajax call to pass updated player_id
// Get player ID
var xhr = new XMLHttpRequest();
var id_return_text;
xhr.onload = function(){
if(xhr.status === 200) {
id_return_text = JSON.parse(xhr.responseText);
player_id = id_return_text[player_name].id;
demo.innerHTML = id_return_text[player_name].name +', your player ID is: '+player_id;
}
};
xhr.open('GET', name_url, true);
xhr.send();
// Search stats with ID
var xhr_2 = new XMLHttpRequest();
var stats_return_text;
xhr.done = function(){
stats_url = "URL.pre"+player_id+"URL.end";
if(xhr_2.status == 200) {
stats_return_text = JSON.parse(xhr_2.responseText);
demo.innerHTML += stats_return_text['playerStatsSummaries'].playerStatType;
}
};
xhr_2.open("GET",stats_url, true);
xhr_2.send();
});
<div id="container">
<img id="duck" src="duck.png" alt="duck">
<div class="form_wrapper">
<h1 id="app_header">*QUACK* What's Your player ID?</h1>
<form>
<input
type="text"
id="player_name"
placeholder="Summoner Name">
<input type="button" id="search" value="Search">
</form>
</div>
<p id="demo"></p>
</div>
<script type="text/javascript" src="script.js"></script>

So your primary error was that if you need to make CORS requests (or any AJAX requests, really), you need to run the code from a server (even localhost).
Google (and most browsers) will freak out at you if your page's protocol is "file:///" and you're trying to load things from the internet (or vice versa). And "file:///" cannot make requests for other files, either.
Future reference: you also can't make "http" requests from an "https" page.
That out of the way, the second issue (the one that was being hidden by CORS security), is that your AJAX requests are being run in parallel right now.
In order to make this work the way you think it should (after the first one returns, run the second one), you would need to:
move all of the code at the bottom, relating to xhr_2 inside of the xhr.onload
move all of the code inside of xhr.done at the bottom inside of the xhr.onload and replace all of the duplicate information (and use the references to the returned results directly)
This results in something like:
var search = document.getElementById('search');
search.addEventListener('click', function(){
var demo = document.getElementById('demo');
var player_name = document.getElementById('player_name').value;
var player_id;
// Interpolated API URLs
var name_url = 'https://na.api.pvp.net/api/lol/na/v1.4/summoner/by-name/'+player_name+'?api_key=<THIS IS THE API KEY>';
var stats_url; //nested in the second ajax call to pass updated player_id
// Get player ID
var xhr = new XMLHttpRequest();
var id_return_text;
xhr.onload = function(){
if(xhr.status === 200) {
id_return_text = JSON.parse(xhr.responseText);
player_id = id_return_text[player_name].id;
demo.innerHTML = id_return_text[player_name].name +', your player ID is: '+player_id;
// Dropped the XHR_2 stuff here
var xhr_2 = new XMLHttpRequest();
var stats_return_text;
stats_url = "https://na.api.pvp.net/api/lol/na/v1.3/stats/by-summoner/"+player_id+"/summary?season=SEASON2016&api_key=<THIS IS THE API KEY>";
// CHANGED THIS TO BE XHR_2.onload -- IN HERE I KNOW XHR_1 IS ALREADY FINISHED
xhr_2.onload = function(){
if(xhr_2.status == 200) {
stats_return_text = JSON.parse(xhr_2.responseText);
demo.innerHTML += stats_return_text['playerStatsSummaries'].playerStatType;
}
};
xhr_2.open("GET",stats_url, true);
xhr_2.send();
}
};
xhr.open('GET', name_url, true);
xhr.send();
});
That should solve practically all of your woes.
The point of this is that onload is a callback which gets fired long after the program has been run, but xhr_2 was firing immediately after you requested data for xhr_1 (not after it was returning the data).
As such, player_id was undefined.
We want to wait until after we know we have player_id, and we know we have it (or some error) when we're inside the callback to xhr_1.onload.
This gets terribly confusing and very nested, and while I think that Promises and Async Functions / Generators are brilliant solutions for managing that complexity, that's way beyond the scope of this; so instead, I'd suggest looking at some functional composition, to simplify all of this:
function noop () { } // do nothing
function getJSON (url, onload, onerror) {
var xhr = new XMLHttpRequest();
onload = onload || noop; // what I've been given or nothing
onerror = onerror || noop; // " "
xhr.onload = function () {
var data;
var error;
try {
// it's possible for parse to throw on malformed JSON
data = JSON.parse(xhr.responseText);
} catch (e) {
error = e;
}
return error ? onerror(error) : onload(data); // fire one or the other (don't fall into the handler, if onload throws)
};
xhr.onerror = onerror;
xhr.open("GET", url);
xhr.send();
}
// localize URL construction
function buildPlayerIdUrl (name) { return "https://______" + name + "_____"; }
function buildPlayerStatsUrl (id) { return "https://______" + id + "_____"; }
// gets player by name and runs a function after the player has been loaded
function getPlayer (player_name, done, error) {
var id_url = buildPlayerIdUrl(player_name);
function buildPlayer (response) {
var player = response[player_name];
return player;
}
function onload (response) {
done(buildPlayer(response));
}
// Load the JSON, build the player, pass the player to done()
getJSON(url, onload, error);
}
// get stats by player id and runs a function after the stats have been loaded
function getPlayerStats (player_id, done, error) {
var stats_url = buildPlayerStatsUrl(player_id);
function buildStats (response) {
var summary = response.playerStatsSummaries;
return summary;
}
function onload (response) {
done(buildStats(response));
}
// Load the JSON, build the stats, pass the stats to done()
getJSON(stats_url, onload, error);
}
// perform a search by player name
// note: All changes in step-number (1, 2, 3) are asynchronous,
// and thus, must be nested in callbacks of some sort
function search (player_name) {
// Step 1: load the player
getPlayer(playerName, function (player) {
// Step 2a: update the DOM with the player name/id
updatePlayerDom(player);
// Step 2b: load the player stats
getPlayerStats(player.id, function (stats) {
// Step 3: update the DOM with the stats
updateStatsDom(stats);
});
});
}
// player DOM update; keeping it nice and simple
function updatePlayerDom (player) {
document.querySelector(".Player-id").textContent = player.id;
document.querySelector(".Player-name").textContent = player.name;
}
// stats DOM update; same as above
function updateStatsDom (stats) {
document.querySelector(".Player-stats").textContent = stats.playerStatType;
}
// bootstrap yourself to your UI
some_button.onclick = function () {
var player_name = some_input.value;
search(player_name); // kick the whole thing off
};
It's definitely more code, but it's also simpler to make edits to each individual piece, without stepping on the toes of other pieces.
It's (hopefully) also easier to see the _eventual timeline_ of all of the pieces, and how they flow, inside of the search( ) itself.

javascript web workers - how do I pass arguments?

Found the answer some some of my problems, html5 web workers!!!
How do I pass an argument to a web worker though using this basic example?
contents of worker.js:
function doSomething() {
postMessage( ' done');
}
setTimeout ( "doSomething()", 3000 );
js code:
var worker = new Worker('worker.js');
worker.onmessage = function (event) {
alert(event.data);
};

As you can see you have the same mechanism for both worker-to-main and main-to-worker messages.
the postMessage method for sending messages
the onmessage member for defining the handler that receives the messages
In the main script:
worker.postMessage(data);
In the worker script:
self.addEventListener("message", function(e) {
// the passed-in data is available via e.data
}, false);
... or just...
onmessage = function(e) {
// the passed-in data is available via e.data
};
It may be that data has to be a string... (Firefox 3.5+ supports passing in JSON-compatible objects)

var worker = new Worker(window.App.baseUrl + '/Scripts/signature/GetCurrenProductWorker.js');
worker.postMessage(window.App.baseUrl)
var _base_url = ''
var xhr = new XMLHttpRequest();
onmessage = function (e) {
_base_url = e.data
xhr.open("GET", _base_url + "/api/product/Get");
xhr.onload = function () {
postMessage(xhr.responseText);
};
xhr.send();
};
this work for me