We Keep Coding

Chrome extension: Get the text of a web page from given url

First, I'm completly newbie making chrome extension, then in a part of the chrome extension I will receive differents urls and I want to store the text of the web page to process it later, resulting in an array of boolean variables, each associated with the given url. Schematically it would be something like this:
var result;
function process(text){
if something -> result.push(true);
if not -> result.push(false);
}
function main(){
for (i...){
url = given[i];
text = getHTMLText(url);
process(text);
}
final();//when the loop finish activate another function that use the global variable: result
}
I have problems with main function, first I have tried with synchronous XMLHttpRequest, although it works it's very slow and chrome always give the warning that synchronous XMLHttpRequest is deprecated.
for (var i = 0; i < urls.length; i++){
url = urls[i];
var req = new XMLHttpRequest();
req.open('GET', url, false);
req.send(null);
if (req.status == 200) detecting(req.responseText);
};
Other solution that I find was use fetch(url), but the code that I find I don't fully understand. Although the returned text works correctly but then the proccess function give different results on each page update.
for (var i = 0; i < urls.length; i++){
url = urls[i];
fetch(url).then(function(response) {
response.text().then(function(text) {
detecting(text);
});
});
};
Other problem, but this is because of the little knowledge I have of fetch(), was that I can't store the text out of the fetch(), every time I do console.log give undefined, this greatly complicates the processing of the text for me.
I have seen that maybe it can be done through extension APIs of chrome but I can't see how to do it.

The algorithm shown in your main pseudocode can be implemented easily by using async/await and Promise.all, without a for loop:
(async () => {
const results = await Promise.all(urls.map(processUrl));
console.log(results);
// further processing must be also inside this IIFE
})();
async function processUrl(url) {
try {
const text = await (await fetch(url)).text();
return {url, text, status: detecting(text)};
} catch (error) {
return {url, error};
}
}

nodelist sometimes loads after the function is finished

I have an issue where I sometimes am able to load the nodelist before it is being called but at the same time it sometimes loads after it is being called(Causing an error of the list being undefined).
This is what I wish would appear all the time
Sorry, this is the right image now. This is the error I receive sometimes.
I believe this is the issue but I do not know how to fix it
I have done some searching online and I think it is related to the code being async or synchrous..(I have not learned about this so I am unsure if I am correct). Here's my code. Context: the getNeighbourhoodData() is being onloaded to the body of my html page.
function getNeighbourhoodData(){
var request = new XMLHttpRequest();
request.open('GET', neighbourhood_url, true);
//This function will be called when data returns from the web api
request.onload = function() {
//get all the restaurant records into our neighbourhood array
neighbourhood_array = JSON.parse(request.responseText);
//get User data
displayNeighbourhoods();
};
//This command starts the calling of the restaurant web api
request.send();
}
function displayNeighbourhoods() {
var list = document.getElementsByName("neiList");
console.log(list);
num=0;
alphabet_array=["A","B","C","D","E","F","G","H","I","J","K","L","M","N","O","P","Q","R","S","T","U","V","W","X","Y","Z"];
console.log(alphabet_array);
for (var count = 0; count < neighbourhood_array.length; count++) {
var neighbourhood = neighbourhood_array[count].Neighbourhood;
if(neighbourhood_array[count].Neighbourhood.startsWith(alphabet_array[num])== true ){
var cell = '<li><a class="a--grey" href="/restByNeighbourhood.html" onclick="getName(this)" name="Paya Lebar">'+ neighbourhood +'</a></li>';
list[num].insertAdjacentHTML('beforeend', cell);
if(count >= neighbourhood_array.length-1 && num <= 25){
num+=1;
count=-1;
console.log(num);
}
}
else if(count >= neighbourhood_array.length - 1 && num <= 25){
num+=1;
count=-1;
console.log(num);
}
else if(num >= 26){
break;
}
else{
continue;
}
}
}

JavaScript is Single threaded, which means only one thing can happen at a time. However, with async calls you can "act" like a multy-threaded language.
For example the build-in fetch() funktion returns a Promise that you can await.
async function loadURLodContent() { //
const result = await fetch(/* url-path */);
}
So you can await Promises and write async funktions that return promises.
But this topic isnt an easy one. I'd really recomend getting into Promises and Async calls as soon as possible because you're gonna encounter them if you develop in the Web sooner or later.
But to your Problem.... at least from my point of view you're not giving enough information. Tracer69 hase a good proposal for that in the comments.

Struggling with Q.js promises

I am sure I am missing something obvious but I can't seem to make heads or tails of this problem. I have a web page that is being driven by javascript. The bindings are being provided by Knockout.js, the data is coming down from the server using Breeze.js, I am using modules tied together with Require.js. My goal is to load the html, load the info from Breeze.js, and then apply the bindings to show the data to the user. All of these things appear to be happening correctly, just not in the correct order which is leading to weird binding errors. Now on to the code.
I have a function that gets called after the page loads
function applyViewModel() {
var vm = viewModel();
vm.activate()
.then(
applyBindings(vm)
);
}
This should call activate, wait for activate to finish, then apply bindings....but it appears to be calling activate, not waiting for it to finish and then runs applybindings.
activate -
function activate() {
logger.log('Frames Admin View Activated', null, 'frames', false);
return datacontext.getAllManufacturers(manufacturers)
.then(function () {
manufacturer(manufacturers()[0]);
}).then(function () {
datacontext.getModelsWithSizes(modelsWithSizes, manufacturers()[0].manufacturerID())
.then(datacontext.getTypes(types));
});
}
datacontext.getAllManufacturers -
var getAllManufacturers = function (manufacturerObservable) {
var query = entityQuery.from('Manufacturers')
.orderBy('name');
return manager.executeQuery(query)
.then(querySucceeded)
.fail(queryFailed);
function querySucceeded(data) {
if (manufacturerObservable) {
manufacturerObservable(data.results);
}
log('Retrieved [All Manufacturer] from remote data source',
data, true);
}
};
datacontext.getModelsWithSizes -
var getModelsWithSizes = function (modelsObservable, manufacturerId) {
var query = entityQuery.from('Models').where('manufactuerID', '==', manufacturerId)
.orderBy('name');
return manager.executeQuery(query)
.then(querySucceeded)
.fail(queryFailed);
function querySucceeded(data) {
if (modelsObservable) {
for (var i = 0; i < data.results.length; i++) {
datacontext.getSizes(data.results[i].sizes, data.results[i].modelID());
// add new size function
data.results[i].addNewSize = function () {
var newValue = createNewSize(this.modelID());
this.sizes.valueHasMutated();
return newValue;
};
}
modelsObservable(data.results);
}
log('Retrieved [Models With Sizes] from remote data source',
data, false);
}
};
Any help on why this promise isn't working would be appreciated, as would any process to figure it out so I can help myself the next time I run into this.

A common mistake when working with promises is instead of specifying a callback, you specify the value returned from a callback:
function applyViewModel() {
var vm = viewModel();
vm.activate()
.then( applyBindings(vm) );
}
Note that when the callback returns a regular truthy value (number, object, string), this should cause an exception. However, if the callback doesn't return anything or it returns a function, this can be tricky to locate.
To correct code should look like this:
function applyViewModel() {
var vm = viewModel();
vm.activate()
.then(function() {
applyBindings(vm);
});
}

Concept - Designing a collapsible queue for asynchronous resources

I've noticed that the size of a file requested will effect how long the response takes for ajax calls. So if I fire 3 ajax GET requests for files of varying size, they may arrive in any order. What I want to do is guarantee the ordering when I append the files to the DOM.
How can I set up a queue system so that when I fire A1->A2->A3. I can guarantee that they are appeneded as A1->A2->A3 in that order.
For example, suppose A2 arrives before A1. I would want the action to wait upon the arrival and loading of A1.
One idea is to create a status checker using a timed callback as such
// pseudo-code
function check(ready, fund) {
// check ready some how
if (ready) {
func();
} else {
setTimeout(function () {
check(ready, fund);
}, 1); // check every msec
}
}
but this seems like a resource heavy way, as I fire the same function every 1msec, until the resources is loaded.
Is this the right path to complete this problem?

status checker using a 1msec-timed callback - but this seems like a resource heavy way; Is this the right path to complete this problem?
No. You should have a look at Promises. That way, you can easily formulate it like this:
var a1 = getPromiseForAjaxResult(ressource1url);
var a2 = getPromiseForAjaxResult(ressource2url);
var a3 = getPromiseForAjaxResult(ressource3url);
a1.then(function(res) {
append(res);
return a2;
}).then(function(res) {
append(res);
return a3;
}).then(append);
For example, jQuery's .ajax function implements this.

You can try something like this:
var resourceData = {};
var resourcesLoaded = 0;
function loadResource(resource, callback) {
var xhr = new XMLHttpRequest();
xhr.onload = function() {
var state = this.readyState;
var responseCode = request.status;
if(state == this.DONE && responseCode == 200) {
callback(resource, this.responseText);
}
};
xhr.open("get", resource, true);
xhr.send();
}
//Assuming that resources is an array of path names
function loadResources(resources) {
for(var i = 0; i < resources.length; i++) {
loadResource(resources[i], function(resource, responseText) {
//Store the data of the resource in to the resourceData map,
//using the resource name as the key. Then increment the
//resource counter.
resourceData[resource] = responseText;
resourcesLoaded++;
//If the number of resources that we have loaded is equal
//to the total number of resources, it means that we have
//all our resources.
if(resourcesLoaded === resources.length) {
//Manipulate the data in the order that you desire.
//Everything you need is inside resourceData, keyed
//by the resource url.
...
...
}
});
}
}
If certain components must be loaded and executed before (like certain JS files) others, you can queue up your AJAX requests like so:
function loadResource(resource, callback) {
var xhr = new XMLHttpRequest();
xhr.onload = function() {
var state = this.readyState;
var responseCode = request.status;
if(state == this.DONE && responseCode == 200) {
//Do whatever you need to do with this.responseText
...
...
callback();
}
};
xhr.open("get", resource, true);
xhr.send();
}
function run() {
var resources = [
"path/to/some/resource.html",
"path/to/some/other/resource.html",
...
"http://example.org/path/to/remote/resource.html"
];
//Function that sequentially loads the resources, so that the next resource
//will not be loaded until first one has finished loading. I accomplish
//this by calling the function itself in the callback to the loadResource
//function. This function is not truly recursive since the callback
//invocation (even though it is the function itself) is an independent call
//and therefore will not be part of the original callstack.
function load(i) {
if (i < resources.length) {
loadResource(resources[i], function () {
load(++i);
});
}
}
load(0);
}
This way, the next file will not be loaded until the previous one has finished loading.
If you cannot use any third-party libraries, you can use my solution. However, your life will probably be much easier if you do what Bergi suggested and use Promises.

There's no need to call check() every millisecond, just run it in the xhr's onreadystatechange. If you provide a bit more of your code, I can explain further.

I would have a queue of functions to execute and each of them checks the previous result has completed before executing.
var remoteResults[]
function requestRemoteResouse(index, fetchFunction) {
// the argument fetchFunction is a function that fetches the remote content
// once the content is ready it call the passed in function with the result.
fetchFunction(
function(result) {
// add the remote result to the list of results
remoteResults[index] = result
// write as many results as ready.
writeResultsWhenReady(index);
});
}
function writeResults(index) {
var i;
// Execute all functions at least once
for(i = 0; i < remoteResults.length; i++) {
if(!remoteResults[i]) {
return;
}
// Call the function that is the ith result
// This will modify the dom.
remoteResults[i]();
// Blank the result to ensure we don't double execute
// Store a function so we can do a simple boolean check.
remoteResults[i] = function(){};
}
}
requestRemoteResouse(0, [Function to fetch the first resouse]);
requestRemoteResouse(1, [Function to fetch the second resouse]);
requestRemoteResouse(2, [Function to fetch the thrid resouse]);
Please note that this is currently O(n^2) for simplicity, it would get faster but more complex if you stored an object at every index of remoteResults, which had a hasRendered property. Then you would only scan back until you found a result that had not yet occurred or one that has been rendered.

Casper JS waitForResource with a restful API

We are having a little problem with a functional test with casper.js.
We request the same resource twice, first with the GET and then with POST method.
Now when waiting for the second resource (POST) it matches the first resource and directly goes to the "then" function.
We would like to be able to check for the HTTP method in the "test" function, that way we can identify the resource properly. For now we use the status code (res.status), but that doesn't solve our problem fully, we really need the http method.
// create new email
this.click(xPath('//div[#id="tab-content"]//a[#class="button create"]'));
// GET
this.waitForResource('/some/resource',
function then() {
this.test.assertExists(xPath('//form[#id="email_edit_form"]'), 'Email edit form is there');
this.fill('form#email_edit_form', {
'email_entity[email]': 'test.bruce#im.com',
'email_entity[isMain]': 1
}, true);
// POST
this.waitForResource(
function test(res) {
return res.url.search('/some/resource') !== -1 && res.status === 201;
},
function then() {
this.test.assert(true, 'Email creation worked.');
},
function timeout() {
this.test.fail('Email creation did not work.');
}
);
},
function timeout() {
this.test.fail('Email adress creation form has not been loaded');
});
Or maybe there is a better way to test this scenario? Although since this is a functional test we need to keep all those steps in one test.

You can try to alter the form action url to add some query string, therefore generating a new resource appended to the stack. Could be done this way:
casper.thenEvaluate(function() {
var form = __utils__.findOne('#email_edit_form');
form.setAttribute('action', form.getAttribute('action') + '?plop');
});
That's a hack though, and functional testing should never be achieved that way. Let's hope more information will be added to the response objects in the future.

The res parameter that is passed to the test function has an ID. I created a helper that tests against this ID and blacklists it, so the same resource won't get accepted a second time.
var blackListedResourceIds = [],
testUniqueResource = function (resourceUrl, statusCode) {
return function (res) {
// check if resource was already loaded
var resourceFound = res.url.search(resourceUrl) !== -1;
// check statuscode
if (statusCode !== undefined) {
resourceFound = resourceFound && res.status === statusCode;
}
// check blacklisting
if (!resourceFound || blackListedResourceIds[res.id] !== undefined) {
return false;
} else {
blackListedResourceIds[res.id] = true;
return true;
}
};
};