We Keep Coding

Eloquent Javavascript, Chapter 11 (Asynchronous Programming), Question on path finding algorithm

I'm having a problem understanding a line of code from Eloquent Javascript ebook, Chapter 11 (Message Routing section). In it the author tries to explain how message routing in a supposed network might work (by incorporating promises and other async concepts). He constructs different types of functions that handle different actions (sending request, receiving it, responding,...). But then there is this implementation of route finding algorithm that I think I don't quite understand.
//SECTION THAT CREATES A KIND OF NEIGHBOUR MAP THAT EVERY NEST (COMPUTER) HAS
requestType("connections", (nest, {name, neighbors},
source) => {
let connections = nest.state.connections;
if (JSON.stringify(connections.get(name)) ==
JSON.stringify(neighbors)) return;
connections.set(name, neighbors);
broadcastConnections(nest, name, source);
});
function broadcastConnections(nest, name, exceptFor = null) {
for (let neighbor of nest.neighbors) {
if (neighbor == exceptFor) continue;
request(nest, neighbor, "connections", {
name,
neighbors: nest.state.connections.get(name)
});
}
}
everywhere(nest => {
nest.state.connections = new Map();
nest.state.connections.set(nest.name, nest.neighbors);
broadcastConnections(nest, nest.name);
});
//PATH FINDING FUNCTION
function findRoute(from, to, connections) {
let work = [{at: from, via: null}];
for (let i = 0; i < work.length; i++) {
let {at, via} = work[i];
for (let next of connections.get(at) || []) {
if (next == to) return via;
if (!work.some(w => w.at == next)) {
work.push({at: next, via: via || next});
}
}
}
return null;
}
//THEN THERE ARE FUNCTIONS THAT HANDLE THE ACTUAL MESSAGE SENDING/ROUTING
function routeRequest(nest, target, type, content) {
if (nest.neighbors.includes(target)) {
return request(nest, target, type, content);
} else {
let via = findRoute(nest.name, target,
nest.state.connections);
if (!via) throw new Error(`No route to ${target}`);
return request(nest, via, "route",
{target, type, content});
}
}
requestType("route", (nest, {target, type, content}) => {
return routeRequest(nest, target, type, content);
});
My question is, in the findRoute function, why is there || [] in the inner for loop? Is it there for appropriate consequent error handling (in case somehow there is no nest specified as having neighbours in the connections property, but is regardless of that listed as someones neighbouring nest)?

connections.get(at) may return null or undefined, depending on the api, and you can't do a for...of loop over null or undefined, so he replaces that value with an empty array in that case

how to compare objects values in 2 identical lists in javascript

i have two collections in mongo db that hold reports, in report there is list matches, so what i want is to run on production reports and for ech one check in staging reports and check that if the matches are the same length if the personId and addressId are also the same...
is there a a good way to do this?
i came up with something like this:
db.production_reports.find({}).forEach((prodRep)=> {
db.reports.find({_id: prodRep._id}).forEach((stagingRep)=> {
if (prodRep.matches.length == stagingRep.matches.length) {
prodRep.matches.forEach((match)=> {
var res = stagingRep.matches.filter(element => element.personId == match.personId && element.addressId == match.addressId);
if (res) {
print("yay")
} else {
print("nay")
}
});
}
});
});
i want for each report the script to tell me "yes, all matches equal", or print the reportId that have non equal matches
thanks

I would draft something like this :
return new Promise((resolve) => {
const data = {
// contains an array of _id of missing production_reports
missing: [],
different: [],
};
// Look at each entry in production_reports
db.production_reports.find({})
.cursor()
.eachAsync(async (x) => {
// get the similar data on reports
const copy = await db.reports.find({
_id: x._id,
});
// If the data doesn't exists into reports
if (!copy || !copy.length) {
data.missing.push(x._id);
return;
}
// If it exists, compare the inner values
// if the size isn't the same, it's obviously different
if (x.matches.length !== copy.length) {
data.different.push(x._id);
return;
}
// Check every element of match one by one
if (x.matches.some(y => !copy.matches.some(z => z.personId === y.personId))) {
data.different.push(x._id);
}
}, {
// How many items do we look at same time
parallel: 250,
}, () => {
// When we are done processing all items
resolve(data);
});
});
NOTE : It won't give you missing documents that exists in reports but not in production_reports

array.forEach works, but not when I nest another inside

I've got two pages I'm working on, and both return an array of objects. When I use the following code, the new results work:
this.adminService.waiversGetAll()
.subscribe((data: Waiver[]) => {
this.waivers = data;
this.waivers.forEach((e) => {
if(e.has_signed === true) {
e.url = `View`
} else {
e.url = `${e.message}`;
}
return e;
});
console.log(this.waivers);
})
}
But when I try to do the same thing with a different array (where I need to update the values of an array nested inside) I don't get updated values:
this.adminService.GetUnsignedWaivers()
.subscribe((data: Player[]) => {
console.log("data",data);
data.forEach(e => {
let record: Object = {};
for(let i = 0; i < e.waivers.length; i++) {
console.log(e.waivers[i].has_signed);
if (e.waivers[i].has_signed === true) {
e.waivers[i].url = e.waivers[i].signatureUrl;
console.log(e.waivers[i].url);
e.waivers[i].message = "View Waiver";
} else {
e.waivers[i].url = e.waivers[i].url;
e.waivers[i].message = e.waivers[i].message;
}
console.log(e.waivers[i].message);
return;
};
return e;
});
this.size = this.players.length;
console.log(this.players);
})
}
When I look at the console.log of e.waivers[i].has_signed, the data is correct, but after that it's not right.
What do I have to do to make this work? I've tried using a for loop inside the foreach, and a bunch of other stuff.
The data supplied to the loop provides info like:
{
buyer: "email#someaddress.edu"
event: "COED A"
field: "Main"
net: null
player: {shirtSize: null, avp_id: 12345678, adult: true, …}
team: null
waivers: [{
email: "someemail#gmail.com",
has_signed: true,
message: "Liability Waiver",
signatureUrl: "https://somelink.pdf",
url: "https://somelink.com/somekeyidentifier"
}
IF the player has signed the waiver, there will be a signatureUrl field and the message should say "View Waiver" instead of the message telling me what type of waiver they will sign. I want the url to be set to signatureUrl if they signed, so I can use it in a table that doesn't like manipulation of data.
A visual of what is returned in my table:
All I get is 1600 records showing the url as though everyone hasn't signed, but when I console.log has_signed in the inner loop, it's showing TRUE for the ones that should show a signatureUrl instead.

Quickly looking at it, you have a return statement within your for loop, which would stop it from running after the first iteration.

First of all drop all the return statements in your code. Next, use map instead of forEach as the former returns you the new manipulated array and the latter is used just for iteration purpose.
Your code within subscribe then becomes:
data.waivers = data.waivers.map((waiver) => {
if (waiver.has_signed) {
// your logic goes here...
waiver.url = waiver.signatureUrl;
waivers.message = "View Waiver";
}
// No else is required as you are just reassigning with same values
});
this.playerDetails = data;
At last bind this modified data in your template.

How to check if image exists?

I have application where images should be set in some of the pages. These images are on the server and each image has unique ID appended. Here is example of the image names:
AA_image1.jpg
AB_image1.jpg
AC_image1.jpg
AD_image1.jpg
AE_image1.jpg
I seen what previous developers did and how they checked the image existance. They used JSON file that has id and image name. Here is example of that code:
var images = [{
"id": "AA",
"image": "AA_image1.jpg"
},
{
"id": "AB",
"image": "AB_image1.jpg"
},
{
"id": "AC",
"image": "AC_image1.jpg"
},
{
"id": "AD",
"image": "AD_image1.jpg"
},
{
"id": "AE",
"image": "AE_image1.jpg"
}
];
var imgID = "AC";
var imgPrimary = "AC_image1.jpg";
var found = false;
var imgDefault = "default.jpg";
for (i = 1; i < images.length; i++) {
if (images[i].id == imgID && (images[i].image).toLowerCase() == (imgID + '_image1.jpg').toLowerCase()) {
found = true;
break;
}
}
if (found === true) {
console.log(imgPrimary);
} else {
console.log(imgDefault);
}
The example above seems pretty simple but my concern is what if image get removed from the folder and JSON file is not updated? In that case we would load the image that do not exist instead of default image. I'm wondering if this approach would be better:
var imgID = "AC";
var imgNames = [imgID + '_image1','default'];
var imgResults = {};
for(var i = 0; i < imgNames.length; i++){
checkImage( imgNames[i] );
}
function checkImage( imgName, keyName ) {
$.ajax({
type: "GET",
async: true,
url: "images/"+imgName+".jpg",
}).done(function(message,text,jqXHR){
imgResults[imgName] = true;
}).fail(function(jqXHR, textStatus, errorThrown){
imgResults[imgName] = false;
});
}
Here is example of imgResults after the process was completed:
console.log(imgResults);
Console result:
{
"default": true,
"AG_image1": false
}
The only problem I'm experiencing with the second example is that if I try to check the result based on the key I'm getting undefined. Here is example:
console.log(imgResults["AG_image1"]);
This is result in the console:
undefined
Between the two I'm not sure which one is better and more secure. If anyone have suggestions please let me know.

This is probably the shortest possible code to handle lost images in JS. imagePad is a reference to the DOM element in which you want to show the image.
var img = new Image();
img.addEventListener('error', function (e) {
// Image not found, show the default
this.src = iconBase + 'gen.png';
});
img.addEventListener('load', function () {
imagePad.appendChild(img);
}
img.src = 'the_given_src';

AJAX request are asynchronous by default
The reason that you're unable to search the imgResults object when you are trying is because AJAX requests are asynchronous, and the request has not completed when you're trying to access the results. You need to wait for the request to complete before continuing.
The reason that console.log shows the results is because console.log is lazy and doesn't evaluate the object until you expand it in dev tools, console.dir will show you an empty object at that point.
Furthermore, as you have multiple requests that you want to wait for, you'll want to create an array of promises where each promise corresponds to the load/failure of each request, then use Promise.all to wait for all the promises to complete before operating on the results.
Verifying that a resource exists
To verify that a resource exists, you can use a HEAD request instead of a GET request, such as to prevent loading the entire image for no reason. There's also no need to depend on a massive library such as jQuery for AJAX requests as XMLHttpRequest is very well supported in this day and age.
class ResourceValidator extends EventTarget {
constructor(target) {
super()
this.target = target
this._ready = false
this._valid = false
this.validate()
}
validate() {
const request = new XMLHttpRequest()
request.addEventListener('load', event => {
this._ready = true
this._valid = true
this.dispatchEvent(new Event('ready'))
})
request.addEventListener('error', event => {
this._ready = true
this.dispatchEvent(new Event('ready'))
})
request.open('HEAD', this.target)
request.send()
}
get ready() {
return new Promise(resolve => {
if(this._ready === true) resolve(true)
else this.addEventListener('ready', _ => resolve(true))
})
}
get valid() {
return new Promise(resolve => {
if(this._ready === true) resolve(this._valid)
else this.addEventListener('ready', _ => resolve(this._valid))
})
}
}
async function validateImageSources(sources) {
const results = {}
const promises = []
for(let source of sources) {
const validator = new ResourceValidator(source)
const promise = validator.valid
promise.then(valid => results[source] = valid)
promises.push(promise)
}
await Promise.all(promises)
return results
}
validateImageSources([
'https://picsum.photos/200',
'https://nosuchaddress.io/image.png'
]).then(results => {
console.log(results)
console.log(results['https://picsum.photos/200'])
})

Unreliable behaviour in Node.js

I have a Node.js application that, upon initialisation, reads two tables from an SQL database and reconstructs their relationship in memory. They're used for synchronously looking up data that changes (very) infrequently.
Problem: Sometimes I can't access the data, even though the application reports successfully loading it.
Code:
constants.js
module.exports = {
ready: function () { return false; }
};
var log = sysLog('core', 'constants')
, Geo = require('../models/geo.js');
var _ready = false
, _countries = []
, _carriers = [];
function reload() {
_ready = false;
var index = Object.create(null);
return Geo.Country.find().map(function (country) {
var obj = country.toPlainObject()
, id = obj.id;
delete obj.id;
index[id] = obj;
return Object.freeze(obj);
}).then(function (countries) {
log.debug('Loaded ' + countries.length + ' countries');
_countries = countries;
return Geo.Carrier.Descriptor.find().map(function (carrier) {
var obj = carrier.toPlainObject();
if (obj.country) {
obj.country = index[obj.country];
}
return Object.freeze(obj);
}).then(function (carriers) {
log.debug('Loaded ' + carriers.length + ' carriers');
_carriers = carriers;
});
}).finally(function () {
_ready = true;
});
}
reload().catch(function (err) {
log.crit({ message: 'Could not load constants', reason: err });
process.exit(-42);
}).done();
module.exports = {
reload : reload,
ready : function () { return _ready; },
countries : function () { return _countries; },
carriers : function () { return _carriers; }
};
utils.js
var log = sysLog('core', 'utils')
, constants = require('./constants');
module.exports = {
getCountryByISO: function(iso) {
if (!iso) {
return;
}
if ('string' != typeof iso) {
throw new Error('getCountryByISO requires a string');
}
if (!constants.ready()) {
throw new UnavailableError('Try again in a few seconds');
}
switch (iso.length) {
case 2:
return _.findWhere(constants.countries(), { 'iso2' : iso.toUpperCase() });
case 3:
return _.findWhere(constants.countries(), { 'iso3' : iso.toUpperCase() });
default:
throw new Error('getCountryByISO requires a 2 or 3 letter ISO code');
}
},
getCarrierByCode: function(code) {
if (!code) {
return;
}
if ('string' != typeof code) {
throw new Error('getCarrierByCode requires a string');
}
if (!constants.ready()) {
throw new UnavailableError('Try again in a few seconds');
}
return _.findWhere(constants.carriers(), { 'code' : code });
},
getCarrierByHandle: function(handle) {
if (!handle) {
return;
}
if ('string' != typeof handle) {
throw new Error('getCarrierByHandle requires a string');
}
if (!constants.ready()) {
throw new UnavailableError('Try again in a few seconds');
}
return _.findWhere(constants.carriers(), { 'handle' : handle });
}
};
Use case
if (data.handle) {
carrier = utils.getCarrierByHandle(data.handle);
if (_.isEmpty(carrier)) {
throw new InternalError('Unknown carrier', { handle: data.handle });
}
}
What's going on: All errors are logged; as soon as I see an error (i.e. "Unknown carrier") in the logs, I check the SQL database to see if it should've been recognised. That has always been the case so far, so I check the debug log to see if data was loaded. I always see "Loaded X countries" and "Loaded Y carriers" with correct values and no sign of "Could not load constants" or any other kind of trouble.
This happens around 10% of the time I start the application and the problem persists (i.e. didn't seem to go away after 12+ hours) and seems to occur regardless of input, leading me to think that the data isn't referenced correctly.
Questions:
Is there something wrong in constants.js or am I doing something very obviously wrong? I've tried setting it up for cyclical loading (even though I am not aware of that happening in this case).
Why can't I (sometimes) access my data?
What can I do to figure out what's wrong?
Is there any way I can work around this? Is there anything else I could to achieve the desired behaviour? Hard-coding the data in constants.js is excluded.
Additional information:
constants.reload() is never actually called from outside of constants.js.
constants.js is required only in utils.js.
utils.js is required in app.js (application entry); all files required before it do not require it.
SQL access is done through an in-house library built on top of knex.js and bluebird; so far it's been very stable.
Versions:
Node.js v0.10.33
underscore 1.7.0
bluebird 2.3.11
knex 0.6.22

}).finally(function () {
_ready = true;
});
Code in a finally will always get called, regardless of if an error was thrown up the promise chain. Additionally, your reload().catch(/* ... */) clause will never be reached, because finally swallows the error.
Geo.Country.find() or Geo.Carrier.Descriptor.find() could throw an error, and _ready would still be set to true, and the problem of your countries and carriers not being set would persist.
This problem would not have occurred if you had designed your system without a ready call, as I described in my previous post. Hopefully this informs you that the issue here is really beyond finally swallowing a catch. The real issue is relying on side-effects; the modification of free variables results in brittle systems, especially when asynchrony is involved. I highly recommend against it.

Try this
var log = sysLog('core', 'constants');
var Geo = require('../models/geo.js');
var index;
var _countries;
var _carriers;
function reload() {
index = Object.create(null);
_countries = Geo.Country.find().map(function (country) {
var obj = country.toPlainObject();
var id = obj.id;
delete obj.id;
index[id] = obj;
return Object.freeze(obj);
});
_carriers = _countries.then(function(countries) {
return Geo.Carrier.Descriptor.find().map(function (carrier) {
var obj = carrier.toPlainObject();
if (obj.country) {
obj.country = index[obj.country];
}
return Object.freeze(obj);
});
});
return _carriers;
}
reload().done();
module.exports = {
reload : reload,
countries : function () { return _countries; },
carriers : function () { return _carriers; }
};

constants.reload() is never actually called from outside of
constants.js.
That's your issue. constants.reload() reads from a database, which is an aysnchronous process. Node's require() is a synchronous process. At the time constants.js is required in utils.js and the module.exports value is returned, your database query is still running. And at whatever point in time that app.js reaches the point where it calls a method from the utils module, that query could still be running, resulting in the error.
You could say that requiring utils.js has the side-effect of requiring constants.js, which has the side-effect of executing a database query, which has the side-effect of concurrently modifying the free variables _countries and _carriers.
Initialize _countries and _carriers as unresolved promises. Have reload() resolve them. Make the utils.js api async.
promises.js:
// ...
var Promise = require('bluebird');
var countriesResolve
, carriersResolve;
var _ready = false
, _countries = new Promise(function (resolve) {
countriesResolve = resolve;
})
, _carriers = new Promise(function (resolve) {
carriersResolve = resolve;
});
function reload() {
_ready = false;
var index = Object.create(null);
return Geo.Country.find().map(function (country) {
// ...
}).then(function (countries) {
log.debug('Loaded ' + countries.length + ' countries');
countriesResolve(countries);
return Geo.Carrier.Descriptor.find().map(function (carrier) {
// ...
}).then(function (carriers) {
log.debug('Loaded ' + carriers.length + ' carriers');
carriersResolve(carriers);
});
}).finally(function () {
_ready = true;
});
}
reload().catch(function (err) {
log.crit({ message: 'Could not load constants', reason: err });
process.exit(-42);
}).done();
module.exports = {
reload : reload,
ready : function () { return _ready; },
countries : function () { return _countries; },
carriers : function () { return _carriers; }
};
utils.js
getCarrierByHandle: function(handle) {
// ...
return constants.carriers().then(function (carriers) {
return _.findWhere(carriers, { 'handle' : handle });
});
}
Use case:
utils.getCarrierByHandle(data.handle).then(function (carrier) {
if (_.isEmpty(carrier)) {
throw new InternalError('Unknown carrier', { handle: data.handle });
}
}).then(function () {
// ... next step in application logic
});
This design will also eliminate the need for a ready method.
Alternatively, you could call constants.reload() on initialization and hang all possibly-dependent operations until it completes. This approach would also obsolete the ready method.
What can I do to figure out what's wrong?
You could have analyzed your logs and observed that "Loaded X countries" and "Loaded Y carriers" were sometimes written after "Unknown carrier", helping you realize that the success of utils.getCarrierByHandle() was a race condition.