We Keep Coding

How to use loops with fetch and Promise.all?

So I was following along an MDN article on promises and was wondering how to modify the following code to be able to work for any number of files (not just 3).
function fetchAndDecode(url) {
return fetch(url).then(response => {
if(!response.ok) {
throw new Error(`HTTP error! status: ${response.status}`);
} else {
if(response.headers.get("content-type") === "image/jpeg") {
return response.blob();
} else if(response.headers.get("content-type") === "text/plain") {
return response.text();
}
}
})
.catch(e => {
console.log(`There has been a problem with your fetch operation for resource "${url}": ` + e.message);
})
.finally(() => {
console.log(`fetch attempt for "${url}" finished.`);
})
}
let coffee = fetchAndDecode('coffee.jpg');
let tea = fetchAndDecode('tea.jpg');
let description = fetchAndDecode('description.txt');
Promise.all([coffee, tea, description]).then(values => {
console.log(values);
// Store each value returned from the promises in separate variables; create object URLs from the blobs
let objectURL1 = URL.createObjectURL(values[0]);
let objectURL2 = URL.createObjectURL(values[1]);
let descText = values[2];
// Display the images in <img> elements
let image1 = document.createElement('img');
let image2 = document.createElement('img');
image1.src = objectURL1;
image2.src = objectURL2;
document.body.appendChild(image1);
document.body.appendChild(image2);
// Display the text in a paragraph
let para = document.createElement('p');
para.textContent = descText;
document.body.appendChild(para);
});
MDN specifically notes that "If you were improving this code, you might want to loop through a list of items to display, fetching and decoding each one, and then loop through the results inside Promise.all(), running a different function to display each one depending on what the type of code was. This would make it work for any number of items, not just three." I'm not sure how to do this though, and would appreciate help. Thanks.

The second part of the code could be generalised as follows:
let urls = ['coffee.jpg', 'tea.jpg', 'description.txt'];
Promise.all(urls.map(fetchAndDecode)).then(values => {
let elem;
for (let value of values) {
if (value instanceof Blob) {
elem = document.createElement('img');
elem.src = URL.createObjectURL(value);
} else if (typeof value === "string") {
elem = document.createElement('p');
elem.textContent = value;
} else {
console.log("unexpected value type");
continue;
}
document.body.appendChild(elem);
}
});

const resources = ['coffee.jpg', 'tea.jpg', 'description'];
const resourceRequests = resources.map(fetchAndDecode);
Promise.all(resourceRequests.then(values => {
...
Is one way to implement the suggestion. This approach allows for easier modification of the list of resources, but doesn't really change any of the Promise code.
The .map code above is equivalent to (resource => fetchAndDecode(resource)) since fetchAndDecode takes only the first argument that .map would pass to it.

Javascript object retaining "old" properties, can't override?

I have the following code:
const readDataFromSql = () => {
// going to have to iterate through all known activities + load them here
let sql = "[...]"
return new Promise((resolve, reject) => {
executeSqlQuery(sql).then((dict) => {
let loadedData = [];
for (let key in dict) {
let newItemVal = new ItemVal("reading hw", 7121, progress.DONE);
loadedData.push(newItemVal);
}
resolve(loadedData);
});
});
}
ItemVal implementation:
class ItemVal {
constructor(name, time, type) {
this.name = name
this.time = time
this.type = type
}
}
Let's assume that newItemVal = "reading hwj", 5081, progress.PAUSED when readDataFromSql() first runs.
readDataFromSql() is then again called after some state changes -- where it repulls some information from a database and generates new values. What is perplexing, however, is that when it is called the second time, newItemVal still retains its old properties (attaching screenshot below).
Am I misusing the new keyword?

From what I can see in your example code, you are not mutating existing properties but creating a new object with the ItemVal constructor function and adding them to an array, that you then return as a resolved promise. Are you sure the examples you give a correct representation of what you are actually doing
Given that, I'm not sure what could be causing the issue you are having, but I would at least recommend a different structure for your code, using a simpler function for the itemVal.
Perhaps with this setup, you might get an error returned that might help you debug your issue.
const itemVal = (name, time, type) => ({ name, time, type })
const readDataFromSql = async () => {
try {
const sql = "[...]"
const dict = await executeSqlQuery(sql)
const loadedData = dict.map((key) =>
ItemVal("reading hw", 7121, progress.DONE)
)
return loadedData
} catch (error) {
return error
}
};
If the issue is not in the function, then I would assume that the way you handle the data, returned from the readDataFromSql function, is where the issue lies. You need to then share more details about your implementation.

const readDataFromSql = async () => {
let sql = "[...]"
------> await executeSqlQuery(sql).then((dict) => {
Use the await keyword instead of creating a new promise.

I did some modification and found that below code is working correctly, and updating the new values on each call.
const readDataFromSql = () => {
return new Promise((resolve, reject) => {
let loadedData = [];
let randomVal = Math.random();
let newItemVal = new ItemVal(randomVal*10, randomVal*100, randomVal*1000);
loadedData.push(newItemVal);
resolve(loadedData);
});
}
Could you recheck if you are using below line in the code, as it will instantiate object with same properties again and again.
let newItemVal = new ItemVal("reading hw", 7121, progress.DONE);

You can modify your code as below to simplify the problem.
const readDataFromSql = async () => {
// going to have to iterate through all known activities + load them here
let sql = "[...]" // define sql properly
let result = await executeSqlQuery(sql);
let loadedData = [];
for (let row in result) {
let newItemVal = new ItemVal(row.name, row.time, row.type);
loadedData.push(newItemVal);
}
return loadedData;
}
class ItemVal {
constructor(name, time, type) {
this.name = name
this.time = time
this.type = type
}
}
What you are talking about is an issue related to Object mutation in Redux, however, you didn't add any redux code. Anyway, you might be making some mistake while recreating(not mutating) the array.
General solution is the use spread operator as:
loadedData = [ ...loadedData.slice(0) , ...newloadedData]

In Dropdown.js line 188 instead of console.log-ing your variable write debugger;
This will function as a breakpoint. It will halt your code and you can inspect the value by hovering your mouse over the code BEFORE the newItemVal is changed again.
I can see in your screenshot that the newItemVal is modified again after you log it.

Continue on Null Value of Result (Nodejs, Puppeteer)

I'm just starting to play around with Puppeteer (Headless Chrome) and Nodejs. I'm scraping some test sites, and things work great when all the values are present, but if the value is missing I get an error like:
Cannot read property 'src' of null (so in the code below, the first two passes might have all values, but the third pass, there is no picture, so it just errors out).
Before I was using if(!picture) continue; but I think it's not working now because of the for loop.
Any help would be greatly appreciated, thanks!
for (let i = 1; i <= 3; i++) {
//...Getting to correct page and scraping it three times
const result = await page.evaluate(() => {
let title = document.querySelector('h1').innerText;
let article = document.querySelector('.c-entry-content').innerText;
let picture = document.querySelector('.c-picture img').src;
if (!document.querySelector('.c-picture img').src) {
let picture = 'No Link'; } //throws error
let source = "The Verge";
let categories = "Tech";
if (!picture)
continue; //throws error
return {
title,
article,
picture,
source,
categories
}
});
}

let picture = document.querySelector('.c-picture img').src;
if (!document.querySelector('.c-picture img').src) {
let picture = 'No Link'; } //throws error
If there is no picture, then document.querySelector() returns null, which does not have a src property. You need to check that your query found an element before trying to read the src property.
Moving the null-check to the top of the function has the added benefit of saving unnecessary calculations when you are just going to bail out anyway.
async function scrape3() {
// ...
for (let i = 1; i <= 3; i++) {
//...Getting to correct page and scraping it three times
const result = await page.evaluate(() => {
const pictureElement = document.querySelector('.c-picture img');
if (!pictureElement) return null;
const picture = pictureElement.src;
const title = document.querySelector('h1').innerText;
const article = document.querySelector('.c-entry-content').innerText;
const source = "The Verge";
const categories = "Tech";
return {
title,
article,
picture,
source,
categories
}
});
if (!result) continue;
// ... do stuff with result
}
Answering comment question: "Is there a way just to skip anything blank, and return the rest?"
Yes. You just need to check the existence of each element that could be missing before trying to read a property off of it. In this case we can omit the early return since you're always interested in all the results.
async function scrape3() {
// ...
for (let i = 1; i <= 3; i++) {
const result = await page.evaluate(() => {
const img = document.querySelector('.c-picture img');
const h1 = document.querySelector('h1');
const content = document.querySelector('.c-entry-content');
const picture = img ? img.src : '';
const title = h1 ? h1.innerText : '';
const article = content ? content.innerText : '';
const source = "The Verge";
const categories = "Tech";
return {
title,
article,
picture,
source,
categories
}
});
// ...
}
}
Further thoughts
Since I'm still on this question, let me take this one step further, and refactor it a bit with some higher level techniques you might be interested in. Not sure if this is exactly what you are after, but it should give you some ideas about writing more maintainable code.
// Generic reusable helper to return an object property
// if object exists and has property, else a default value
//
// This is a curried function accepting one argument at a
// time and capturing each parameter in a closure.
//
const maybeGetProp = default => key => object =>
(object && object.hasOwnProperty(key)) ? object.key : default
// Pass in empty string as the default value
//
const getPropOrEmptyString = maybeGetProp('')
// Apply the second parameter, the property name, making 2
// slightly different functions which have a default value
// and a property name pre-loaded. Both functions only need
// an object passed in to return either the property if it
// exists or an empty string.
//
const maybeText = getPropOrEmptyString('innerText')
const maybeSrc = getPropOrEmptyString('src')
async function scrape3() {
// ...
// The _ parameter name is acknowledging that we expect a
// an argument passed in but saying we plan to ignore it.
//
const evaluate = _ => page.evaluate(() => {
// Attempt to retrieve the desired elements
//
const img = document.querySelector('.c-picture img');
const h1 = document.querySelector('h1')
const content = document.querySelector('.c-entry-content')
// Return the results, with empty string in
// place of any missing properties.
//
return {
title: maybeText(h1),
article: maybeText(article),
picture: maybeSrc(img),
source: 'The Verge',
categories: 'Tech'
}
}))
// Start with an empty array of length 3
//
const evaluations = Array(3).fill()
// Then map over that array ignoring the undefined
// input and return a promise for a page evaluation
//
.map(evaluate)
// All 3 scrapes are occuring concurrently. We'll
// wait for all of them to finish.
//
const results = await Promise.all(evaluations)
// Now we have an array of results, so we can
// continue using array methods to iterate over them
// or otherwise manipulate or transform them
//
results
.filter(result => result.title && result.picture)
.forEach(result => {
//
// Do something with each result
//
})
}

Try-catch worked for me:
try {
if (await page.$eval('element')!==null) {
const name = await page.$eval('element')
}
}catch(error){
name = ''
}

How can I use a Javascript object/map as queue

Right now I have a queue (JS array) that is used to store players waiting for a game. I need the FIFO property of a queue so that players who were added to the queue first, get put in a new game first. The problem with a queue is that it doesnt have constant time lookup. It would be great if I could have a map that kept track of the order of insertion (i know that relying on a map to do this is JS is not reliable). If I give the property a value for its insertion order, it would need to be updated if someone leaves the queue, so that isnt helpful either. Anyway around this? A way to get constant lookup and maintain insertion order?

If you don't have memory constraints, maybe you can maintain a map with the queue implemented as a double linked list. Here is a sample implementation:
function Queue() {
var oldestRequest,
newestRequest,
map = {};
this.addUser = function(userID) {
var newRequest = { userID: userID };
map[userID] = newRequest;
// Set this as the oldest request if it is the first request
if (!oldestRequest) {
oldestRequest = newRequest;
}
// If this isn't the first request, add it to the end of the list
if (newestRequest) {
newestRequest.next = newRequest;
newRequest.previous = newestRequest;
}
newestRequest = newRequest;
};
this.nextUser = function() {
// If we don't have any requests, undefined is returned
if (oldestRequest) {
var request = oldestRequest;
oldestRequest = request.next;
delete map[request.userID];
// Make sure we don't hang on to references to users
// that are out of the queue
if (oldestRequest) {
delete oldestRequest.previous;
}
// This is the last request in the queue so "empty" it
if (request === newestRequest) {
newestRequest = undefined;
}
return request;
}
};
this.removeUser = function(userID) {
var request = map[userID];
delete map[userID];
if (request.previous) {
request.previous.next = request.next;
}
if (request.next) {
request.next.previous = request.previous;
}
};
return this;
}

You can use a map together with a queue to provide constant time access. Below is the implementation in TypeScript 4.2. Map is used instead of Object to provide better performance in addition and removal of values.
// TypeScript typing
export type KeyValuePair<K, V> = [ K, V ]
interface ValueData<V> {
value: V
refCount: number
}
// Public classes
export class MapQueue<K, V> {
readonly #queue: Array<KeyValuePair<K, V>>
readonly #map: Map<K, ValueData<V>>
constructor () {
this.#queue = []
this.#map = new Map()
}
get length (): number {
return this.#queue.length
}
unshiftOne (pair: KeyValuePair<K, V>): number {
const [key, value] = pair
const valueData = this.#map.get(key)
if (valueData !== undefined) {
if (valueData.value !== value) {
throw new Error(`Key ${String(key)} with different value already exists`)
}
valueData.refCount++
} else {
this.#map.set(key, {
value,
refCount: 1
})
}
return this.#queue.unshift(pair)
}
pop (): KeyValuePair<K, V> | undefined {
const result = this.#queue.pop()
if (result !== undefined) {
const valueData = this.#map.get(result[0])
if (valueData !== undefined) {
valueData.refCount--
if (valueData.refCount === 0) {
this.#map.delete(result[0])
}
}
}
return result
}
get (key: K): V | undefined {
return this.#map.get(key)?.value
}
}

Combine Observables A and B: Should only trigger if A has data; take from B if data or default

I have an application communicating with a device via serial port. Every sent command is answered by a data event containing the status/answer. Basically there are commands changing the device and a command which just returns the status. Every time the last command has been answered (so upon receiving data) the app should send the next command or as a default query the status. I'm trying to model this with rxjs.
My idea here is that there is a command observable and a data observable derived from the data event. These two should be combined in such a way, that the resulting observable only emits values, when there is data and combine it with a command or the default command (request status), if no command came down the command stream.
data: ---------d---d-----d---------d------d-------
command: --c1---c2----------------------c3-----------
______________________________________________________
combined ---------c1--c2----dc--------dc-----c3
dc is the default command. Also no commands should be lost.
Currently I have an implementation with a anonymous subject, implementing the observable and observer myself. Collecting commands from the command stream in an array, subscribing to the data event, publish the data by hand with onNext and sending the next command from the array or the default. This works, but I have the feeling this could be expressed more elegantly with rxjs.
One approach was to have a separate default_command stream, repeating the default command every 100ms. This was merged with the command stream and then zipped with the data stream. The problem here was the merged command stream, because it piled up default commands, but the default command should only apply, if there is no other command.

Only thing I can think of is to:
subscribe to the command stream and queue the results (in an array)
Apply a mapping operation to the data stream which will pull from the queue (or use default if the queue is empty).
We can wrap this up into a generic observable operator. I'm bad with names, so I'll call it zipWithDefault:
Rx.Observable.prototype.zipWithDefault = function(bs, defaultB, selector) {
var source = this;
return Rx.Observable.create(function(observer) {
var sourceSubscription = new Rx.SingleAssignmentDisposable(),
bSubscription = new Rx.SingleAssignmentDisposable(),
subscriptions = new Rx.CompositeDisposable(sourceSubscription, bSubscription),
bQueue = [],
mappedSource = source.map(function(value) {
return selector(value, bQueue.length ? bQueue.shift() : defaultB);
});
bSubscription.setDisposable(bs.subscribe(
function(b) {
bQueue.push(b);
},
observer.onError.bind(observer)));
sourceSubscription.setDisposable(mappedSource.subscribe(observer));
return subscriptions;
});
};
And use it like so:
combined = dataStream
.zipWithDefault(commandStream, defaultCommand, function (data, command) {
return command;
});

I think the sample operator would be your best bet. Unfortunately, it does not come with a built in default value, so you would have to roll your own from the existing operator:
Rx.Observable.prototype.sampleWithDefault = function(sampler, defaultValue){
var source = this;
return new Rx.AnonymousObservable(function (observer) {
var atEnd, value, hasValue;
function sampleSubscribe() {
observer.onNext(hasValue ? value : defaultValue);
hasValue = false;
}
function sampleComplete() {
atEnd && observer.onCompleted();
}
return new Rx.CompositeDisposable(
source.subscribe(function (newValue) {
hasValue = true;
value = newValue;
}, observer.onError.bind(observer), function () {
atEnd = true;
}),
sampler.subscribe(sampleSubscribe, observer.onError.bind(observer), sampleComplete)
);
}, source);
}
You can achieve the queuing behavior using the controlled operator. Thus your final data chain would look like so:
var commands = getCommandSource().controlled();
var pipeline = commands
.sampleWithDefault(data, defaultCommand)
.tap(function() { commands.request(1); });
Here is a full example:
Rx.Observable.prototype.sampleWithDefault = function(sampler, defaultValue) {
var source = this;
return new Rx.AnonymousObservable(function(observer) {
var atEnd, value, hasValue;
function sampleSubscribe() {
observer.onNext(hasValue ? value : defaultValue);
hasValue = false;
}
function sampleComplete() {
atEnd && observer.onCompleted();
}
return new Rx.CompositeDisposable(
source.subscribe(function(newValue) {
hasValue = true;
value = newValue;
}, observer.onError.bind(observer), function() {
atEnd = true;
}),
sampler.subscribe(sampleSubscribe, observer.onError.bind(observer), sampleComplete)
);
}, source);
}
var scheduler = new Rx.TestScheduler();
var onNext = Rx.ReactiveTest.onNext;
var onCompleted = Rx.ReactiveTest.onCompleted;
var data = scheduler.createHotObservable(onNext(210, 18),
onNext(220, 17),
onNext(230, 16),
onNext(250, 15),
onCompleted(1000));
var commands = scheduler.createHotObservable(onNext(205, 'a'),
onNext(210, 'b'),
onNext(240, 'c'),
onNext(400, 'd'),
onCompleted(800))
.controlled(true, scheduler);
var pipeline = commands
.sampleWithDefault(data, 'default')
.tap(function() {
commands.request(1);
});
var output = document.getElementById("output");
pipeline.subscribe(function(x) {
var li = document.createElement("li");
var text = document.createTextNode(x);
li.appendChild(text);
output.appendChild(li);
});
commands.request(1);
scheduler.start();
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/2.5.2/rx.all.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/2.5.2/rx.testing.js"></script>
<div>
<ul id="output" />
</div>

This can be solved by using the scan function. In the accumulated value the commands are stored for which no data response has been received yet.
var result = Rx.Observable
.merge(data, command)
.scan(function (acc, x) {
if (x === 'd') {
acc.result = acc.commands.length > 0 ? acc.commands.shift() : 'dc';
} else {
acc.result = '';
acc.commands.push(x);
}
return acc;
}, {result: '', commands: []})
.map(function (x) {
return x.result;
})
.filter(function (x) {
return x !== '';
});
Please find a complete more detail here: http://jsbin.com/tubade/edit?html,js,console