I'm working on converting a legacy callback-based API into an async library. But I just can't wrap my head around getting a "resultset" to work as a generator (Node 10.x).
The original API works like this:
api.prepare((err, rs) => {
rs.fetchRows(
(err, row) => {
// this callback is called as many times as rows exist
console.log("here's a row:", row);
},
() => {
console.log("we're done, data exausted");
}
);
});
But here is how I want to use it:
const wrapped = new ApiWrapper(api);
const rs = await wrapped.prepare({});
for (let row of rs.rows()) {
console.log("here's a row:", row);
}
let row;
while(row = await rs.next()) {
console.log("here's a row:", row);
}
I thought I had it under control with generators, but it looks like you cannot use yield inside a callback. It actually seems logical if you think about.
class ApiWrapper {
constructor(api) {
this.api = api;
}
prepare() {
return new Promise((resolve, reject) => {
this.api.prepare((err, rs) => {
if (err) {
reject(err);
} else {
resolve(rs);
}
});
});
}
*rows() {
this.api.fetchRows((err, row) => {
if (err) {
throw err;
} else {
yield row; // nope, not allowed here
}
});
}
next() { ... }
}
So what alternatives do I have?
Important: I don't want to store anything in an array then iterate that, we're talking giga-loads of row data here.
Edit
I'm able to simulate the behavior I want using stream.Readable but it warns me that it's an experimental feature. Here's a simplified array-based version of the issue I'm trying to solve using stream:
const stream = require('stream');
function gen(){
const s = new stream.Readable({
objectMode: true,
read(){
[11, 22, 33].forEach(row => {
this.push({ value: row });
});
this.push(null)
}
});
return s;
}
for await (let row of gen()) {
console.log(row);
}
// { value: 11 }
// { value: 22 }
// { value: 33 }
(node:97157) ExperimentalWarning: Readable[Symbol.asyncIterator] is an experimental feature. This feature could change at any time
I finally realized I needed something similar to Go's channels that were async/await compatible. Basically the answer is to synchronize an async iterator and a callback, making them wait for each other as next() iterations are consumed.
The best (Node) native solution I found was to use a stream as an iterator, which is supported in Node 10.x but tagged experimental. I also tried to implement it with the p-defer NPM module, but that turned out to be more involved than I expected. Finally ran across the https://www.npmjs.com/package/#nodeguy/channel module, which was exactly what I needed:
const Channel = require('#nodeguy/channel');
class ApiWrapper {
// ...
rows() {
const channel = new Channel();
const iter = {
[Symbol.asyncIterator]() {
return this;
},
async next() {
const val = await channel.shift();
if (val === undefined) {
return { done: true };
} else {
return { done: false, value: val };
}
}
};
this.api.fetchRows(async (err, row) => {
await channel.push(row);
}).then(() => channel.close());
return iter;
}
}
// then later
for await (let row of rs.rows()) {
console.log(row)
}
Note how each iterating function core, next() and rows(), have a await that will throttle how much data can be pushed across the channel, otherwise the producing callback could end up pushing data uncontrollably into the channel queue. The idea is that the callback should wait for data to be consumed by the iterator next() before pushing more.
Here's a more self-contained example:
const Channel = require('#nodeguy/channel');
function iterating() {
const channel = Channel();
const iter = {
[Symbol.asyncIterator]() {
return this;
},
async next() {
console.log('next');
const val = await channel.shift();
if (val === undefined) {
return { done: true };
} else {
return { done: false, value: val };
}
}
};
[11, 22, 33].forEach(async it => {
await channel.push(it);
console.log('pushed', it);
});
console.log('returned');
return iter;
}
(async function main() {
for await (let it of iterating()) {
console.log('got', it);
}
})();
/*
returned
next
pushed 11
got 11
next
pushed 22
got 22
next
pushed 33
got 33
next
*/
Like I said, Streams and/or Promises can be used to implement this, but the Channel module solves some of the complexity that make it more intuitive.
The original question has two nested callback taking async functions
api.prepare((err,res) => ...)
rs.fetchRows((err,res) => ...)
The first one runs the callback only once so just promisifying it as follows is sufficient.
function promisfied(f){
return new Promise((v,x) => f(x,v));
}
However the second function will invoke it's callback multiple times and we wish to generate an async iterable from this function such that we can consume it in a for await of loop.
This is also possible by employing async generators as follows;
async function* rowEmitterGenerator(rs){
let _v, // previous resolve
_x, // previous reject
_row = new Promise((v,x) => (_v = v, _x = x));
rs.fetchRows((err, row) => ( err ? _x(err) : _v(row)
, _row = new Promise((v,x) => (_v = v, _x = x))
));
while(true){
try {
yield _row;
}
catch(e){
console.log(e);
}
}
}
Then putting all together in a top level await context;
const rows = await promisified(api.prepare),
rowEmitter = rowEmitterGenerator(rows);
for await (let row of rowEmitter){
console.log(`Received row: ${row}`);
// do something with the row
}
Related
This loop is going to run an arbitrary amount of times, and I want to get the result out of it after them all. Anything I try (promisifying, async/await, nesting functions, et al) seems to be a dead end. I don't get why I cannot just stick a .then on the API call, or on the function I made here. But I suspect the problem is more fundamental with my understanding, because I can't seem to even get just the "data" to return...same inability to wait on the API call. Wrapping it in a promise loses the "data" and pulling it with the "for loop" inside there doesn't work either. This is making me question my entire progress with JS/implementing other people's APIs.
const gotPeeps = () => {
challongeClient.tournaments.show({
id: tournamentURL,
callback: (err, data) => {
//return data //doesnt return "data" from gotPeeps?
for (const [key, value] of Object.entries(data.tournament.matches)) {
if (value.match.state === 'open') {
peepsList.push(value.match.player1Id, value.match.player2Id)
console.log(peepsList)
}}}})}
gotPeeps()
EDIT
To the comments:
I'm trying to get the results after the for loop is complete.
The "loop" I was referring to is the "for of" over the data Object.
"Putting the code after the loop but inside the callback" does not work. I have a previous question in my week of failing to solve this:
How to solve race condition in Javascript?
Here is the whole thing, with some past versions commented out.:
const tournamentModel = require('../models/tournamentSchema')
require('dotenv').config()
const challonge = require('challonge');
module.exports = {
name: 'getmatches',
aliases: ['gm'],
cooldown: 0,
description: 'Get Challonge data into console.',
execute(message, args, cmd, client, Discord, profileData) {
let peep = ''
let peepsList = ''
const tournamentURL = 'TESTING_Just_Sign_Up_If_You_See_This850587786533011496'
const challongeClient = challonge.createClient({
apiKey: process.env.CHALLONGE_API_KEY,
})
const getPlayer = (playerXId) => {
return new Promise((resolve, reject) => {
challongeClient.participants.show({
id: tournamentURL,
participantId: playerXId,
callback: (err, data) => {
if (err) {
reject(err);
return;
}
peep = data.participant.misc
peepsList.push(peep)
console.log(peepsList)
console.log('RUNNING GET PLAYER', playerXId, playerIndexCount)
resolve(peepsList);
}
});
});
}
const peepsList = []
const matchList = []
const gotPeeps = () => {
challongeClient.tournaments.show({
id: tournamentURL,
include_participants: 1,
include_matches: 1,
callback: (err, data) => {
for (const [key, value] of Object.entries(data.tournament.matches)) {
if (value.match.state === 'open') {
peepsList.push(value.match.player1Id, value.match.player2Id)
console.log(peepsList)
}
}
}
/*// GET PLAYERS
getPlayer(value.match.player1Id)
.then(() => {
getPlayer(value.match.player2Id)
})
}
*/
}
)
}
gotPeeps()
}}
You can make this function return a promise and await the function (as long as your function is an async function)
const gotPeeps = () => {
return new Promise((resolve, reject) => {
const peepsList = []; // declare the empty array to e filled
challongeClient.tournaments.show({
id: tournamentURL,
callback: (err, data) => {
for (const [key, value] of Object.entries(data.tournament.matches)) {
if (value.match.state === "open") {
peepsList.push(value.match.player1Id, value.match.player2Id);
}
}
resolve(peepsList); // resolve the promise with the filled array
// TODO: handle reject
},
});
})
};
(async () => {
try {
const result = await gotPeeps();
} catch (error) {
// TODO: handle error
}
})();
I am trying to make use of async.parallelLimit to update records in my db. I need to do this in a batches of 10 and each time the total records will be 1000. I am trying to understand how can i make use of this asyn.parallelLimit in my code. I looked at some example interpreation and tried but it is running only 10 times and after giving back 10 responses it does not give me next ones. I am trying to query the db by executing the filter and then sending the records into async.parallelLimit and get response of all those records. For now i am just trying to understand the parallelLimit and its working. Here is my code
const async = require("async");
module.exports = async (server) => {
async function data(server) {
return await resolveData(server);
}
function resolveData(server) {
return new Promise((resolve) => {
let parcel = server.models["Parcel"].find({
order: "createdAt ASC",
include: [
{
relation: "parcelStatuses",
},
{
relation: "customerData",
scope: {
fields: ["firstName", "lastName", "cityId", "customerId"],
},
},
],
limit: 1000,
skip: 200,
});
resolve(parcel);
});
}
// console.log(await data(server));
var parcelData = await data(server);
for (var i = 1; i <= parcelData.length; i++) {
parcelData[i - 1] = (function (i) {
return function () {
console.log(i);
};
})(parcelData[i]);
}
async.parallelLimit(parcelData, 10, function (err, results) {
if (results) {
console.log("This is resilt", results);
} else {
console.log(err);
}
});
};
I need my parallelLimit function to just return me the records that my query fetch. I will run the update commands later. Since its returning me only 10 records i want to know what i am doing wrong
I'm not sure what the for loop is supposed to do. I'm guessing you're trying to create functions for each elements of the parcelData. An easy way to do that is just map over the parcelData and return an async function:
let async = require("async");
// Your data is here but I used a dummy array with 100 elements
let parcelData = Array(100).fill(0).map((_, i) => i);
// Just a dummy function to wait
function delay(ms) {
return new Promise(r => setTimeout(r, ms));
}
// map over array and return an `async` function;
let tasks = parcelData.map(p => {
return async () => {
await delay(1000);
console.log(p);
}
});
// Finally execute the tasks
async.parallelLimit(tasks, 10, (err, result) => {
if (err) console.error('error: ', err)
else console.log('done');
});
You can run the code on repl.it to see it in action. Here's one I made.
Can an async generator be somehow broadcast or multicast, so that all its iterators ("consumers"? subscribers?) receive all values?
Consider this example:
const fetchMock = () => "Example. Imagine real fetch";
async function* gen() {
for (let i = 1; i <= 6; i++) {
const res = await fetchMock();
yield res.slice(0, 2) + i;
}
}
const ait = gen();
(async() => {
// first "consumer"
for await (const e of ait) console.log('e', e);
})();
(async() => {
// second...
for await (const é of ait) console.log('é', é);
})();
Iterations "consume" a value, so only one or the other gets it.
I would like for both of them (and any later ones) to get every yielded value, if such a generator is possible to create somehow. (Similar to an Observable.)
This is not easily possible. You will need to explicitly tee it. This is similar to the situation for synchronous iterators, just a bit more complicated:
const AsyncIteratorProto = Object.getPrototypeOf(Object.getPrototypeOf(async function*(){}.prototype));
function teeAsync(iterable) {
const iterator = iterable[Symbol.asyncIterator]();
const buffers = [[], []];
function makeIterator(buffer, i) {
return Object.assign(Object.create(AsyncIteratorProto), {
next() {
if (!buffer) return Promise.resolve({done: true, value: undefined});
if (buffer.length) return buffer.shift();
const res = iterator.next();
if (buffers[i^1]) buffers[i^1].push(res);
return res;
},
async return() {
if (buffer) {
buffer = buffers[i] = null;
if (!buffers[i^1]) await iterator.return();
}
return {done: true, value: undefined};
},
});
}
return buffers.map(makeIterator);
}
You should ensure that both iterators are consumed at about the same rate so that the buffer doesn't grow too large.
Here's a solution using Highland as an intermediary. From the docs:
A stream forked to multiple consumers will pull values, one at a time, from its source as only fast as the slowest consumer can handle them.
import _ from 'lodash'
import H from 'highland'
export function fork<T>(generator: AsyncGenerator<T>): [
AsyncGenerator<T>,
AsyncGenerator<T>
] {
const source = asyncGeneratorToHighlandStream(generator).map(x => _.cloneDeep(x));
return [
highlandStreamToAsyncGenerator<T>(source.fork()),
highlandStreamToAsyncGenerator<T>(source.fork()),
];
}
async function* highlandStreamToAsyncGenerator<T>(
stream: Highland.Stream<T>
): AsyncGenerator<T> {
for await (const row of stream.toNodeStream({ objectMode: true })) {
yield row as unknown as T;
}
}
function asyncGeneratorToHighlandStream<T>(
generator: AsyncGenerator<T>
): Highland.Stream<T> {
return H(async (push, next) => {
try {
const result = await generator.next();
if (result.done) return push(null, H.nil);
push(null, result.value);
next();
} catch (error) {
return push(error);
}
});
}
Usage:
const [copy1, copy2] = fork(source);
Works in Node, browser untested.
I built a library to do this here: https://github.com/tjenkinson/forkable-iterator
Means you can do something like:
import { buildForkableIterator, fork } from 'forkable-iterator';
function* Source() {
yield 1;
yield 2;
return 'return';
}
const forkableIterator = buildForkableIterator(Source());
console.log(forkableIterator.next()); // { value: 1, done: false }
const child1 = fork(forkableIterator);
// { value: 2, done: false }
console.log(child1.next());
// { value: 2, done: false }
console.log(forkableIterator.next());
// { value: 'return', done: true }
console.log(child1.next());
// { value: 'return', done: true }
console.log(forkableIterator.next());
If you no longer need to keep consuming from a fork providing you loose references to it there also shouldn’t be a memory leak.
I'm using async await inside of for loop as below.
for (let i = 0; i < result.length; i += 1) {
try{
const client = await axios.get(
`${process.env.user}/client/${result[i].id}`
);
} catch(error){
console.log(error)
}
if (client.data.success === true) {
result[i].Name = rider.data.client.Name;
result[i].PhoneNumber = rider.data.client.Number;
}
}
But I want to make this using 'new Promise' and 'promiss.all' to make it asynclously.
But I don'k know how to make this correctly doing error handle well.
Could you recommend some advice for this? Thank you for reading it.
This can be a basic solution, i think
let playList = []
for (let i = 0; i < result.length; i += 1) {
playList.push(axios.get(
`${process.env.user}/client/${result[i].id}`
).then(res => {
if (res.data.success === true) {
result[i].Name = rider.data.client.Name;
result[i].PhoneNumber = rider.data.client.Number;
}
}).catch(ex => console.log(ex)));
}
await Promise.all(playList)
This can also be done by using a foreach loop.
The for/foreach loop can be simplified by using a map function.
Js map function is equivalent of c# select linq function.
The fat arrow in js map function is not bound to return a value unlike c# select inner function which must return a value.
await Promise.all(result.map(async r => {
let client;
try {
client = await axios.get(`${process.env.user}/client/${r.id}`);
} catch (error) {
console.log(error)
}
if (client.data.success === true) {
r.Name = rider.data.client.Name;
r.PhoneNumber = rider.data.client.Number;
}
}));
Try this
var promises = result.map(r => axios.get(`${process.env.user}/client/${r.id}`);
Promise.all(promises).then(function(values) {
console.log('All promises done');
});
The idea is that if you are awaiting something, that is promise, you can await it, or call it to get promise
Example:
function Foo()
{
return new Promise(...); // Promise of int for example
}
you can do
var p = Foo(); //you will get promise
Or
var v = await Foo(); // you will get int value when promise resolved
This is how you do it with async/await + Promise.all:
const myResult = await Promise.all(result.map(({ id }) => {
return axios.get(`${process.env.user}/client/${id}`);
}));
// deal with the result of those requests
const parsed = myResult.map(data => /* your code here */);
Here is an example using Array.map to call your function along with Promise.all. I wrapped the axios request in a function so if one of your request fails, it wont stop every other requests. If you don't mind stopping when you got an issue, look at others answers to your question.
function fakeRequest() {
return new Promise((resolve, reject) => {
setTimeout(() => {
resolve({
data: {
success: true,
client: {
Name: 'Todd',
Number: 5,
},
},
});
}, 300);
});
}
(async() => {
const result = [{}, {}, {}];
await Promise.all(result.map(async(x, xi) => {
try {
const client = await fakeRequest();
if (client.data.success === true) {
result[xi].Name = client.data.client.Name;
result[xi].PhoneNumber = client.data.client.Number;
}
} catch (err) {
console.log(err)
}
}));
console.log(result);
})();
UPDATE
I have read over a dozen articles on this topic and not one of them addresses this fundamental question. I am going to start listing a resources section at the end of this post.
ORIGINAL POST
My understanding of an async function is it returns a promise.
MDN docs: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function
Inside my program I could write something like:
function testPromise() {
return new Promise((resolve, reject) => {
// DO WORK
reject() // IF WORK FAILS
resolve() // IF WORK IS SUCCESSFUL
})
}
async function mainFunction() {
let variable
try {
variable = await testPromise()
} catch(e) {
throw e
}
return variable
}
I could also write testPromise as an async function and await that in the same context.
async function testAsyncFunction() {
//DO WORK AND THROW ERROR IF THEY OCCUR
}
async function mainFunction() {
let variable
try {
variable = await testAsyncFunction()
} catch(e) {
throw e
}
return variable
}
Which would be considered best practice? If I wish to create asynchronous operation, should the function use return New Promise and awaited in a async function or is awaiting an async function from an async function the same difference?
RESOURCES
JavaScript ES 2017: Learn Async/Await by Example
https://codeburst.io/javascript-es-2017-learn-async-await-by-example-48acc58bad65
Javascript — ES8 Introducing async/await Functions
https://medium.com/#reasoncode/javascript-es8-introducing-async-await-functions-7a471ec7de8a
6 Reasons Why JavaScript’s Async/Await Blows Promises Away (Tutorial)
https://hackernoon.com/6-reasons-why-javascripts-async-await-blows-promises-away-tutorial-c7ec10518dd9
----------------------CURRENT----------------------
export default function time_neo4jUpdate({ store, action, change, args, }) {
return new Promise(async (resolve, reject) => {
try {
const {
thing: { type },
nonValidatedArgs: { leapYear, root, start, end },
_neo4j,
_cypherReducers,
_neo4jCreateReduce,
_timetreeSteps: { update }
} = store.getState()
let results = []
for (let i = 0; i < _neo4jCreateReduce.length; i++) {
const result = await _neo4j.session(
_neo4jCreateReduce[i],
_cypherReducers.runQuery(update, i, root, start, end))
results = [...results, result]
}
resolve({
store,
action: 'NEO4J_UPDATE',
change: results,
args
})
} catch (e) {
const m = `NEO4J TIMETREE UPDATE: Unable to complete the UPDATE step for the timetree. WHAT: ${e}`
reject(m)
}
})
}
----------------------AS ASYNC FUNCTION----------------------
export default async function time_neo4jUpdate({ store, action, change, args, }) {
try {
const {
thing: { type },
nonValidatedArgs: { leapYear, root, start, end },
_neo4j,
_cypherReducers,
_neo4jCreateReduce,
_timetreeSteps: { update }
} = store.getState()
let results = []
for (let i = 0; i < _neo4jCreateReduce.length; i++) {
const result = await _neo4j.session(
_neo4jCreateReduce[i],
_cypherReducers.runQuery(update, i, root, start, end))
results = [...results, result]
}
return {
store,
action: 'NEO4J_UPDATE',
change: results,
args
}
} catch (e) {
const m = `NEO4J TIMETREE UPDATE: Unable to complete the UPDATE step for the timetree. WHAT: ${e}`
throw m
}
}
Even without the availability of async/await, you should very rarely need to use new Promise(). If you're using it a lot, it's typically a code smell.
The whole point of async/await is that it allows you to avoid a lot of the situations where you would otherwise need to work with promises explicitly.
So if it's supported in the environment you're targeting (Internet Explorer does not support async/await) or you're using a transpiler, go ahead and use it anywhere you can. That's what it's for.
Bear in mind that this is pointless:
catch(e) {
throw e;
}
There's no point in catching an error just to rethrow it. So if you're not actually doing anything with the caught error, don't catch it:
async function testAsyncFunction() {
//DO WORK AND THROW ERROR IF THEY OCCUR
return value
}
Edit: Now that you've provided an example of your code, I can answer with more certainty: If your function is based upon existing promises, then by all means, it is good to use async/await and you usually should not use new Promise():
export default async function time_neo4jUpdate({
store,
action,
change,
args,
}) {
try {
const {
thing: {
type
},
nonValidatedArgs: {
leapYear,
root,
start,
end
},
_neo4j,
_cypherReducers,
_neo4jCreateReduce,
_timetreeSteps: {
update
}
} = store.getState()
const results = await _neo4jCreateReduce.reduce(async function (acc, el) {
const result = await _neo4j.session(
el,
_cypherReducers.runQuery(
update,
i,
root,
start,
end
)
)
return [...(await acc), result]
}, []);
return {
store,
action: 'NEO4J_UPDATE',
change: results,
args
};
} catch (e) {
const m = `NEO4J TIMETREE UPDATE: Unable to complete the UPDATE step for the timetree. WHAT: ${e}`
throw m;
}
}