We Keep Coding

How to rewrite this request using async/await syntax?

Here's the task:
You need to make a GET request for the resource: https://jsonplaceholder.typicode.com/posts using fetch method
Save the response to response.json file
Save only those items, where id < 20
What I wrote:
const fetch = require('node-fetch');
const fs = require('fs');
const path = require('path');
const filePath = path.join(__dirname, 'response.json');
fetch('https://jsonplaceholder.typicode.com/posts')
.then(res => res.json())
.then(data => {
const refined = data.filter(item => item.id < 20);
const stringified = JSON.stringify(refined);
fs.appendFile(filePath, stringified, err => {
if (err) {
throw err;
}
});
});
How to write the same fetch, but with async/await syntax?

await keyword can only be used inside an async function, so you need to write an async function that makes the API request to fetch the data
async function fetchData() {
const response = await fetch('https://jsonplaceholder.typicode.com/posts');
const data = await response.json();
const refined = data.filter(item => item.id < 20);
const stringified = JSON.stringify(refined);
// promise version of appendFile function from fs.promises API
await fs.appendFile(filePath, stringified);
}
fs module of nodeJS has functions that use promises instead of callbacks. if you don't want to use callback version, you will need to use promise version of appendFile function.
You can import the promise version of fs module as require('fs').promises or require('fs/promises').
To handle errors, make sure that code that calls this function has a catch block to catch and handle any errors that might be thrown from this function. You could also wrap the code in this function with try-catch block to handle the errors inside this function.
Side tip: If you want to write data in the file in easily readable format, change
const stringified = JSON.stringify(refined);
to
const stringified = JSON.stringify(refined, null, 4);

Below snippet could help you (tested in node v14)
const fetch = require("node-fetch")
const fs = require("fs")
const path = require("path")
const filePath = path.join(__dirname, "response.json")
async function execute() {
const res = await fetch("https://jsonplaceholder.typicode.com/posts")
const data = await res.json()
const refined = data.filter((item) => item.id < 20)
const stringified = JSON.stringify(refined)
fs.appendFile(filePath, stringified, (err) => {
if (err) {
throw err
}
})
}
execute()

console.log not logging with await variable

I am trying to log the data of a promise to my console but it's not showing. i have tried defining then in then and on top of functions and tried with let and redefining the before executing the algorithm but. no response
sample
var trade;
const getTrades = async () => {
return await axios({
method: 'get',
url: bUrl + tradeQuery
})
}
const getSOrders = async () => {
return await axios({
method: 'get',
url: bUrl + mOrderQuery
})
}
const postOrder = async() => {
const binanceRest = new api.BinanceRest({
...
}
)
binanceRest.newOrder({
...
})
.then(async(data) => {
const trades = await getTrades()
const mOrders = await getSOrders()
console.log(data)
console.log(trades)
})
.catch((err) => {
console.error(err)
})
}
(
postOrder(),
async () => {
const trades = await getTrades()
const mOrders = await getSOrders()
const sells = mOrders.data.asks
const buys = mOrders.data.bids
while (true/*while order is in */) {
trade = trades.data[trades.data.length - 1]
console.log(sells)
}
}
)()

Your code is stuck in a an infinite while loop. Since node is single threaded, it will continue to execute the loop until broken and never execute the .then code in the original promise. You should be able to easily see this by commenting out the while loop.
Specifically:
binanceRest.newOrder is async call. It's true that postOrder is called first, but depending on the how nodejs decides to optimize, all of the awaits are hit at the same time. When the awaits resolve, if Nodejs decides that postOrders will resume your console.log will write, however if the while loop it hit first, then your program will be stuck there.

I'm sorry to say this, but I can see a lot of confusing things in this code.
while(true) {
trade = trades.data[trades.data.length - 1];
console.log(sells);
}
this will never exits. If you omitted part of your code is not a good idea as this makes us harder to give you the actual answer to your question. If this is your real code, I would change in
while(trades.data.length) {
trade = trades.data.pop();
console.log(sells);
}
postOrder is an async function which deals with Promise, and this is confusing: I would rewrite it as
const postOrder = async() => {
try {
const binanceRest = new api.BinanceRest({ ... });
const data = await binanceRest.newOrder({ ... });
const trades = await getTrades()
const mOrders = await getSOrders()
console.log(data)
console.log(trades)
}
catch(err) {
console.error(err)
}
}
Last postOrder is an async function which is called without await and this is source of confusion as well.
I would start cleaning your code, probably many problem will be solved as well.
Hope this helps.

Is not an "await" problem
I've added a bunch of console.log instructions at your code and there's no problem printing anything. Try adding the same to your original code, assuming your API's are working fine there's no reason for failure.
The following snippet mocks your API's, please take a look at the result. Your error must be at your logic, api's or syntax.
Also add a safe break inside your while, something like this: if (calls++ > 5) { break; }, to me it seems that the infinite loop is the problem.
/* MOCK API'S */
let test = 0, calls = 0;
const fakePromise = (arg, apiName) => { return { then: (f1, f2) => { console.log('mocked result for ' + apiName); return f1(arg); } }; };
const axios = (args) => { console.log('-> axios is called', args); return fakePromise({data: { asks: ['ask'], bids: ['bid'], length: 1, 0: 'fake index'} }, args.url); };
const api = { BinanceRest: function(){ return { newOrder: () => fakePromise(' -> newOrder result <- ', 'newOrder')}; } };
const bUrl = 'bUrl/', mOrderQuery = 'mOrderQuery', tradeQuery = 'tradeQuery';
/* YOUR CODE STARTS HERE */
var trade;
const getTrades = async () => {
console.log('-> getTrades is called');
return await axios({
method: 'get',
url: bUrl + tradeQuery
});
}
const getSOrders = async () => {
console.log('-> getSOrders is called');
return await axios({
method: 'get',
url: bUrl + mOrderQuery
})
}
const postOrder = async() => {
console.log('step 1');
const binanceRest = new api.BinanceRest({
// ...
});
console.log('step 2');
binanceRest.newOrder({
// ...
})
.then(async(data) => {
console.log('step 3');
const trades = await getTrades()
const mOrders = await getSOrders()
console.log('step 4.2');
console.log('data: ', data)
console.log('trades: ', trades)
console.log('mOrders', mOrders)
})
.catch((err) => {
console.error(err)
})
}
// test
(
postOrder(),
async () => {
console.log('step 4.1');
const trades = await getTrades()
const mOrders = await getSOrders()
console.log('step 5', mOrders);
const sells = mOrders.data.asks
const buys = mOrders.data.bids
console.log('step 5.0');
while (true/*while order is in */) {
console.log('step 5.' + (test++));
trade = trades.data[trades.data.length - 1]
console.log('sells: ', sells)
if (calls++ > 5) {
console.log('way too much calls, break! ');
break;
}
}
}
)()

Take a closer look at your postOrder func:
const postOrder = async() => {
const binanceRest = new api.BinanceRest({ ... })
binanceRest.newOrder({ ... })
.then(...)
.catch(...)
}
Now imagine the scenario when a typo has been made, f.e. it should be new api.FinanceRest(...) and as a result you end up with error:
Uncaught (in promise) TypeError: api.BinanceRest is not a function
Rest of code in postOrder is simply skipped, no console.logs. So the main point here: if in new api.BinanceRest({ ... }) something leads to an error (any error) than you'll end up with situation you have right now.
And also worth to mention that an async anonymous function with while loop still gets executed and that's because postOrder is an async func which means it returns a pending Promise.
Try running this in you browsers console to get more sense of what's goin' on:
function delay(sec) {
return new Promise((res, rej) => {
setTimeout(() => res("delay " + sec + "sec"), sec * 1000);
});
}
async function getTrades() {
return await delay(0.1);
};
async function getSOrders () {
return await delay(0.2);
};
async function postOrder() {
console.log("postOrder");
const api = {};
const binanceRest = api.BinanceRest()
delay(0.4)
.then(async (data) => {
console.log("postOrder result")
const trades = await getTrades()
const mOrders = await getSOrders()
console.log(trades)
console.log(mOrders)
})
.catch(err => {
console.error(err);
});
};
(postOrder(),
async () => {
const trades = await getTrades();
const mOrders = await getSOrders();
console.log("gfdfg");
})();

Problem with fs.writeFile in reduce with fetch

I need some help with this helper I'm writing. For some reason using reduction within an async on a readFile, when trying to write results to a file it won't advance to the next item of the array. However, if I use a console.log, it works just fine.
const neatCsv = require('neat-csv');
const fetch = require('node-fetch');
const fs = require('fs');
fs.readFile('./codes.csv', async (err, data) => {
if (err) { throw err; }
let baseUrl = 'https://hostname/orders?from=2019-10-21T00:00:00.001Z&to=2019-12-31T23:59:59.000Z&promo=';
const starterPromise = Promise.resolve(null);
const promos = await neatCsv(data);
const logger = (item, result) => console.log(item, result);
function write (item, result) {
return new Promise((resolve, reject) => {
fs.writeFile(`./output/${item.PROMO}.json`, JSON.stringify(result), (err) => {
if (err) { throw err; }
console.log(`Wrote file ${item.PROMO}`);
});
})
}
function asyncFetch(item) {
console.log(`runTask <---------${item.PROMO}---------`);
return fetch(`${baseUrl}${item.PROMO}`, { headers: { 'x-apikey': 'xyz' }})
.then(res => (res.json())
.then(json => json))
}
await promos.reduce(
(p, item) => p.then(() => asyncFetch(item).then(result => write(item, result))),
starterPromise
)
});
The csv file is just a basic layout like so..
PROMO
12345
56789
98765
...
The goal is to iterate over these, make a REST call to get the json results and write those to a file with the name of the current promo, then move to the next, making a new call and saving that one into a different file with its respective code.
In the reduce, if you call logger instead of write, it works fine. Calling write, it just makes the same call over and over and overwriting to the same file, forcing me to kill it. Please help, I'm losing my mind here...

You might have a better time using async functions everywhere, the fs promises API and a simple while loop to consume the CSV items. Dry-coded, naturally, since I don't have your CSV or API.
(Your original problem is probably due to the fact you don't resolve/reject in the write function, but the reduce hell isn't needed either...)
const neatCsv = require("neat-csv");
const fetch = require("node-fetch");
const fsp = require("fs").promises;
const logger = (item, result) => console.log(item, result);
const baseUrl = "https://hostname/orders?from=2019-10-21T00:00:00.001Z&to=2019-12-31T23:59:59.000Z&promo=";
async function asyncFetch(item) {
console.log(`runTask <---------${item.PROMO}---------`);
const res = await fetch(`${baseUrl}${item.PROMO}`, { headers: { "x-apikey": "xyz" } });
const json = await res.json();
return json;
}
async function write(item, result) {
await fsp.writeFile(`./output/${item.PROMO}.json`, JSON.stringify(result));
console.log(`Wrote file ${item.PROMO}`);
}
async function process() {
const data = await fsp.readFile("./codes.csv");
const promos = await neatCsv(data);
while (promos.length) {
const item = promos.shift();
const result = await asyncFetch(item);
await write(item, result);
}
}
process().then(() => {
console.log("done!");
});
A version that uses mock data and the JSON Placeholder service, works just fine:
const fetch = require("node-fetch");
const fsp = require("fs").promises;
const baseUrl = "https://jsonplaceholder.typicode.com/comments/";
async function asyncFetch(item) {
console.log(`runTask <---------${item.PROMO}---------`);
const res = await fetch(`${baseUrl}${item.PROMO}`);
return await res.json();
}
async function write(item, result) {
const data = JSON.stringify(result);
await fsp.writeFile(`./output/${item.PROMO}.json`, data);
console.log(`Wrote file ${item.PROMO}: ${data}`);
}
async function getItemList() {
return [
{PROMO: '193'},
{PROMO: '197'},
{PROMO: '256'},
];
}
async function process() {
const promos = await getItemList();
while (promos.length) {
const item = promos.shift();
const result = await asyncFetch(item);
await write(item, result);
}
}
process().then(() => {
console.log("done!");
});

function is being executed twice when called only once

I am making a program that will read content of files inside nested folders.For now I am just trying to log the content of the file in console. But I am getting two logs instead of only one.Here is what I have done till now
const fs = require('fs');
const path = require('path');
const { promisify } = require('util');
const getStats = promisify(fs.stat);
const readdir = promisify(fs.readdir);
const http = require('http');
handle_files = async (req, res) => {
let files = await scanDir("logs_of_109");
let result = await read_content(files)
check_file_content(result)
res.writeHead(200, { 'Content-Type': 'text/html' });
// console.log(result)
res.write("Hello");
res.end();
};
check_file_content = (file_data) => {
console.log(file_data[1])
}
async function read_content(files) {
let file_data = []
files.map(file => {
let start_index = file.toString().lastIndexOf('.') + 1
let ext = file.substring(start_index, file.length)
if (ext == 'data') {
file_data.push(fs.readFileSync(file, { encoding: 'utf-8' }))
}
})
return file_data
}
http.createServer(handle_files).listen(8080)
async function scanDir(dir, fileList = []) {
// fetch list of files from the giver directory
let files = await readdir(dir);
// loop through all the files
for (let file of files) {
// join new folder name after the parent folder
// logs_of_109/24
let filePath = path.join(dir, file);
try {
//
let stats = await getStats(filePath);
if (!stats.isDirectory()) {
// add the filepath to the array
fileList.push(filePath);
}
if (stats.isDirectory()) {
await scanDir(filePath, fileList);
}
} catch (err) {
// Drop on the floor..
}
}
return fileList;
}
I expect the file content to be logged only once but it is logging twice on my console. Why is this happening and how do I stop this?

Your browser is making two requests to your server, most likely one for the URL you put in the address bar and another for favicon.ico. (You can quickly tell by opening the dev tools on your browser and going to the Network tab.)
handleFiles should look at req (specifically its url property) and act according to what's being requested. (This is something the code should be doing anyway.)
Side note 1: You're passing an async function into something (createServer) that won't do anything with the promise it returns. If you do that, it's important to catch any errors in the function locally within the function, since (again) nothing else is going to handle them. E.g.:
handle_files = async (req, res) => {
try {
let files = await scanDir("logs_of_109");
let result = await read_content(files)
check_file_content(result)
res.writeHead(200, { 'Content-Type': 'text/html' });
// console.log(result)
res.write("Hello");
res.end();
} catch (e) {
// ...handle error here...
}
};
Side note 2: That code is falling prey to The Horror of Implicit Globals¹. Declare your variables in the appropriate scope. Not declaring them, in loose mode, makes them globals. (Also recommend using strict mode, so you get an error for this.)
¹ (that's a post on my anemic little blog)

The answer above is correct.
My approach is to solve it via 'routing' of any kind.
Here is small basic example of how it can be done
const fs = require('fs');
const path = require('path');
const { promisify } = require('util');
const getStats = promisify(fs.stat);
const readdir = promisify(fs.readdir);
const http = require('http');
handle_routes = async (req, res) => {
switch(req.url) {
case '/files':
handle_files(req, res);
default:
console.log('for default page');
}
}
handle_files = async (req, res) => {
let files = await scanDir("logs_of_109");
let result = await read_content(files)
check_file_content(result)
res.writeHead(200, { 'Content-Type': 'text/html' });
res.write("Hello");
res.end();
};
check_file_content = (file_data) => {
console.log(file_data[1])
}
async function read_content(files) {
let file_data = []
files.map(file => {
let start_index = file.toString().lastIndexOf('.') + 1
let ext = file.substring(start_index, file.length)
if (ext == 'data') {
file_data.push(fs.readFileSync(file, { encoding: 'utf-8' }))
}
})
return file_data
}
http.createServer(handle_routes).listen(8080)
async function scanDir(dir, fileList = []) {
// fetch list of files from the giver directory
let files = await readdir(dir);
// loop through all the files
for (let file of files) {
// join new folder name after the parent folder
// logs_of_109/24
let filePath = path.join(dir, file);
try {
//
let stats = await getStats(filePath);
if (!stats.isDirectory()) {
// add the filepath to the array
fileList.push(filePath);
}
if (stats.isDirectory()) {
await scanDir(filePath, fileList);
}
} catch (err) {
// Drop on the floor..
}
}
return fileList;
}
This gives you possibility to call handle_files function by going to localhost:8080/files url

How to set variable = a value from a function result inside async function

Inside a function, I would like to set the value of a variable (foldersInDir) to the results of getting the contents of a directory using fs.readdir();
I thought using await would force the console.log line to wait for a response, but it's not.
How can I set foldersInDir = the return value?
/*Begin function*/
const listContents = async (myPath) => {
var fs = require('fs');
let foldersInDir = await fs.readdir(myPath, function(err, items) {
console.log(items); //works
return items;
});
console.log(foldersInDir); //does not work, undefined
}

You need to convert readdir to a promise, e.g.:
const foldersPromised = (path) =>
new Promise((resolve, reject) =>
fs.readdir(path, (err, items) =>
err !== undefined ? reject(err) : resolve(items)
)
);
try {
let foldersInDir = await foldersPromised(myPath);
} catch(err) {
console.log(err);
}

const fs = require('fs');
const test = () => {
let folders = fs.readdirSync('.');
return folders;
}
console.log(test());

Edit: sorry, need to promisify() the function
const fs = require('fs');
const { promisify } = require('util') // available in node v8 onwards
const readdir = promisify(fs.readdir)
async function listContents() {
try { // wrap in try-catch in lieu of .then().catch() syntax
const foldersInDir = await readdir(myPath) // call promised function
console.log('OK, folders:', foldersInDir) // success
} catch (e) {
console.log('FAIL reading dir:', e) // fail
}
}
listContents('path/to/folder') // run test

I recommend using the promisify function provided by Node.js to fix the problem. This function will convert a callback-based function to a promise-based function, which can then be used using the await keyword.
const fs = require('fs');
const {
promisify
} = require('util');
const readdirAsync = promisify(fs.readdir);
/*Begin function*/
const listContents = async(myPath) => {
let foldersInDir = await readdirAsync(myPath);
console.log(foldersInDir);
}