I have a button that makes an http.get on click, and then displays some data, although my async/awaits seem to not be synchronized correctly.
For a reason I haven't quite figured out yet, this.tabRows = file.contents will be called before loadDateFolderFileContents() completes. Am I missing an await somewhere?
async ngAfterViewInit() {
if (!this.drawn) {
if(!file.contents){
await this.dataLakeService.loadDateFolderFileContents(file.parentFolder);
}
this.tabRows = file.contents;
this.redraw();
}
}
public async loadDateFolderFileContents(dateFolder: folder) {
await date.files.map(async file => {
file.contents = await this.getFileContents(dateFolder.name, file.name);
});
}
public async loadDateFolderFileContents(dateName: string, fileName: string): Promise<any> {
var url = this.buildUrl(dateName, fileName);
var fileContents = {};
await this.http.get<any>(url).toPromise()
.then(contents => {
fileContents = contents;
})
.catch(e => {
console.log(`Error retreiving file contents from url ${url}. Exception:${e}`);
});
return fileContents;
}
If you use await in a map, map will always return an array of promise. This is because asynchronous functions always return promises.
Since map always return promises (if you use await), you have to wait for the array of promises to get resolved. You can do this with await Promise.all(arrayOfPromises), after all promises are resolved, Promise.all returns a single promise.
below is your updated code
async ngAfterViewInit() {
if (!this.drawn) {
if(!file.contents){
await this.dataLakeService.loadDateFolderFileContents(file.parentFolder);
}
this.tabRows = file.contents;
this.redraw();
}
}
public async loadDateFolderFileContents(dateFolder: folder) {
await Promise.all(date.files.map(async file => {
file.contents = await this.getFileContents(dateFolder.name, file.name);
}));
}
public async loadDateFolderFileContents(dateName: string, fileName: string): Promise<any> {
var url = this.buildUrl(dateName, fileName);
var fileContents = {};
await this.http.get<any>(url).toPromise()
.then(contents => {
fileContents = contents;
})
.catch(e => {
console.log(`Error retreiving file contents from url ${url}. Exception:${e}`);
});
return fileContents;
}
map is not async so instead of:
await date.files.map(async file => {
...
try
await Promise.all(date.files.map(async file => {
...)
I think this is what you are trying to do:
async ngAfterViewInit() {
if (!this.drawn) {
if(!file.contents){
await this.dataLakeService.loadDateFolderFileContents(file.parentFolder);
}
this.tabRows = file.contents;
this.redraw();
}
}
public async loadDateFolderFileContents(dateFolder: folder) {
await Promise.allSettled(date.files.map(async file => {
file.contents = await this.getFileContents(dateFolder.name, file.name);
return file;
}));
}
public async loadDateFolderFileContents(dateName: string, fileName: string): Promise<any> {
var url = this.buildUrl(dateName, fileName);
var fileContents = {};
try {
const contents = await this.http.get<any>(url).toPromise();
fileContents = contents;
return fileContents;
} catch(e){
console.log(`Error retreiving file contents from url ${url}. Exception:${e}`);
}
}
Related
so i need a way to delay the return statement in the controller until the foreach loop is done.
i just need to properly handle the asynchronous operation and i don't know how else
the response returns an empty array because the array is populated after the response is returned
uploads.controller.ts
import {
Controller,
Post,
UseInterceptors,
UploadedFiles,
} from '#nestjs/common';
import { FilesInterceptor } from '#nestjs/platform-express';
import { UploadsService } from './uploads.service';
import { v2 as cloudinary } from 'cloudinary';
#Controller('attachments')
export class UploadsController {
constructor(private readonly uploadsService: UploadsService) {}
#Post('images')
#UseInterceptors(FilesInterceptor('attachment'))
async uploadFile(#UploadedFiles() attachment: Array<Express.Multer.File>) {
cloudinary.config({
cloud_name: process.env.CLOUDINARY_CLOUD_NAME,
api_key: process.env.CLOUDINARY_API_KEY,
api_secret: process.env.CLOUDINARY_API_SECRET,
secure: true,
});
const response = [];
await attachment.forEach(async (file) => {
const fileResponse = await this.uploadsService.uploadImage(file);
response.push(fileResponse.secure_url);
console.log('1', response);
});
console.log('2', response);
return await response;
}
}
uploads.service.ts
import { UploadApiErrorResponse, UploadApiResponse, v2 } from 'cloudinary';
import toStream = require('buffer-to-stream');
#Injectable()
export class UploadsService {
async uploadImage(
file: Express.Multer.File,
): Promise<UploadApiResponse | UploadApiErrorResponse> {
return new Promise((resolve, reject) => {
const upload = v2.uploader.upload_stream((error, result) => {
if (error) return reject(error);
resolve(result);
});
toStream(file.buffer).pipe(upload);
});
}
}
The problem is here.
await attachment.forEach(async (file) => {
const fileResponse = await this.uploadsService.uploadImage(file);
response.push(fileResponse.secure_url);
console.log('1', response);
});
Don't pass async function to Array.forEach. Check out this question.
This will work.
const response = await Promise.all(attachment.map(async (file) => {
const fileResponse = await this.uploadsService.uploadImage(file);
return fileResponse.secure_url;
}));
How to read files asynchronously in node js, here is a simple function.
There are a lot of convoluted answers on the internet, does anyone agree if this is the simplest?
export default async function handler(req, res) {
let data = await readFiles('data/companies/');
res.status(200).json(data);
}
// async file reader
function readFiles(dirname) {
return new Promise(function (resolve, reject) {
let data = {}
fs.readdir(dirname, async function(err, filenames) {
filenames.forEach(function(filename) {
fs.readFile(dirname + filename, 'utf-8', function(err, content) {
if (err) {
reject(err)
}
data[filename] = content;
if (filenames.length === Object.keys(data).length) {
resolve(data)
}
});
});
});
})
}
A bit cleaner and easier using the built in promise support in fs.promises:
const fs = require('fs');
const fsp = fs.promises;
const path = require('path');
// async file reader
async function readFiles(dirname) {
const data = {};
const files = await fsp.readdir(dirname);
for (const filename of files) {
const full = path.join(dirname, filename);
const content = await fsp.readFile(full, {encoding: 'utf8'});
data[filename] = content;
}
return data;
}
Or, if you want to run your file operations in parallel (at least to the limit of the thread pool), you might get slightly faster end-to-end performance like this:
// async file reader
async function readFiles(dirname) {
const data = {};
const files = await fsp.readdir(dirname);
await Promise.all(files.map(async filename => {
const full = path.join(dirname, filename);
const content = await fsp.readFile(full, {encoding: 'utf8'});
data[filename] = content;
}));
return data;
}
Also, this:
res.status(200).json(data);
can be replaced with:
res.json(data);
200 is already the default status so there is no reason to specify it.
I am new to node and stuck with this issue. Here' the file:
I am running 'startProcess' function and I want to run 'downloadFiles' and wait until it's completed and save the files before executing any code after it.
This code always ends up running 'runVideoUploadEngine' even before the download has been completed?
const downloadAndSaveFiles = async ({ url, dir }) => {
try {
https.get(url, (res) => {
// File will be stored at this path
console.log('dir: ', dir);
var filePath = fs.createWriteStream(dir);
res.pipe(filePath);
filePath.on('finish', () => {
filePath.close();
console.log('Download Completed');
});
});
return true;
} catch (e) {
console.log(e);
throw e;
}
};
const downloadFiles = async ({ data }) => {
try {
mediaUrl = data.mediaUrl;
thumbnailUrl = data.thumbnailUrl;
const mediaExt = path.extname(mediaUrl);
const thumbExt = path.extname(thumbnailUrl);
mediaDir = `${__dirname}/temp/${'media'}${mediaExt}`;
thumbDir = `${__dirname}/temp/${'thumb'}${thumbExt}`;
await downloadAndSaveFiles({ url: mediaUrl, dir: mediaDir });
await downloadAndSaveFiles({ url: thumbnailUrl, dir: thumbDir });
return { mediaDir, thumbDir };
} catch (e) {
console.log(e);
throw e;
}
};
module.exports = {
startProcess: async ({ message }) => {
//check if message is proper
data = JSON.parse(message.Body);
//download video and thumbnail and store in temp.
console.log('starting download..');
const { mediaDir, thumbDir } = await downloadFiles({ data });
console.log('dir:- ', mediaDir, thumbDir);
pageAccessToken =
'myRandomToken';
_pageId = 'myRandomPageID';
console.log('running engine');
await runVideoUploadEngine({ pageAccessToken, _pageId, mediaDir, thumbDir });
//start videoUploadEngine
//on success: delete video/thumbnail
},
};
What am I doing wrong?
downloadAndSaveFiles returns a promise (because the function is async) but that promise doesn't "wait" for https.get or fs.createWriteStream to finish, and therefore none of the code that calls downloadAndSaveFiles can properly "wait".
If you interact with callback APIs you cannot really use async/await. You have to create the promise manually. For example:
const downloadAndSaveFiles = ({ url, dir }) => {
return new Promise((resolve, reject) => {
// TODO: Error handling
https.get(url, (res) => {
// File will be stored at this path
console.log('dir: ', dir);
var filePath = fs.createWriteStream(dir);
filePath.on('finish', () => {
filePath.close();
console.log('Download Completed');
resolve(); // resolve promise once everything is done
});
res.pipe(filePath);
});
});
};
I have a async function which makes a face_detection command line call. It's working fine otherwise, but I can't make it to wait for the response. Here is my function:
async uploadedFile(#UploadedFile() file) {
let isThereFace: boolean;
const foo: child.ChildProcess = child.exec(
`face_detection ${file.path}`,
(error: child.ExecException, stdout: string, stderr: string) => {
console.log(stdout.length);
if (stdout.length > 0) {
isThereFace = true;
} else {
isThereFace = false;
}
console.log(isThereFace);
return isThereFace;
},
);
console.log(file);
const response = {
filepath: file.path,
filename: file.filename,
isFaces: isThereFace,
};
console.log(response);
return response;
}
isThereFace in my response I return is always undefined because the response is sent to client before the response from face_detection is ready. How could I make this work?
You can either use the child_process.execSync call, which will wait for the exec to finish. But executing sync calls is discouraged ...
Or you can wrap child_process.exec with a promise
const result = await new Promise((resolve, reject) => {
child.exec(
`face_detection ${file.path}`,
(error: child.ExecException, stdout: string, stderr: string) => {
if (error) {
reject(error);
} else {
resolve(stdout);
}
});
});
I think you must convert child.exec into a Promise and use it with await. Otherwise the async function is not waiting for child.exec result.
To make it easy you can use Node util.promisify method:
https://nodejs.org/dist/latest-v8.x/docs/api/util.html#util_util_promisify_original
import util from 'util';
const exec = util.promisify(child.exec);
const result = await exec(`my command`);
A one-liner that does it:
const execute = async (command: string) => await new Promise(resolve => exec(command, resolve))
const result = await execute(`my command`);
here is my Promise Function, I go through each blob in Azure BlobStorage, then I read each blob. console.log(download) delivers the values as JSON.
But to close the return new Promise function, I want that resolve should return the JSON data from the reading the blobstream. But here in my case, resolve leads to nothing.
In Angular Service.ts file the code looks like this:
getData(): Promise<JsonData[]> {
return new Promise(async resolve => {
const containerName = "blobcontainer";
const containerClient = this.blobServiceClient.getContainerClient(containerName);
//list blobs
let i = 1;
async function main() {
i = 1;
for await (const blob of containerClient.listBlobsFlat()) {
console.log(`Blob ${i++}: ${blob.name}`);
const blockBlobClient = containerClient.getBlockBlobClient(blob.name);
//console.log(blockBlobClient)
const downloadBlockBlobResponse = await blockBlobClient.download(0);
const download = await blobToString(await downloadBlockBlobResponse.blobBody)
//console.log(downloadBlockBlobResponse)
console.log(download)
}
}
async function blobToString(blob: Blob): Promise<string> {
const fileReader = new FileReader();
return new Promise((resolve, reject) => {
fileReader.onloadend = (ev: any) => {
JSON.parse(ev.target!.result)
resolve(JSON.parse(ev.target!.result));
};
fileReader.onerror = reject;
fileReader.readAsText(blob);
});
}
const _blob = await main().catch((err) => {
console.error('message'); return null
});
resolve(_blob) //resolve should return downloaded JSON file, but it didn't
})
}
Then in the component file, I want to retrieve the data from the resolve, which should return the JSON string variables like "name", "timestamp", "value"- But in my case, you receive metadata from the blob and not the contents. Means the service.ts file isn't correctly programmed:
xy.component.ts
export class xyComponent implements OnInit {
#Input() title: string;
//jsondatas: Array<JsonData> = [];
jsondata: JsonData;
name: String;
timestamp: string;
value: number;
//constructor() { }
private jsonlistService: JsonDataService;
jsondatas: JsonData[]=null;
constructor(private jsonService: JsonDataService) {
this.jsonlistService = jsonService;
}
ngOnInit(): void {
this.jsonlistService.getData()
.then(results => this.jsondatas = results);
console.log(this.jsonService)
}
}
EDIT:
Even if I return download at the main function, resolve from main() doesn't deliver the json string.
Second EDIT:
here is the snippets how to return data:
async function main() {
i = 1;
for await (const blob of containerClient.listBlobsFlat()) {
console.log(`Blob ${i++}: ${blob.name}`);
const blockBlobClient = containerClient.getBlockBlobClient(blob.name);
//console.log(blockBlobClient)
const downloadBlockBlobResponse = await blockBlobClient.download(0);
const download = await blobToString(await downloadBlockBlobResponse.blobBody)
//console.log(downloadBlockBlobResponse)
console.log(download)
return download
}
}
But I didn't receive the downloaded file, error is still the same.
Would be very nice if you could help me
You doesn't return anything from main. Just return the answer.