[EDIT - SOLVED] An important piece of information not in the original question is that this function is running on Zeit Now serverless platform. Zeit functions stop processing after a res.end to keep uptime short. It was confusing because console.logs and other code was being run after the res.end but I assume because it wasn't taking any time to evaluate.
In this simple Node function, why isn't the await function running? I res.end early then want the rest of the function to run but it stops at the await function.
const fs = require("fs")
const bent = require("bent")
const buffer = bent("buffer")
const file_url = "http://klippr.s3.amazonaws.com/assets/mask.png"
module.exports = async (req, res) => {
console.log("REQUEST RECEIVED")
res.end("JOB ACCEPTED")
console.log("STARTING JOB")
console.log("BUT NOTHING AFTER THIS RUNS")
fs.writeFileSync("/tmp/mask.png", await buffer(file_url))
console.log("FILE DOWNLOADED")
// do more stuff with the downloaded file..
}
When I call this function the file is never downloaded and the FILE DOWNLOADED log line doesn't run. Log:
REQUEST RECEIVED
STARTING JOB
BUT NOTHING AFTER THIS RUNS
If I change the fs.writeFileSync to a regular fs.writeFile and use a callback everything works as expected:
const fs = require("fs")
const bent = require("bent")
const buffer = bent("buffer")
const file_url = "http://klippr.s3.amazonaws.com/assets/mask.png"
module.exports = async (req, res) => {
console.log("REQUEST RECEIVED")
res.end("JOB ACCEPTED")
console.log("STARTING JOB")
console.log("BUT NOTHING AFTER THIS RUNS")
// fs.writeFileSync("/tmp/mask.png", await buffer(file_url))
fs.writeFile("/tmp/mask.png", buffer(file_url), () => {
console.log("FILE DOWNLOADED")
})
console.log("FILE DOWNLOADED")
// do more stuff with the downloaded file..
}
Now the file is downloaded and both FILE DOWNLOADED lines log to the console. But I like the sync method because it avoids callback soup.
So my question is why can't an await function run after res.end?
Have you tried to do fs.writeFileSync without awaiting the buffer function like you do in fs.writeFile?
Related
I have a JavaScript API, but this API doesn't let the other API get called or call itself again during the time that it runs. Specifically, this API involves sleeping for a duration of time and while this is sleeping I expect that other API calls or same API call should also happen.
I have created a sample program for the same, and it behaves analogously. Looking for your comments and suggestions, thanks.
'use strict';
// importing required libraries
const sleep = require("sleep");
const express = require('express');
const app = express();
const port = 1234;
const server = require("http").createServer(app);
const bodyParser = require("body-parser");
app.use(bodyParser.urlencoded({ extended: false }));
app.use(bodyParser.json());
// this is the function that just sleeps for 30 seconds and after that is resolves the output
// because of this function the API calls needs to wait till it is completed
const promisified = async function () {
console.log("Promisified started")
return new Promise((resolve, reject) => {
let count = 0
console.log("promisified hit")
while (true) {
console.log("Sleeping for 30 seconds. Try: ", count);
sleep.sleep(30);
if (count >= 1) {
console.log("Timeout breaking");
return resolve("Broke out");
}
count += 1;
}
})
}
// this is the function that is being called on the API hit, which internally calls promisified function
const justafunction = async function (req, res) {
console.log("Just a function api hit");
res.status(200).send("api hit")
promisified()
.then(data => {
console.log("This then is workd...",data);
})
.catch(err => {
console.log("Error is :", err);
});
let x = await promisified();
console.log("X is here...",x);
}
// this is the API that we will call
app.get("/test", justafunction);
server.listen(port, function () {
console.log(" Server Successfully Started ", port);
});
Ideally, in python and C++ would open up a thread to do the job independently but I don't have interal working of JavaScript much. Hence, any comment in that context will be much appreciated.
JavaScript is single threaded: the event loop manager thread executes JavaScript code and hence can do nothing else until the JavaScript code being executed runs to completion (i.e. returns to the event loop).
No sleep function exists in JavaScript. To run synchronously this function call
sleep.sleep();
must record the time on entry and enter a polling loop that reads the time until the system clock has advanced sufficiently to return. Until such time the function continues to loop without returning to the event loop and blocks the thread.
An alternative using async and await could be:
function sleep( msec) {
return new Promise(resolve=>setTimeout(resolve, msec));
}
// calling example
async function test() {
console.log("sleep for 3000 msec");
await sleep(3000);
console.log("sleep expired");
}
document.querySelector("button").addEventListener("click", test);
Sleep for 3 seconds: <button type = "button">ok</button>
Note the version of sleep returning a promise must be called from an async function in JavaScript after an await operator, or have have fulfillment handler added to it using a then handler. The key to it working with awaitis that the await operator returns to the event loop until its operand promise becomes settled, allowing other code to be executed in the main thread.
I am trying to make a http request with node module http on an azure function with javascript and for some reason http.request is not receiving data(no error printed its like the requested is blocked). There is anything wrong with azure configuration or the code?Am i missing something really obvious?
Code is running fine on local js file
context.log('JavaScript timer trigger function ran!', timeStamp);
printed as expected but context.log("SUCCESS", JSON.parse(data)); not printed at all.
Also tryed different libraries (request, axios) with no success
module.exports = async function (context, myTimer) {
var timeStamp = new Date().toISOString();
if (myTimer.IsPastDue)
{
context.log('JavaScript is running late!');
}
context.log('JavaScript timer trigger function ran!', timeStamp);
const http = require('http');
http.get('http://myAPIurl', (resp) => {
let data = '';
// A chunk of data has been recieved.
resp.on('data', (chunk) => {
data += chunk;
});
// The whole response has been received. Print out the result.
resp.on('end', () => {
context.log("SUCCESS", JSON.parse(data));
});
}).on("error", (err) => {
context.log("ERROR: " + err.message);
});
}
I rewrote your code using Axios. It has support for async/await out of the box, and simplifies a lot of the code to function as your would expect since it makes asynchronous code perform like synchronous code.
I think the main issue you may have been running into it that everything in JavaScript is async. As a result, the Azure Function runtime was exiting before your async function finished since it didn't block like synchronous code would when making the HTTP request. If you want to use callbacks, you need to call context.done() inside your callback function so the Azure Function doesn't exit before the callback is completed. By using async/await, you can guarantee that your code will block on the HTTP request until it receives a response or timeout. In the example below, Axios will return a response object that includes data as an element. I'm extracting data with a deconstruction operation, which allows me to log data.
const axios = require('axios');
const url = 'https://google.com';
module.exports = async function (context, myTimer) {
try {
const { data } = await axios.get(url);
context.log(data);
// do something with the data
return data;
} catch (err) {
context.log(err);
// do something with the error
}
context.done();
}
To Install a Package on an Azure Function
Go to the Azure Portal; select your Azure Function App
Select Platform Features. Then Advanced Tools (Kudu) under Development Tools
Using the file explorer, navigate to site/wwwroot/. You should see a package.json file in there, and several folders each with the name of your functions.
From that directory, run npm install --save axios
To confirm it worked, edit your package.json with the pencil icon. axios should be listed under dependencies json element
I've kept your code callback based.
I removed the async moniker from the definition and added a call to context.done (this signals the functions host when your function has ended) in your resp.end handler
module.exports = function (context, myTimer) {
var timeStamp = new Date().toISOString();
if (myTimer.IsPastDue)
{
context.log('JavaScript is running late!');
}
context.log('JavaScript timer trigger function ran!', timeStamp);
const https = require('https');
https.get('https://raw.githubusercontent.com/LearnWebCode/json example/master/animals-1.json', (resp) => {
let data = '';
// A chunk of data has been recieved.
resp.on('data', (chunk) => {
data += chunk;
});
// The whole response has been received. Print out the result.
resp.on('end', () => {
context.log("SUCCESS", JSON.parse(data));
context.done(null, data);
});
}).on("error", (err) => {
context.log("ERROR: " + err.message);
});
}
Another option would be to keep the function as async but you'd need to replace the callbacks with promise based calls. In some scenarios this can be achieved by wrapping them using util.promisify and then calling them with the await keyword
I have a file, and I want to read it line by line, and for every line extracted I perform some expensive analyzes and then save the results to the database. In short, I have something like this:
const fs = require('fs');
const path = require('path');
const readline = require('readline');
async function analyzeAndSave(url) {
// Removed for brevity, but this function takes a minute or so finsh.
}
async function run() {
try {
const dataPath = path.join(path.dirname(require.main.filename), 'data/urls.txt');
const rl = readline.createInterface({
input: fs.createReadStream(dataPath),
});
let line_no = 0;
rl.on('line', async (url) => {
line_no++;
logger.info(`Analyzing: ${url}`);
await analyzeAndSave(url);
});
} catch (err) {
// Error caught.
logger.error(err);
}
}
run();
The problem with this is that, I notice that it doesn't wait for the analyzes of one line to finish, it kind of tries to execute multiple of the analyzes instances. I can see this as initially it prints all the lines with logger.info('Analyzing: ' + url);`. So, it is not executed sequentially. How can I make sure that one line finishes before moving onto the next?
The readline interface is emitting the "on" events asynchronously and doing an await inside one of them doesn't stop other from being fired. Instead you can buffer up the lines in an array like this:
r.on('line', url => urls.push(url));
r.on('close', async () => {
for (const url of urls) {
await analyzeAndSave(url);
}
});
where urls is initialized to an empty array before the readline interface is created.
I think this is going to be helpful to you, exampled and mentioned here.
Nodejs - read line by line from file, perform async action for each line and reusme
Someone stated you can use a library for big files which is titled: line-by-line
#JavierFerrero stated a solution as such.
var LineByLineReader = require('line-by-line'),
lr = new LineByLineReader('big_file.txt');
lr.on('error', function (err) {
// 'err' contains error object
});
lr.on('line', function (line) {
// pause emitting of lines...
lr.pause();
// ...do your asynchronous line processing..
setTimeout(function () {
// ...and continue emitting lines.
lr.resume();
}, 100);
});
lr.on('end', function () {
// All lines are read, file is closed now.
});
You can also pass it ass a callback, waiting for the operation to
finish.
const fs = require('fs');
function run(path, cb) {
try {
fs.readFile(path, 'utf8', function(err, data){
if(err) throw err;
cb(data);
});
} catch (err) {
// Error caught.
}
}
run('./test.txt', (response) => {
// We are done, now continue
console.log(response)
})
I am reading a CSV using CSV-parser npm module and have to perform some operation on the data I get from the CSV (for each line).
const readstream = fs.createReadStream('src/working_file.csv');
const stream = readstream.pipe(parser());
stream.on('data', async data => {
// data is a JSON object of the row in CSV.
// Now i am calling another async function from user using the data in the JSON
console.log('before calling');
const writeToFile = await getImage(data.searchKey);
console.log('after calling');
// do other stuff
}
async function getImage(searchKey){
// im doing web scraping here using puppeeter
// it has some await calls too
console.log('in getimage');
const results = await scrapper.run().catch(err => {
console.error(err);
process.exit(1);
});
}
let say my csv has 2 rows then, my output is coming like below
before calling
in getimage
before calling
in getimage
after calling
after calling
but when I am doing this all callings are happening at a time though I used await. If I have 10 rows in the CSV all 10 rows calling the function is happening at the same time. but I want it to happen one by one. Only when the operation with the first row completes then I want the operate the second row.
my problem is all calls are happening at once rather than once by one.
Try this code.
var fs = require('fs');
var parse = require('csv-parse');
var async = require('async');
var inputFile='src/working_file.csv';
var parser = parse({delimiter: ','}, function (err, data) {
async.eachSeries(data, function (line, callback) {
// do something with the line
doSomething(line).then(function() {
// when processing finishes invoke the callback to move to the next one
callback();
});
})
});
fs.createReadStream(inputFile).pipe(parser);
You can also use fast-csv
The following code outputs the content of the index.html (it just contains the text hello world) to the browser. However, when I replace readFile() with readFileSync(), the request times out.
What am I missing? Is a different kind of buffer required? I am using node 0.61 and express 2.4.
var express = require('express');
var fs = require('fs');
var app = express.createServer(express.logger());
app.get('/', function(request, response) {
fs.readFile('index.html', function(err, data){
response.send(data.toString());
});
});
var port = process.env.PORT || 5000;
app.listen(port, function() {
console.log("Listening on " + port);
});
fs.readFile takes a call back which calls response.send as you have shown - good. If you simply replace that with fs.readFileSync, you need to be aware it does not take a callback so your callback which calls response.send will never get called and therefore the response will never end and it will timeout.
You need to show your readFileSync code if you're not simply replacing readFile with readFileSync.
Also, just so you're aware, you should never call readFileSync in a node express/webserver since it will tie up the single thread loop while I/O is performed. You want the node loop to process other requests until the I/O completes and your callback handling code can run.
'use strict'
var fs = require("fs");
/***
* implementation of readFileSync
*/
var data = fs.readFileSync('input.txt');
console.log(data.toString());
console.log("Program Ended");
/***
* implementation of readFile
*/
fs.readFile('input.txt', function (err, data) {
if (err) return console.error(err);
console.log(data.toString());
});
console.log("Program Ended");
For better understanding run the above code and compare the results..
readFileSync() is synchronous and blocks execution until finished. These return their results as return values.
readFile() are asynchronous and return immediately while they function in the background. You pass a callback function which gets called when they finish.
let's take an example for non-blocking.
following method read a file as a non-blocking way
var fs = require('fs');
fs.readFile(filename, "utf8", function(err, data) {
if (err) throw err;
console.log(data);
});
following is read a file as blocking or synchronous way.
var data = fs.readFileSync(filename);
LOL...If you don't want readFileSync() as blocking way then take
reference from the following code. (Native)
var fs = require('fs');
function readFileAsSync(){
new Promise((resolve, reject)=>{
fs.readFile(filename, "utf8", function(err, data) {
if (err) throw err;
resolve(data);
});
});
}
async function callRead(){
let data = await readFileAsSync();
console.log(data);
}
callRead();
it's mean behind scenes readFileSync() work same as above(promise) base.