We Keep Coding

AWS S3 CreateReadStream in a loop only reads and writes 1 file

I am trying to retrieve multiple files from S3 using a readstream, and insert them into a single file locally.
Below, the 'output' variable is the single writestream I wish to append to using the downloaded S3 file data.
I am looping through days where the nextDay variable is used for the S3 key. The fileservice.s3Handler.getS3Obj returns an S3 object which allows a readstream for a single file and appending to the output file.
However, no other files are being read and are not showing the console either using the on('data', ()) method.
I tried to wrap the readstream in a promise to try to wait until the read was finished but it is running the same error.
More recently I keep get this error: "ERR_STREAM_WRITE_AFTER_END"
Not sure what is going wrong here.
async fetchCSV(req, res) {
const output = fs.createWriteStream(outputPathWithFile, {
'flags': 'a'});
let nextDay = startDate;
while (nextDay !== endDate) {
const s3path = path.join(`${req.params.stationId}`, `${nextDay}.csv`);
const file = await this.fileService.s3Handler.getS3Obj(s3path);
await this.completePipe(file, output);
nextDay = await getTomorrow(nextDay);
}
}
completePipe(file, output) {
return new Promise((resolve) => {
file.createReadStream().on('finish', () => {
resolve();
}).on('error', (err) => {
resolve();
}).on('data', (data) => {
console.log(data.toString());
}).pipe(output);
})
}
}
getS3Obj(file) {
return new Promise(async (resolve) => {
const getParams = {
Bucket: this.bucket,
Key: file
};
resolve(this.s3.getObject(getParams, (err) => {
if (err) {
console.log('Error in getS3 object')
}
}));
})
}
Please help me?

Solved it.
Did a couple things:
Added a tag to the pipe method.
stream.pipe(output, {end: false})
Instead of creating a new function for the promise I just put this code in instead:
await new Promise((resolve) => {
stream.once('finish', () => {
resolve();
});
});
But the tag was what made it work, the promise was just a tidy up.
Yay.

Firebase Google Cloud Function: createReadStream results in empty file

I try to process a Video file (stored in Google Firebase storage) through a Google Cloud Function. I have working code that download the entire video files into the NodeJS Google cloud function: await bucket.file(filePath).download({ destination: tempFile }).
But the goal is only to read the framerate, therefore the headers of the videofile would suffice. But createReadStream gives me an empty tempFile. Any advise much appreciated!
exports.checkFramerate = functions.region('europe-west1').storage.object().onFinalize(async (object, context) => {
const bucket = admin.storage().bucket(object.bucket); // Bucket class
const filePath = object.name; // videos/xbEXdMNFb1Blbd9r2E8m/comp_test.mp4
const fileName = filePath.split('/').pop(); // comp_test.mp4
const bucketDir = path.dirname(filePath); // videos/xbEXdMNFb1Blbd9r2E8m
const tempFile = path.join(os.tmpdir(), 'temp.mp4')
fs.closeSync(fs.openSync(tempFile, 'w'))
console.log("tempFile size1", fs.statSync(tempFile).size)
// await bucket.file(filePath).download({ destination: tempFile }); // this works: tempFile size2 = 3180152
await bucket.file(filePath).createReadStream({ // this does not work: tempFile size2 = 0
start: 10000,
end: 20000
})
.on('error', function(err) {console.log(err)})
.pipe(fs.createWriteStream(tempFile));
console.log("tempFile size2", fs.statSync(tempFile).size)
mi(tempFile).then(data => {
console.log("frameRate", data[0].general.frame_rate[0])
return data[0].general.frame_rate[0];
}).catch(e => {console.error(e)});
});
I tried implementing even the example of https://googleapis.dev/nodejs/storage/latest/File.html#createReadStream but to no avail. remoteFile.download works beautifully but remoteFile.createReadStream gives me empty files...
const remoteFile = bucket.file(filePath);
const localFilename = tempFile;
remoteFile.createReadStream()
.on('error', function(err) {})
.on('response', function(response) {})
.on('end', function() {})
.pipe(fs.createWriteStream(localFilename));
fs.stat(localFilename, (err, stats) => {
if (err) {console.log(err)}
return console.log("stats async",stats.size)
})

as mentioned, promise should be used
reading json file example
let buf = '';
const loadData = async () => {
return await new Promise((resolve, reject) => {
storage.bucket('bucket-name').file('test-config.json')
.createReadStream()
.on('error', reject)
.on('data', function(d) {
buf += d;
}).on('end', function() {
resolve(buf)
});
})
}
const data = await loadData()

Your problem is that the stream API isn't promisifed. So, the await does nothing, and your function continues before the stream is piped, and the file is still zero-length when you stat it the second time.
The download method works just fine because it returns a Promise.
This answer outlines the general approach you need to take. In summary though, you basically want the section of your code that does the piping to read like this:
const stream = bucket.file(filePath).createReadStream({
start: 10000,
end: 20000
})
.pipe(fs.createWriteStream(tempFile));
await new Promise((resolve, reject) => {
stream.on('finish', resolve);
stream.on('error', reject);
});
console.log("tempFile size2", fs.statSync(tempFile).size)
Your function will then wait until the finish event occurs when the piping is complete and the stream is closed. Obviously you probably want to do something more clever with the error handler too, but this is the general form of what you need.

Javascript how the better way to code nested callback?

I have 3 layer callbacks like this :
app.post('/', (req, res) => {
var filename = `outputs/${Date.now()}_output.json`;
let trainInput = req.files.trainInput;
let trainOutput = req.files.trainInput;
let testInput = req.files.trainInput;
//first
trainInput.mv(`inputs/${req.body.caseName}/train_input.csv`, function (err) {
if (err) return res.status(500).send(err);
//second
trainOutput.mv(`inputs/${req.body.caseName}/train_output.csv`, function (err) {
if (err) return res.status(500).send(err);
//third
testInput.mv(`inputs/${req.body.caseName}/test_input.csv`, function (err) {
if (err) return res.status(500).send(err);
res.send('success');
});
});
});
});
In this case, there are only 3 file uploads. In another case, I have more than 10 file uploads, and it makes 10 layer callbacks. I know it because of JavaScript asynchronous.
Is there any way, with this case, to make a beautiful code? This is because when it 10 layer callbacks, the code looks horizontally weird.
Thanks

You can use the following code to make you code look better and avoid callback hell
app.post('/', async (req, res) => {
var filename = `outputs/${Date.now()}_output.json`;
let trainInput = req.files.trainInput;
let trainOutput = req.files.trainInput;
let testInput = req.files.trainInput;
try {
var result1 = await trainInput.mv(`inputs/${req.body.caseName}/train_input.csv`);
var result2 = await trainInput.mv(`inputs/${req.body.caseName}/train_output.csv`);
var result2 = await testInput.mv(`inputs/${req.body.caseName}/test_input.csv`);
res.send('success');
}
catch (error) {
res.status(500).send(error);
}
});

You can make the functions return a Promise
I advice to make one function because you do the same thing 3 times. In this case I called the function 'save' but you can call it what ever you want. The first parameter is the file end the second the output filename.
function save(file, output) = return new Promise((resolve, reject) => {
file.mv(`inputs/${req.body.caseName}/${output}`, err =>
if (err) return reject(err)
resolve()
})
Promise.all([
save(req.files.trainInput, 'train_input.csv'),
save(req.files.trainInput, 'train_output.csv'),
save(req.files.trainInput, 'test_input.csv')
])
.then(_ => res.send(200))
.catch(err => res.send(400);

What version of Node you using? If async/await is available that cleans it up a bunch.
const moveCsv = (file, dest) => {
return new Promise((resolve, reject) => {
//third
file.mv(dest, function (err) {
if (err) reject(err);
resolve();
});
})
}
app.post('/', async(req, res) => {
try {
var filename = `outputs/${Date.now()}_output.json`;
const {
trainInput,
trainOutput,
testInput
} = req.files;
const prefix = `inputs/${req.body.caseName}`;
await moveCsv(trainInput, `${prefix}/train_input.csv`);
await moveCsv(trainOutput, `${prefix}/train_output.csv`);
await moveCsv(testInput, `${prefix}/test_input.csv`);
res.send('success');
} catch(err) {
res.status(500).send(err);
}
});
I'm also assuming here that your trainInput, trainOutput, testOutput weren't all meant to be req.files.trainInput.
Just be careful since the synchronous nature of the await calls are thread blocking. If that writer function takes ages you could also looking at putting those calls onto a worker thread. Won't really matter if your requests to that server endpoint are fast and non-frequent.

You can add RXJS to your project and use Observables.forkJoin()
Solution with Observables(assuming that trainInput.mv() returns Observable):
/* Without a selector */
var source = Rx.Observable.forkJoin(
trainInput.mv(`inputs/${req.body.caseName}/train_input.csv`),
trainInput.mv(`inputs/${req.body.caseName}/train_output.csv`),
trainInput.mv(`inputs/${req.body.caseName}/test_input.csv`)
);
var subscription = source.subscribe(
function (x) {
// On success callback
console.log('Success: %s', x);
},
function (err) {
// Error callback
console.log('Error');
},
function () {
// Completed - runs always
console.log('Completed');
});
// => Success: [result_1, result_2, result_3] or Error
// => Completed

Download txt file via FTP with Async & Promises in Node and AWS Lambda

I am using a single Node module basic-ftp to download a txt file in AWS Lambda and place it in the /tmp/ directory within the Lambda function.
I then want to work with the txt file and its contents outside of the FTP function.
I am using Async and Promises and have got a bit lost with the code. The current error returned in AWS Lambda is
module initialization error: ReferenceError
await finalData = (() => {
^^^^^^^^^^^^^^^
resulting from this line await finalData = (() => {
Can anyone help fix this and help me access the finalData outside of the FTP function?
var fs = require('fs');
var ftp = require("basic-ftp");
var path = require('path');
exports.handler = async (event, context, callback) => {
var fullPath = event.line_items[0].meta_data[2].value.tmp_name; // File path on Linux server -------
var myFileNameWithExtension = path.basename(fullPath); // Uploaded filename with the file extension eg. filename.txt
// FTP Function - Download from FTP and write to /tmp/ within AWS Lambda function
example()
//example().then(finalData=> callback(finalData))
async function example() {
var finalData = '';
const client = new ftp.Client()
client.ftp.verbose = true
try {
await client.access({
host: "XXXX",
user: "XXXX",
password: "XXXX",
})
let writeStream = fs.createWriteStream('/tmp/' + myFileNameWithExtension);
await client.download(writeStream, myFileNameWithExtension)
await finalData = (() => {
return new Promise((resolve, reject) => {
writeStream
.on('finish', () => {
fs.readFile("/tmp/" + myFileNameWithExtension, function (err, data) {
if (err) {
reject(err)
} else {
console.log('Contents of AWS Lambda /tmp/ directory', data);
resolve(data);
}
});
})
.on('error', (err) => {
console.log(err);
reject(err);
})
})
})();
}
catch (err) {
console.log(err)
}
client.close();
return finalData;
}
// Output contents of downloaded txt file into console and use in later code outside of the FTP function
console.log("Raw text:\n" + finalData.Body.toString('ascii'));
};

finalData is only defined within example which returns it, but you're not assigning this to anything. Combined with Luca Kiebel's comment, try adding
const finalData = await example();
then log that out.
Because finalData is defined within the function example, it is only available within that function and any function defined within that function.
You Don't Know JS explains this better than I can

Accessing the Node ReadStream

I have a piece of code which is wrapped in a promise. That piece of code reads an image form http, does various things, and at the end sends it to aws.s3.putObject. It looks like this (simplified):
Please note form is multiparty object.
form.on('part', (part) => {//form is multiparty
fileCount++;
let tmpFile = path.join(os.tmpDir(), `${userId}_${timePrefix}_${path.basename(part.filename)}`);
part.pipe(fs.createWriteStream(tmpFile));
part.on('end', () => {
resolve(fs.createReadStream(tmpFile));
});
part.once('error', (error) => {
handleError(error);
});
});
form.once('error', (error) => {
handleError(error);
});
form.parse(request);
}).then((imageStream) => {
//here is a call to AWS S3.putObject. Which return a promise
}).then(() => {
return new Ok();
});
In essence a stream is made on a created image and sent to AWS. I wanted to do some manipulation on binary level (read file signature, to check if it is an image). I got it to work like this:
form.on('part', (part) => {
fileCount++;
let tmpFile = path.join(os.tmpDir(), `${userId}_${timePrefix}_${path.basename(part.filename)}`);
part.pipe(fs.createWriteStream(tmpFile));
part.on('end', () => {
let chunk = [];
let file = fs.createReadStream(tmpFile);
let isFirstChunkSet = false;
file.on('data', function(chunks) {
if (!isFirstChunkSet){
chunk = chunks;
isFirstChunkSet = true;
}
});
file.on('close', function() {
let magicNumber = chunk.toString('hex', 0, 2);
if (imageProperties.allowed_file_types.includes(magicNumber)) {
resolve(fs.createReadStream(tmpFile));
} else {
error.message = 'wrong file type';
handleError(error);
}
});
});
part.once('error', (error) => {
handleError(error);
});
});
form.once('error', (error) => {
handleError(error);
});
form.parse(request);
}).then((imageStream) => {
//here is a call to AWS S3.putObject. Which return a promise
}).then(() => {
return new Ok();
});
Basically I attached two event listeners to the existing stream to access the data, and so some checking on the header file.
What bothers me is the feeling that I am overdoing things here. I would like to avoid those two listeners (data and close) and read a stream if possible.
To be more precise, this section of code receives the stream, and inside it I would like to access the data before sending it to AWS. This stream is ready, and simply put how do I read it, without using events?
form.parse(request);
}).then((imageStream) => {
//How can I access the imageStream here, without event listeners.
//Assumption is the stream is ready to be accessed.
//here is a call to AWS S3.putObject. Which return a promise
}).then(() => {
return new Ok();
});