We Keep Coding

Uploading images to AWS S3 bucket - specifying file path for randomly generated file names

I'm working on an automated testing project using PuppeteerJS in headless Chrome and trying to integrate existing screenshot functionality with AWS-SDK to upload images to an AWS S3 bucket on test failure.
The problem i'm having is the sub directories in a screenshots folder and the image file names are generated randomly in another file based on the current date and test environment, and run every time a test runs. The format of the generated directories/files is "screenshots/year/month/day/randomname.png".
The next step in the test is after the screenshots are generated, the folder containing the newly created images should be uploaded to AWS, and I've tried to achieve this using a glob to get every subdirectory and file with a png extension, like "screenshots/**/**/**/*.png", but i get a "no such file or directory" error". The folders/file names will be different everytime the tests run.
I've just started using AWS and I haven't been able to find a specific answer to my problem while researching.
import { PutObjectCommand } from "#aws-sdk/client-s3";
import { s3Client } from "../libs/s3Client.js";
import path from "path";
import fs from "fs";
const file = "../../screenshots/**/**/**/*.png";
const fileStream = fs.createReadStream(file);
// Set the parameters
export const uploadParams = {
Bucket: "bucket-name",
Key: path.basename(file),
// Add the required 'Body' parameter
Body: fileStream,
};
// Upload file to specified bucket.
export const run = async () => {
try {
const data = await s3Client.send(new PutObjectCommand(uploadParams));
console.log("Success", data);
return data; // For unit tests.
} catch (err) {
console.log("Error", err);
}
};
run();

Worked this out with the help of Jarmod. I needed to use the nodeJS:fs module to get the file paths recursively and returns a string which can be passed into the AWS fileStream variable for it to be uploaded to AWS. Jarmod shared the webmound article and i found the coder rocket fuel article hepful also.
https://www.webmound.com/nodejs-get-files-in-directories-recursively/
https://coderrocketfuel.com/article/recursively-list-all-the-files-in-a-directory-using-node-js

How do I write image to a specific directory with fs module in nodejs?

This is my current code:
await createCollage(pfparray, collageWidth).then((imageBuffer) => {
pfpcollage = new MessageAttachment(imageBuffer, 'collage.png');
fs.writeFile(`./images/${params.msg.author.id}.png`, imageBuffer, () => console.log('image downloaded'));
}).catch(e => params.msg.channel.send({embed: {title: "Could not generate collage",description: e + " | error!"}}));
}
So when run, its supposed to save the image file in directory "images" with the author's id as name. However it console logs but does not actually save anything in the directory. This happens when I try saving to any specific directory. When i just try to save "${params.msg.author.id}.png" without any directory it saves it in the same directory correctly. What could be the issue?

As #h-sifat showed me, it seems the path was not being resolved. I imported the path module (Typescript [TS1259] hit an error here), I had to do
import * as path from 'path'
After, I updated my code to this:
fs.writeFile(path.resolve(__dirname,`./images/${params.msg.author.id}.png`), imageBuffer, () => console.log('image downloaded'));
This seemed to do the trick!

Is it possible to append code into JavaScript files using File System in Node.JS?

I am creating a generator that should insert certain lines of code into a .JS file based on the users choices. In my experience with File System for Node.JS I have only ever written to .txt or .json files.
How would I go about appending this code into the file? Is this possible? Should I use specific encoding?
UPDATE
I did some lightweight experimentation by setting up an application like so:
-app.js
-index.js
-snippet.js
-index.html
Inside app.js I have the following code.
const fs = require("fs");
fs.readFile("snippet.js", function read(err, data) {
if (err) {
throw err;
}
var file_content = data.toString();
fs.writeFile("index.js", file_content, (err)=>{
if (err) throw err;
console.log("File Written");
});
});
Then I am taking a simple alert("File System Insert") from snippet.js and writing it to index.js and once index.html is loaded the alert functions as expected. So yes, it is possible to pass javascript code using File System.
I guess my next step would be to figure out how to template it without the use of template engine file extensions.

You can use a templating engine for this purpose.
For example, take a look at Mustache.
You can have a template like this (file.template):
let greetings = 'hello world!'
{{ optionalCode }}
console.log(greetings)
Then use Mustache and compile the template, populate the {{ optionalCode }} with your desired JS code and get the final JS file.
Read the Mustache's documentations for Node.js and JavaScript.
There are other templating libraries as well. Check the following link for more:
https://www.sitepoint.com/overview-javascript-templating-engines/

how to remove templateCache with gulp?

i use angular-template-cache.
follow code exist for remove template cache in app module but i need to remove all templateCache with gulp on dev machine.
myApp.run(function($rootScope, $templateCache) {
$rootScope.$on('$viewContentLoaded', function() {
$templateCache.removeAll();
});
});

The best way to avoid template caching is revisioning your files.
Since you are using gulp, you can revision your files using gulp-rev or gulp-rev-all.
What is revisioning?
Static asset revisioning by appending content hash to filenames unicorn.css → unicorn-d41d8cd98f.css.
i.e., On every builds the filename changes and that way avoiding template caching.
You can revision every file including .html, .css, .js, images, videos etc.
Since gulp-rev-all is the latest and forked from gulp-rev, let's talk about gulp-rev-all only.
Revisioning using gulp-rev-all:
var revAll = require('gulp-rev-all');
if you want to neglect some files from revisioning, you can do that like this.
var rev = new revAll({dontRenameFile: [/^\/favicon.ico$/g, /^\/index.html/g]})
Consider all your files are in the folder dist and save the new revisioned files in the folder www.(You can save them in dist also. Considering www is your build directory.)
return gulp.src('dist/**')
.pipe(rev.revision())
.pipe(gulp.dest('www'))
Next, create a manifest file to map your files with the revisioned one. for that use .manifestFile() function. which returns a transform function that will filter out any existing files going through the pipe and will emit a new manifest file. Must be called after .revision().
.pipe(rev.manifest())
.pipe(gulp.dest('www/manifest'));
An asset manifest, mapping the original paths to the revisioned paths, will be written to www/manifest/rev-manifest.json:
{
"css/unicorn.css": "css/unicorn.098f6bcd.css",
"js/unicorn.js": "js/unicorn.273c2cin.js"
.....
.....
}
Complete code:
gulp.task('rev', () => {
var revAll = require('gulp-rev-all'),
rev = new revAll({dontRenameFile: [/^\/favicon.ico$/g, /^\/index.html/g]});
return gulp.src('dist/**')
.pipe(rev.revision())
.pipe(gulp.dest('www'))
.pipe(rev.manifest())
.pipe(gulp.dest('www/manifest'));
});
Read more about gulp-rev-all here

In Node.js, reading a directory of .html files and searching for element attributes inside them?

I can't even begin to think about how this would be done. Basically, imagine a folder and it has a static website in it. It has all the images, styles and html files etc. With my Node application, I want to look inside this folder, get just the .html files only and then pick just the .html files that have the attribute 'data-template="home"' inside them.
I know this seems a little odd maybe, but it's for a project that requires the user to upload their static website files and then my Node app does things to them files.
Anyhow, was just curious about iterating over certain filetypes and then looking inside them... Any help with approaching this would really help me.
Many thanks, James

This piece of code will scan for all files in a directory, then read the contents of .html files and then look for a string data-template="home" in them.
var fs = require('fs');
fs.readdir('/path/to/html/files', function(err, files) {
files
.filter(function(file) { return file.substr(-5) === '.html'; })
.forEach(function(file) { fs.readFile(file, 'utf-8', function(err, contents) { inspectFile(contents); }); });
});
function inspectFile(contents) {
if (contents.indexOf('data-template="home"') != -1) {
// do something
}
}
If you need more flexibility, you could also use the cheerio module to look for an element in the html file with that attribute:
var cheerio = require('cheerio');
function inspectFile(contents) {
var $ = cheerio.load(contents);
if ($('html[data-template="home"]').length) {
// do something
}
}

Take a look at the nodejs filesystem module
http://nodejs.org/docs/v0.5.3/api/fs.html
You could use fs.readdir() to get the names of all the files, then read the .html ones to find 'data-template=home'.