We Keep Coding

folders and files are not visible after uploading file though multer

I am working on a small project. discussing Step by step
At first I am uploading zip files though multer
extracting those files (How can I call extract function after completing upload using multer?)
After extracting those I am trying to filter those files
after filtering those files I want to move some files to another directory
in my main index.js I have
A simple route to upload files which is working
// MAIN API ENDPOINT
app.post("/api/zip-upload", upload, async (req, res, next) => {
console.log("FIles - ", req.files);
});
Continuous checking for if there is any zip file that needs to unzip but the problem is after uploading it's not showing any files or dir
// UNZIP FILES
const dir = `${__dirname}/uploads`;
const files = fs.readdirSync("./uploads");
const filesUnzip = async () => {
try {
if (fs.existsSync(dir)) {
console.log("files - ", files);
for (const file of files) {
console.log("file - ", file);
try {
const extracted = await extract("./uploads/" + file, { dir: __dirname + "/uploads/" });
console.log("Extracted - ",extracted);
// const directories = await fs.statSync(dir + '/' + file).isDirectory();
} catch (bufErr) {
// console.log("------------");
console.log(bufErr.syscall);
}
};
// const directories = await files.filter(function (file) { return fs.statSync(dir + '/' + file).isDirectory(); });
// console.log(directories);
}
} catch (err) {
console.log(err);
}
return;
}
setInterval(() => {
filesUnzip();
}, 2000);
Moving files to static directory but here is the same problem no directory found
const getAllDirs = async () => {
// console.log(fs.existsSync(dir));
// FIND ALL DIRECTORIES
if (fs.existsSync(dir)) {
const directories = await files.filter(function (file) { return fs.statSync(dir + '/' + file).isDirectory(); });
console.log("Directories - ",directories);
if (directories.length > 0) {
for (let d of directories) {
const subdirFiles = fs.readdirSync("./uploads/" + d);
for (let s of subdirFiles) {
if (s.toString().match(/\.xml$/gm) || s.toString().match(/\.xml$/gm) !== null) {
console.log("-- ", d + "/" + s);
const move = await fs.rename("uploads/" + d + "/" + s, __dirname + "/static/" + s, (err) => { console.log(err) });
console.log("Move - ", move);
}
}
}
}
}
}
setInterval(getAllDirs, 3000);

There are so many issues with your code, I don't know where to begin:
Why are you using fs.xxxSync() methods if all your functions are async? Using xxxSync() methods is highly discouraged because it's blocking the server (ie parallel requests can't/won't be accepted while a sync reading is in progress). The fs module supports a promise api ...
Your "Continuous checking" for new files is always checking the same (probably empty) files array because it seems you are executing files = fs.readdirSync("./uploads"); only once (probably at server start, but I can't tell for sure because there isn't any context for that snippet)
You shouldn't be polling that "uploads" directory. Because as writing a file (if done properly) is an asynchronous process, you may end up reading incomplete files. Instead you should trigger the unzipping from your endpoint handler. Once it is hit, body.files contains the files that have been uploaded. So you can simply use this array to start any further processing instead of frequently polling a directory.
At some points you are using the callback version of the fs API (for instance fs.rename(). You cannot await a function that expects a callback. Again, use the promise api of fs.
EDIT
So I'm trying to address your issues. Maybe I can't solve all of them because of missing infomation, but you should get the general idea.
First of all, you shuld use the promise api of the fs module. And also for path manipulation, you should use the available path module, which will take care of some os specific issues.
const fs = require('fs').promises;
const path = require('path');
Your API endpoint isn't currently returning anything. I suppose you stripped away some code, but still. Furthermore, you should trigger your filehandling from here, so you don't have to do directory polling, which is
error prone,
wasting resources and
if you do it synchronously like you do blocks the server
app.post("/api/zip-upload", upload, async (req, res, next) => {
console.log("FIles - ", req.files);
//if you want to return the result only after the files have been
//processed use await
await handleFiles(req.files);
//if you want to return to the client immediately and process files
//skip the await
//handleFiles(req.files);
res.sendStatus(200);
});
Handling the files seems to consist of two different steps:
unzipping the uploaded zip files
copying some of the extracted files into another directory
const source = path.join(".", "uploads");
const target = path.join(__dirname, "uploads");
const statics = path.join(__dirname, "statics");
const handleFiles = async (files) => {
//a random folder, which will be unique for this upload request
const tmpfolder = path.join(target, `tmp_${new Date().getTime()}`);
//create this folder
await fs.mkdir(tmpfolder, {recursive: true});
//extract all uploaded files to the folder
//this will wait for a list of promises and resolve once all of them resolved,
await Promise.all(files.map(f => extract(path.join(source, f), { dir: tmpfolder })));
await copyFiles(tmpfolder);
//you probably should delete the uploaded zipfiles and the tempfolder
//after they have been handled
await Promise.all(files.map(f => fs.unlink(path.join(source, f))));
await fs.rmdir(tmpfolder, { recursive: true});
}
const copyFiles = async (tmpfolder) => {
//get all files/directory names in the tmpfolder
const allfiles = await fs.readdir(tmpfolder);
//get their stats
const stats = await Promise.all(allfiles.map(f => fs.stat(path.join(tmpfolder, f))));
//filter directories only
const dirs = allfiles.filter((_, i) => stats[i].isDirectory());
for (let d of dirs) {
//read all filenames in the subdirectory
const files = await fs.readdir(path.join(tmpfolder, d)));
//filter by extension .xml
const xml = files.filter(x => path.extname(x) === ".xml");
//move all xml files
await Promise.all(xml.map(f => fs.rename(path.join(tmpfolder, d, f), path.join(statics, f))));
}
}
That should do the trick. Of course you may notice there is no error handling with this code. You should add that.
And I'm not 100% sure about your paths. You should consider the following
./uploads refers to a directory uploads in the current working directory (whereever that may be)
${__dirname}/uploads refers to a directory uploads which is in the same directory as the script file currently executing Not sure if that is the directory you want ...
./uploads and ${__dirname}/uploads may point to the same folder or to completely different folders. No way knowing that without additional context.
Furthermore in your code you extract the ZIP files from ./uploads to ${__dirname}/uploads and then later try to copy XML files from ./uploads/xxx to ${__dirname}/statics, but there won't be any directory xxx in ./uploads because you extracted the ZIP file to a (probably) completely different folder.

How do I get access to the variables / methods in static files in Node.js + Expressjs at the server-side without using POST request?

More Specifically, I have a blob initialized and processed in a .js file in the 'public' (static folder) folder. Since that has been processed at the client-side, I want to know if there's a way I can somehow get access to the blob at the server-side without using a POST request. The file we talking about has been processed and is stored in a variable in a static file (script.js), Now I need to upload that variable/blob onto the Database. But, in a static file, I don't have access to the Database and can't even export the variable to the server. How do I get access to that variable which is within the static file? Someone, please edit if they have understood my requirement.
What my program does is that it records audio through the microphone of the client And that audio file has to be uploaded onto the Database. Now, I can add the functionality of 'Download' for the client and let the client download the file and then the client uses <input> tag to send a POST request to the server, But, now the client can upload any audio file into that input tag, Basically, this is a web app for live monitoring students writing exam, So that don't they don't cheat I capture their audio and save it to the DB. Please refer to my folder Structure for more details and then read the question again.
My Folder Structure:
--DBmodels
---AudioModels.js
---ImageModel.js
--node_modules
--public
---scipt.js (This contains the code for audio processing)
--views
---test.ejs (Main HTML page)
--index.js (server file)
--package.json
Here is a small diagram for reference :
Diagram
And here is my Folder Structure :
Folder Structure
One way of doing is this to download it on the client-side and then ask the client to upload but that doesn't work for me due to some reasons.
Here is my script.js, But I don't have access of the variables such as chunk_audio[] on the server.
const chunks_audio = [];
mediaRecorder.onstop = async function (e) {
console.log("DONE WITH AUDIO");
const blob = new Blob(chunks_audio, {
'type': "audio/ogg codecs=opus"
});
const audioURL = window.URL.createObjectURL(blob);
console.log(audioURL);
var link = document.getElementById("downloadaudio");
link.href = audioURL;
var audioMIMEtypes = ["audio/aac", "audio/mpeg", "audio/ogg", "audio/opus", "audio/wav"]
const audio = blob
const audiodb = new AudioSchema({
name : "Audio"+Date.now().toString()[5]
});
saveAudio(audiodb,audio)
try{
const new_audio = await audiodb.save();
console.log("AUDIO UPLOADED" + new_audio);
}catch (err){
console.log(err);
}
function saveAudio(audiodb, audioEncoded) {
if (audioEncoded == null) return;
console.log("before parse: " + audioEncoded);
const audio = JSON.parse(audioEncoded);
console.log("JSON parse: " + audio);
if (audio != null && audioMIMEtypes.includes(audio.type)) {
audiodb.audio = new Buffer.from(audio.data, "base64");
audiodb.audioType = audio.type;
}
}
// module.exports = chunks_audio; (This doesn't work for obvious reasons)
Here is my server file (index.js) , I tried to use POST request where the user posts the audio file after it gets downloaded, But the user could post any other file in the <input> tag, So that doesn't match with my requirement:
var audioMIMEtypes = ["audio/aac", "audio/mpeg", "audio/ogg", "audio/opus", "audio/wav"]
app.post('/', async ( req, res, next)=>{
const audio = blob // 'blob' is a variable in script.js , Hence don't have access here
const audiodb = new AudioSchema({
name : "Audio"+Date.now().toString()[5]
});
saveAudio(audiodb,audio)
try{
const new_audio = await audiodb.save();
console.log("AUDIO UPLOADED" + new_audio);
}catch (err){
console.log(err);
}
function saveAudio(audiodb, audioEncoded) {
if (audioEncoded == null) return;
console.log("before parse: " + audioEncoded);
const audio = JSON.parse(audioEncoded);
console.log("JSON parse: " + audio);
if (audio != null && audioMIMEtypes.includes(audio.type)) {
audiodb.audio = new Buffer.from(audio.data, "base64");
audiodb.audioType = audio.type;
}
}
})

The file I converted with FFMPEG in Firebase Cloud function is not accessable

The following is my cloud function code.
exports.increaseVolume = functions.storage.object().onFinalize(async (object) => {
const fileBucket = object.bucket; // The Storage bucket that contains the file.
const filePath = object.name; // File path in the bucket.
const contentType = object.contentType; // File content type.
// Exit if this is triggered on a file that is not an audio.
if (!contentType.startsWith('video/mp4')) {
console.log('This is not an audio.');
return null;
}
// Get the file name.
const fileName = path.basename(filePath);
// Exit if the audio is already converted.
if (fileName.endsWith('_output.mp4')) {
console.log('Already a converted audio.');
return null;
}
// Download file from bucket.
const bucket = gcs.bucket(fileBucket);
const tempFilePath = path.join(os.tmpdir(), fileName);
// We add a '_output.flac' suffix to target audio file name. That's where we'll upload the converted audio.
const targetTempFileName = fileName.replace(/\.[^/.]+$/, '') + '_output.mp4';
const targetTempFilePath = path.join(os.tmpdir(), targetTempFileName);
const targetStorageFilePath = path.join(path.dirname(filePath), targetTempFileName);
await bucket.file(filePath).download({destination: tempFilePath});
console.log('Audio downloaded locally to', tempFilePath);
// Convert the audio to mono channel using FFMPEG.
let command = ffmpeg(tempFilePath)
.audioFilters([
{
filter: 'volume',
options: '5dB'
},
{
filter: 'afftdn'
}
])
.format('mp4')
.output(targetTempFilePath);
await promisifyCommand(command);
console.log('Output audio created at', targetTempFilePath);
// Uploading the audio.
await bucket.upload(targetTempFilePath, {destination: targetStorageFilePath});
console.log('Output audio uploaded to', targetStorageFilePath);
// Once the audio has been uploaded delete the local file to free up disk space.
fs.unlinkSync(tempFilePath);
fs.unlinkSync(targetTempFilePath);
return console.log('Temporary files removed.', targetTempFilePath);
});
This is how the file in my storage bucket shows. Where do I get the download link or how can I access the file?. When I typed the link in the browser it returns a JSON saying 403 - unauthorized access.

You need to use the getSignedUrl() method as follows:
const uploadResp = await bucket.upload(targetTempFilePath, {destination: targetStorageFilePath});
const file = uploadResp[0];
const options = {
action: 'read',
expires: '03-17-2025'
};
const getSignedUrlResponse = await file.getSignedUrl(options);
const url = getSignedUrlResponse[0];
//Do wathever you want with this url: save it in Firestore for example

The message you are facing is about permissions. Who should have access to what?
First of all you have to think about your business logic. Who should have access to that file? Is it good if that file should be public? Is there a time-frame in which the file should be accessible? The data should remain in the Bucket or it can be deleted?
I think there are two options for you:
Create a Bucket with public data, in such case the data would accessible for everyone who has access to the specific filename in the Bucket.
If the above is not allowed, then you can create a SignedURL, as mentioned by #Renaud Tarnec link example. You should keep in mind that the SignedURL has time-limited access and everyone who has the URL of it, will be able to get the object. Once the time is expired, the object will not be longer accessible.
Once you have defined this, you can either delete the object in your Bucket programatically or you can set a Lifecycle Management. There, you can set configurations which contains a set of rules, for example, delete objects created by their age (in days).

copy selected file from an input on electron app form

I have this form
<form>
<input type="file" name="idp" id="idp" onchange="uploadFiles();"/>
</form>
When the user chooses a picture I need to copy it to a specified folder and store its full name into a variable to save it into the database.
I've tried this code:
const electron = require('electron');
const { dialog } = electron; // electron.remote; (if in renderer process)
const fs = require('fs'); // module that interacts with the file system
const path = require("path");
function uploadFiles(){
// opens a window to choose file
dialog.showOpenDialog({properties: ['openFile']}).then(result => {
// checks if window was closed
if (result.canceled) {
console.log("No file selected!")
} else {
// get first element in array which is path to file selected
const filePath = result.filePaths[0];
// get file name
const fileName = path.basename(filePath);
// path to app data + fileName = "C:\Users\John\AppData\Roaming\app_name\picture.png"
imgFolderPath = path.join(app.getPath('userData'), fileName);
// copy file from original location to app data folder
fs.copyFile(filePath, imgFolderPath, (err) => {
if (err) throw err;
console.log(fileName + ' uploaded.');
});
}
});
};
and it gives me this error message on the console:
Uncaught TypeError: Cannot read property 'showOpenDialog' of undefined
at uploadFiles
at HTMLInputElement.onchange

const { dialog } = electron; // electron.remote; (if in renderer process)
It appears that you are calling this from a renderer process. Therefore you need to use electron.remote in order to access the dialog module (as the code example you are using above indicates).
It's worth taking the time to read up on themain and renderer processes in Electron – they are fundamental concepts that you will encounter over and over and they are especially important in IPC messaging

why does this JavaScript function stop execution?

I have the following JS function which serves as a first prototype for a mozilla thunderbird extension.
The goal is to connect to a server and download a sample file, then unzipping it and storing the contents in the thunderbird profile folder.
Now this all works fine, except that the execution of the function stops after creating the zip file on the file system. So i have to restart the function again, in order to get the second part of the function executed which extracts the user.js file from the zip file.
Any ideas what the problem could be?
function downloadFile(httpLoc) {
// get profile directory
var file = Components.classes["#mozilla.org/file/directory_service;1"].
getService(Components.interfaces.nsIProperties).
get("ProfD", Components.interfaces.nsIFile);
var profilePath = file.path;
// change profile directory to native style
profilePath = profilePath.replace(/\\/gi , "\\\\");
profilePath = profilePath.toLowerCase();
// download the zip file
try {
//new obj_URI object
var obj_URI = Components.classes["#mozilla.org/network/io-service;1"].getService(Components.interfaces.nsIIOService).newURI(httpLoc, null, null);
//new file object
var obj_TargetFile = Components.classes["#mozilla.org/file/local;1"].createInstance(Components.interfaces.nsILocalFile);
//set to download the zip file into the profil direct
obj_TargetFile.initWithPath(profilePath + "\/" + "test.zip");
//if file the zip file doesn't exist, create it
if(!obj_TargetFile.exists()) {
alert("zip file wird erstellt");
obj_TargetFile.create(0x00,0644);
}
//new persitence object
var obj_Persist = Components.classes["#mozilla.org/embedding/browser/nsWebBrowserPersist;1"].createInstance(Components.interfaces.nsIWebBrowserPersist);
// with persist flags if desired ??
const nsIWBP = Components.interfaces.nsIWebBrowserPersist;
const flags = nsIWBP.PERSIST_FLAGS_REPLACE_EXISTING_FILES;
obj_Persist.persistFlags = flags | nsIWBP.PERSIST_FLAGS_FROM_CACHE;
//save file to target
obj_Persist.saveURI(obj_URI,null,null,null,null,obj_TargetFile);
} catch (e) {
alert(e);
} finally {
// unzip the user.js file to the profile direc
// creat a zipReader, open the zip file
var zipReader = Components.classes["#mozilla.org/libjar/zip-reader;1"]
.createInstance(Components.interfaces.nsIZipReader);
zipReader.open(obj_TargetFile);
//new file object, thats where the user.js will be extracted
var obj_UnzipTarget = Components.classes["#mozilla.org/file/local;1"].createInstance(Components.interfaces.nsILocalFile);
//set path for the user.js
obj_UnzipTarget.initWithPath(profilePath + "\/" + "user.js");
// if user.js doesn't exist, create it
if(!obj_UnzipTarget.exists()) {
alert("user.js wird erstellt");
obj_UnzipTarget.create(0x00,0644);
}
// extract the user.js out of the zip file, to the specified path
zipReader.extract("user.js", obj_UnzipTarget);
zipReader.close();
}
}
var hello = {
click: function() {
downloadFile("http://pse2.iam.unibe.ch/profiles/profile.zip");
},
};

saveURI is asynchronous, so you need to set the progress listener on the persist object to know when it has finished. Here's an example of setting a progress listener and later on there's a check to see whether the transfer has finished.