I am new to Typescript and node.
I have this function
sftp.connect(config) //CONNECT TO STFP
.then(() => {
sftp.list(remoteFilePath) //LIST THE FILES IN THE FILEPATH
.then((list) => {
list.forEach((index) => { //FOR EVERY FILE IN THE FOLDER, DOWNLOAD IT
const fileName = remoteFilePath + index.name;
console.log(fileName);
sftp.fastGet(fileName, "/Users/Bob/" + index.name)
.then((value) => {
console.log(value);
sftp.end();
})
})
})
})
// .then(() => {
// // sftp.end();
// })
.catch(err => {
console.error(err.message);
});
and using the ssh2-sftp-client library. My question is that is it possible for this library to get the contents of the file as opposed to downloading it? I plan on making this function into a lambda function.
At the moment, the variable value contains a text telling me that the file has been downloaded to my designated path.
If you want to get the contents of the file you can read it using the fs module after downloading it
// using the ES6 module syntax
import { readFileSync } from "fs"
const data = readFileSync("./file.txt")
If you want to get the contents of the file without downloading them to the disk you have to pass a different destination. Use the ssh2-sftp-client get method instead, it accepts a Stream or a Buffer as the destination. You can use a Stream but you have to pipe it somewhere. Here's an example using process.stdout which is a writable stream:
// ...
stfp.get(
fileName, "/Users/Bob/" + index.name,
process.stdout
)
Related
I am trying to upload file to s3, before that I am altering the name of the file. Now I am accepting 2 files from request form-data object, renaming the filename, and uploading the file to s3. And end of the task I need to return the renamed file list which is uploaded successfully.
I am using S3.upload() function. But the problem is, the variable which is assigned as empty array initially, that will contain the renamed file list. But the array is returning empty response. The s3.upload() is taking much time. is there any probable solution where I can store the file name if upload is successful and return those names in response.
Please help me to fix this. The code looks like this,
if (formObject.files.document && formObject.files.document.length > 0) {
const circleCode = formObject.fields.circleCode[0];
let collectedKeysFromAwsResponse = [];
formObject.files.document.forEach(e => {
const extractFileExtension = ".pdf";
if (_.has(FILE_EXTENSIONS_INCLUDED, _.lowerCase(extractFileExtension))) {
console.log(e);
//change the filename
const originalFileNameCleaned = "cleaning name logic";
const _id = mongoose.Types.ObjectId();
const s3FileName = "s3-filename-convension;
console.log(e.path, "", s3FileName);
const awsResponse = new File().uploadFileOnS3(e.path, s3FileName);
if(e.hasOwnProperty('ETag')) {
collectedKeysFromAwsResponse.push(awsResponse.key.split("/")[1])
}
}
});
};
use await s3.upload(params).promise(); is the solution.
Use the latest code - which is AWS SDK for JavaScript V3. Here is the code you should be using
// Import required AWS SDK clients and commands for Node.js.
import { PutObjectCommand } from "#aws-sdk/client-s3";
import { s3Client } from "./libs/s3Client.js"; // Helper function that creates Amazon S3 service client module.
import {path} from "path";
import {fs} from "fs";
const file = "OBJECT_PATH_AND_NAME"; // Path to and name of object. For example '../myFiles/index.js'.
const fileStream = fs.createReadStream(file);
// Set the parameters
export const uploadParams = {
Bucket: "BUCKET_NAME",
// Add the required 'Key' parameter using the 'path' module.
Key: path.basename(file),
// Add the required 'Body' parameter
Body: fileStream,
};
// Upload file to specified bucket.
export const run = async () => {
try {
const data = await s3Client.send(new PutObjectCommand(uploadParams));
console.log("Success", data);
return data; // For unit tests.
} catch (err) {
console.log("Error", err);
}
};
run();
More details can be found in the AWS JavaScript V3 DEV Guide.
I am trying to make an image organizer app , which searches images using tag's ,
So I want the user to select the image they want, so far I have done this by the following code
// renderer process
$("#uploadImage).on("click", (e) => {
ipcRenderer.send('dialoguploadImage')
});
this is the main process
ipcMain.on('dialoguploadImage', (e) => {
dialog.showOpenDialog({
properties: ['openFile']
}).then(result => {
sendBackimagePathFromMain(result.filePaths[0])
}).
catch(err => {
console.log(err)
})
});
function sendBackimagePathFromMain(result) {
mainWindow.webContents.send('imagePathFromMain',result)
}
so I have the image path, and the only thing I want to know is
how can I duplicate this image, rename it, cerate a new folder and save the image in that folder
like for example to this folder
('./currentDirectory/imageBackup/dognothapppy.jpg')
You can use fs.mkdirSync() to make the folder and fs.copyFileSync() to 'duplicate and rename' the file (in a file system, you don't need to duplicate and rename a file in two different steps, you do both at once, which is copying a file), or their async functions.
const { mkdirSync, copyFileSync } = require('fs')
const { join } = require('path')
const folderToCreate = 'folder'
const fileToCopy = 'selectedFile.txt'
const newFileName = 'newFile.txt'
const dest = join(folderToCreate, newFileName)
mkdirSync(folderToCreate)
copyFileSync(fileToCopy, dest)
I was trying to write to a file using Deno.writeFile
await Deno.writeFile('./file.txt', 'some content')
But got the following cryptic error:
error: Uncaught TypeError: arr.subarray is not a function
at Object.writeAll ($deno$/buffer.ts:212:35)
at Object.writeFile ($deno$/write_file.ts:70:9)
What's the right way to write files in Deno?
There are multiple ways to write a file in Deno, all of them require --allow-write flag and will throw if an error occurred, so you should handle errors correctly.
Using Deno.writeFile
This API takes a Uint8Array, not a string, the reason why you get that error. It also takes an optional WriteFileOptions object
const res = await fetch('http://example.com/image.png');
const imageBytes = new Uint8Array(await res.arrayBuffer());
await Deno.writeFile('./image.png', imageBytes);
There's also the synchronous API (it blocks the event loop as it does in Node.js).
Deno.writeFileSync('./image.png', imageBytes);
Writing strings
The easiest way is to use Deno.writeTextFile
await Deno.writeTextFile('./file.txt', 'some content');
You can also use Deno.writeFile with TextEncoder.
const encoder = new TextEncoder(); // to convert a string to Uint8Array
await Deno.writeFile('./file.txt', encoder.encode('some content'));
Streaming
Deno.open returns a FsFile which contains a WritableStream in .writable property, so you can just pipe a stream directly to it.
const res = await fetch('https://example.com/csv');
const file = await Deno.open('./some.csv', { create: true, write: true })
await res.body.pipeTo(file.writable);
file.close();
If you have a Reader instead of a ReadableStream you can convert it to a ReadableStream using readableStreamFromReader from std/streams:
import { readableStreamFromReader } from "https://deno.land/std#0.156.0/streams/mod.ts?s=readableStreamFromReader";
// ...
const readable = readableStreamFromReader(someReader);
await readable.pipeTo(file.writeable)
Low-level APIs
Using Deno.open and Deno.writeAll (or Deno.writeAllSync)
const file = await Deno.open('./image.png', { write: true, create: true });
/* ... */
await Deno.writeAll(file, imageBytes);
file.close(); // You need to close it!
See OpenOptions here. If you want to append you would do:
{ append: true }
And you can also use even lower-level APIs such as Deno.write or Writer.write
You can use ensureDir to safely write files to possibly non-existent directories:
import { ensureDir } from "https://deno.land/std#0.54.0/fs/ensure_dir.ts";
ensureDir("./my/dir")
.then(() => Deno.writeTextFile("./my/dir/file.txt", "some content"));
The containing file directory can be derived via dirname:
import { dirname } from "https://deno.land/std#0.54.0/path/mod.ts";
const file = "./my/dir/file.txt";
ensureDir(dirname(file)).then(() => Deno.writeTextFile(file, "some content"));
An alternative is ensureFile to assert file existence:
import { ensureFile } from "https:deno.land/std/fs/ensure_file.ts";
ensureFile(file).then(/* your file write method */)
This variant is slightly less verbose, with the cost of one additional write operation (file creation, if not exists).
I have been trying to upload a file to Firebase storage using a callable firebase cloud function.
All i am doing is fetching an image from an URL using axios and trying to upload to storage.
The problem i am facing is, I don't know how to save the response from axios and upload it to storage.
First , how to save the received file in the temp directory that os.tmpdir() creates.
Then how to upload it into storage.
Here i am receiving the data as arraybuffer and then converting it to Blob and trying to upload it.
Here is my code. I have been missing a major part i think.
If there is a better way, please recommend me. Ive been looking through a lot of documentation, and landed up with no clear solution. Please guide. Thanks in advance.
const bucket = admin.storage().bucket();
const path = require('path');
const os = require('os');
const fs = require('fs');
module.exports = functions.https.onCall((data, context) => {
try {
return new Promise((resolve, reject) => {
const {
imageFiles,
companyPIN,
projectId
} = data;
const filename = imageFiles[0].replace(/^.*[\\\/]/, '');
const filePath = `ProjectPlans/${companyPIN}/${projectId}/images/${filename}`; // Path i am trying to upload in FIrebase storage
const tempFilePath = path.join(os.tmpdir(), filename);
const metadata = {
contentType: 'application/image'
};
axios
.get(imageFiles[0], { // URL for the image
responseType: 'arraybuffer',
headers: {
accept: 'application/image'
}
})
.then(response => {
console.log(response);
const blobObj = new Blob([response.data], {
type: 'application/image'
});
return blobObj;
})
.then(async blobObj => {
return bucket.upload(blobObj, {
destination: tempFilePath // Here i am wrong.. How to set the path of downloaded blob file
});
}).then(buffer => {
resolve({ result: 'success' });
})
.catch(ex => {
console.error(ex);
});
});
} catch (error) {
// unknown: 500 Internal Server Error
throw new functions.https.HttpsError('unknown', 'Unknown error occurred. Contact the administrator.');
}
});
I'd take a slightly different approach and avoid using the local filesystem at all, since its just tmpfs and will cost you memory that your function is using anyway to hold the buffer/blob, so its simpler to just avoid it and write directly from that buffer to GCS using the save method on the GCS file object.
Here's an example. I've simplified out a lot of your setup, and I am using an http function instead of a callable. Likewise, I'm using a public stackoverflow image and not your original urls. In any case, you should be able to use this template to modify back to what you need (e.g. change the prototype and remove the http response and replace it with the return value you need):
const functions = require('firebase-functions');
const axios = require('axios');
const admin = require('firebase-admin');
admin.initializeApp();
exports.doIt = functions.https.onRequest((request, response) => {
const bucket = admin.storage().bucket();
const IMAGE_URL = 'https://cdn.sstatic.net/Sites/stackoverflow/company/img/logos/so/so-logo.svg';
const MIME_TYPE = 'image/svg+xml';
return axios.get(IMAGE_URL, { // URL for the image
responseType: 'arraybuffer',
headers: {
accept: MIME_TYPE
}
}).then(response => {
console.log(response); // only to show we got the data for debugging
const destinationFile = bucket.file('my-stackoverflow-logo.svg');
return destinationFile.save(response.data).then(() => { // note: defaults to resumable upload
return destinationFile.setMetadata({ contentType: MIME_TYPE });
});
}).then(() => { response.send('ok'); })
.catch((err) => { console.log(err); })
});
As a commenter noted, in the above example the axios request itself makes an external network access, and you will need to be on the Blaze or Flame plan for that. However, that alone doesn't appear to be your current problem.
Likewise, this also defaults to using a resumable upload, which the documentation does not recommend when you are doing large numbers of small (<10MB files) as there is some overhead.
You asked how this might be used to download multiple files. Here is one approach. First, lets assume you have a function that returns a promise that downloads a single file given its filename (I've abridged this from the above but its basically identical except for the change of INPUT_URL to filename -- note that it does not return a final result such as response.send(), and there's sort of an implicit assumption all the files are the same MIME_TYPE):
function downloadOneFile(filename) {
const bucket = admin.storage().bucket();
const MIME_TYPE = 'image/svg+xml';
return axios.get(filename, ...)
.then(response => {
const destinationFile = ...
});
}
Then, you just need to iteratively build a promise chain from the list of files. Lets say they are in imageUrls. Once built, return the entire chain:
let finalPromise = Promise.resolve();
imageUrls.forEach((item) => { finalPromise = finalPromise.then(() => downloadOneFile(item)); });
// if needed, add a final .then() section for the actual function result
return finalPromise.catch((err) => { console.log(err) });
Note that you could also build an array of the promises and pass them to Promise.all() -- that would likely be faster as you would get some parallelism, but I wouldn't recommend that unless you are very sure all of the data will fit inside the memory of your function at once. Even with this approach, you need to make sure the downloads can all complete within your function's timeout.
I am trying to upload a file from mobile to google bucket using ionic 4. Although a file can upload into the could. I am struggling to get the file properties out of file object.
Here is my method,
async selectAFile() {
const uploadFileDetails = {
name: '',
contentLength: '',
size: '',
type: '',
path: '',
};
this.fileChooser.open().then(uri => {
this.file.resolveLocalFilesystemUrl(uri).then(newUrl => {
let dirPath = newUrl.nativeURL;
const dirPathSegments = dirPath.split('/');
dirPathSegments.pop();
dirPath = dirPathSegments.join('/');
(<any>window).resolveLocalFileSystemURL(
newUrl.nativeURL,
function(fileEntry) {
uploadFileDetails.path = newUrl.nativeURL;
const file: any = getFileFromFileEntry(fileEntry);
//log 01
console.log({ file });
uploadFileDetails.size = file.size;
uploadFileDetails.name = `${newUrl.name
.split(':')
.pop()}.${file.type.split('/').pop()}`;
uploadFileDetails.type = file.type;
async function getFileFromFileEntry(fileEntry) {
try {
return await new Promise((resolve, reject) =>
fileEntry.file(resolve, reject)
);
} catch (err) {
console.log(err);
}
}
},
function(e) {
console.error(e);
}
);
});
});
// here uploadFileDetails is simller to what I declared at the top ;)
// I wan't this to be populated with file properties
// console.log(uploadFileDetails.name) --> //''
const uploadUrl = await this.getUploadUrl(uploadFileDetails);
const response: any = this.uploadFile(
uploadFileDetails,
uploadUrl
);
response
.then(function(success) {
console.log({ success });
this.presentToast('File uploaded successfully.');
this.loadFiles();
})
.catch(function(error) {
console.log({ error });
});
}
even though I can console.log the file in log 01. I am unable to get file properties like, size, name, type out of the resolveLocalFileSystemURL function. basically, I am unable to populate uploadFileDetails object. What am I doing wrong? Thank you in advance.
you actually need 4 Ionic Cordova plugins to upload a file after getting all the metadata of a file.
FileChooser
Opens the file picker on Android for the user to select a file, returns a file URI.
FilePath
This plugin allows you to resolve the native filesystem path for Android content URIs and is based on code in the aFileChooser library.
File
This plugin implements a File API allowing read/write access to files residing on the device.
File Trnafer
This plugin allows you to upload and download files.
getting the file's metadata.
file.resolveLocalFilesystemUrl with fileEntry.file give you all the metadata you need, except the file name. There is a property called name in the metadata but it always contains value content.
To get the human readable file name you need filePath. But remember you can't use returning file path to retrieve metadata. For that, you need the original url from fileChooser.
filePathUrl.substring(filePathUrl.lastIndexOf('/') + 1) is used to get only file name from filePath.
You need nativeURL of the file in order to upload it. Using file path returning from filePath is not going to work.
getFileInfo(): Promise<any> {
return this.fileChooser.open().then(fileURI => {
return this.filePath.resolveNativePath(fileURI).then(filePathUrl => {
return this.file
.resolveLocalFilesystemUrl(fileURI)
.then((fileEntry: any) => {
return new Promise((resolve, reject) => {
fileEntry.file(
meta =>
resolve({
nativeURL: fileEntry.nativeURL,
fileNameFromPath: filePathUrl.substring(filePathUrl.lastIndexOf('/') + 1),
...meta,
}),
error => reject(error)
);
});
});
});
});
}
select a file from the file system of the mobile.
async selectAFile() {
this.getFileInfo()
.then(async fileMeta => {
//get the upload
const uploadUrl = await this.getUploadUrl(fileMeta);
const response: Promise < any > = this.uploadFile(
fileMeta,
uploadUrl
);
response
.then(function(success) {
//upload success message
})
.catch(function(error) {
//upload error message
});
})
.catch(error => {
//something wrong with getting file infomation
});
}
uploading selected file.
This depends on your backend implementation. This is how to use File Transfer to upload a file.
uploadFile(fileMeta, uploadUrl) {
const options: FileUploadOptions = {
fileKey: 'file',
fileName: fileMeta.fileNameFromPath,
headers: {
'Content-Length': fileMeta.size,
'Content-Type': fileMeta.type,
},
httpMethod: 'PUT',
mimeType: fileMeta.type,
};
const fileTransfer: FileTransferObject = this.transfer.create();
return fileTransfer.upload(file.path, uploadUrl, options);
}
hope it helps. :)