I am retriving a file from my bucket.
I get the file and want to read it's contents but I do not want to download it to my local project
i just want to read the contents, take the data and do other operations with it.
my code:
export const fileManager = async () => {
try {
const source = 'upload/';
const options = { prefix: source, delimiter: '/' };
const remoteFile = st.bucket(bName).file('myData.csv');
const readFileData;
remoteFile
.createReadStream()
.on('error', err => {
console.log('error');
})
.on('response', response => {
readFileData = response;
console.log('success');
// Server connected and responded with the specified status and headers.
})
.on('end', () => {
console.log('end');
// The file is fully downloaded.
});
console.log("data", readFileData)
} catch (error) {
console.log('Error Is', error);
}
};
readFileData is undefined.
Is this possible? Every example I find envolves me downloading the file.
createReadStream is asynchronous and returns immediately. You have to use the callbacks to find out when the download to memory is complete. Right now, your code is always going to print "data undefined" because it's trying to print the response before it's available.
createReadStream is definitely the right way to go, but you will have to understand how node streams work in order to process the results correctly. There is a whole section in the linked documentation for reading streams, which is what you want to do here. The way you deal with the stream is not specific to Cloud Storage - it's the same for all node streams.
You might be helped by the answers to these questions that deal with reading node streams:
Node.js: How to read a stream into a buffer?
Convert stream into buffer?
How do I read the contents of a Node.js stream into a string variable?
Related
I use XMLHttpRequest in Node.js and when handling request in server.js (asynchronous POST request) I use this code:
const filePath = './input.pdf';
fsPromises.readFile(filePath)
.then(res => {
const pdfBytes = new Uint8Array(res).buffer;
})
.catch(err => {
console.log('Could not read file-template');
});
then I get ArrayBuffer through this code. But I want to move this code into module and call export function there with this code. This change disrupted reading file and devtools network returns error in my request with message 'Provisional headers are shown'.
How can I make it work in module, not in server.js?
I attached images with messages in devtools img1, img2
I Googled this but couldn't find an answer but it must be a common problem. This is the same question as Node request (read image stream - pipe back to response), which is unanswered.
How do I send an image file as an Express .send() response? I need to map RESTful urls to images - but how do I send the binary file with the right headers? E.g.,
<img src='/report/378334e22/e33423222' />
Calls...
app.get('/report/:chart_id/:user_id', function (req, res) {
//authenticate user_id, get chart_id obfuscated url
//send image binary with correct headers
});
There is an api in Express.
res.sendFile
app.get('/report/:chart_id/:user_id', function (req, res) {
// res.sendFile(filepath);
});
http://expressjs.com/en/api.html#res.sendFile
a proper solution with streams and error handling is below:
const fs = require('fs')
const stream = require('stream')
app.get('/report/:chart_id/:user_id',(req, res) => {
const r = fs.createReadStream('path to file') // or any other way to get a readable stream
const ps = new stream.PassThrough() // <---- this makes a trick with stream error handling
stream.pipeline(
r,
ps, // <---- this makes a trick with stream error handling
(err) => {
if (err) {
console.log(err) // No such file or any other kind of error
return res.sendStatus(400);
}
})
ps.pipe(res) // <---- this makes a trick with stream error handling
})
with Node older then 10 you will need to use pump instead of pipeline.
I have written a simple function to handle upload of files in my sails.js app.
let upload = file.upload((err, uploadedFiles) => {
if (err) {
return res.serverError(err);
} else {
return res.send({ data: uploadedFiles });
}
});
When the upload is complete I am redirected to a page displaying raw json, which contains the uploaded file information (including the path).
raw json response
What I am expecting when I console.log(upload) is the same information, however I am getting the writestream instead.
console.log output
This is a problem for me because I would like to be able to extract the file name from the object and use it in another part of my program, but I can't do this because all I am able to access is the writestream.
I have tried using async/await and callbacks and can't seem to fix my issue.
Hopefully someone can help me!
Thanks
A helpful person on the sails Gitter suggested that I use this package, which supports async/await: https://www.npmjs.com/package/sails-hook-uploads
I tested it out with the following code and it works:
let upload = await sails
.uploadOne(file, {
maxBytes: 3000000,
})
.intercept('E_EXCEEDS_UPLOAD_LIMIT', 'tooBig')
.intercept(
(err) => new Error('The photo upload failed: ' + util.inspect(err))
);
My task is to download json-file from website (pubchem) using only the query string (h2o for example) and JS. I know it's possible to do with parsing, but this is too much code because of number of pages i need to parse for getting destination. Is there any other options to solve the problem?
Using google didnt give me any of idea ):
You will still need to do some parsing if you really want to automate this, since only using a query parameter will get you to the main page that lists the 'articles' and you need to go in to find the URL that will give you the JSON format. But! I think you can "reverse engineer" it since the URLS for the article and its JSON format are very similar.
I checked out the website and tried to download one of the files that they have for https://pubchem.ncbi.nlm.nih.gov/compound/3076959 and it turns out to get the JSON representation this was the URL https://pubchem.ncbi.nlm.nih.gov/rest/pug_view/data/compound/748328/JSON/
As you can see they are very similar and you might be able to figure out how different topics such as compound for example construct the JSON output endpoint.
To download the JSON files using NodeJS is to use the node-fetch module or axios library to send your http requests to the JSON endpoint and from there you can save the response to a file on your machine.
Here is an example of how you can do this with axios and the NodeJS fs module in order to save the file to your machine.
const fs = require("fs");
const fetch = require("node-fetch");
async function downloadASJson(url, fileName) {
const response = await fetch(url);
const jsonContent = await response.buffer();
fs.writeFile(`${fileName}.json`, jsonContent, "utf8", function (err) {
if (err) {
console.log("An error occured while writing JSON Object to File.");
return console.log(err);
}
console.log("JSON file has been saved.");
});
}
try {
downloadASJson(
"https://pubchem.ncbi.nlm.nih.gov/rest/pug_view/data/compound/748328/JSON/",
"2-Methyl-3-(5'-bromobenzofuroyl-2')-4-dimethylaminomethyl-5-hydroxybenzofuran HCl H20"
);
} catch (err) {
console.log(error);
}
You save the following code in a file called app.js for example, and you can use node app.js to run it. Don't forget to install the dependencies.
I've tried all sorts to get this to work. I'm trying to request a PDF from an API on node, then send this back to the client who called it to begin with.
For the minute I just want to successfully save and view the PDF on the node server.
The issue is the PDF file is always empty when I open it (Even though it has a size of 30kb).
The basic flow is like this (removed a few bits, but the below code works and returns me the PDF fine)
// We pass through session ID's, request dates etc through in body
app.post("/getPayslipURL", function(client_request, res) {
// create request, which will simply pass on the data to the database (In order to get the NI number we need for the pay API)
const NI_NUMBER_REQUEST = db_api.createRequestTemplate({
body: JSON.stringify(client_request.body)
});
// Create a chain of HTTPS Requests, Starting with our call to the DB
requestPromise(NI_NUMBER_REQUEST)
.then((db_response) => {
const PAY_API_OPTIONS = /*Code to generate options based on furhter DB info (Includes dates etc)*/
return requestPromise(PAY_API_OPTIONS); // Call pay API
})
.then((pay_pdf_data) => {
console.log(typeof pay_pdf_data); // It's a string
// At this point I can log pay_pdf_data, But if I try to save it to file it's always empty
// No matter how I encode it etc
fs.writeFile("./test.pdf", pay_pdf_data, 'binary', function(err) {
if(err) {
return console.log(err);
}
console.log("The file was saved!");
});
})
.catch(err => `Error caught: ${console.log}`) // Catch any errors on our request chain
});
}
I've tried saving with / without the binary flag as suggested in other posts in both the file save aswell as within the requests itself. Also various types of decoding methods have been tried, I always get an empty PDF saved.
My return data looks like this (is much bigger, when saved as test.pdf I get a 30kb file as before mentioned)
%PDF-1.4
%����
1 0 obj
0 obj
<
I've found a post which says about piping the data all the way through, I have a feeling my pdf_data is corrupted when getting converted to a string
Any ideas how would I go about doing this with the current setup?
e/ RequestPromise is a library, could also use the standards request library if it's easier
https://github.com/request/request-promise -
https://github.com/request/request
Thanks!
Your code doesn't work because the underlying request library (used by request-promise) requires the option encoding set to null for binary data - see https://github.com/request/request#requestoptions-callback.
Here's how you download binary data using that module -
app.post("/getPayslipURL", function(client_request, res) {
const NI_NUMBER_REQUEST = db_api.createRequestTemplate({
body: JSON.stringify(client_request.body),
encoding: null
});
requestPromise(NI_NUMBER_REQUEST)
.then((db_response) => {
const PAY_API_OPTIONS = /*Code to generate options based on furhter DB info (Includes dates etc)*/
return requestPromise(PAY_API_OPTIONS); // Call pay API
})
.then((pay_pdf_data) => {
fs.writeFile("./test.pdf", pay_pdf_data, 'binary', (err) => {
if(err) {
return console.log(err);
}
console.log("The file was saved!");
});
})
.catch(err => `Error caught: ${console.log}`) // Catch any errors on our request chain
});
}