Node.js Express send huge data to client vanilla JS - javascript

In my application I read huge data of images, and send the whole data to the client:
const imagesPaths = await getFolderImagesRecursive(req.body.rootPath);
const dataToReturn = await Promise.all(imagesPaths.map((imagePath) => new Promise(async (resolve, reject) => {
try {
const imageB64 = await fs.readFile(imagePath, 'base64');
return resolve({
filename: imagePath,
imageData: imageB64,
});
} catch {
return reject();
}
})));
return res.status(200).send({
success: true,
message: 'Successfully retreived folder images data',
data: dataToReturn,
});
Here is the client side:
const getFolderImages = (rootPath) => {
return fetch('api/getFolderImages', {
method: 'POST',
headers: { 'Content-type': 'application/json' },
body: JSON.stringify({ rootPath }),
});
};
const getFolderImagesServerResponse = await getFolderImages(rootPath);
const getFolderImagesServerData = await getFolderImagesServerResponse.json();
When I do send the data I get failure due to the huge data. Sending the data just with res.send(<data>) is impossible. So, then, how can I bypass this limitation - and how should I accept the data in the client side with the new process?

The answer to your problem requires some read :
Link to the solution
One thing you probably haven’t taken full advantage of before is that webserver’s http response is a stream by default.
They just make it easier for you to pass in synchron data, which is parsed to chunks under the hood and sent as HTTP packages.
We are talking about huge files here; naturally, we don’t want them to be stored in any memory, at least not the whole blob. The excellent solution for this dilemma is a stream.
We create a readstream with the help of the built-in node package ‘fs,’ then pass it to the stream compatible response.send parameter.
const readStream = fs.createReadStream('example.png');
return response.headers({
'Content-Type': 'image/png',
'Content-Disposition': 'attachment; filename="example.png"',
}).send(readStream);
I used Fastify webserver here, but it should work similarly with Koa or Express.
There are two more configurations here: naming the header ‘Content-Type’ and ‘Content-Disposition.’
The first one indicates the type of blob we are sending chunk-by-chunk, so the frontend will automatically give the extension to it.
The latter tells the browser that we are sending an attachment, not something renderable, like an HTML page or a script. This will trigger the browser’s download functionality, which is widely supported. The filename parameter is the download name of the content.
Here we are; we accomplished minimal memory stress, minimal coding, and minimal error opportunities.
One thing we haven’t mentioned yet is authentication.
For the fact, that the frontend won’t send an Ajax request, we can’t expect auth JWT header to be present on the request.
Here we will take the good old cookie auth approach. Cookies are set automatically on every request header that matches the criteria, based on the cookie options. More info about this in the frontend implementation part.
By default, cookies arrive as semicolon separated key-value pairs, in a single string. In order to ease out the parsing part, we will use Fastify’s Cookieparser plugin.
await fastifyServer.register(cookieParser);
Later in the handler method, we simply get the cookie that we are interested in and compare it to the expected value. Here I used only strings as auth-tokens; this should be replaced with some sort of hashing and comparing algorithm.
const cookies = request.cookies;
if (cookies['auth'] !== 'authenticated') {
throw new APIError(400, 'Unauthorized');
}
That’s it. We have authentication on top of the file streaming endpoint, and everything is ready to be connected by the frontend.

Related

Why do I get a 400 error trying to get data from an html page using fetch API?

Hi I'm doing a coding challenge and I'm trying to fetch data from an html page but I keep getting the 400 error. I don't see where my syntax would be wrong. The Promise returns an empty string as PromiseResult. Why do I not get the data from https://adventofcode.com/2021/day/2/input?
fetch('https://adventofcode.com/2021/day/2/input', {
method: 'GET',
mode: 'no-cors',
credentials: 'omit',
headers: {
'Content-Type': 'text/plain'
}
})
.then((response) => {
response.text();
})
.then((html) => {
var parser = new DOMParser();
var doc = parser.parseFromString(html, 'text/html');
})
.catch((err) => { console.log(err) })
After visting https://adventofcode.com/2021/day/2/input I get information that
Puzzle inputs differ by user. Please log in to get your puzzle input.
Please check if you are providing your login data correctly.
mode: 'no-cors',
This says that you are not going to do anything that requires permission to be granted using CORS. Anything that does require CORS will be quietly ignored.
You need CORS permission to read data across origins. Since you have said you aren't going to use CORS, you don't have permission, so you don't get any data.
This is why you get an empty string.
credentials: 'omit',
The input endpoint requires your credentials so that it can give you your input, which is unique to each user. Since told the browser not to send any, the end point doesn't understand the request.
As an aside:
headers: {
'Content-Type': 'text/plain'
}
This is just nonsense.
It claims that the request includes a body consisting of plain text.
You are making a GET request. There is no body on the request at all.
Advent of code expects you to manually download your input data. It doesn't expect your solution to fetch it from the AoC website.
JS solutions are generally run in Node.js (rather than a web browser) where they can use the fs module to read the local copy of the input file. (Tip: I find it handy to copy/paste the sample data into a sample file to test my results against the worked example on each day).

Route that is executed within another route in Node.js is not being executed

Good Evening,
I have a function that contains a route that is a call to the Auth0 API and contains the updated data that was sent from the client. The function runs, but the app.patch() does not seem to run and I am not sure what I am missing.
function updateUser(val) {
app.patch(`https://${process.env.AUTH0_BASE_URL}/api/v2/users/${val.id}`,(res) => {
console.log(val);
res.header('Authorization: Bearer <insert token>)
res.json(val);
})
app.post('/updateuser', (req, ) => {
const val = req.body;
updateUser(val);
})
app.patch() does NOT send an outgoing request to another server. Instead, it registers a listener for incoming PATCH requests. It does not appear from your comments that that is what you want to do.
To send a PATCH request to another server, you need to use a library that is designed for sending http requests. There's a low level library built into the nodejs http module which you could use an http.request() to construct a PATCH request with, but it's generally a lot easier to use a higher level library such as any of them listed here.
My favorite in that list is the got() library, but many in that list are popular and used widely.
Using the got() library, you would send a PATCH request like this:
const got = require('got');
const options = {
headers: {Authorization: `Bearer ${someToken}`},
body: someData
};
const url = `https://${process.env.AUTH0_BASE_URL}/api/v2/users/${val.id}`;
got.patch(url, options).then(result => {
console.log(result);
}).catch(err => {
console.log(err);
});
Note: The PATCH request needs body data (the same that a POST needs body data)

AWS SDK JS S3 getObject Stream Metadata

I have code similar to the following to pipe an S3 object back to the client as the response using Express, which is working perfectly.
const s3 = new AWS.S3();
const params = {
Bucket: 'myBucket',
Key: 'myImageFile.jpg'
};
s3.getObject(params).createReadStream().pipe(res);
Problem is, I want to be able to access some of the properties in the response I get back from S3, such as LastModified, ContentLength, ETag, and more. I want to use those properties to send as headers in the response to the client, as well as for logging information.
Due to the fact that it is creating a stream I can't figure out how to get those properties.
I have thought about doing a separate s3.headObject call to get the data I need, but that seems really wasteful, and will end up adding a large amount of cost at scale.
I have also considered ditching the entire stream idea and just having it return a buffer, but again, that seems really wasteful to be using extra memory when I don't need it, and will probably add more cost at scale due to the extra servers needed to handle requests and the extra memory needed to process each request.
How can I get back a stream using s3.getObject along with all the metadata and other properties that s3.getObject normally gives you back?
Something like the following would be amazing:
const s3 = new AWS.S3();
const params = {
Bucket: 'myBucket',
Key: 'myImageFile.jpg',
ReturnType: 'stream'
};
const s3Response = await s3.getObject(params).promise();
s3Response.Body.pipe(res); // where `s3Response.Body` is a stream
res.set("ETag", s3Response.ETag);
res.set("Content-Type", s3Response.ContentType);
console.log("Metadata: ", s3Response.Metadata);
But according to the S3 documentation it doesn't look like that is possible. Is there another way to achieve something like that?
I've found and tested the following. it works.
as per: Getting s3 object metadata then creating stream
function downloadfile (key,res) {
let stream;
const params = { Bucket: 'xxxxxx', Key: key }
const request = s3.getObject(params);
request.on('httpHeaders', (statusCode, httpHeaders) => {
console.log(httpHeaders);
stream.pipe(res)
stream.on('end', () => {
console.log('were done')
})
})
stream = request.createReadStream()
}

How to extract the Content-Type of a file sent via multipart/form-data

I receive a Request in my cloudflare worker and want to upload the data to Google cloud storage. My problem is that I can't extract the content type from the multipart/form-data data I receive in order to upload it with the correct content type to GCS.
When I read the request with await req.formData() I can get('file')from the formData and it returns the raw file data that I need for the GCS, but I can't seem to see anywhere the file content-type that I need (I can see it only when looking at the raw Request body).
Here is my (striped down) code :
event.respondWith((async () => {
const req = event.request
const formData = await req.formData()
const file = formData.get('file')
const filename = formData.get('filename')
const oauth = await getGoogleOAuth()
const gcsOptions = {
method: 'PUT',
headers: {
Authorization: oauth.token_type + ' ' + oauth.access_token,
'Content-Type': 'application/octet-stream' //this should by `'Content-Type': file.type`
},
body: file,
}
const gcsRes = await fetch(
`https://storage.googleapis.com/***-media/${filename}`,
gcsOptions,
)
if (gcsRes.status === 200) {
return new Response(JSON.stringify({filename}), gcsRes)
} else {
return new Response('Internal Server Error', {status: 500, statusText: 'Internal Server Error'})
}
})())
Reminder - the code is part of our cloudflare worker code.
It seems to me this should be straight forward, determining the type of file you extract from the multipart/form-data data.
Am I missing something?
Unfortunately, as of this writing, the Cloudflare Workers implementation of FormData is incomplete and does not permit extracting the Content-Type. In fact, it appears our implementation currently interprets all entries as text and return strings, which means binary content will be corrupted. This is a bug which will require care to fix since we don't want to break already-deployed scripts that might rely on the buggy behavior.
Thanks Kenton for your response.
What I ended up doing:
As the Cloudflare Workers don't support the multipart/form-data of Blob or any type other than String, I ended up using the raw bytes in the ArrayBuffer data type. After converting it to an Uint8Array I parsed it byte by byte to determine the file type and the start and end indexes of the file data. Once I found the start and end of the transferred file I was able create an array of the file data, add it to the request and send it to the GCS as I showed above.

cordova-plugin-file-transfer: How do you upload a file to S3 using a signed URL?

I am able to upload to S3 using a file picker and regular XMLHttpRequest (which I was using to test the S3 setup), but cannot figure out how to do it successfully using the cordova file transfer plugin.
I believe it is either to do with the plugin not constructing the correct signable request, or not liking the local file uri given. I have tried playing with every single parameter from headers to uri types, but the docs aren't much help, and the plugin source is bolognese.
The string the request needs to sign match is like:
PUT
1391784394
x-amz-acl:public-read
/the-app/317fdf654f9e3299f238d97d39f10fb1
Any ideas, or possibly a working code example?
A bit late, but I just spent a couple of days struggling with this so in case anybody else is having problems, this is how managed to upload an image using the javascript version of the AWS SDK to create the presigned URL.
The key to solving the problem is in the StringToSign element of the XML SignatureDoesNotMatch error that comes back from Amazon. In my case it looked something like this:
<StringToSign>
PUT\n\nmultipart/form-data; boundary=+++++org.apache.cordova.formBoundary\n1481366396\n/bucketName/fileName.jpg
</StringToSign>
When you use the aws-sdk to generate a presigned URL for upload to S3, internally it will build a string based on various elements of the request you want to make, then create an SHA1 hash of it using your AWS secret. This hash is the signature that gets appended to the URL as a parameter, and what doesn't match when you get the SignatureDoesNotMatch error.
So you've created your presigned URL, and passed it to cordova-plugin-file-transfer to make your HTTP request to upload a file. When that request hits Amazon's server, the server will itself build a string based on the request headers etc, hash it and compare that hash to the signature on the URL. If the hashes don't match then it returns the dreaded...
The request signature we calculated does not match the signature you provided. Check your key and signing method.
The contents of the StringToSign element I mentioned above is the string that the server builds and hashes to compare against the signature on the presigned URL. So to avoid getting the error, you need to make sure that the string built by the aws-sdk is the same as the one built by the server.
After some digging about, I eventually found the code responsible for creating the string to hash in the aws-sdk. It is located (as of version 2.7.12) in:
node_modules/aws-sdk/lib/signers/s3.js
Down the bottom at line 168 there is a sign method:
sign: function sign(secret, string) {
return AWS.util.crypto.hmac(secret, string, 'base64', 'sha1');
}
If you put a console.log in there, string is what you're after. Once you make the string that gets passed into this method the same as the contents of StringToSign in the error message coming back from Amazon, the heavens will open and your files will flow effortlessly into your bucket.
On my server running node.js, I originally created my presigned URL like this:
var AWS = require('aws-sdk');
var s3 = new AWS.S3(options = {
endpoint: 'https://s3-eu-west-1.amazonaws.com',
accessKeyId: "ACCESS_KEY",
secretAccessKey: "SECRET_KEY"
});
var params = {
Bucket: 'bucketName',
Key: imageName,
Expires: 60
};
var signedUrl = s3.getSignedUrl('putObject', params);
//return signedUrl
This produced a signing string like this, similar to the OP's:
PUT
1481366396
/bucketName/fileName.jpg
On the client side, I used this presigned URL with cordova-plugin-file-transfer like so (I'm using Ionic 2 so the plugin is wrapped in their native wrapper):
let success = (result: any) : void => {
console.log("upload success");
}
let failed = (err: any) : void => {
let code = err.code;
alert("upload error - " + code);
}
let ft = new Transfer();
var options = {
fileName: filename,
mimeType: 'image/jpeg',
chunkedMode: false,
httpMethod:'PUT',
encodeURI: false,
};
ft.upload(localDataURI, presignedUrlFromServer, options, false)
.then((result: any) => {
success(result);
}).catch((error: any) => {
failed(error);
});
Running the code produced the signature doesn't match error, and the string in the <StringToSign> element looks like this:
PUT
multipart/form-data; boundary=+++++org.apache.cordova.formBoundary
1481366396
/bucketName/fileName.jpg
So we can see that cordova-plugin-file-transfer has added in its own Content-Type header which has caused a discrepancy in the signing strings. In the docs relating to the options object that get passed into the upload method it says:
headers: A map of header name/header values. Use an array to specify more than one value. On iOS, FireOS, and Android, if a header named Content-Type is present, multipart form data will NOT be used. (Object)
so basically, if no Content-Type header is set it will default to multipart form data.
Ok so now we know the cause of the problem, it's a pretty simple fix. On the server side I added a ContentType to the params object passed to the S3 getSignedUrl method:
var params = {
Bucket: 'bucketName',
Key: imageName,
Expires: 60,
ContentType: 'image/jpeg' // <---- content type added here
};
and on the client added a headers object to the options passed to cordova-plugin-file-transfer's upload method:
var options = {
fileName: filename,
mimeType: 'image/jpeg',
chunkedMode: false,
httpMethod:'PUT',
encodeURI: false,
headers: { // <----- headers object added here
'Content-Type': 'image/jpeg',
}
};
and hey presto! The uploads now work as expected.
I run into such issues with this plugin
The only working way I found to upload a file with a signature is the method of Christophe Coenraets : http://coenraets.org/blog/2013/09/how-to-upload-pictures-from-a-phonegap-app-to-amazon-s3/
With this method you will be able to upload your files using the cordova-plugin-file-transfer
First, I wanted to use the aws-sdk on my server to sign with getSignedUrl()
It returns the signed link and you only have to upload to it.
But, using the plugin it always end with 403 : signatures don't match
It may be related to the content length parameter but I didn't found for now a working solution with aws-sdk and the plugin

Categories