I have code similar to the following to pipe an S3 object back to the client as the response using Express, which is working perfectly.
const s3 = new AWS.S3();
const params = {
Bucket: 'myBucket',
Key: 'myImageFile.jpg'
};
s3.getObject(params).createReadStream().pipe(res);
Problem is, I want to be able to access some of the properties in the response I get back from S3, such as LastModified, ContentLength, ETag, and more. I want to use those properties to send as headers in the response to the client, as well as for logging information.
Due to the fact that it is creating a stream I can't figure out how to get those properties.
I have thought about doing a separate s3.headObject call to get the data I need, but that seems really wasteful, and will end up adding a large amount of cost at scale.
I have also considered ditching the entire stream idea and just having it return a buffer, but again, that seems really wasteful to be using extra memory when I don't need it, and will probably add more cost at scale due to the extra servers needed to handle requests and the extra memory needed to process each request.
How can I get back a stream using s3.getObject along with all the metadata and other properties that s3.getObject normally gives you back?
Something like the following would be amazing:
const s3 = new AWS.S3();
const params = {
Bucket: 'myBucket',
Key: 'myImageFile.jpg',
ReturnType: 'stream'
};
const s3Response = await s3.getObject(params).promise();
s3Response.Body.pipe(res); // where `s3Response.Body` is a stream
res.set("ETag", s3Response.ETag);
res.set("Content-Type", s3Response.ContentType);
console.log("Metadata: ", s3Response.Metadata);
But according to the S3 documentation it doesn't look like that is possible. Is there another way to achieve something like that?
I've found and tested the following. it works.
as per: Getting s3 object metadata then creating stream
function downloadfile (key,res) {
let stream;
const params = { Bucket: 'xxxxxx', Key: key }
const request = s3.getObject(params);
request.on('httpHeaders', (statusCode, httpHeaders) => {
console.log(httpHeaders);
stream.pipe(res)
stream.on('end', () => {
console.log('were done')
})
})
stream = request.createReadStream()
}
Related
My goal is to upload a Readable stream to S3.
The problem is that AWS api seems to accept only a ReadStream as a stream argument.
For instance, the following snippet works just fine:
const readStream = fs.createReadStream("./file.txt") // a ReadStream
await s3.putObject({
Bucket: this.bucket,
Key: this.key,
Body: readStream,
ACL: "bucket-owner-full-control"
}
Problem starts when I try to do the same with a Readable (ReadStream extends stream.Readable).
The following snippet fails
const { Readable } = require("stream")
const readable = Readable.from("data data data data") // a Readable
await s3.putObject({
Bucket: this.bucket,
Key: this.key,
Body: readable,
ACL: "bucket-owner-full-control"
}
the error I get from AWS sdk is: NotImplemented: A header you provided implies functionality that is not implemented
*** Note that I prefer a Readable rather than a ReadStream since I'd like to allow passing streams that are not necessarily originated from a file - an in-memory string for instance.
So a possible solution could be converting a Readable to a Readstream to work with the SDK.
Any help will be much appreciated!
Given the Readable is no longer a file that has metadata such as MIME type and content length you'd need to update your putObject to include those values:
const { Readable } = require("stream")
const readable = Readable.from("data data data data") // a Readable
await s3.putObject({
Bucket: this.bucket,
Key: this.key,
Body: readable,
ACL: "bucket-owner-full-control",
ContentType: "text/plain",
ContentLength: 42 // calculate length of buffer
}
Hopefully that helps!
I'm uploading a video to S3 using aws-sdk in a reaction environment.
And Use an accelerated endpoint for faster data transfers.
the endpoint is bucket-name.s3-accelerate.amazonaws.com
And I changed the option to 'Enabled' for accelerating transmission in bucket properties.
Below is my code for uploading file objects to s3.
import AWS from "aws-sdk";
require("dotenv").config();
const AWS_ACCESS_KEY = process.env.REACT_APP_AWS_ACCESS_KEY;
const AWS_SECRET_KEY = process.env.REACT_APP_AWS_SECRET_KEY;
const BUCKET = process.env.REACT_APP_BUCKET_NAME;
const REGION = process.env.REACT_APP_REGION;
AWS.config.update({
accessKeyId: AWS_ACCESS_KEY,
secretAccessKey: AWS_SECRET_KEY,
region: REGION,
useAccelerateEndpoint: true, //----> the options is here.
});
async function uploadFileToS3(file) {
const params = {
Bucket: BUCKET,
Key: file.name,
ContentType: 'multipart/form-data',
Body: file
};
const accelerateOption = {
Bucket: BUCKET,
AccelerateConfiguration: { Status: 'Enabled'},
ExpectedBucketOwner: process.env.REACT_APP_BUCKET_OWNER_ID,
};
const s3 = new AWS.S3();
try {
s3.putBucketAccelerateConfiguration(accelerateOption, (err, data) => {
if (err) console.log(err)
else console.log(data, 'data put accelerate') //----> this is just object {}
});
s3.upload(params)
.on("httpUploadProgress", progress => {
const { loaded, total } = progress;
const progressPercentage = parseInt((loaded / total) * 100);
console.log(progressPercentage);
})
.send((err, data) => {
console.log(data, 'data data');
});
} catch (err) {
console.log(err);
}
}
There is definitely s3-accelerate in url in the location property of the data object. (in console.log)
{
Bucket: "newvideouploada2f16b1e1d6a4671947657067c79824b121029-dev"
ETag: "\"d0b40e4afca23137bee89b54dd2ebcf3-8\""
Key: "Raw Run __ Race Against the Storm.mp4"
Location: "https://newvideouploada2f16b1e1d6a4671947657067c79824b121029-dev.s3-accelerate.amazonaws.com/Raw+Run+__+Race+Aga
}
However, the object URL of the property of the video I uploaded does not exist.
Is this how it's supposed to be?
Am I using Transfer Acceleration wrong way?
I saw the documentation and AWS said using putBucketAccelerateConfiguration.
But When I console.log there is noting responsed.
Please let me know How to use Transfer Acceleration in Javascript awd-sdk.
If you are running this code on some AWS compute (EC2, ECS, EKS, Lambda), and the bucket is in the same region as your compute, then consider using VPC Gateway endpoints for S3. More information here. If the compute and the bucket are in different regions, consider using VPC Endpoints for S3 with inter region VPC peering. Note: VPC Gateway endpoints are free while VPC Endpoints are not.
After you enable BucketAccelerate it takes at least half an hour to take effect. You don't need to call this every time you upload a file unless you are also suspending bucket acceleration after you are done.
Bucket acceleration helps when you want to use the AWS backbone network to upload data faster(may be user is in region 'A' and bucket is in region 'B' or you want to upload bigger files in which case it goes to the nearest edge location and then uses the AWS backbone network). You can use this tool to check the potential improvement in terms of speed for various regions.
Also there is additional cost when you use this feature. Check the Data Transfer section on the S3 pricing page.
I want my users to upload profile pictures. Right now I can choose a file and pass it into a POST request body. I'm working in Node.js + JavaScript.
I am using DigitalOcean's Spaces object storage service to store my images, which is S3-compatible.
Ideally, my storage stores the file as an actual image. Instead, it is storing as a strange file with Content-Type application/octet-stream. I don't know how I'm supposed to work with this -- normally to display the image, I simply reference the URL that hosts the image, but in this case the URL is pointing to this strange file. It is named something like VMEDFS3Q65JV4B7YQKLS (no extension). The size of the file is 14kb which seems right and it appears to hold the file data. It looks like this:
etc...
I know I'm grabbing the right image and I know the database is hooked up properly as it's posting to the exact right place, I'm just unhappy with the file type.
Request on front end:
fetch('/api/image/profileUpload', {
method: 'PUT',
body: { file: event.target.files[0] },
'Content-Type': 'image/jpg',
})
Code in backend:
const AWS = require('aws-sdk')
let file = req.body;
file = JSON.stringify(file);
AWS.config.update({
region: 'nyc3',
accessKeyId: process.env.SPACES_KEY,
secretAccessKey: process.env.SPACES_SECRET,
});
const s3 = new AWS.S3({
endpoint: new AWS.Endpoint('nyc3.digitaloceanspaces.com')
});
const uploadParams = {
Bucket: process.env.SPACES_BUCKET,
Key: process.env.SPACES_KEY,
Body: file,
ACL: "public-read",
ResponseContentType: 'image/jpg'
};
s3.upload(uploadParams, function(err, data) {
if (err) console.log(err, err.stack);
else console.log(data);
});
What I've tried:
Adding content-type header in request and content-type parameters in back
Other methods of fetching the data in the backend -- they all result in the same thing
Not stringifying the file after grabbing it from the req.body
Changing POST request to PUT request
Would appreciate any insight into either a) converting this octet-stream file into an image or b) getting this image to upload as an image. Thank you
I solved this problem somewhat -- I changed my parameters to this:
const uploadParams = {
Bucket: 'oscarexpert',
Key: 'asdff',
Body: image,
ContentType: "image/jpeg",
ACL: "public-read",
};
^^added ContentType.
It still doesn't store the actual image but that's sort of a different issue.
In my application I read huge data of images, and send the whole data to the client:
const imagesPaths = await getFolderImagesRecursive(req.body.rootPath);
const dataToReturn = await Promise.all(imagesPaths.map((imagePath) => new Promise(async (resolve, reject) => {
try {
const imageB64 = await fs.readFile(imagePath, 'base64');
return resolve({
filename: imagePath,
imageData: imageB64,
});
} catch {
return reject();
}
})));
return res.status(200).send({
success: true,
message: 'Successfully retreived folder images data',
data: dataToReturn,
});
Here is the client side:
const getFolderImages = (rootPath) => {
return fetch('api/getFolderImages', {
method: 'POST',
headers: { 'Content-type': 'application/json' },
body: JSON.stringify({ rootPath }),
});
};
const getFolderImagesServerResponse = await getFolderImages(rootPath);
const getFolderImagesServerData = await getFolderImagesServerResponse.json();
When I do send the data I get failure due to the huge data. Sending the data just with res.send(<data>) is impossible. So, then, how can I bypass this limitation - and how should I accept the data in the client side with the new process?
The answer to your problem requires some read :
Link to the solution
One thing you probably haven’t taken full advantage of before is that webserver’s http response is a stream by default.
They just make it easier for you to pass in synchron data, which is parsed to chunks under the hood and sent as HTTP packages.
We are talking about huge files here; naturally, we don’t want them to be stored in any memory, at least not the whole blob. The excellent solution for this dilemma is a stream.
We create a readstream with the help of the built-in node package ‘fs,’ then pass it to the stream compatible response.send parameter.
const readStream = fs.createReadStream('example.png');
return response.headers({
'Content-Type': 'image/png',
'Content-Disposition': 'attachment; filename="example.png"',
}).send(readStream);
I used Fastify webserver here, but it should work similarly with Koa or Express.
There are two more configurations here: naming the header ‘Content-Type’ and ‘Content-Disposition.’
The first one indicates the type of blob we are sending chunk-by-chunk, so the frontend will automatically give the extension to it.
The latter tells the browser that we are sending an attachment, not something renderable, like an HTML page or a script. This will trigger the browser’s download functionality, which is widely supported. The filename parameter is the download name of the content.
Here we are; we accomplished minimal memory stress, minimal coding, and minimal error opportunities.
One thing we haven’t mentioned yet is authentication.
For the fact, that the frontend won’t send an Ajax request, we can’t expect auth JWT header to be present on the request.
Here we will take the good old cookie auth approach. Cookies are set automatically on every request header that matches the criteria, based on the cookie options. More info about this in the frontend implementation part.
By default, cookies arrive as semicolon separated key-value pairs, in a single string. In order to ease out the parsing part, we will use Fastify’s Cookieparser plugin.
await fastifyServer.register(cookieParser);
Later in the handler method, we simply get the cookie that we are interested in and compare it to the expected value. Here I used only strings as auth-tokens; this should be replaced with some sort of hashing and comparing algorithm.
const cookies = request.cookies;
if (cookies['auth'] !== 'authenticated') {
throw new APIError(400, 'Unauthorized');
}
That’s it. We have authentication on top of the file streaming endpoint, and everything is ready to be connected by the frontend.
I'm trying to generate a pre-signed URL then upload a file to S3 through a browser. My server-side code looks like this, and it generates the URL:
let s3 = new aws.S3({
// for dev purposes
accessKeyId: 'MY-ACCESS-KEY-ID',
secretAccessKey: 'MY-SECRET-ACCESS-KEY'
});
let params = {
Bucket: 'reqlist-user-storage',
Key: req.body.fileName,
Expires: 60,
ContentType: req.body.fileType,
ACL: 'public-read'
};
s3.getSignedUrl('putObject', params, (err, url) => {
if (err) return console.log(err);
res.json({ url: url });
});
This part seems to work fine. I can see the URL if I log it and it's passing it to the front-end. Then on the front end, I'm trying to upload the file with axios and the signed URL:
.then(res => {
var options = { headers: { 'Content-Type': fileType } };
return axios.put(res.data.url, fileFromFileInput, options);
}).then(res => {
console.log(res);
}).catch(err => {
console.log(err);
});
}
With that, I get the 403 Forbidden error. If I follow the link, there's some XML with more info:
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>
The request signature we calculated does not match the signature you provided. Check your key and signing method.
</Message>
...etc
Your request needs to match the signature, exactly. One apparent problem is that you are not actually including the canned ACL in the request, even though you included it in the signature. Change to this:
var options = { headers: { 'Content-Type': fileType, 'x-amz-acl': 'public-read' } };
Receiving a 403 Forbidden error for a pre-signed s3 put upload can also happen for a couple of reasons that are not immediately obvious:
It can happen if you generate a pre-signed put url using a wildcard content type such as image/*, as wildcards are not supported.
It can happen if you generate a pre-signed put url with no content type specified, but then pass in a content type header when uploading from the browser. If you don't specify a content type when generating the url, you have to omit the content type when uploading. Be conscious that if you are using an upload tool like Uppy, it may attach a content type header automatically even when you don't specify one. In that case, you'd have to manually set the content type header to be empty.
In any case, if you want to support uploading any file type, it's probably best to pass the file's content type to your api endpoint, and use that content type when generating your pre-signed url that you return to your client.
For example, generating a pre-signed url from your api:
const AWS = require('aws-sdk')
const uuid = require('uuid/v4')
async function getSignedUrl(contentType) {
const s3 = new AWS.S3({
accessKeyId: process.env.AWS_KEY,
secretAccessKey: process.env.AWS_SECRET_KEY
})
const signedUrl = await s3.getSignedUrlPromise('putObject', {
Bucket: 'mybucket',
Key: `uploads/${uuid()}`,
ContentType: contentType
})
return signedUrl
}
And then sending an upload request from the browser:
import Uppy from '#uppy/core'
import AwsS3 from '#uppy/aws-s3'
this.uppy = Uppy({
restrictions: {
allowedFileTypes: ['image/*'],
maxFileSize: 5242880, // 5 Megabytes
maxNumberOfFiles: 5
}
}).use(AwsS3, {
getUploadParameters(file) {
async function _getUploadParameters() {
let signedUrl = await getSignedUrl(file.type)
return {
method: 'PUT',
url: signedUrl
}
}
return _getUploadParameters()
}
})
For further reference also see these two stack overflow posts: how-to-generate-aws-s3-pre-signed-url-request-without-knowing-content-type and S3.getSignedUrl to accept multiple content-type
If you're trying to use an ACL, make sure that your Lambda IAM role has the s3:PutObjectAcl for the given Bucket and also that your bucket allows for the s3:PutObjectAcl for the uploading Principal (user/iam/account that's uploading).
This is what fixed it for me after double checking all my headers and everything else.
Inspired by this answer https://stackoverflow.com/a/53542531/2759427
1) You might need to use S3V4 signatures depending on how the data is transferred to AWS (chunk versus stream). Create the client as follows:
var s3 = new AWS.S3({
signatureVersion: 'v4'
});
2) Do not add new headers or modify existing headers. The request must be exactly as signed.
3) Make sure that the url generated matches what is being sent to AWS.
4) Make a test request removing these two lines before signing (and remove the headers from your PUT). This will help narrow down your issue:
ContentType: req.body.fileType,
ACL: 'public-read'
Had the same issue, here is how you need to solve it,
Extract the filename portion of the signed URL.
Do a print that you are extracting your filename portion correctly with querystring parameters. This is critical.
Encode to URI Encoding of the filename with query string parameters.
Return the url from your lambda with encoded filename along with other path or from your node service.
Now post from axios with that url, it will work.
EDIT1:
Your signature will also be invalid, if you pass in wrong content type.
Please ensure that the content-type you have you create the pre-signed url is same as the one you are using it for put.
Hope it helps.
As others have pointed out the solution is to add the signatureVerision.
const s3 = new AWS.S3(
{
apiVersion: '2006-03-01',
signatureVersion: 'v4'
}
);
There is very detailed discussion around the same take a look https://github.com/aws/aws-sdk-js/issues/468
This code was working with credentials and a bucket I created several years ago, but caused a 403 error on recently created credentials/buckets:
const s3 = new AWS.S3({
region: region,
accessKeyId: process.env.AWS_ACCESS_KEY,
secretAccessKey: process.env.AWS_SECRET_KEY,
})
The fix was simply to add signatureVersion: 'v4'.
const s3 = new AWS.S3({
signatureVersion: 'v4',
region: region,
accessKeyId: process.env.AWS_ACCESS_KEY,
secretAccessKey: process.env.AWS_SECRET_KEY,
})
Why? I don't know.
TLDR: Check that your bucket exists and is accessible by the AWS Key that is generating the Signed URL..
All of the answers are very good and most likely are the real solution, but my issue actually stemmed from S3 returning a Signed URL to a bucket that didn't exist.
Because the server didn't throw any errors, I had assumed that it must be the upload that was causing the problems without realizing that my local server had an old bucket name in it's .env file that used to be the correct one, but has since been moved.
Side note: This link helped https://aws.amazon.com/premiumsupport/knowledge-center/s3-troubleshoot-403/
It was while checking the uploading users IAM policies that I discovered that the user had access to multiple buckets, but only 1 of those existed anymore.
Did you add the CORS policy to the S3 bucket? This fixed the problem for me.
[
{
"AllowedHeaders": [
"*"
],
"AllowedMethods": [
"PUT"
],
"AllowedOrigins": [
"*"
],
"ExposeHeaders": []
}
]
I encountered the same error twice with different root causes / solutions:
I was using generate_presigned_url.
The solution for me was switching to generate_presigned_post (doc) which returns a host of essential information such as
"url":"https://xyz.s3.amazonaws.com/",
"fields":{
"key":"filename.ext",
"AWSAccessKeyId":"ASIAEUROPRSWEDWOMM",
"x-amz-security-token":"some-really-long-string",
"policy":"another-long-string",
"signature":"the-signature"
}
Add these fields to your request headers, don't forget to keep file last!
That time I forgot to give proper permissions to the Lambda. Interestingly, Lambda can create good looking signed upload URLs which you won't have permission to use. The solution is to enrich the policy with S3 actions:
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::my-own-bucket/*"
]
}
using python boto3 when you upload a file the permissions are private by default. you can make the object public using ACL='public-read'
s3.put_object_acl(
Bucket='gid-requests', Key='potholes.csv', ACL='public-read')
I did all that's mentioned here and allowed these permissions for it to work: