We Keep Coding

AWS S3 presigned url generated successfully but then using it via curl or Postman returns InvalidAccessKeyId

I have this code in my express server that generates an S3 presigned url using NodeJS AWS SDK:
const { S3Client } = require("#aws-sdk/client-s3");
const { getSignedUrl } = require("#aws-sdk/s3-request-presigner");
app.get('/getS3PresignedUrl', async (req, res) => {
const s3 = new S3Client({
region: 'ap-southeast-2',
credentials: {
accessKeyId: accessKey.data.access_key,
secretAccessKey: accessKey.data.secret_key
}
});
const presignedS3Url = await getSignedUrl(s3, new PutObjectCommand({
Bucket: 'xxx',
Key: 'test.txt',
})
);
res.send({
status: true,
message: 'success',
data: {
s3Url: presignedS3Url
}
});
})
Once I get the URL, I use that and put it in either curl or Postman but both gets 403 InvalidAccessKeyId.
For example:
curl
Postman
I don't understand why it fails with 403 when the S3 url is generated successfully? Note, I use that url right away and does not exceed the 900 seconds expire limit.
Any ideas what is the issue here?

How to use an S3 pre-signed POST url?

Actually, this is the first time that I'm using s3 for uploading files. I have heard about pre-signed urls But apparently, I can't set a limitation for file size so I found "pre-signed post urls" but it's a little bit wierd!! Surprisingly I didn't find any example. maybe it's not what I want.
I'm getting pre-signed post url from the server:
const { S3 } = require("aws-sdk");
const s3 = new S3({
accessKeyId: accessKey,
secretAccessKey: secretKey,
endpoint: api,
s3ForcePathStyle: true,
signatureVersion: "v4",
});
app.post("/get-url", (req, res) => {
const key = `user/${uuri.v4()}.png`;
const params = {
Bucket: "bucketName",
Fields: {
Key: key,
ContentType: "image/png",
},
};
s3.createPresignedPost(params, function (err, data) {
if (err) {
console.error("Presigning post data encountered an error", err);
} else {
res.json({ url: data.url });
}
});
});
The weird thing is that the url that I get is not like a pre-signed url. it's just the endpoint followed by the bucket name. no query parameter. no option.
As you might guess, i can't use this url:
await axios.put(url, file, {
headers: {
"Content-Type": "image/png",
},
});
I do not even know if I should use post or two requests.
I tried both, Nothing happens. Maybe the pre-signed post url is not like pre-signed url!
At least show me an example! I can't found any.

You are on the right track, but you need to change the method you are invoking. The AWS S3 API docs of the createPresignedPost() that you are currently using states that:
Get a pre-signed POST policy to support uploading to S3 directly from an HTML form.
Try change this method to either getSignedUrl():
Get a pre-signed URL for a given operation name.
const params = { Bucket: 'bucket', Key: 'key' };
s3.getSignedUrl('putObject', params, function (err, url) {
if (err) {
console.error("Presigning post data encountered an error", err);
} else {
res.json({ url });
}
});
or synchronously:
const params = { Bucket: 'bucket', Key: 'key' };
const url = s3.getSignedUrl('putObject', params)
res.json({ url });
Alternatively, use a promise by executing getSignedUrlPromise():
Returns a 'thenable' promise that will be resolved with a pre-signed URL for a given operation name.
const params = { Bucket: 'bucket', Key: 'key' };
s3.getSignedUrlPromise('putObject', params)
.then(url => {
res.json({ url });
}, err => {
console.error("Presigning post data encountered an error", err);
});
Please also read the notes parts of the API documentation to make sure that you understand the limitations of each method.

Accessing S3 from an external location using IAM user access keys

I have used the javascript AWS-SDK to put a file on S3 via the putObject call. This works fine if I set the bucket to be public, but as soon as I turn off public access, it no longer works and I'm given a 403 error in the response.
I have created a security key against an IAM user, the IAM user is myself and I have sufficient access to S3 via the aws console, so I think my permission are correct.
Here is my code snippet, which works if the bucket is public;
const options = {
region: AWS_REGION,
accessKeyId: AWS_ACCESS_KEY,
secretAccessKey: AWS_SERECT_KEY,
};
const filesAwaitingProcessing = getFilesAwaitingProcessing(FOLDER_ID);
filesAwaitingProcessing.forEach((fileId) => {
const dataFile = file.load({
id: fileId
});
if (dataFile) {
const s3 = new AWS.S3(options);
let error = false;
s3.putObject({
Bucket: BUCKET,
ACL: 'authenticated-read',
ContentEncoding: 'UTF-8',
ContentType: 'application/json',
Key: `${BUCKET_DIRECTORY}/${dataFile.name}`,
Body: dataFile.getContents()
}, (err, data) => {
if (err) {
error = true;
log.error(JSON.stringify(err), JSON.stringify(err));
} else {
log.debug(data);
}
});
s3.getObject({
Bucket: BUCKET,
Key: `${BUCKET_DIRECTORY}/${dataFile.name}`
}, (err, data) => {
if (err) {
error = true;
log.error(err, err.stack);
} else {
log.debug(data);
}
});
Am I passing the key up correctly?
Or am I doing this completely the wrong way for secure access?
This is my bucket policy;
{
"Version": "2012-10-17",
"Id": "Policy1602780209612",
"Statement": [
{
"Sid": "Stmt1602780204129",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::619425574045:user/myUserName"
},
"Action": "s3:*",
"Resource": "arn:aws:s3:::dawson-group/processing"
}
]
}
And then I have the full S3 policy against my user as well as a policy I created that related to just the bucket in question

So it appears that there was no issue with the actual code and bucket set up, the issue was the application we were trying to access S3 from. It is stripping the Authentication headers

AWS S3 bucket returns access denied on ec2 instance usage, even with attached policies and permissions

I have hosted a loopback api on ec-2 server which will download the contents of an s3 bucket folder. When the bucket policy has
"Principal": {
"AWS": "*"
},
The download will work this way,
but on changing the principle to
"Principal": {
"AWS": "arn:aws:iam::<account-number>:role/<ec2-role>"
},
the server throws, access denied error with code 403.
My bucket policy is as:
{
"Version": "2012-10-17",
"Id": "Policy1566920715903",
"Statement": [
{
"Sid": "Stmt1566920713528",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<account-number>:role/<ec2-role>"
},
"Action": "s3:*",
"Resource": "arn:aws:s3:::<bucket-name>/*"
}
]
}
The "block all public access" on my bucket is turned off.
The ec2 role that I have attached to my instance has full s3 access.
The uploader of the bucket object I am trying to access is my account itself.
The ACL for bucket owner has all the permissions turned on.
The IAM user running the ec2 instance also has s3 full access.
I have read various questions here on stack overflow including :
AWS bucket policy- permission denied
AWS S3 Bucket Access from EC2
Grant EC2 instance access to S3 Bucket
also,
https://aws.amazon.com/premiumsupport/knowledge-center/s3-troubleshoot-403/
https://aws.amazon.com/blogs/security/iam-policies-and-bucket-policies-and-acls-oh-my-controlling-access-to-s3-resources/
and checked on countless AWS documentation, but nothing seems to work.
Error on the server:
message: 'Access Denied',
code: 'AccessDenied',
region: null,
time: 2019-08-29T11:14:30.083Z,
requestId: '98B0580B0D2A00F3',
extendedRequestId:'s5K0fPsew96Mf8c2d3R8xj0M85ICY/gNL5wu0ZthpTwO1jgLAccfVee/J7QXZDSXXLmXioVNQwE=',
cfId: undefined,
statusCode: 403,
retryable: false,
retryDelay: 89.70255005732741 }
The code to download from s3 bucket:
module.exports = function(app) {
var myParser = require('body-parser');
app.use(myParser.urlencoded({extended: true}));
app.post('/downloadZip', function(request, response) {
const s3Zip = require('s3-zip');
const AWS = require('aws-sdk');
const region = 'us-east-1';
const bucket = '<bucket-name';
const filename = new Date().toISOString().replace(/:/g,'-').replace(/T|Z/g,'_') + '_recordings_dump.zip';
const folder =request.body.userId + '/';
const fileArray = JSON.parse(request.body.files);
console.log('Request body=', request.body);
response.set('content-type', 'application/zip');
response.set('Content-Disposition', 'attachment; filename=' + filename);
s3Zip.archive({region: region, bucket: bucket, preserveFolderStructure: true, debug: true },folder, fileArray).pipe(response);
});
};
I dont want to keep the bucket items have public access.

AWS SDK Presigned URL + Multipart upload

Is there a way to do a multipart upload via the browser using a generated presigned URL?

Angular - Multipart Aws Pre-signed URL
Example
https://multipart-aws-presigned.stackblitz.io/
https://stackblitz.com/edit/multipart-aws-presigned?file=src/app/app.component.html
Download Backend:
https://www.dropbox.com/s/9tm8w3ujaqbo017/serverless-multipart-aws-presigned.tar.gz?dl=0
To upload large files into an S3 bucket using pre-signed url it is necessary to use multipart upload, basically splitting the file into many parts which allows parallel upload.
Here we will leave a basic example of the backend and frontend.
Backend (Serveless Typescript)
const AWSData = {
accessKeyId: 'Access Key',
secretAccessKey: 'Secret Access Key'
};
There are 3 endpoints
Endpoint 1: /start-upload
Ask S3 to start the multipart upload, the answer is an UploadId associated to each part that will be uploaded.
export const start: APIGatewayProxyHandler = async (event, _context) => {
const params = {
Bucket: event.queryStringParameters.bucket, /* Bucket name */
Key: event.queryStringParameters.fileName /* File name */
};
const s3 = new AWS.S3(AWSData);
const res = await s3.createMultipartUpload(params).promise()
return {
statusCode: 200,
headers: {
'Access-Control-Allow-Origin': '*',
'Access-Control-Allow-Credentials': true,
},
body: JSON.stringify({
data: {
uploadId: res.UploadId
}
})
};
}
Endpoint 2: /get-upload-url
Create a pre-signed URL for each part that was split for the file to be uploaded.
export const uploadUrl: APIGatewayProxyHandler = async (event, _context) => {
let params = {
Bucket: event.queryStringParameters.bucket, /* Bucket name */
Key: event.queryStringParameters.fileName, /* File name */
PartNumber: event.queryStringParameters.partNumber, /* Part to create pre-signed url */
UploadId: event.queryStringParameters.uploadId /* UploadId from Endpoint 1 response */
};
const s3 = new AWS.S3(AWSData);
const res = await s3.getSignedUrl('uploadPart', params)
return {
statusCode: 200,
headers: {
'Access-Control-Allow-Origin': '*',
'Access-Control-Allow-Credentials': true,
},
body: JSON.stringify(res)
};
}
Endpoint 3: /complete-upload
After uploading all the parts of the file it is necessary to inform that they have already been uploaded and this will make the object assemble correctly in S3.
export const completeUpload: APIGatewayProxyHandler = async (event, _context) => {
// Parse the post body
const bodyData = JSON.parse(event.body);
const s3 = new AWS.S3(AWSData);
const params: any = {
Bucket: bodyData.bucket, /* Bucket name */
Key: bodyData.fileName, /* File name */
MultipartUpload: {
Parts: bodyData.parts /* Parts uploaded */
},
UploadId: bodyData.uploadId /* UploadId from Endpoint 1 response */
}
const data = await s3.completeMultipartUpload(params).promise()
return {
statusCode: 200,
headers: {
'Access-Control-Allow-Origin': '*',
'Access-Control-Allow-Credentials': true,
// 'Access-Control-Allow-Methods': 'OPTIONS,POST',
// 'Access-Control-Allow-Headers': 'Content-Type',
},
body: JSON.stringify(data)
};
}
Frontend (Angular 9)
The file is divided into 10MB parts
Having the file, the multipart upload to Endpoint 1 is requested
With the UploadId you divide the file in several parts of 10MB and from each one you get the pre-signed url upload using the Endpoint 2
A PUT is made with the part converted to blob to the pre-signed url obtained in Endpoint 2
When you finish uploading each part you make a last request the Endpoint 3
In the example of all this the function uploadMultipartFile

I was managed to achieve this in serverless architecture by creating a Canonical Request for each part upload using Signature Version 4. You will find the document here AWS Multipart Upload Via Presign Url

from the AWS documentation:
For request signing, multipart upload is just a series of regular requests, you initiate multipart upload, send one or more requests to upload parts, and finally complete multipart upload. You sign each request individually, there is nothing special about signing multipart upload request
So I think you should have to generate a presigned url for each part of the multipart upload :(
what is your use case? can't you execute a script from your server, and give s3 access to this server?