We Keep Coding

Is Transfer Acceleration working? response location and object URL is different

I'm uploading a video to S3 using aws-sdk in a reaction environment.
And Use an accelerated endpoint for faster data transfers.
the endpoint is bucket-name.s3-accelerate.amazonaws.com
And I changed the option to 'Enabled' for accelerating transmission in bucket properties.
Below is my code for uploading file objects to s3.
import AWS from "aws-sdk";
require("dotenv").config();
const AWS_ACCESS_KEY = process.env.REACT_APP_AWS_ACCESS_KEY;
const AWS_SECRET_KEY = process.env.REACT_APP_AWS_SECRET_KEY;
const BUCKET = process.env.REACT_APP_BUCKET_NAME;
const REGION = process.env.REACT_APP_REGION;
AWS.config.update({
accessKeyId: AWS_ACCESS_KEY,
secretAccessKey: AWS_SECRET_KEY,
region: REGION,
useAccelerateEndpoint: true, //----> the options is here.
});
async function uploadFileToS3(file) {
const params = {
Bucket: BUCKET,
Key: file.name,
ContentType: 'multipart/form-data',
Body: file
};
const accelerateOption = {
Bucket: BUCKET,
AccelerateConfiguration: { Status: 'Enabled'},
ExpectedBucketOwner: process.env.REACT_APP_BUCKET_OWNER_ID,
};
const s3 = new AWS.S3();
try {
s3.putBucketAccelerateConfiguration(accelerateOption, (err, data) => {
if (err) console.log(err)
else console.log(data, 'data put accelerate') //----> this is just object {}
});
s3.upload(params)
.on("httpUploadProgress", progress => {
const { loaded, total } = progress;
const progressPercentage = parseInt((loaded / total) * 100);
console.log(progressPercentage);
})
.send((err, data) => {
console.log(data, 'data data');
});
} catch (err) {
console.log(err);
}
}
There is definitely s3-accelerate in url in the location property of the data object. (in console.log)
{
Bucket: "newvideouploada2f16b1e1d6a4671947657067c79824b121029-dev"
ETag: "\"d0b40e4afca23137bee89b54dd2ebcf3-8\""
Key: "Raw Run __ Race Against the Storm.mp4"
Location: "https://newvideouploada2f16b1e1d6a4671947657067c79824b121029-dev.s3-accelerate.amazonaws.com/Raw+Run+__+Race+Aga
}
However, the object URL of the property of the video I uploaded does not exist.
Is this how it's supposed to be?
Am I using Transfer Acceleration wrong way?
I saw the documentation and AWS said using putBucketAccelerateConfiguration.
But When I console.log there is noting responsed.
Please let me know How to use Transfer Acceleration in Javascript awd-sdk.

If you are running this code on some AWS compute (EC2, ECS, EKS, Lambda), and the bucket is in the same region as your compute, then consider using VPC Gateway endpoints for S3. More information here. If the compute and the bucket are in different regions, consider using VPC Endpoints for S3 with inter region VPC peering. Note: VPC Gateway endpoints are free while VPC Endpoints are not.
After you enable BucketAccelerate it takes at least half an hour to take effect. You don't need to call this every time you upload a file unless you are also suspending bucket acceleration after you are done.
Bucket acceleration helps when you want to use the AWS backbone network to upload data faster(may be user is in region 'A' and bucket is in region 'B' or you want to upload bigger files in which case it goes to the nearest edge location and then uses the AWS backbone network). You can use this tool to check the potential improvement in terms of speed for various regions.
Also there is additional cost when you use this feature. Check the Data Transfer section on the S3 pricing page.

Saving an S3 buffer to file through AWS Javascript SDK

I'm using the AWS Javascript SDK to download a file from S3
var s3 = new AWS.S3();
var params = {
Bucket: "MYBUCKET",
Key: file
};
s3.getObject(params, function(err, data) {
if (err) console.log(err, err.stack); // an error occurred
else {
//code to save file from data's byte array here
}
});
This feels like it should be easier than I'm making it out to be. Basically I want to trigger the native file download for the browser. Every resource I've found on the internet is for node's file system. I can't just use the file's URL to download as it is stored encrypted via KMS, so that is why I am going about it this way.
Thanks for the help!

I ended up changing how I was storing files. Instead of encrypting them with KMS, I moved them to a private bucket and then based the retrieval off of the logged in cognito user's ID. Then, I switched to using getSignedURL to appropriately pass in the cognito user ID.
var s3 = new AWS.S3();
var params = {
Bucket: "MYBUCKET",
Key: cognitoUser.username + "/" + file
};
var url = s3.getSignedUrl('getObject', params);
window.open(url);

aws-sdk: NoSuchKey: The specified key does not exist?

In my nodejs project, I am using aws-sdk to download all the images from my s3 bucket, But I got this error- NoSuchKey: The specified key does not exist. But keys are correct and I can upload images with these keys.
My code is:
var AWS = require('aws-sdk');
s3 = new AWS.S3();
var params = {
Bucket: config.get("aws.s3.bucket"),
Key: config.get("aws.credentials.secretAccessKey")
};
s3.getObject(params, function (err, data) {
console.log("data");
if (err) console.log(err, err.stack); // an error occurred
else console.log(data);
});
}
Can anyone please tell me where I am doing wrong?

There are problems related to how to use aws-sdk and it should be as following example:
var aws = require('aws-sdk');
aws.config.update({
accessKeyId: {{AWS_ACCESS_KEY}},
secretAccessKey: {{AWS_SECRET_KEY}}
});
var s3 = new aws.S3();
var s3Params = {
Bucket: {{bucket name}},
Key: {{path to dedicated S3 Object (folder name + file/object
name)}}
};
s3.getObject(s3Params, function (err, data) {
//Continue handling the returned results.
});
replace the strings inside {{}} with correct data and it should work well.

This is because that img url doesnot exist with same user.Means that u put a img alone.jpeg in your postman when you upload image in aws.But in getobject you are posting image.jpg with same user.conclusion is that image should be same which you are uploading and then getting with same user. [when you are getting a getObject,you post this image with same user]
when you are uploading image to aws
[1]: https://i.stack.imgur.com/WLh5v.png
but when you use getObject with another image with same user(which user's token you are using ),it will give the same error.
When you user image.jpg instead [1]: https://i.stack.imgur.com/WLh5v.png
So use same image key.
use image key which is coming from aws's response instead of url.

AWS javascript API example to list all user pools associated with an account

Hi I need AWS cognito javascript API examples which will
1. return me the list of user-pools associated with the account.
2. create a new user pool.
I have searched through most of the documentation, but unable to find any relevant answers. Can anyone please point me to a link/article or help me out with a sample code here.

For list User Pools (you basically require the aws sdk, configure credentials, instantiate the client and call the specific operation):
var aws = require('aws-sdk');
aws.config.update({accessKeyId: 'akid', secretAccessKey: 'secret'});
var CognitoIdentityServiceProvider = aws.CognitoIdentityServiceProvider;
var client = new CognitoIdentityServiceProvider({ apiVersion: '2016-04-19', region: 'us-east-1' });
client.listUserPools({MaxResults: 10}, function(err, data) {
if (!err) {
console.log('successful' + JSON.stringify(data));
} else {
console.log('error ' + err);
}
});
Note that there might be different ways in which you can configure the AWS credentials to call the operation. You do need credentials as this is an authenticated operation. CreateUserPool is similar, you just need to pass the appropriate parameters as JSON in the call.

cannot seem to delete key from AWS 3S using javascript

I'm trying to delete a 'folder' called js from an Amazon AWS S3 bucket. I've followed numerous tutorials and the Identity I'm using has AmazonS3FullAccess permissions.
By all I've gathered, the following should work - but it doesn't. I get no errors, I merely get console output of {}.
I have a method that can upload to Amazon S3 using the same credentials, so I know they check out right. This is a listing of how the credentials are configured on my IAM;
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": "*"
}
]
}
and this is the actual code I'm trying to run to delete the content;
var AWS = require('aws-sdk');
AWS.config = {
'accessKeyId': `{HIDDEN}`,
'secretAccessKey': `{HIDDEN}`,
'region': `us-east-1`
};
var rmAWS = function() {
var BUCKET_NAME = `{HIDDEN}`; var s3 = new AWS.S3();
var params = {
Bucket: BUCKET_NAME,
Key: 'js'
};
s3.deleteObject(params, function(err, data) {
if (err) console.log(err, err.stack); // an error occurred
else console.log(data); // successful response
});
}
rmAWS();

AWS S3 doesn't understand the concept of directory. So, when you try to delete js it looks for a file who's key is js. The reality is that there are several files who's keys are js/foo. You should try deleting all those files, that the "folder" will be "deleted".
To do so, you should use the deleteObjects method (http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#deleteObjects-property).
You can't delete objects using a wildcard such as js/*, but you can list the objects in the bucket and then create an array with only the ones you want to delete.
Try using the listObjects method (http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#listObjects-property). Assign the prefix to js/ that it should only return the objects you'd like to delete. Then use this result as the array for the deleteObjects method.

Are there multiple "js" "folders" that need to be deleted? Or just one?
var params = {
Bucket: BUCKET_NAME,
Delimiter: '/',
Prefix: 'FOLDERNAME/FOLDERNAME/js/'
};
Make sure to try it out with listing the objects instead of deleting them first.