We Keep Coding

S3 GetObject Returning Weird Encoding for HTML file

I am attempting to read an HTML file into a Lambda Edge function and then return it in the response body, but for some reason, I cannot get it to return the contents of the HTML file correctly.
Here is my code (simplified):
const AWS = require('aws-sdk');
const S3 = new AWS.S3({
signatureVersion: 'v4'
});
const { Body } = await S3.getObject({ Bucket: 'my-bucket', Key: 'index.html' }).promise();
console.log(Body.toString());
Instead of seeing <html... in the console log, I am seeing the dreaded question mark characters which implies (I think), bad encoding:
��Y�r#��x�`�,b�J�ٲ��NR�yIٮ�!!���"���n���޴��Is�>}�n4pF�_���de�nq�~�]� f�����v��۔*�㺮Ý� Hdž�<�! �c�5�1B��,#|Ŵ;ֶ�U����z� �Qi��j�0��V ���H���...etc
I have literally tried everything including, but not limited to:
Body.toString('utf-8');
Body.toString('ascii');
Body.toString('base64');
decoder.write(Body.toString('base64'));
and a lot more...
I think I must be missing something really obvious here as I cannot find anyone else facing the same issue. I thought it might be to do with the encryption but my other Lambda Edge function reads an image file without issues so I assume it has to be something to do with encoding that I haven't thought of.
UPDATE
I believe the issue may be related to the fact that the file is gzipped.
Here is a print of the response from S3:
{
AcceptRanges: 'bytes',
LastModified: 2023-02-17T19:44:41.000Z,
ContentLength: 1598,
ETag: 'some-key',
CacheControl: 'max-age=31536000',
ContentEncoding: 'gzip',
ContentType: 'text/html; charset=UTF-8',
ServerSideEncryption: 'AES256',
Metadata: { etag: 'some-key' },
Body: <Buffer 1f 8b 08 00 00 00 00 00 00 03 cd 59 db 72 23 b7 11 fd 15 78 f2 60 bb 2c 62 ee b7 8d c8 4a b2 d9 b2 b7 ca 4e 52 bb 79 49 d9 ae 14 06 e8 21 21 cd 0c a6 ... 1548 more bytes>
}

The issue was caused by the fact that the index.html file was gzipped. The solution to handling this is as follows:
const AWS = require('aws-sdk');
const zlib = require('zlib');
const S3 = new AWS.S3({
signatureVersion: 'v4'
});
const { Body } = await S3.getObject({ Bucket: 'my-bucket', Key: 'index.html' }).promise();
const body = zlib.unzipSync(Body).toString();
console.log(body);
However, given that this code is being deployed within a Lambda Edge function, speed is essential! With that in mind, it would/should be faster to actually utilise https.get as this will automatically unzip the contents of the file prior to returning the Buffer stream.

How to display playable Video Url from Blob

I am trying to download videos from a webDav server to my server. After that the videos should be viewable as well as playable on my client. For downloading I use the webdav library from npm.
The download of the videos works on my server, as well as the transmission of the data to my client. My Problem is, I can't play those videos on the generated website.
app.post("/someroute/api", async (req, res) => {
let result = [];
try {
const { username, password, targetUrl, data } = req.body;
const dataFull = `somedata_${data}`;
const client = createClient(targetUrl, {
username: username,
password: password,
});
const filesToDownload = await client.getDirectoryContents(`/${dataFull}`);
const downloadPromises = filesToDownload.map(async (file) => {
const fileBuffer = await client.getFileContents(file.filename);
result.push({name: file.basename, buffer: fileBuffer});
});
await Promise.all(downloadPromises);
} catch (error) {
console.error(error);
res.status(500).json({ error: "Error retrieving directory contents" });
}
console.log(result);
res.send(result);});
The result looks like this :
{ name: 'butterfly slow motion [oAI-OqTmJj4].mp4',
buffer: <Buffer 00 00 00 18 66 74 79 70 6d 70 34 32 00 00 00 00 69 73 6f 6d 6d 70 34 32 00 00 40 d7 6d 6f 6f 76 00 00 00 6c 6d 76 68 64
00 00 00 00 d8 cd f6 85 d8 cd ... 4073099 more bytes> }
On the client side I have this function to request the videos and store them in a useState.
useEffect(() => {
if (selectedData) {
getDataVideos();
}
}, [selectedData]);
const getDataVideos = async () => {
try {
const response = await fetch("/someroute/api", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
username: username,
password: password,
targetUrl: targetUrl,
data: selectedData,
}),
});
const content = await response.json();
console.log(content);
setSelectedDataVideos(content);
} catch (error) {
console.error(error);
}
};
the console.log() of the selectedDataVideos is :
Array (1) 0 Object
buffer: {type: "Buffer", data: Array}
name: "butterfly slow motion [oAI-OqTmJj4].mp4"
Later I transform the videos into a blob using this method and safe the result into a usestate
const [tempUrls, settempUrls] = useState([]);
, here a snapshot of it :
const blob = new Blob([video.buffer.data], { type: "video/mp4" });
console.log("Videoslide 2 blob")
console.log(blob)
const url = URL.createObjectURL(blob);
return {name:video.name,url:url};
The blobs are saved in a usestate const [tempUrls, settempUrls] = useState([]);
The blob itself looks like this :
[Log] Blob {size: 14484621, type: "video/mp4", slice: function,
stream: function, text: function, …}
An index the tempUrls looks finished like this :
[Log] {name: "butterfly slow motion [oAI-OqTmJj4].mp4", url:
"blob:http:// localhost:8081/ed893b3d-298f-40c4-b4c0-1c3ac688aa43"}
The space in http:// localhost is for removing the hyperlink here.
For display the video and to play it I have this :
{tempUrls.map(video =>
<div key={video.name} style={{ flex: 2, padding: "15px 20px" }}>
<video
src={video.url}
controls
type="video/mp4"
>
</video>
Only the control placeholder is displayed but I can't play the video. Also, when I open the web page in Firefox this is displayed. "No video with supported format and MIME type found". But I am not clear where exactly I made the mistake when converting. Actually I converted the array buffer to a blob of type mp4/video and then to a URL. Then saved it as object {filename, url} in a usestate and from this usestate the videos should be played/displayed.
Does anyone here possibly see the error, I've been sitting on it a bit longer without finding it. The video itself is not corrupted, ive stored it in a docker next cloud server and can play it there without problems.

Solved it by using
import { Buffer } from "buffer";
from npm to convert the arraybuffer.
let buffer = Buffer.from(item.buffer.data);
const blob = new Blob([buffer], { type: "video/mp4" });
const url = URL.createObjectURL(blob);
result.push({ name: item.name, url: url });

NodeJS Server - Unable to read POST request body

I'm trying to create a simple account system on a Nodejs HTTP Server.
My Server-side code:
var server=http.createServer(function(req,res){
if(req.url.startsWith("/api/")){
console.log(" - // - API CALL - // -")
switch (req.url.substr(4)) {
case '/createAccount':
if(req.method=="POST"){
const requestBody = [];
req.on('data', (chunks)=>{
requestBody.push(chunks);
console.log(chunks)
});
req.on('end', ()=>{
console.log(requestBody)
const parsedData = JSON.stringify(Buffer.concat(requestBody));
console.log(parsedData)
res.writeHead(200,{"Content-Type":"application/json","Access-Control-Allow-Origin": "*"});
res.end(JSON.stringify({'error':false}));
});
}else{
res.writeHead(405,{"Content-Type":"application/json","Access-Control-Allow-Origin": "*"});
res.end(JSON.stringify({'error':true,'statuscode':'405','errortext':'You need to send a POST request to /createAccount.'}));
}
break;
default:
res.writeHead(404,{"Content-Type":"application/json","Access-Control-Allow-Origin": "*"});
res.end(JSON.stringify({'error':true,'statuscode':'404','errortext':'This is not a valid API address.'}));
}
}else{
console.log(" - // - NORMAL WEB PAGE - // -")
res.writeHead(200,{"Content-Type":"text/html","Access-Control-Allow-Origin": "*"});
let accountstate="not logged in"
page="<html><body>This is my homepage!<br>You are "+accountstate+".</body></html>";
res.end(page);
}
});
server.listen(7000);
console.log(`Listening at http://localhost:7000`);
The request I'm sending:
fetch("http://localhost:7000/api/createAccount",{
method:"POST",
body:{
AccountName:"test",
Password:"THISSHOULDBEENCRYPTED"
}
}).then(r=>r.json()).then(r=>console.log(r))
The console output after sending the request:
- // - API CALL - // -
<Buffer 5b 6f 62 6a 65 63 74 20 4f 62 6a 65 63 74 5d>
[ <Buffer 5b 6f 62 6a 65 63 74 20 4f 62 6a 65 63 74 5d> ]
{"type":"Buffer","data":[91,111,98,106,101,99,116,32,79,98,106,101,99,116,93]}
Does anyone have an idea why this is the output and how I can read the plain request body? (Expected output: {AccountName:"test",Password:"THISSHOULDBEENCRYPTED"})
I have looked for answers on many sites, but all solutions failed in the same error.
Thanks in advance!

Here is my fixed code:
var server=http.createServer(function(req,res){
if(req.url.startsWith("/api/")){
console.log(" - // - API CALL - // -")
switch (req.url.substr(4)) {
case '/createAccount':
if(req.method=="POST"){
req.setEncoding('utf8');
const rb = [];
req.on('data', (chunks)=>{
rb.push(chunks);
});
req.on('end', ()=>{
const body=JSON.parse(rb.join(""));
console.log(body)
res.writeHead(200,{"Content-Type":"application/json","Access-Control-Allow-Origin": "*"});
res.end(JSON.stringify({'error':false,'recieved':body}));
});
}else{...}
break;
default:
...
}
}else{...}
});
server.listen(7000);
console.log(`Listening at http://localhost:7000`);
And the fixed request:
fetch("http://localhost:7000/api/createAccount",{
method:"POST",
Headers:{'Content-Type': 'application/json'},
body:JSON.stringify({
AccountName:'test',
Password:'THISSHOULDBEENCRYPTED'
})
}).then(r=>r.json()).then(r=>console.log(r))
Basically, setting the Encoding to UTF8 and sending the body as a String will fix this issue.

Problems with converting buffer to JS object

I'm using the mosca library for a MQTT broker and this for MQTT client.
The client code looks like this:
client.on('connect', function () {
client.subscribe('presence')
var json = {
someVar: 888,
someNum: 234
}
client.publish('presence', JSON.stringify(json));
});
client.on('message', function (topic, message) {
var json = JSON.parse(message.toString())
console.log(json.someVar);
client.end();
});
The broker, where each published message is handled is here:
server.on('published', function(packet, client) {
console.log('Package received:')
console.log('Published', packet);
var stringBuf = packet.payload.toString('utf-8')
console.log('buffer to string:', stringBuf)
var stringify = JSON.stringify(packet.payload)
console.log('stringify:',stringify);
var json = JSON.parse(stringily);
}
The output for buffer to string: looks like this:
{"someVar":888,"someNum":234}
Which should be valid json. But whenever I try to access the json-object (after using JSON.parse()) it just returns undefined for the variables.
I've also tried to do
var buf = json.data;
console.log('buffer data:', but);
But that just returns a buffer array, which can't seem to convert either.
In the client function I can easily parse the string to a JS object, so I'm a bit lost as to what is happening.
Any suggestions? Thanks.

Like stdob-- said, you're trying to parse all incoming messages, example of output for your example:
Package received:
Published mqttjs_05cac308
Package received:
Published {"clientId":"mqttjs_05cac308","topic":"presence"}
Package received:
Published <Buffer 7b 22 73 6f 6d 65 56 61 72 22 3a 38 38 38 2c 22 73 6f 6d 65 4e 75 6d 22 3a 32 33 34 7d>
Package received:
Published {"clientId":"mqttjs_05cac308","topic":"presence"}
Package received:
Published mqttjs_05cac308
You can check for the topic to handle the presence message you need:
server.on('published', function (packet, client) {
if (packet.topic == 'presence') {
var stringBuf = packet.payload.toString('utf-8');
var obj = JSON.parse(stringBuf);
console.log(obj);
}
});
/* output: { someVar: 888, someNum: 234 } */

You receive an error because tries to parse all incoming messages, including service messages that do not contain the pure JSON data. Try this:
server.on('published', function(packet, client) {
console.log('Package received:')
console.log('Published', packet);
var stringBuf = packet.payload.toString('utf-8')
try {
var json = JSON.parse(stringBuf);
console.log( json );
} catch (e) {
console.log( stringBuf );
}
}

Pipe multipart form uploads to another server

I am trying to handle a POST request on my Node Express server to deal with multipart form uploads, in my case the user is uploading images.
I want to pipe the upload to another server via my Express app which is currently setup to use body parser, which I also see does not support multipart bodes and instead recommends using some other libraries.
I have seen multiparty but I am unsure how to use this with my client side application.
In my client side code I am posting a FormData object like so:
function create(data, name) {
var formData = new FormData();
formData.append('file', data, name);
return this.parentBase.one('photos').withHttpConfig({transformRequest: angular.identity}).customPOST(formData, undefined, undefined, {'Content-Type': undefined});
}
Note: I am using the Restangular library for AngularJS as documented here
So from what I understand looking at the multiparty docs, I have to handle the form upload events and act upon it further once the form has finished uploading.
The thing is, I was hoping I could just pipe the upload directly to another server. Beforehand my client side app was making direct calls to this other server, but I am now trying to get everything routed through Express, is this possible, or do I have to use something like multiparty?
The request documentation gives an example of using formData, but I am unsure how this would work with the multiparty examples I have seen. For example once the upload completes in Express using mutliparty, do I then have to construct another formData object to then make a further request with, or would I have to pipe each part to the other server?
I'm confused, please can someone help clear this up for me?
Thanks
EDIT
OK, I have taken a look at multer following #yarons comments and this seems to be the kind of thing I want to be using, I have attempted to use this with my express router setup as per below:
routes.js
var express = require('express'),
router = express.Router(),
customers = require('./customers.controller.js'),
multer = require('multer'),
upload = multer();
router.post('/customers/:customerId/photos/', upload.single('file'), customers.createPhoto);
controller.js
module.exports.createPhoto = function(req, res) {
console.log(req.file);
var options = prepareCustomersAPIHeaders(req);
options.formData = req.file;
request(options).pipe(res);
};
Logging out the req.file property in the above controller I see this:
{ fieldname: 'file',
originalname: '4da2e703044932e33b8ceec711c35582.jpg',
encoding: '7bit',
mimetype: 'image/png',
buffer: <Buffer 89 50 4e 47 0d 0a 1a 0a 00 00 00 0d 49 48 44 52 00 00 00 fa 00
00 00 fa 08 06 00 00 00 88 ec 5a 3d 00 00 20 00 49 44 41 54 78 5e ac bd f9 8f e
6 e9 7a ... >,
size: 105868 }
Which is what I am posting through from the client side code using:
var formData = new FormData();
formData.append('file', data, name);
return this.parentBase.one('photos').withHttpConfig({transformRequest: angular.identity}).customPOST(formData, undefined, undefined, {'Content-Type': undefined});
Is what I have tried sensible? Only it doesn't work, I get an error back from the server I'm trying post to. Beforehand where I was making this post request directly to the server it all worked fine, so I must have something wrong in my Express\Multer setup
EDIT 2
Ok, so after more hunting around I came across this article using multiparty which I have manager to get working in my setup like so:
var request = require('request'),
multiparty = require('multiparty'),
FormData = require('form-data');
module.exports.createPhoto = function(req, res) {
//console.log(req.file);
var options = prepareCustomersAPIHeaders(req),
form = new multiparty.Form();
options.headers['Transfer-Encoding'] = 'chunked';
form.on('part', function(part){
if(part.filename) {
var form = new FormData(), r;
form.append(part.name, part, {filename: part.filename, contentType: part['content-type']});
r = request(options, function(err, response, body){
res.status(response.statusCode).send(body);
});
r._form = form
}
});
form.on('error', function(error){
console.log(error);
});
form.parse(req);
};
This is now uploading the files for me as expected to my other server, whilst this solution works, I dont like the line:
r._form = form
Seems to be assigning a private form variable to the request object, plus I can't see anything that is documented in this way on multiparty pages
Can anyone offer any comments on this possible solution?

We use something like the following:
CLIENT
//HTML
<input type="file" ng-file-select uploader="info.uploadPath" />
//DIRECTIVES
// It is attached to <input type="file" /> element
.directive('ngFileSelect', function() {
return {
link: function($scope, $element) {
$element.bind('change', function() {
$scope.$emit('file:add', this.files ? this.files : this);
});
}
};
})
//OTHER
var uploadPath = '/api/things/' + $stateParams.thingId + '/add_photo'
var uploadInfo = {
headers: {
'Authorization': authToken
},
form: {
title: scope.info.name
}
}
//SERVICE:
$rootScope.$on('file:add', function(event, items) {
this.addToQueue(items);
}.bind(this));
...
addToQueue: function(items) {
var length = this.queue.length;
angular.forEach(items.length ? items : [items], function(item) {
var isValid = !this.filters.length ? true : !!this.filters.filter(function(filter) {
return filter.apply(this, [item]);
}, this).length;
if (isValid) {
item = new Item({
url: this.url,
alias: this.alias,
removeAfterUpload: this.removeAfterUpload,
uploader: this,
file: item
});
this.queue.push(item);
}
}, this);
this.uploadAll();
},
getNotUploadedItems: function() {
return this.queue.filter(function(item) {
return !item.isUploaded;
});
},
/**
* Upload a item from the queue
* #param {Item|Number} value
*/
uploadItem: function(value, uploadInfo) {
if (this.isUploading) {
return;
}
var index = angular.isObject(value) ? this.getIndexOfItem(value) : value;
var item = this.queue[index];
var transport = item.file._form ? '_iframeTransport' : '_xhrTransport';
this.isUploading = true;
this[transport](item, uploadInfo);
},
uploadAll: function(uploadInfo) {
var item = this.getNotUploadedItems()[0];
this._uploadNext = !!item;
this._uploadNext && this.uploadItem(item, uploadInfo);
},
_xhrTransport: function(item, uploadInfo) {
var xhr = new XMLHttpRequest();
var form = new FormData();
var that = this;
form.append(item.alias, item.file);
angular.forEach(uploadInfo.form, function(value, name) {
form.append(name, value);
});
xhr.upload.addEventListener('progress', function(event) {
var progress = event.lengthComputable ? event.loaded * 100 / event.total : 0;
that._scope.$emit('in:progress', item, Math.round(progress));
}, false);
xhr.addEventListener('load', function() {
xhr.status === 200 && that._scope.$emit('in:success', xhr, item);
xhr.status !== 200 && that._scope.$emit('in:error', xhr, item);
that._scope.$emit('in:complete', xhr, item);
}, false);
xhr.addEventListener('error', function() {
that._scope.$emit('in:error', xhr, item);
that._scope.$emit('in:complete', xhr, item);
}, false);
xhr.addEventListener('abort', function() {
that._scope.$emit('in:complete', xhr, item);
}, false);
this._scope.$emit('beforeupload', item);
xhr.open('POST', item.url, true);
angular.forEach(uploadInfo.headers, function(value, name) {
xhr.setRequestHeader(name, value);
});
xhr.send(form);
},
SERVER
//things.router
app.route('/api/things/:thingId/add_photo')
.post(things.uploadPhoto);
//things.controller
exports.uploadPhoto = function(req, res) {
var formidable = require('formidable');
var form = new formidable.IncomingForm();
form.parse(req, function(err, fields, files) {
var data = files.qqfile;
//actual file is at data.path
fs.createReadStream(data.path).pipe(request.put(uploadUrl));
}
}