We Keep Coding

S3 GetObject Returning Weird Encoding for HTML file

I am attempting to read an HTML file into a Lambda Edge function and then return it in the response body, but for some reason, I cannot get it to return the contents of the HTML file correctly.
Here is my code (simplified):
const AWS = require('aws-sdk');
const S3 = new AWS.S3({
signatureVersion: 'v4'
});
const { Body } = await S3.getObject({ Bucket: 'my-bucket', Key: 'index.html' }).promise();
console.log(Body.toString());
Instead of seeing <html... in the console log, I am seeing the dreaded question mark characters which implies (I think), bad encoding:
��Y�r#��x�`�,b�J�ٲ��NR�yIٮ�!!���"���n���޴��Is�>}�n4pF�_���de�nq�~�]� f�����v��۔*�㺮Ý� Hdž�<�! �c�5�1B��,#|Ŵ;ֶ�U����z� �Qi��j�0��V ���H���...etc
I have literally tried everything including, but not limited to:
Body.toString('utf-8');
Body.toString('ascii');
Body.toString('base64');
decoder.write(Body.toString('base64'));
and a lot more...
I think I must be missing something really obvious here as I cannot find anyone else facing the same issue. I thought it might be to do with the encryption but my other Lambda Edge function reads an image file without issues so I assume it has to be something to do with encoding that I haven't thought of.
UPDATE
I believe the issue may be related to the fact that the file is gzipped.
Here is a print of the response from S3:
{
AcceptRanges: 'bytes',
LastModified: 2023-02-17T19:44:41.000Z,
ContentLength: 1598,
ETag: 'some-key',
CacheControl: 'max-age=31536000',
ContentEncoding: 'gzip',
ContentType: 'text/html; charset=UTF-8',
ServerSideEncryption: 'AES256',
Metadata: { etag: 'some-key' },
Body: <Buffer 1f 8b 08 00 00 00 00 00 00 03 cd 59 db 72 23 b7 11 fd 15 78 f2 60 bb 2c 62 ee b7 8d c8 4a b2 d9 b2 b7 ca 4e 52 bb 79 49 d9 ae 14 06 e8 21 21 cd 0c a6 ... 1548 more bytes>
}

The issue was caused by the fact that the index.html file was gzipped. The solution to handling this is as follows:
const AWS = require('aws-sdk');
const zlib = require('zlib');
const S3 = new AWS.S3({
signatureVersion: 'v4'
});
const { Body } = await S3.getObject({ Bucket: 'my-bucket', Key: 'index.html' }).promise();
const body = zlib.unzipSync(Body).toString();
console.log(body);
However, given that this code is being deployed within a Lambda Edge function, speed is essential! With that in mind, it would/should be faster to actually utilise https.get as this will automatically unzip the contents of the file prior to returning the Buffer stream.

How to display playable Video Url from Blob

I am trying to download videos from a webDav server to my server. After that the videos should be viewable as well as playable on my client. For downloading I use the webdav library from npm.
The download of the videos works on my server, as well as the transmission of the data to my client. My Problem is, I can't play those videos on the generated website.
app.post("/someroute/api", async (req, res) => {
let result = [];
try {
const { username, password, targetUrl, data } = req.body;
const dataFull = `somedata_${data}`;
const client = createClient(targetUrl, {
username: username,
password: password,
});
const filesToDownload = await client.getDirectoryContents(`/${dataFull}`);
const downloadPromises = filesToDownload.map(async (file) => {
const fileBuffer = await client.getFileContents(file.filename);
result.push({name: file.basename, buffer: fileBuffer});
});
await Promise.all(downloadPromises);
} catch (error) {
console.error(error);
res.status(500).json({ error: "Error retrieving directory contents" });
}
console.log(result);
res.send(result);});
The result looks like this :
{ name: 'butterfly slow motion [oAI-OqTmJj4].mp4',
buffer: <Buffer 00 00 00 18 66 74 79 70 6d 70 34 32 00 00 00 00 69 73 6f 6d 6d 70 34 32 00 00 40 d7 6d 6f 6f 76 00 00 00 6c 6d 76 68 64
00 00 00 00 d8 cd f6 85 d8 cd ... 4073099 more bytes> }
On the client side I have this function to request the videos and store them in a useState.
useEffect(() => {
if (selectedData) {
getDataVideos();
}
}, [selectedData]);
const getDataVideos = async () => {
try {
const response = await fetch("/someroute/api", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
username: username,
password: password,
targetUrl: targetUrl,
data: selectedData,
}),
});
const content = await response.json();
console.log(content);
setSelectedDataVideos(content);
} catch (error) {
console.error(error);
}
};
the console.log() of the selectedDataVideos is :
Array (1) 0 Object
buffer: {type: "Buffer", data: Array}
name: "butterfly slow motion [oAI-OqTmJj4].mp4"
Later I transform the videos into a blob using this method and safe the result into a usestate
const [tempUrls, settempUrls] = useState([]);
, here a snapshot of it :
const blob = new Blob([video.buffer.data], { type: "video/mp4" });
console.log("Videoslide 2 blob")
console.log(blob)
const url = URL.createObjectURL(blob);
return {name:video.name,url:url};
The blobs are saved in a usestate const [tempUrls, settempUrls] = useState([]);
The blob itself looks like this :
[Log] Blob {size: 14484621, type: "video/mp4", slice: function,
stream: function, text: function, …}
An index the tempUrls looks finished like this :
[Log] {name: "butterfly slow motion [oAI-OqTmJj4].mp4", url:
"blob:http:// localhost:8081/ed893b3d-298f-40c4-b4c0-1c3ac688aa43"}
The space in http:// localhost is for removing the hyperlink here.
For display the video and to play it I have this :
{tempUrls.map(video =>
<div key={video.name} style={{ flex: 2, padding: "15px 20px" }}>
<video
src={video.url}
controls
type="video/mp4"
>
</video>
Only the control placeholder is displayed but I can't play the video. Also, when I open the web page in Firefox this is displayed. "No video with supported format and MIME type found". But I am not clear where exactly I made the mistake when converting. Actually I converted the array buffer to a blob of type mp4/video and then to a URL. Then saved it as object {filename, url} in a usestate and from this usestate the videos should be played/displayed.
Does anyone here possibly see the error, I've been sitting on it a bit longer without finding it. The video itself is not corrupted, ive stored it in a docker next cloud server and can play it there without problems.

Solved it by using
import { Buffer } from "buffer";
from npm to convert the arraybuffer.
let buffer = Buffer.from(item.buffer.data);
const blob = new Blob([buffer], { type: "video/mp4" });
const url = URL.createObjectURL(blob);
result.push({ name: item.name, url: url });

Nodejs: make a GET request with users parameters

I'm new in nodejs. I'm making a express server. I recive some parameter by url, and I wnat to make a request to another server with the parameters in the recived url.
var express = require("express"),
app = express(),
bodyParser = require("body-parser"),
methodOverride = require("method-override");
request = require('request');
Client = require('node-rest-client').Client;
app.use(bodyParser.urlencoded({ extended: false }));
app.use(bodyParser.json());
app.use(methodOverride());
var router = express.Router();
app.use(router);
router.get('/get_metric_names_by_pattern', function(req, res) {
console.log('get_metric_names_by_pattern '+req.url);
console.log(req.query.pattern);
//console.log(req.parameters);
res.header('Access-Control-Allow-Origin', '*');
var client = new Client();
var salida;
console.log(req.params.name);
//Creamos la variable para los parametros
var args = {
parameters: {pattern: "OEE"}
//headers: { "test-header": "client-api" }
};
console.log(args);
console.log(args.parameters.pattern);
client.get('http:// localhost:8000/get_metric_names_by_pattern/?pattern=:pattern', req.query.pattern,
//http:// localhost:8000/get_metric_names_by_pattern/?pattern=OEE
function (data, response) {
//console.log(args.parameters.pattern);
console.log(data);
//var pattern = req.query.pattern;
//data = data.toString('utf8'); //Transformamos los datos a string para que los reconozca el navegador
//console.log(data);
salida = JSON.parse(data);
//console.log("Datos:", salida.date);
res.send(salida); //Lo enviamos al navegador
//console.log(salida);
});
});
I do recive the paramteter from the client (in this case 'OEE') but I find no way to insert that string into client.get('http:// localhost:8000/get_metric_names_by_pattern/?pattern=:pattern', req.query.pattern, .
The request should look like the commented line under that one ( //http://localhost:8000/get_metric_names_by_pattern/?pattern=OEE) and from that request I'm recibing a json, just like this :
["PanelPC0_OEE", "empacadora1.OEE", "empacadora1.OEEE", "empacadora1.OEE_Avaible", "empacadora1.OEE_Performance", "empacadora1.OEE_quality"]
The console.log look like this:
node servicio.js
Servidor corriendo en: http:// localhost:3000
get_metric_names_by_pattern /get_metric_names_by_pattern/?pattern=OEE
OEE
undefined
{ parameters: { pattern: 'OEE' } }
OEE
Buffer 5b 5d
and it should look like :
node servicio.js
Servidor corriendo en: http:// localhost:3000
get_metric_names_by_pattern /get_metric_names_by_pattern/?pattern=OEE
OEE
undefined
{ parameters: { pattern: 'OEE' } }
OEE
Buffer 5b 22 50 61 6e 65 6c 50 43 30 5f 4f 45 45 22 2c 20 22 74 65 61 6d 66 6f
6f 64 73 2e 62 6f 67 6f 74 61 2e 6d 61 72 67 61 72 69 6e 61 73 2e 65 6d 70 61 ..
.
Please, I need to know how to insert that parameter into the new url. Sorry if this post is to long or obvius. Thanks in advance

According to the documentation, to insert a parameter or argument you should use:
${param}
like this example:
var args = {
path: { "id": 120, "arg1": "hello", "arg2": "world" },
parameters: { arg1: "hello", arg2: "world" },
headers: { "test-header": "client-api" }
};
client.get("http://remote.site/rest/json/${id}/method?arg1=${arg1}&arg2=${arg2}", args,
function (data, response) {
console.log(data);
console.log(response);
});
so in your case you should try to use:
var args = {
parameters: { pattern: req.query.pattern },
};
client.get('http:// localhost:8000/get_metric_names_by_pattern/?pattern=${pattern}', args,
function(data, response){
});

although it is not pretty... the comment of #mrwillihog gave me this idea, wich actually works!
urln = 'http://localhost:8000/get_metric_names_by_pattern/?pattern='+req.query.pattern;
console.log(urln);
client.get(urln, function (data, response) { });

Problems with converting buffer to JS object

I'm using the mosca library for a MQTT broker and this for MQTT client.
The client code looks like this:
client.on('connect', function () {
client.subscribe('presence')
var json = {
someVar: 888,
someNum: 234
}
client.publish('presence', JSON.stringify(json));
});
client.on('message', function (topic, message) {
var json = JSON.parse(message.toString())
console.log(json.someVar);
client.end();
});
The broker, where each published message is handled is here:
server.on('published', function(packet, client) {
console.log('Package received:')
console.log('Published', packet);
var stringBuf = packet.payload.toString('utf-8')
console.log('buffer to string:', stringBuf)
var stringify = JSON.stringify(packet.payload)
console.log('stringify:',stringify);
var json = JSON.parse(stringily);
}
The output for buffer to string: looks like this:
{"someVar":888,"someNum":234}
Which should be valid json. But whenever I try to access the json-object (after using JSON.parse()) it just returns undefined for the variables.
I've also tried to do
var buf = json.data;
console.log('buffer data:', but);
But that just returns a buffer array, which can't seem to convert either.
In the client function I can easily parse the string to a JS object, so I'm a bit lost as to what is happening.
Any suggestions? Thanks.

Like stdob-- said, you're trying to parse all incoming messages, example of output for your example:
Package received:
Published mqttjs_05cac308
Package received:
Published {"clientId":"mqttjs_05cac308","topic":"presence"}
Package received:
Published <Buffer 7b 22 73 6f 6d 65 56 61 72 22 3a 38 38 38 2c 22 73 6f 6d 65 4e 75 6d 22 3a 32 33 34 7d>
Package received:
Published {"clientId":"mqttjs_05cac308","topic":"presence"}
Package received:
Published mqttjs_05cac308
You can check for the topic to handle the presence message you need:
server.on('published', function (packet, client) {
if (packet.topic == 'presence') {
var stringBuf = packet.payload.toString('utf-8');
var obj = JSON.parse(stringBuf);
console.log(obj);
}
});
/* output: { someVar: 888, someNum: 234 } */

You receive an error because tries to parse all incoming messages, including service messages that do not contain the pure JSON data. Try this:
server.on('published', function(packet, client) {
console.log('Package received:')
console.log('Published', packet);
var stringBuf = packet.payload.toString('utf-8')
try {
var json = JSON.parse(stringBuf);
console.log( json );
} catch (e) {
console.log( stringBuf );
}
}

Pipe multipart form uploads to another server

I am trying to handle a POST request on my Node Express server to deal with multipart form uploads, in my case the user is uploading images.
I want to pipe the upload to another server via my Express app which is currently setup to use body parser, which I also see does not support multipart bodes and instead recommends using some other libraries.
I have seen multiparty but I am unsure how to use this with my client side application.
In my client side code I am posting a FormData object like so:
function create(data, name) {
var formData = new FormData();
formData.append('file', data, name);
return this.parentBase.one('photos').withHttpConfig({transformRequest: angular.identity}).customPOST(formData, undefined, undefined, {'Content-Type': undefined});
}
Note: I am using the Restangular library for AngularJS as documented here
So from what I understand looking at the multiparty docs, I have to handle the form upload events and act upon it further once the form has finished uploading.
The thing is, I was hoping I could just pipe the upload directly to another server. Beforehand my client side app was making direct calls to this other server, but I am now trying to get everything routed through Express, is this possible, or do I have to use something like multiparty?
The request documentation gives an example of using formData, but I am unsure how this would work with the multiparty examples I have seen. For example once the upload completes in Express using mutliparty, do I then have to construct another formData object to then make a further request with, or would I have to pipe each part to the other server?
I'm confused, please can someone help clear this up for me?
Thanks
EDIT
OK, I have taken a look at multer following #yarons comments and this seems to be the kind of thing I want to be using, I have attempted to use this with my express router setup as per below:
routes.js
var express = require('express'),
router = express.Router(),
customers = require('./customers.controller.js'),
multer = require('multer'),
upload = multer();
router.post('/customers/:customerId/photos/', upload.single('file'), customers.createPhoto);
controller.js
module.exports.createPhoto = function(req, res) {
console.log(req.file);
var options = prepareCustomersAPIHeaders(req);
options.formData = req.file;
request(options).pipe(res);
};
Logging out the req.file property in the above controller I see this:
{ fieldname: 'file',
originalname: '4da2e703044932e33b8ceec711c35582.jpg',
encoding: '7bit',
mimetype: 'image/png',
buffer: <Buffer 89 50 4e 47 0d 0a 1a 0a 00 00 00 0d 49 48 44 52 00 00 00 fa 00
00 00 fa 08 06 00 00 00 88 ec 5a 3d 00 00 20 00 49 44 41 54 78 5e ac bd f9 8f e
6 e9 7a ... >,
size: 105868 }
Which is what I am posting through from the client side code using:
var formData = new FormData();
formData.append('file', data, name);
return this.parentBase.one('photos').withHttpConfig({transformRequest: angular.identity}).customPOST(formData, undefined, undefined, {'Content-Type': undefined});
Is what I have tried sensible? Only it doesn't work, I get an error back from the server I'm trying post to. Beforehand where I was making this post request directly to the server it all worked fine, so I must have something wrong in my Express\Multer setup
EDIT 2
Ok, so after more hunting around I came across this article using multiparty which I have manager to get working in my setup like so:
var request = require('request'),
multiparty = require('multiparty'),
FormData = require('form-data');
module.exports.createPhoto = function(req, res) {
//console.log(req.file);
var options = prepareCustomersAPIHeaders(req),
form = new multiparty.Form();
options.headers['Transfer-Encoding'] = 'chunked';
form.on('part', function(part){
if(part.filename) {
var form = new FormData(), r;
form.append(part.name, part, {filename: part.filename, contentType: part['content-type']});
r = request(options, function(err, response, body){
res.status(response.statusCode).send(body);
});
r._form = form
}
});
form.on('error', function(error){
console.log(error);
});
form.parse(req);
};
This is now uploading the files for me as expected to my other server, whilst this solution works, I dont like the line:
r._form = form
Seems to be assigning a private form variable to the request object, plus I can't see anything that is documented in this way on multiparty pages
Can anyone offer any comments on this possible solution?

We use something like the following:
CLIENT
//HTML
<input type="file" ng-file-select uploader="info.uploadPath" />
//DIRECTIVES
// It is attached to <input type="file" /> element
.directive('ngFileSelect', function() {
return {
link: function($scope, $element) {
$element.bind('change', function() {
$scope.$emit('file:add', this.files ? this.files : this);
});
}
};
})
//OTHER
var uploadPath = '/api/things/' + $stateParams.thingId + '/add_photo'
var uploadInfo = {
headers: {
'Authorization': authToken
},
form: {
title: scope.info.name
}
}
//SERVICE:
$rootScope.$on('file:add', function(event, items) {
this.addToQueue(items);
}.bind(this));
...
addToQueue: function(items) {
var length = this.queue.length;
angular.forEach(items.length ? items : [items], function(item) {
var isValid = !this.filters.length ? true : !!this.filters.filter(function(filter) {
return filter.apply(this, [item]);
}, this).length;
if (isValid) {
item = new Item({
url: this.url,
alias: this.alias,
removeAfterUpload: this.removeAfterUpload,
uploader: this,
file: item
});
this.queue.push(item);
}
}, this);
this.uploadAll();
},
getNotUploadedItems: function() {
return this.queue.filter(function(item) {
return !item.isUploaded;
});
},
/**
* Upload a item from the queue
* #param {Item|Number} value
*/
uploadItem: function(value, uploadInfo) {
if (this.isUploading) {
return;
}
var index = angular.isObject(value) ? this.getIndexOfItem(value) : value;
var item = this.queue[index];
var transport = item.file._form ? '_iframeTransport' : '_xhrTransport';
this.isUploading = true;
this[transport](item, uploadInfo);
},
uploadAll: function(uploadInfo) {
var item = this.getNotUploadedItems()[0];
this._uploadNext = !!item;
this._uploadNext && this.uploadItem(item, uploadInfo);
},
_xhrTransport: function(item, uploadInfo) {
var xhr = new XMLHttpRequest();
var form = new FormData();
var that = this;
form.append(item.alias, item.file);
angular.forEach(uploadInfo.form, function(value, name) {
form.append(name, value);
});
xhr.upload.addEventListener('progress', function(event) {
var progress = event.lengthComputable ? event.loaded * 100 / event.total : 0;
that._scope.$emit('in:progress', item, Math.round(progress));
}, false);
xhr.addEventListener('load', function() {
xhr.status === 200 && that._scope.$emit('in:success', xhr, item);
xhr.status !== 200 && that._scope.$emit('in:error', xhr, item);
that._scope.$emit('in:complete', xhr, item);
}, false);
xhr.addEventListener('error', function() {
that._scope.$emit('in:error', xhr, item);
that._scope.$emit('in:complete', xhr, item);
}, false);
xhr.addEventListener('abort', function() {
that._scope.$emit('in:complete', xhr, item);
}, false);
this._scope.$emit('beforeupload', item);
xhr.open('POST', item.url, true);
angular.forEach(uploadInfo.headers, function(value, name) {
xhr.setRequestHeader(name, value);
});
xhr.send(form);
},
SERVER
//things.router
app.route('/api/things/:thingId/add_photo')
.post(things.uploadPhoto);
//things.controller
exports.uploadPhoto = function(req, res) {
var formidable = require('formidable');
var form = new formidable.IncomingForm();
form.parse(req, function(err, fields, files) {
var data = files.qqfile;
//actual file is at data.path
fs.createReadStream(data.path).pipe(request.put(uploadUrl));
}
}