I am using node.js (aws lambda) to perform a simple get request for a .png image on an external server. The data that I'm getting back seems to be encoded incorrectly, but I cant figure out how or why to get the correct encoding. the length and structure of the data seems to be the same (as shown in the images), but I just cant figure out why I dont get the same data...ive tried converting to base64, utf-8, etc.
my function:
http.get(FinalURL, function(res) {
res.on('data', function(d) {
body += d;
});
res.on('end', function() {
resolve(body);
});
res.on('error', function(e) {
resolve(e.message);
});
});
INCORRECT ENCODING
CORRECT ENCODING
I was able to get it to work by downloading it as binary using
res.setEncoding('binary');
i think this simplifies the characters and then its able to understand what its downloading
Related
Can someone please explain why decoding Base64 giving a broken pdf?
I need to find the way how to decode Base64 and get pdf out.
When i use this service
https://emn178.github.io/online-tools/base64_decode_file.html
I am able to pass Base64 and get file out without problem.
But when i do same in node.js I am getting empty (broken) file consistently.
I tried different packages like:
js-base64,
atob
and none of them worked, getting same empty file as the result.
Link to my code:
https://repl.it/#afiliptsov/FaroffGloriousFormula
You get a corrupted PDF, because:
According to the officially documentation, the
Base64.decode() function decodes Base64 value to UTF-8 string. As
you can see, this is the wrong function, because you need to decode
value as binary data.
The Base64.atob() function does exactly what you need, but you
make a mistake when saving data, because, according to the
officially documentation, by default the fs.writeFile()
function saves data as UTF-8, while you want to save binary data.
To properly decode Base64 value and store it as binary data, depending on your needs, you can choose one of the following methods:
require('js-base64').Base64.atob()
Decode the Base64 value using Base64.atob() and specify binary encoding when saving the file. This is useful only if you need to handle binary data. Unlike other methods you must install and load the "js-base64" module.
var bin = Base64.atob(stringToDecode);
// Your code to handle binary data
fs.writeFile('result_binary.pdf', bin, 'binary', error => {
if (error) {
throw error;
} else {
console.log('binary saved!');
}
});
Buffer.from
Convert the Base64 value to buffer using Buffer.from() and save it into file without specifying encoding. This is useful only if you need to handle buffer.
var buf = Buffer.from(stringToDecode, 'base64');
// Your code to handle buffer
fs.writeFile('result_buffer.pdf', buf, error => {
if (error) {
throw error;
} else {
console.log('buffer saved!');
}
});
The encoding option
If you do not need to read/modify the binary data or the buffer, just specify encoding option when saving file. This method is the simplest one and may be the fastest and most memory efficient.
fs.writeFile('result_base64.pdf', stringToDecode, 'base64', error => {
if (error) {
throw error;
} else {
console.log('base64 saved!');
}
});
A related issue for me which was solved by reading #victor 's answer is where an Express.js app get's a bas64 encoded PDF from an API and wants to return it to the client as a 'proper' pdf:
res.set({
'Content-Disposition' : 'attachment; filename='+ data.fileName,
'Content-Type': 'application/pdf',
});
res.send(Buffer.from(data.content, 'base64'));
Simple is the best! Just use fs package to save the base64 string to a file, remember that you have to set base64 for encoding option.
fs.writeFile('result_document.pdf', stringToDecode, 'base64', (error) => {
if (error) throw error;
console.log("Doc saved!");
});
I am trying to send an image encoded in base64 to a socket.io server. Simply emitting it doesn't seem to work. However, sending shorter strings works fine, the problem is with sending a really long base64 encoded image.
// this works
this.socket.emit('imageUpload', {captureType: "documentBack", data: "simple text"})
// this doesn't
this.socket.emit('imageUpload', {captureType: "documentBack", data: "really long base64 encoded image"})
I have a frontend built with React JS and backend built with Node.
Following the comment by #Zac, I've implemented socket.io-stream like the documentation instructs but I still can't see the request on the backend. The data appears to have been emitted from the client but I can't see the data on the event handler.
This is what I have so far:
// Client (React.js)
import ss from 'socket.io-stream';
stream = ss.createStream();
ss(this.socket).emit('imageUpload', this.stream, {
captureType: 'documentBack',
data: imageData,
});
// Server (Node.js)
socket.on("imageUpload", (stream, imageData) => {
console.log("Image uploaded");
// console.log(stream);
// console.log(imageData);
socket.broadcast.to(sessionId).emit("imageDownload", imageData);
});
I am working on a simple AWS Lambda function in Javascript (Node 6.x) which should 'proxy' an RSS of an italian news provider.
This is the code of the function:
var http = require("http")
exports.handler = (event, context, callback) => {
http.get("http://www.milanotoday.it/rss/", (response) => {
response.setEncoding("utf8")
let xml = ""
response.on("data", (chunk) => { xml += chunk })
response.on("end", () => { callback(null, xml) })
})
}
It works, or at least it loads the response inside the xml variable.
I can't get why the string is something like this:
"<?xml version=\"1.0\" encoding=\"UTF-8\" ?>\n<rss version=\"2.0\" xmlns:content=\"http://purl.org/rss/1.0/modules/content/\"...
It seems to be sort of 'escaped'.
Can somebody help me?
Thank you in advance...
http.get is probably working just fine. If you run your code outside of the AWS ecosystem, you'll see that you are getting unescaped XML from http.get. The problem is that Lambda wants you to return JSON so when you pass the string to the callback it gets escaped.
If you are running this function through API Gateway you can do a transform in the integration response. The way to do this is to return an object from your lambda:
callback(null, {myXML: xml}))
Then in the API Gateway go to the integration response area under your GET (or POST) resource and click the arrow next to the 200 response. This should reveal an area for body mappings. You want to add a mapping for application/xml and then add something like:
#set($inputRoot = $input.path('$'))
$inputRoot.myXML
This should get you nice clean XML. It's a little hard to describe so I'll post a screen shot that might help:
Double quotes have to be escaped inside a double quoted string (Makes sense, right?). Now you just have to parse the XML, using for instance xml2js
I am using a third party api to get some images the response gives me this. I don't think this is base64?
"����\u0000\u0010JFIF\u0000\u0001\u0001\u0000\u0000\u0001\u0000\u0001\u0000\u0000��\u0000C\u0000\b\u0006\u0006\u0007\u0006\u0005\b\u0007\u0007\u0007\t\t\b\n\f\u0014\r\f\u000b\u000b\f\u0019\u0012\u0013\u000f\u0014\u001d\u001a\u001f\u001e\u001d\u001a\u001c\u001c $.' \",#\u001c\u001c(7),01444\u001f'9=82<.342��\u0000C\u0001\t\t\t\f\u000b\f\u0018\r\r\u00182!\u001c!22222222222222222222222222222222222222222222222222��\u0000\u0011\b\u0002!\u0002&\u0003\u0001\"\u0000\u0002\u0011\u0001\u0003\u0011\u0001��\u0000\u001f\u0000\u0000\u0001\u0005\u0001\u0001\u0001\u0001\u0001\u0001\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0001\u0002\u0003\u0004\u0005\u0006\u0007\b\t\n\u000b��\u0000"
The code that makes the request.
unirest.get("MYAPIROUTE")
.header("X-Mashape-Key", "MYKEY")
.end(function (result) {
console.log(result.status, result.headers, result.body);
res.send(result.body);
});
My question is, with node.js how do I decode this so I can send the client a proper image?
Resolution:
unirest.get("MYAPIROUTE")
.header("X-Mashape-Key", "MYKEY")
.end(function (result) {
console.log(result.status, result.headers, result.body);
if(result.status==200) {
var buffer = (new Buffer(result.body.toString()));
res.end(buffer.toString("base64")); // output content as response body
require('fs').writeFileSync('/some/public/folder/md5HashOfRequestedUrl.jpg', buffer); // also write it to file
delete buffer;
return;
}
res.writeHead(result.status, result.headers);
res.write(result.body);
res.end();
});
reference: http://nodejs.org/api/buffer.html
let's say it's an image. so why not to try to set
<img src="http://your-site.com/some/public/folder/md5HashOfRequestedUrl.jpg">
also You can write response to file in temporary public folder to avoid doing same requests to somewhere.
The 3rd-party API should have the data-type of what the expected response type is, so that we can figure out what the format is that needs to be decoded...
My guess is that its returning a UTF-8 string, try this modules' decode function to decode the string: https://www.npmjs.com/package/utf8
is there a way to identify binary messages from a server by tagging them with some kind of type attribute?
Background: I'm working with node.js and I'm sending binary messages to my client. At the moment these are only pictures sent as blobs. So on clientside I'm testing incoming messages with instance of Blob and then do something with the pics.
But now I additionally want to send other files like .txt over the websocket protocol (for downloads). I feel like this is much simpler than HTTP requests etc. But now I have to make a difference on incoming binary messages if they are images or textfiles or sth. else.
The Chrome Developertools show me that my incoming images (blobs) have a type attribute that is an empty string. As I read this attribute is read only, so I'm looking for a solution to identify my binary messages like I can do it with JSON Objects...
Thanks in advance for every idea :)
Edit:
Here is one example for sending images to the client. I'm using the "ws" module for nodejs.
fs.readFile(path, function (err, data)
{
if (!err)
{
connection.send(data, {binary: true}, function(err)
{
if (!err) console.log("Server finished sending image to client.");
else console.log("Error while sending imageto client.");
});
}
else console.log("Error while loading image");
});
Where could I add some kind of metadata to the binary data? Base64 encoding is no possibility for me.
If it's only about the format, you might use the content-type header to specify what kind of file type it is: text/plain or image/png or whatever you want to use ...
Of course, you can extend this idea and use a custom-header to transport any information you would like to transfer.
Or, which is the most flexible solution: Send back a real JSON object, embed all the metadata you need, and provide the binary content encoded within the JSON (using any encoding you like and that is compatible to JSON, such as Base64), such as:
{
"filename": "foo.txt",
"content-type": "image/png",
"binary-data": "A5EBC7FFEFDCD8975BC3..."
}
To use your code snippet as base, you might implement it as follows:
fs.readFile(path, function (err, data) {
if (err) { throw err; }
connection.send({
filename: 'foo.txt',
contentType: 'image/png',
data: JSON.stringify(data)
}, { binary: false }, function (err) {
if (err) { throw err; }
console.log('Server finished sending image to client.');
});
});
Please note that I did not test the code, but your final code should look somewhat similar to this one.
Hope this helps.