We Keep Coding

Converting Base64 to File causes Excel to be Corrupted [duplicate]

Can someone please explain why decoding Base64 giving a broken pdf?
I need to find the way how to decode Base64 and get pdf out.
When i use this service
https://emn178.github.io/online-tools/base64_decode_file.html
I am able to pass Base64 and get file out without problem.
But when i do same in node.js I am getting empty (broken) file consistently.
I tried different packages like:
js-base64,
atob
and none of them worked, getting same empty file as the result.
Link to my code:
https://repl.it/#afiliptsov/FaroffGloriousFormula

You get a corrupted PDF, because:
According to the officially documentation, the
Base64.decode() function decodes Base64 value to UTF-8 string. As
you can see, this is the wrong function, because you need to decode
value as binary data.
The Base64.atob() function does exactly what you need, but you
make a mistake when saving data, because, according to the
officially documentation, by default the fs.writeFile()
function saves data as UTF-8, while you want to save binary data.
To properly decode Base64 value and store it as binary data, depending on your needs, you can choose one of the following methods:
require('js-base64').Base64.atob()
Decode the Base64 value using Base64.atob() and specify binary encoding when saving the file. This is useful only if you need to handle binary data. Unlike other methods you must install and load the "js-base64" module.
var bin = Base64.atob(stringToDecode);
// Your code to handle binary data
fs.writeFile('result_binary.pdf', bin, 'binary', error => {
if (error) {
throw error;
} else {
console.log('binary saved!');
}
});
Buffer.from
Convert the Base64 value to buffer using Buffer.from() and save it into file without specifying encoding. This is useful only if you need to handle buffer.
var buf = Buffer.from(stringToDecode, 'base64');
// Your code to handle buffer
fs.writeFile('result_buffer.pdf', buf, error => {
if (error) {
throw error;
} else {
console.log('buffer saved!');
}
});
The encoding option
If you do not need to read/modify the binary data or the buffer, just specify encoding option when saving file. This method is the simplest one and may be the fastest and most memory efficient.
fs.writeFile('result_base64.pdf', stringToDecode, 'base64', error => {
if (error) {
throw error;
} else {
console.log('base64 saved!');
}
});

A related issue for me which was solved by reading #victor 's answer is where an Express.js app get's a bas64 encoded PDF from an API and wants to return it to the client as a 'proper' pdf:
res.set({
'Content-Disposition' : 'attachment; filename='+ data.fileName,
'Content-Type': 'application/pdf',
});
res.send(Buffer.from(data.content, 'base64'));

Simple is the best! Just use fs package to save the base64 string to a file, remember that you have to set base64 for encoding option.
fs.writeFile('result_document.pdf', stringToDecode, 'base64', (error) => {
if (error) throw error;
console.log("Doc saved!");
});

http.get nodejs lambda wrong encoding for png image

I am using node.js (aws lambda) to perform a simple get request for a .png image on an external server. The data that I'm getting back seems to be encoded incorrectly, but I cant figure out how or why to get the correct encoding. the length and structure of the data seems to be the same (as shown in the images), but I just cant figure out why I dont get the same data...ive tried converting to base64, utf-8, etc.
my function:
http.get(FinalURL, function(res) {
res.on('data', function(d) {
body += d;
});
res.on('end', function() {
resolve(body);
});
res.on('error', function(e) {
resolve(e.message);
});
});
INCORRECT ENCODING
CORRECT ENCODING

I was able to get it to work by downloading it as binary using
res.setEncoding('binary');
i think this simplifies the characters and then its able to understand what its downloading

May I have a res.json and res.sendFile together in the same api endpoint in express?

I want upload the content of an excel file into the server in order to get its data and do some stuff...
I came up with the following code, however it seems like it is not working properly as the following error displays in the console Error: Can't set headers after they are sent.
The file is getting uploaded into the folder and the json message is being displayed... However I do not know if I am going to face any issue in the future...
Actually I just need the excel data no need for the excel being uploaded... Maybe you could give me a workaround, guys...
const router = express.Router();
const storage = multer.diskStorage({
destination(req, file, cb) {
cb(null, 'uploads/');
},
filename(req, file, cb) {
cb(
null,
`${file.fieldname}-${Date.now()}${path
.extname(file.originalname)
.toLowerCase()}`
);
},
});
const excelFilter = (req, file, cb) => {
if (
file.mimetype.includes('excel') ||
file.mimetype.includes('spreadsheetml')
) {
cb(null, true);
} else {
cb('Please upload only excel file.', false);
}
};
const upload = multer({
storage,
fileFilter: excelFilter,
});
router.post('/', upload.single('file'), (req, res) => {
var workbook = XLSX.readFile(req.file.path);
var sheet_name_list = workbook.SheetNames;
var xlData = XLSX.utils.sheet_to_json(workbook.Sheets[sheet_name_list[0]]);
res.json(xlData).sendFile(`/${req.file.path}`, { root: path.resolve() });
});

May I have a res.json and res.sendFile together in the same api endpoint in express?
No, you cannot. Each of those methods, sends a complete http response (including calling res.end() which terminates the http request) and you can only send one http response to each incoming request. The particular error you're getting has to do with the res.sendFile() trying to configure the response that it's getting ready to send and finding that the http response object has already been used for sending a response and can't be used again.
Ordinarily, if you wanted to sent two different pieces of data, you would just combine them into a single Javascript object and just call res.json() on the object that contains both pieces of data.
But, sending a binary file is not something you can easily put in a JSON package. You could construct a multipart response where one part was the JSON and one part was the file. You could JSON encode binary data (though that's inefficient). I presume there are probably some modules that would help you do that, but for most clients, that isn't what they are really expecting or equipped to handle.
The only way to a proper solution is for us to understand what client/server workflow you're trying to implement here and why you're trying to send back the same file that was just uploaded. There would normally not be a reason to do that since the client already has that data (they just uploaded it).

writestream and express for json object?

I might be out of depth but I really need something to work. I think a write/read stream will solve both my issues but I dont quite understand the syntax or whats required for it to work.
I read the stream handbook and thought i understood some of the basics but when I try to apply it to my situation, it seems to break down.
Currently I have this as the crux of my information.
function readDataTop (x) {
console.log("Read "+x[6]+" and Sent Cached Top Half");
jf.readFile( "loadedreports/top"+x[6], 'utf8', function (err, data) {
resT = data
});
};
Im using Jsonfile plugin for node which basically shortens the fs.write and makes it easier to write instead of constantly writing catch and try blocks for the fs.write and read.
Anyways, I want to implement a stream here but I am unsure of what would happen to my express end and how the object will be received.
I assume since its a stream express wont do anything to the object until it receives it? Or would I have to write a callback to also make sure when my function is called, the stream is complete before express sends the object off to fullfill the ajax request?
app.get('/:report/top', function(req, res) {
readDataTop(global[req.params.report]);
res.header("Content-Type", "application/json; charset=utf-8");
res.header("Cache-Control", "max-age=3600");
res.json(resT);
resT = 0;
});
I am hoping if I change the read part to a stream it will allievate two problems. The issue of sometimes receiving impartial json files when the browser makes the ajax call due to the read speed of larger json objects. (This might be the callback issue i need to solve but a stream should make it more consistent).
Then secondly when I load this node app, it needs to run 30+ write files while it gets the data from my DB. The goal was to disconnect the browser from the db side so node acts as the db by reading and writing. This due to an old SQL server that is being bombarded by a lot of requests already (stale data isnt an issue).
Any help on the syntax here?
Is there a tutorial I can see in code of someone piping an response into a write stream? (the mssql node I use puts the SQL response into an object and I need in JSON format).
function getDataTop (x) {
var connection = new sql.Connection(config, function(err) {
var request = new sql.Request(connection);
request.query(x[0], function(err, topres) {
jf.writeFile( "loadedreports/top"+x[6], topres, function(err) {
if(err) {
console.log(err);
} else {
console.log(x[6]+" top half was saved!");
}
});
});
});
};

Your problem is that you're not waiting for the file to load before sending the response. Use a callback:
function readDataTop(x, cb) {
console.log('Read ' + x[6] + ' and Sent Cached Top Half');
jf.readFile('loadedreports/top' + x[6], 'utf8', cb);
};
// ...
app.get('/:report/top', function(req, res) {
// you should really avoid using globals like this ...
readDataTop(global[req.params.report], function(err, obj) {
// setting the content-type is automatically done by `res.json()`
// cache the data here in-memory if you need to and check for its existence
// before `readDataTop`
res.header('Cache-Control', 'max-age=3600');
res.json(obj);
});
});

Giving binary messages a JSON-like type for identification purposes

is there a way to identify binary messages from a server by tagging them with some kind of type attribute?
Background: I'm working with node.js and I'm sending binary messages to my client. At the moment these are only pictures sent as blobs. So on clientside I'm testing incoming messages with instance of Blob and then do something with the pics.
But now I additionally want to send other files like .txt over the websocket protocol (for downloads). I feel like this is much simpler than HTTP requests etc. But now I have to make a difference on incoming binary messages if they are images or textfiles or sth. else.
The Chrome Developertools show me that my incoming images (blobs) have a type attribute that is an empty string. As I read this attribute is read only, so I'm looking for a solution to identify my binary messages like I can do it with JSON Objects...
Thanks in advance for every idea :)
Edit:
Here is one example for sending images to the client. I'm using the "ws" module for nodejs.
fs.readFile(path, function (err, data)
{
if (!err)
{
connection.send(data, {binary: true}, function(err)
{
if (!err) console.log("Server finished sending image to client.");
else console.log("Error while sending imageto client.");
});
}
else console.log("Error while loading image");
});
Where could I add some kind of metadata to the binary data? Base64 encoding is no possibility for me.

If it's only about the format, you might use the content-type header to specify what kind of file type it is: text/plain or image/png or whatever you want to use ...
Of course, you can extend this idea and use a custom-header to transport any information you would like to transfer.
Or, which is the most flexible solution: Send back a real JSON object, embed all the metadata you need, and provide the binary content encoded within the JSON (using any encoding you like and that is compatible to JSON, such as Base64), such as:
{
"filename": "foo.txt",
"content-type": "image/png",
"binary-data": "A5EBC7FFEFDCD8975BC3..."
}
To use your code snippet as base, you might implement it as follows:
fs.readFile(path, function (err, data) {
if (err) { throw err; }
connection.send({
filename: 'foo.txt',
contentType: 'image/png',
data: JSON.stringify(data)
}, { binary: false }, function (err) {
if (err) { throw err; }
console.log('Server finished sending image to client.');
});
});
Please note that I did not test the code, but your final code should look somewhat similar to this one.
Hope this helps.