I'm converting my code from PHP to node.js, and I need to convert a part of my code where there's the gzuncompress() function.
For that I'm using zlib.inflateSync. But I don't know which encoding I should use to create the buffer and so to have the same result of php
Here's what I do with php to decompress a string:
gzuncompress(substr($this->raw, 8))
and here's what I've tried in node.js
zlib.inflateSync(new Buffer(this.raw.substr(8), "encoding"))
So what encoding should I use to make zlib.inflateSync returns the same data as gzuncompress ?
I am not sure about what would be exact encoding here however this repo has some PHP translations for node.js (https://github.com/gamalielmendez/node-fpdf/blob/master/src/PHP_CoreFunctions.js). According to this repo, the following could work:
const gzuncompress = (data) => {
const chunk = (!Buffer.isBuffer(data)) ? Buffer.from(data, 'binary') : data
const Z1 = zlib.inflateSync(chunk)
return Z1.toString('binary')//'ascii'
}
Related
Java code that uses
Apache Commons library to generate signature:
byte[] md5 = DigestUtils.md5("test");
String signature = Base64.encodeBase64String(md5);
System.out.println(signature);
// CY9rzUYh03PK3k6DJie09g==
Javascript code I am trying to write to get the same output:
const md5 = CryptoJS.MD5("test");
const signature = btoa(md5);
console.log(signature);
// MDk4ZjZiY2Q0NjIxZDM3M2NhZGU0ZTgzMjYyN2I0ZjY=
I know how to get the same output if I change the Java code like this:
String md5 = DigestUtils.md5Hex("test");
String signature = Base64.encodeBase64String(md5.getBytes(StandardCharsets.UTF_8));
System.out.println(signature);
// MDk4ZjZiY2Q0NjIxZDM3M2NhZGU0ZTgzMjYyN2I0ZjY=
But unfortunately I am not allowed to modify Java code, so how can I modify Javascript code instead, to get the same output as Java code?
I think I found the solution by myself:
const md5 = CryptoJS.MD5("test");
const signature = md5.toString(CryptoJS.enc.Base64);
console.log(signature);
// CY9rzUYh03PK3k6DJie09g==
I am relatively new to JavaScript and I want to get the hash of a file, and would like to better understand the mechanism and code behind the process.
So, what I need: An MD5 or SHA-256 hash of an uploaded file to my website.
My understanding of how this works: A file is uploaded via an HTML input tag of type 'file', after which it is converted to a binary string, which is consequently hashed.
What I have so far: I have managed to get the hash of an input of type 'text', and also, somehow, the hash of an uploaded file, although the hash did not match with websites I looked at online, so I'm guessing it hashed some other details of the file, instead of the binary string.
Question 1: Am I correct in my understanding of how a file is hashed? Meaning, is it the binary string that gets hashed?
Question 2: What should my code look like to upload a file, hash it, and display the output?
Thank you in advance.
Basically yes, that's how it works.
But, to generate such hash, you don't need to do the conversion to string yourself. Instead, let the SubtleCrypto API handle it itself, and just pass an ArrayBuffer of your file.
async function getHash(blob, algo = "SHA-256") {
// convert your Blob to an ArrayBuffer
// could also use a FileRedaer for this for browsers that don't support Response API
const buf = await new Response(blob).arrayBuffer();
const hash = await crypto.subtle.digest(algo, buf);
let result = '';
const view = new DataView(hash);
for (let i = 0; i < hash.byteLength; i += 4) {
result += view.getUint32(i).toString(16).padStart(2, '0');
}
return result;
}
inp.onchange = e => {
getHash(inp.files[0]).then(console.log);
};
<input id="inp" type="file">
I have a file.txt that I need to get to my script and parse via d3.request.
The content of file is encoded with windows-1250 encoding and has extra lines to be deleted, so that only lines starting with 'Date' and '2017' should pass.
So far I have been using cli solution to grep text file (removing extra lines) and use d3 dsv2json to get clean json which can be loaded.
$ grep -E '^(Date|2017)' file.txt > file.csv
$ dsv2json -r ';' --input-encoding windows-1250 --output-encoding utf-8 < file.csv > file.json
However now i need to do these operations programmatically once txt file is loaded in the script via d3.request.
d3.request('file.txt')
.mimeType('text/csv')
.response(function(response) {
// response.responseText
})
TheresponseText gives me raw data with wrong encoding and extra lines. How to fix it so it will produce clean json at the end?
After further investigation I have found solution.
To decode file I used solution from here with TextDecoder. In order to do this d3.request.response should be set to arraybuffer.
function decode(response) {
const dataView = new DataView(response);
const decoder = new TextDecoder("windows-1250");
const decodedString = decoder.decode(dataView);
return decodedString
}
To filter out extra lines I used following step:
function filterData(rawData) {
return rawData
.split(/\n/)
.filter(row => (row.startsWith('Data') || row.startsWith('2017')))
.join('\n')
}
So finally, in context of d3.request:
d3.request('file.txt')
.header('Content-Type', 'text/csv;charset=windows-1250')
.mimeType('text/csv')
.responseType('arraybuffer')
.response(function(xhr) {
const decoded = decode(xhr.response)
const filtered = filterData(decoded)
const json = d3.dsvFormat(';').parse(filtered)
return json
})
.get()
Over my node.js application I decode base64 encoded images using the following line of code:
const fileDataDecoded = Buffer.from(base64EncodedfileData,'base64');
So far I can write a file with the following piece of code:
const fs = require('fs');
....
const fileDataDecoded = Buffer.from(base64EncodedfileData,'base64');
fs.writeFile("/tmp/test.png", fileDataDecoded, function(err) {
//Handle Error
});
Now what I want to achieve is the decoded files via the buffer to get written into a file via streams in order to acheive better efficiency on the executed application.
In other words I want the filedata to get written at the same time that the data is getting base64-decoded, in order to write large files with an efficient way. If is not possible via a Buffer to stream base64 decoded data then I would like to know how is possible to decode base64 data.
You can create a readable stream and push the image buffer. Then pipe it to a writeable stream like this.
var base64 = 'iVBORw0KGgoAAAANSUhEUgAAAJAAAACQCAYAAADnRuK4AAAMcklEQVR42u2deVBUVxbGGxHQBOPEZdxr3KJJcO+INGsDDXGJC8RWBKMGFIhRGVxYLDCgINAoq9J2I2jiGCPDKGClZvKXmaWmZq1U4lRiJslkU2dMTWapmiQVWzlzz+vXLCrQ0G/vc6q+egVF1Tvvfr8+5977mvd0OgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKisEFgE9RUdEwc0uLL0k+oQfohWqgMV4tGs4lTaGoQE/QG8XCxCXX6+czI+LbmqfFdzTOWH7ZOl1+nZFB8l4zjj16gF7055XsgaXSlVhch32bqaPxTaZPmP7HdIfpezkUd+W0YiTTGNzhPUAv3kRvXPC4PFMMPCy5KJbkB0zQpXa75Ipj51W65BiXHr58YLpsNSoCIlcCse32lO4E7Y7YdpvD1Ga/x37fKY0aO9m5VSfMW4rxcXphc/rCPGK/g8g36rbICpGrFCLNLnhYYncwOanUq9qpXJKNW5sd2xrEtDZAZFNZvDxzIn4mr7el+7FP0XWp4TFpDJ6eLV8qiPAYcfbYR1PNhpE9PZWy+vhEt1pTu9sWVRw1VSR2DofxQi2E1BTuQS+xGEi6p8AOvrFttp87L9bmELfiNDrlRQC5rlk8gOyOmEtWMDQcvsq8HG42m32lbF8+c3NSRzGAPuUu1jlhpqqjomqEnmHLDLWVfjlxRcR49FSSNuaqPsssBU+wBL7h+3YnwaM6iHAlCOHNFd8GZWxeiJ5KUoWMRdz8Z/jS8twgLINiAOR17WoQbU3AiTQHUMRZi+Pp9KRl6Kk+XYJ5EA+Qn75w13xceQkLEEHiXjVqFAyg8KYKx5NbEgycp1IAxJ8kYNGBHYuEBMhELUvaJX8PgGalrAlDT4PMZn/JAFq4N22xUAAREDLMjXoANHPT6nDVAkQQyASRzACNEAIgMl9GiB4EaISqACLTZYZIzQCR2QqASK0AkckKgUiNAJG5CoJIbQDRPo/C9onUBRAZqrgdazUBRGYqsJWpBSC6MarQG7CqAUjsT1iLFaKKiiBiXz5E7FWRWL6YN+YvSxVSA0BSfMKMtRUQan4JQterUCzvqMoSZqZNeoiUDpBUJTrKcpQzI2rTbtj5ShVkl55QvDBPzBfzjiguhKjXa6SHSMkASTnvcQG0IjUH/njtOnx+67bihXlivph3eFEhRJ6t5CGySzcfIoB6A/Tcjjx4/5Mv4Pa//qt4YZ6Yb0+AOIjO14i7X6YGgKReZbgAWptxEG599U/4/o5D8cI8Md/7AXJCVC36pisB9BCA1mUehK++/jfcu3dP8cI8Md+HASQFRIoFyEQACQKQq52JNidSLEAybJRpFaDuibU4qzPFASTXTquWARIVIucTTQggrQMkGkTKAqiRABIRIFEg4jw7rQyA+AcmEUAiAiQ4RAyguCsEkFcB1LU6E2LBohSA4pwJEUASASTYPhEPUESzhQDyNoAEgYgA8m6AXBANuZ0pASB8DjEBJB9A3TvWNg8BSiCAvBWgIa/OCCACyCOI5AcokwBSEECDhkjzALU3wsoLVkg8VQfrT/StxOIyWJuRDynZh+HSW2/DL375O8UL88R8Me91BUcg4ZilT62psoDplAgQaR2gVecaYOuhSkjLq/B6bS2ogPiGwU6s7d4N0Maqaq+AI/UgO+Z3/7ydKT23/IG/W19mGfTqrN99Iq0DlFxepXl4tpQeh9U/tcG6nzTAi4UWBk85NGZlw28ytoJlb36vv006UiHsZiMBpH4lnj0Jsb86C6a3z0ByVQ3szD0KH6eZ4cvURPh1xjZ4KbfMI4D6hYgAUr+Sj1fDs281wSq2YNhafAzSWQXq2JkBH27fAE17smFHXrnHAPU5JyKANCA293mRLRSwfbl+l8mqTlZOCQeTpy2s39UZAeRd8hSgByAigAggjyAigAggjyC6bCOACCCP7uJ34uqMACKAhqSIM5Wdxtdr+a+0EkAE0BAAinz1GITUF8v1hTICSAsALasudMw0LyeACCAPAEoggAggAogAIoAIIAKIACKACCACiAB6/nA5bMoqJvFKPFRCAA3qvy1+nK/Oh4eLpMhd+wkgAmjoiniZACKACCACiAAigAggAogAIoAEAsjwfCaEJOyAZeu2q06YN+bvpQClyQ6QITED9Ku2wKLlm2Dhs0mqE+aN+eN1EEAyAPTM6m2qBOd+4XUQQDIAtHjl5i4T9CtTYHVaNiSk71e8ME/M15U7XoesAJmXa+8Zie4AhOXfZUJ0Ugb8/t334ePPbypemCfm2wU/uw5ZAdLiQzbdASgkMR0Wr3B+kmOSMuFP167Dpzf+rnhhnpgvV31Y/ngd8gF0yHsBcq7CMiB4bSrEv5AF7374N/jH1/9RvDBPzBfzxvzlXIXJ+KV6cZ9UP9h9oJVpOfDeXz9TxSsvMU/MVwn7QKGnSgkgAmhoAEWdq4LwpnJtvq2HABIfoOiL9dp9XxgBJC5ArH11xl4+pWGA9h4kgHoCtPuAwP8bX6vtd6Yaj5ZAqHknAYRi4xBZWCBk9YGY1gaNv7W5xQpReQUQlrIbQpNeHlAr0/PgvY9UAhDLE/N157rCUnZBxL5ciLQL/HgXr3hvfJsNos5VQ2STZUCtOX8Crt24oQqAME/M153rimy2CNu6sPr8zOolAPHvOMdPzEADs+aNk+oCiOUr6KTYXYAu1HI+KRYgZyuTHiICyA14XjsOMWzl5X0AuQERATSwjC31XR4pGiBRWtkAEBFAbrzdkM0pVQOQWBDhA7MfBhEBNNDEuaG3P2oASDyIbA9AtPx8Hbz6599C21/eUbwwT8xXstZ1sf5Bb1QDUHujZBCR3GhdagNItPkQQTSwXjvWvepSM0CxIlWh/uZEXi8274luPdm3J+oCSODbHEPcbPQmRbfU9+uH6gASs5URRH3sNmsNIIJIAnj4l6loFiCCSMwVF/cSFfd8UDNAokPkhRNrXK73ueLSIkBiQ+RNqzOsPIOCRysAiQ+R9veJ+Hd/DX7stQIQQeThaqvNNrRx1xJAYu8TaQ4i3CS8WO/ReMsO0IID6UtMHXaHUADRjrW7tyeO97/DPEiAIporHDM2r42QDCCj0TicO9nu1IUMnm+EBUjcG7BqX+IPeqXlBkBh9rJvf7R+VSh6qtfr/UQHSGc2++LJxoUtnhx7yfoF13ra7PeEBEiuL6UpFhzWsrivZAx1vvOwsWWe4VgY6g7deuzpWbPRU52zOIgew6ZOnTqSHccaL9Rf5QG6KwZABBFfde7/MpgwVf5uTGsDLCnd/wfm5Tje02GSADQhfsGjCFBw5cFcXEKa2hpFA6irpbV7F0T4BXhjywlBq053++LeG383vLkCgvZsKUMvxxuDAqUCyEc3e3YAOz4eMP6xJ8KaLJ9xEHXYHGJCJEY1UuLEGtsVLs9jLp0SZxwRHrb4iblkBX1Zzk3/sYFPoZe66dNHcN5KApBO5zdqzuRx7DhtbkZSJlsK8okxiMT4xIi45FfMEh/BYXlw//Qn1tjx8OB1h9QcgplJq7PRQ95LPykBwslW4MgxY6ay45x5WamlodYj+P/WXGnkhBNrNtMXUyZ8rIwAYhW0M+p8TSc+8kRq4VMy8NzRrQ3ijRM3YXb6goAuqy6EuTuSKtE75uE09JL3VBKAdHyvxJI3NmD06FnsGDT7hbXZz1Tk3gxrLINo1rtNYt8w7aXTEHfFM2G+RtY6sBJIIXy4E7cZyJblUowRfrjDTpeDvmTfrVkbnstBz3jvxvJeDtNJGD58yUNyJ/qPGvUkO84fOWaUYW56cu3iV7LeWWrJux1youi7MFvpXYO1xCGV8JFtQ5WhocQRXFXgWFqRJ44s+Y7g4wWOkLoidq4joo9FmK3kruFk8XfBlfm3FzFP5mzfWMc8wj2f+bxnE3kP/aSsPj2rEE6mf8A0hZ+MzfPz81vEtCRwyoSYiSFLN0yKNCRPCtPLpOAhSJ88wcAULKAMvKS+/sjg5AkhSzeiF8wTPXqDHvFeTeG9C5C6+vSsQr58+cNEJusCAmb7Px44z9/ffwH7eQEPE0lmoRfoCXqDHnFeOT0bwXvoo5MpekL0GNN4JpxYz2CJzmVJP+UfyIAiySfmAfNiDueJ05vxvFeyw3M/RHgj7hE+OZyY/ZDvsZN4TSZJKte4T+S9QE9G8x75KwWenhC5QPLj++pIPtlHeQWSJJVr3B/hvQjgvfHt4ZfiwpXYMF6+JEXI5YdiwRkIKJK8oqCgoKCgoKCgoKCgoKCgoKCg8Cj+Dyqrhq0vA9X1AAAAAElFTkSuQmCC';
var fs = require('fs')
var Readable = require('stream').Readable
const imgBuffer = Buffer.from(base64, 'base64')
var s = new Readable()
s.push(imgBuffer)
s.push(null)
s.pipe(fs.createWriteStream("test.png"));
Just another solution with base-64 & utf8 in case we're using react-native which doesn't have Buffer global object:
function (base64Str) {
return utf8.decode(base64.decode(base64Str))
}
I want to send some string data from Python3 to nodeJs. The string is Korean characters and I am encoding it to utf8.(Cause I don't know other ways sending data safely.) When I send(from python) it is ByteStream and in nodeJs I receive it as Array. I convert this Array to String. But now I cannot decode string back to original Korean characters.
Here are some codes I am using.
python
input = sys.argv[1]
d = bot.get_response(input)
data = str(d).encode('utf8')
print(data)
nodeJs
var utf = require('utf8');
var python = require('python-shell');
var pyt = path.normalize('path/to/my/python.exe'),
scrp = path.normalize('path/to/my/scriptsFolder/'),
var options = {
mode: 'text',
pythonPath: pyt,
pythonOptions: ['-u'],
scriptPath: scrp,
encoding: 'utf8',
args: [message]
};
python.run('test.py', options, function (err, results) {
//here I need to decode 'results'
var originalString = utf.encode(results.toString());// that code is not working for me
});
I have used several libs like utf8 to decode but didn't help.
Can someone please give some idea how to make it work.
EDIT
I have to edit with some more info.
I have tried #smarx approach but did not work.
I have two cases:
1. if I send data as string from python here is what I get in nodeJs b'\xec\x95\x88\xeb\x85\x95\xed\x95\x98\xec\x8b\xad\xeb\x8b\x88\xea\xb9\x8c? \xec\x9d\xb4\xed\x9a\xa8\xec\xa2\x85 \xea\xb3\xa0\xea\xb0\x9d\xeb\x8b\x98! \xeb\x8f\x99\xec\x96\x91\xeb\xa7\xa4\xec\xa7\x81\xec\x9e\x85\xeb\x8b\x88\xeb\x8b\xa4
2. if I encode data and send. I get �ȳ��Ͻʴϱ�? ��ȿ�� ������! �
I had the totally same issue on my project and now I finally found the answer.
I solved my problem by using these codes.
It works on windows (macOS and Linux, their default system encoding it 'utf8' so the issue doesn't happen).
I hope it might help you, too!
#in the python file that your javascript file will call by python-shell module put those code
import sys
sys.stdout.reconfigure(encoding='utf-8')
I found the hints from python-shell description.
feature >Simple and efficient data transfers through stdin and stdout streams
I'm still not sure what python.run does, since you won't share that code, but here's my version of the code, which is working fine:
test.py
print("안녕 세상")
app.js
const { exec } = require('child_process');
exec('python3 test.py', function (err, stdout, stderr) {
console.log(stdout);
});
// Output:
// 안녕 세상
I have the same issue when using python-shell.
Here is my solution:
The string after .encode('utf-8') is a binary string. So you need to print it on stdout directly.
in test.py, it print a utf-8 json which include some chinese char:
sys.stdout.buffer.write(json.dumps({"你好":"世界"}, ensure_ascii=False).encode('utf8'))
print() # print \n at ending to support python-shell in json mode
in main.js
let opt = {mode: 'json', pythonOptions: ['-u'], pythonPath: 'python', encoding: 'utf8'}
let pyshell = new PythonShell('lyric.py', opt);
pyshell.on('message', function (message) {
console.log(message); //*** The console msg may still wrong (still ���)
let json = JSON.stringify(message);
let fs = require('fs');
fs.writeFile('myjsonfile.json', json, 'utf8', function () {
}); //*** The output json file will be correct utf8 output
});
result:
This shows the msg is correctly receive in utf-8, because the json output is correct.
However console.log output apparently failed.
I don't know is there any way to fix console.log output. (Windows 10)
I had same trouble in using data(string) from python in node js.
I solved this problem in this way:
Try change default code page of Windows Console to UTF-8, if your code page of Windows Console is not UTF-8.
(In my case default code page was CP949.)
In my case:
I got message like ������ 2���� ��������.
I tried encoding on online (http://code.cside.com/3rdpage/us/url/converter.html)
then I found my strings encoded cp949 -> decoded utf-8.