We Keep Coding

How to parse Xlsx file and search columns by NLP JSON dictionary

I'm working on a kind of magical xlsx parsing where I need to parse a complex xlsx file (it contains some presentational and useless informations as a header, and a data table).
What I'm trying to do is parsing the file (that's cool), and I'm trying to figure out where the table starts. For that what I want to do, is having a kind on dictionary as JSON.
Since the xlsx's files are dynamics and are all supposed to be some product ordering topics, I want to find the table's header row by "blindly" looking for column names according to the JSON dictionary below.
{
"productName": [
"description",
"product",
"name",
"product name"
],
"unit": [
"unit",
"bundle",
"package",
"packaging",
"pack"
]
}
Here for example, we suppose one column header is going to be "bundle", and another "description". Thanks to my json, I'm suppose to be able to find the key "productName" and "unit". It's a kind of synonym searching actually.
Questions are :
1/ Is there a clean data structure or an efficient for doing doing this search ? The Json could be huge with memory leaks, I want it to be as fast as possible.
2/ I know this is not precise, but it could work. So do you have any suggestion how to to it or how would you do it ?
3/ The operation is going to be costly because I have to parse this xlsx file, and in the mean time I have to parse the JSON to do my dictionary thing, do you have any advice ?
I suppose I should to both parsing in a stream way so I only parse one row of the xlsx at a time, and I make my JSON parsing and research of that row before jumping to the next one ?
Here is the start of the parsing code.
const ExcelJS = require('exceljs');
const { FILE_EXTESTIONS } = require('../constants');
const { allowExtention } = require('../helpers');
const executeXlsxParsing = () => {
const filePath = process.argv.slice(2)[1];
const workbook = new ExcelJS.Workbook();
const isAllowed = allowExtention(FILE_EXTESTIONS.xlsx, filePath)
if (!filePath || !isAllowed) return;
return workbook.xlsx.readFile(filePath).then(() => {
var inboundWorksheet = workbook.getWorksheet(1);
inboundWorksheet.eachRow({ includeEmpty: true }, (row, rowNumber) => {
console.log('Row ' + rowNumber + ' = ' + JSON.stringify(row.values));
});
});
};
module.exports.executeXlsxParsing = executeXlsxParsing();
Thank you all :)

How to replace specific parts of a json file using node.js

Currently, I have a JSON file that looks like this:
"1E10BC5D4EC68EE2916BFD97F23E951C": "Seattle Seahawks",
"E6B87019417436B73B62F7802763A289": "Seaside style. ",
"81EEA9E6400BFEADE161559AF14EE468": " {1}",
"6F02148E4A78B33C1CEB75BC2753CA69": " {EndDate}",
"D89CA2634FFF8FA02D028096BAAE6963": "\"You have received a reward for completing the {Bundle Name} {Number} challenges!",
"Mission": {
"Default Mission Info Description": "Default Mission Description",
"Default Mission Info Name": "Default Mission Name",
"RewardDescription": "You ranked {PositionRank}. For your efforts, you have been awarded:",
"NonParticipationRewardDescription": "Your teammates did a great job! For their efforts, you have been awarded:",
"RewardTitle": "{MissionName} Completed!"
}
It goes on for about 40,000 lines, and I would like to modify all of the strings set by it. Currently, I am using #zuzak/owo to try to accomplish this. My current code looks like this:
const owo = require('#zuzak/owo')
fs = require('fs');
var name = '../jsonfile.json';
var data = fs.readFileSync(name).toString();
fs.writeFileSync('../owo.json', JSON.stringify(owo(data)))
How would I be able to only change the strings, such as "Seaside style. " and not edit any of the string names, such as "81EEA9E6400BFEADE161559AF14EE468" (sorry for any incorrect wording, hopefully you can understand what I am saying.)
In case you need it, here is the main code used in #zuzak/owo:
const addAffixes = (str) => randomItem(prefixes) + str + randomItem(suffixes)
const substitute = (str) => {
const replacements = Object.keys(substitutions)
replacements.forEach((x) => {
str = replaceString(str, x, substitutions[x])
})
return str
}

Parse the JSON, iterate over keys, modify values. If necessary recursively iterate over subobjects too:
function owoifyObject (obj) {
for (const key in obj) {
if (typeof obj[key] === 'string') {
obj[key] = owo(obj[key])
} else if (typeof obj[key] === 'object' && obj[key]) {
owoifyObject(obj[key])
}
}
}
const data = JSON.parse(fs.readFileSync('file.json').toString())
owoifyObject(data)
fs.writeFileSync('file2.json', JSON.stringify(data, null, 4))
Note that the argument 4 in JSON.stringify is purely there for formatting so that the result looks something like your input did, otherwise you'd get all the data in a single line.

Use JSON.parse() to parse the JSON file into an object. Then go through the object calling owo() on each value.
var data = JSON.parse(fs.readFileSync(name).toString());
for (let key in data) {
data[key] = owo(data[key]);
}
fs.writeFileSync("../owo.json", JSON.stringify(data));

Convert Facebook json file sequences like \u00f0\u009f\u0098\u008a to emoji characters

I have downloaded my Facebook data as json files. The json files for my posts contain emojis, which appear something like this in the json file: \u00f0\u009f\u0098\u008a. I want to parse this json file and extract the posts with the correct emojis.
I can't find a way to load this json file into a json object (using JavaScript) then read (and output) the post with the correct emojis.
(Eventually I will upload these posts to WordPress using its REST API, which I've worked out how to do.)
My program is written in JavaScript and run using nodejs from the command line. I've parsed the file using:
const fs = require('fs')
let filetext = fs.readFileSync(filename, 'utf8')
let jsonObj = JSON.parse(filetext)
However, when I output the data (using something like jsonObj.status_updates.data[0].post), I get strange characters for the emoji, like Happy birthday ├░┬ƒ┬ÿ┬è instead of Happy birthday 😊. This is not a Windows 10 console display issue because I've piped the output to a file also.
I've used the answer Decode or unescape \u00f0\u009f\u0091\u008d to 👍 to change the \uXXXX sequences in the json file to actual emojis before parsing the file. However, then JSON.parse does not work. It gives this message:
SyntaxError: Unexpected token o in JSON at position 1
at JSON.parse (<anonymous>)
So I'm in a bind: if I convert the \uXXXX sequences before trying to parse the json file, the JavaScript json parser has an error. If I don't convert the \uXXXX sequences then the parsed file in the form of a json object does not provide the correct emojis!
How can I correctly extract data, including emojis, from the json file?

I believe you should be able to do all this in Node.js, here's an example.
I've tested this using Visual Studio Code.
You can try it here: https://repl.it/repls/BrownAromaticGnudebugger
Note: I've updated processMessageas per #JakubASuplicki's very helpful comments to only look at string properties.
index.js
const fs = require('fs')
let filename = "test.json";
let filetext = fs.readFileSync(filename, "utf8");
let jsonObj = JSON.parse(filetext);
console.log(jsonObj);
function decodeFBString(str) {
let arr = [];
for (var i = 0; i < str.length; i++) {
arr.push(str.charCodeAt(i));
}
return Buffer.from(arr).toString("utf8");
}
function processMessages (messageArray) {
return messageArray.map(processMessage);
}
function processMessage(message) {
return Object.keys(message).reduce((obj, key) => {
obj[key] = (typeof message[key] === "string") ? decodeFBString(message[key]): message[key];
return obj
}, {});
}
let messages = processMessages(jsonObj.messages);
console.log("Input: ", jsonObj.messages);
console.log("Output: ", messages);
test.json
{
"participants": [
{
"name": "Philip Marlowe"
},
{
"name": "Terry Lennox"
}
],
"messages": [
{
"sender_name": "Philip Marlowe",
"timestamp_ms": 1546857175,
"content": "Meet later? \u00F0\u009F\u0098\u008A",
"type": "Generic"
},
{
"sender_name": "Terry Lennox",
"timestamp_ms": 1546857177,
"content": "Excellent!! \u00f0\u009f\u0092\u009a",
"type": "Generic"
}
]
}

Nodejs - Writing a tab delimited file as json object

Is there a npm module which converts a tab delimited file to a JSON object so i could look up the data by certain properties.
Example: The file would looks like below,
name sex age
A M 20
B F 30
C M 40
D F 50
JSON
[
{
"name": "A",
"sex": "M",
"age": "20"
},
{
"name": "B",
"sex": "F",
"age": "30"
},
/* Etc. */
]

Sometimes i prefer not using node modules so that I can run these scripts without any setup....
IE. nodejs ./convert-to-csv.js
Here is a nodejs script in case you choose not to use a node module.
var fs = require("fs");
fs.readFile("./birthrate_poverty.txt","utf8", function(err, data){
var rows = data.split("\n");
var json = [];
var keys = [];
rows.forEach((value, index)=>{
if(index < 1){// get the keys from the first row in the tab space file
keys = value.split("\t");
}else {// put the values from the following rows into object literals
values = value.split("\t");
json[index-1] = values.map((value, index) => {
return {
[keys[index]]: value
}
}).reduce((currentValue, previousValue) => {
return {
...currentValue,
...previousValue
}
});
}
})
// convert array of objects into json str, and then write it back out to a file
let jsonStr = JSON.stringify(json);
fs.writeFileSync("./birthrate_poverty.json", jsonStr, {encoding: "utf8"})
});

Yes, csvtojson and the delimiter can be anything not only commas.
Example:
const csvFilePath='FILE'
const csv=require('csvtojson')
csv({delimiter:"\t"})
.fromFile(csvFilePath)
.on('json',(jsonObj)=>{
console.log(jsonObj);
})
.on('done',(error)=>{
console.log('end');
})

Parse nested JSON

I have the following JSON on a URL:
{
"href":"http:\/\/api.rwlabs.org\/v1\/jobs?limit=10",
"time":18,
"links":
{
"self":
{
"href":"http:\/\/api.rwlabs.org\/v1\/jobs?offset=0&limit=10&preset=minimal"
},
"next":
{
"href":"http:\/\/api.rwlabs.org\/v1\/jobs?offset=10&limit=10&preset=minimal"
}
},
"totalCount":2279,
"count":10,
"data":[
{
"id":"1148141",
"score":1,
"href":"http:\/\/api.rwlabs.org\/v1\/jobs\/1148141",
"fields":
{
"title":"Volunteer Renewable Energy Programmes Coordinator"
}
},
{
"id":"1147901",
"score":1,
"href":"http:\/\/api.rwlabs.org\/v1\/jobs\/1147901",
"fields":
{
"title":"Volunteer Project Management Coordinators \/ Intern"
}
}
/* so on*/
And I want get the information inside "data" and inside "fields".
If I remove the part before data array, I can get some of the values, but fields returns undefined. Although I also need to have a solution without removing the information before data.
JS
var table = '<table><thead><th>id</th><th>title</th></thead><tbody>';
var obj = $.parseJSON(data);
$.each(obj, function(i, val) {
table += '<tr><td>' + this['id'] + '</td><td>' + this['obj[i].title'] + '</td></tr>';
});
table += '</tbody></table>';
document.getElementById("datalist").innerHTML = table;
(I also do not know how to parse the data from the URL, so for now I am copying the data into the script)
[JSFiddle]: http://jsfiddle.net/1v803c3L/1/ Although I have part of the data on the code, even though the entire information is on an URL as seen on the code I posted.

obj.data[i].fields.title where i is an int that you use to index into the data array