I have logs of json files which are objects that look like
{
"logs": [
{
"id": "12321321321321",
"email": "test#email.com",
"message": "ahahaha"
},
{
"id": "12321321312",
"email": "test#email.com",
"message": "hahahaha."
},
"id": "12321321321"
}
I need to return a new object that contains
{
"hello_id": outer id of the json file,
"array": [
{
"email": "test#me.com",
"total": 2
}
]
}
So far I am looping through the json files and have
jsonsInDirectory.forEach((file) => {
const fileData = fs.readFileSync(path.join("./logs", file), "utf8");
const jsonData = JSON.parse(fileData);
}
});
The key is "logs" and "id" and the values are the objects in the "logs" and the value of "id"
How can I count and return a new object at the same time?
You can try this approach: make a hash object that counts emails. Then just map it to an array of objects.
const data = {
logs: [{
id: "89004ef9-e825-4547-a83a-c9e9429e8f95",
email: "noah.sanchez#me.com",
message: "successfully handled skipped operation."
},
{
id: "89004ef9-e825-4547-a83a-c9e9429e8f95",
email: "noah.sanchez#me.com",
message: "successfully handled skipped operation."
},
{
id: "89004ef9-e825-4547-a83a-c9e9429e8f95",
email: "noname#me.com",
message: "successfully handled skipped operation."
}],
id: "56f83bed-3705-4115-9067-73930cbecbc0",
};
const emails = data.logs.reduce((acc, { email }) => {
acc[email] = (acc[email] ?? 0) + 1;
return acc;
}, {});
const tally = Object.entries(emails)
.map(([email, total]) => ({ email, total }));
const result = { logs_id: data.id, tally };
console.log(result)
.as-console-wrapper { max-height: 100% !important; top: 0 }
When you do const jsonData = JSON.parse(fileData);, you get the file data as a JSON and knowing the struct of that JSON you can easily get the info.
I have created a example https://codesandbox.io/s/stackoverflow-logs-count-example-jys2vg?file=/src/index.js
It may not solve exactly wath you want.
To solve this problem with the most time efficiency, you can create a tally object by firstly creating a map of the occurences of each mail with the email as the key and no of occurences as the value, since this will take constant (O(1)) time to execute, afterwhich you can create the tally array from the map as given below
output = []
jsonsInDirectory.forEach((file) => {
const fileData = fs.readFileSync(path.join("./logs", file), "utf8");
const jsonData = JSON.parse(fileData);
var map = {}
jsonData.logs.forEach((log) => {
if(log.email in map){
map[log.email] += 1
}
else {
map[log.email] = 1
}
});
var tally = []
for(var email in map){
tally.push({email: email, total: map[email]})
}
output.push({logs_id: jsonData['id'], tally: tally});
})
Related
I'm new to NodeJS and started to build out a serverless function to
Match custom_alerts criteria to crypto_signals data and send an email to the user.
I'm stuck in my loop trying to for that email combine all matching customAlerts and then create an array that matches customAlerts.signals_selection to cryptoSignals.combo_signal
The issue is here (see all code below for context) since one is an array and one is not
customObject.coins = cryptoSignalList.filter(item => item.combo_signal == x.signals_selection.includes(item))
I'm trying to use filter, reduce stuff for the first time.
Current Output
[
{ name: 'only Very Positive', coins: [] },
{ name: 'Only negative', coins: [] }
]
Output Goal
To send an email to the user with all the coins that match their criteria from custom_alerts. To create the email I'm thinking In the loop create an array that looks like below so I can use that in the body of the email
[
{
"name": customAlerts.name
"coins": [
{
"coin": cryptoSignals.coin,
"combo_signal": cryptoSignals.combo_signal
}
]
},
{
"name": customAlerts.name
"coins": [
{
"coin": cryptoSignals.coin,
"combo_signal": cryptoSignals.combo_signal
}
]
}
]
Data examples
Cryptosignals output example
[{
id: 119,
coin: 'iExec RLC',
coin_abr: 'RLC',
combo_signal: 'Negative',
combo_change: true,
combo_prev_signal: 'Very Negative'
}
]
customAlerts output example
[{
id: 8,
created_at: '2022-03-01T07:58:48.963809+00:00',
user_id: 'test12345',
signals_selection: [ 'Very Positive', 'Positive' ],
categories_selection: [ 'combo_signal' ],
name: 'Only Positive',
profiles: { email: 'email#email.com' }
}
]
My code
const matchAlertSignal = (emailList, customAlertList, cryptoSignalList) => {
const testArray = []
for (let i of emailList) { //loop through the email list
const emailAlertFilter = customAlertList.filter((customAlertList) => customAlertList.profiles.email == i )
const filterArray = []
for(let x of emailAlertFilter) {
const customObject = {}
customObject.name = x.name
customObject.coins = cryptoSignalList.filter(item => item.combo_signal == x.signals_selection.includes(item))
filterArray.push(customObject)
}
testArray.push(filterArray)
}
console.log(testArray)
}
async function main(){
let cryptoSignals = await getCryptoSignals();
let customAlerts = await getCustomAlerts();
let supabaseEmails = userEmail(customAlerts);
let customFilter = matchAlertSignal(supabaseEmails, customAlerts, cryptoSignals);
I see two issues with the current version of your question:
matchAlertSignaldoes not return a value
The filtering on cryptoSignalList seems incorrect as you test equality between combo_signal (string) and the fact that the emailAlertFilter.signals_selection contains the value (boolean).
I would suggest the following:
const matchAlertSignal = (emailList, customAlertList, cryptoSignalList) => {
for (let i of emailList) { //loop through the email list
const emailAlertFilter = customAlertList.filter((customAlertList) => customAlertList.profiles.email === i )
const filterArray = []
const customObject = {}
for(let x of emailAlertFilter) {
customObject.name = x.name
customObject.coins = cryptoSignalList.filter((combo_signal) => x.signals_selection.includes(combo_signal))
filterArray.push(customObject)
}
}
return filterArray
}
I'm trying to loop through a JSON and sort it by the date so I can see the latest date to the oldest date, and then write it to the file.
Here is my code
var reader = JSON.parse(fs.readFileSync('txt.json', 'utf8'));
function sortByDate(a, b) {
return new Date(a.lastUpdated).toJSON() - new Date(b.lastUpdated).toJSON();
}
reader.sort(sortByDate)
JSON Data Example
{
"Data": {
"Contents": [
{
"Key": [
"HelloTest"
],
"lastUpdated": [
"2019-10-25T10:30:50.558Z"
]
},
{
"Key": [
"TestHello"
],
"lastUpdated": [
"2019-03-26T10:30:50.558Z"
]
}
]
}
}
Here are a couple of errors I found in your code:
Your function name has a typo, it should be sortByDate and not sortbyDate.
You need top sort the inner json.Data.Contents array, not the outer json object.
You need to reference the first element of your lastUpdated arrays using lastUpdated[0].
Finally, you do not need to call toJSON() on the date objects in your sorting function, simply convert to date and return the difference.
Also your inner data fields are arrays, which seems strange for a Key and a lastUpdated value.
If you keep your fields as arrays, here is a working example showing how to sort the inner Data.Contents array by date:
const jsonString = `{
"Data": {
"Contents": [{
"Key": ["HelloTest"],
"lastUpdated": ["2019-10-25T10:30:50.558Z"]
}, {
"Key": ["TestHello"],
"lastUpdated": ["2019-03-26T10:30:50.558Z"]
}]
}
}`;
function sortByDate(a, b) {
return new Date(a.lastUpdated[0]) - new Date(b.lastUpdated[0]);
}
const json = JSON.parse(jsonString);
const defaultValue = { Data: { Contents: [] } };
const sortedContents = [...(json || defaultValue).Data.Contents].sort(sortByDate);
const output = { ...json, Data: { Contents: sortedContents } };
console.log(output);
If you change your fields to scalars, which I suggest, here is another example:
const jsonString = `{
"Data": {
"Contents": [{
"Key": "HelloTest",
"lastUpdated": "2019-10-25T10:30:50.558Z"
}, {
"Key": "TestHello",
"lastUpdated": "2019-03-26T10:30:50.558Z"
}]
}
}`;
function sortByDate(a, b) {
return new Date(a.lastUpdated) - new Date(b.lastUpdated);
}
const json = JSON.parse(jsonString);
const defaultValue = { Data: { Contents: [] } };
const sortedContents = [...(json || defaultValue).Data.Contents].sort(sortByDate);
const output = { ...json, Data: { Contents: sortedContents } };
console.log(output);
It looks like you're reading contents from a file, then needs to sort it by date, and then finally write it to a new file. If that is what you're going for, the following should help:
const fs = require('fs');
const path = require('path');
// JSON files can be requied in as objects without parsing
// `arrayOfStuff` will be the var name for `Contents`
const { Data: { Contents: arrayOfStuff } } = require('./data.json');
function sortByDate(el1, el2) {
// access the date from the object and turn into date object
let date1 = new Date(el1.lastUpdated[0]);
let date2 = new Date(el2.lastUpdated[0]);
// compare date objects in revers order to get newest to oldest
return (date2 - date1);
}
// sort `Contents` from the `Data` object and turn into JSON
const sortedContent = arrayOfStuff.sort(sortByDate);
const newDataObj = JSON.stringify({ Data: { Content: sortedContent }}, null, 2);
// create the fully qualified file path with `sortedByDate.json` as the file name
const filePath = path.resolve('./', 'sortedByDate.json');
// write to new file
fs.writeFile(filePath, newDataObj, (err) => {
if(err) {
console.log('Made an oopsie:', err);
}
console.log(`Success!, new JSON file located at: ${filePath}`);
}); // write to file
I got the following problem,
I need to iterate through a big Json object ( child nodes consist of array's, strings and objects with at least 4-5 layers of depth in terms of nested properties ).
In some parts across the big Json file there is a specific object structure, it has a property named "erpCode". I need to scan the Json and find all the objects with that property, take the value use that code to ask a different API for details and once I get the details insert them into the object with the current 'erpCode'.
Just to clarify, in my case the parent node property name in the Json always equals the value in 'typeSysname' field which located on the same 'level' as the erpCode property.
A simple example :
{
"cars": [
{
"name": "X222",
"carType": {
"erpCode": "skoda",
"value": null,
"typeSysName": "carType"
}
}
],
"model": {
"year": 1999,
"details": {
"erpCode": "112"
"value": null,
"typeSysName": "details"
}
}
}
In this example I need to find 2 properties get the values skoda and 112 out of them and get the value and description data from a different API and set it into this Json in the right location.
P.S. Any chance there is a good npm package which can help me with that?
Edit:
I got a solution in C# from a few months ago which runs in a generic way on the Json and handles the complexity of the structure in a generic way.
But I now need to convert this into Javascript and I am a bit lost.
public static string TranslateDocErpCodes(string jsonString, string topRetailerSysName)
{
try
{
var doc = JObject.Parse(jsonString);
var erpCodeList = doc.SelectTokens("$..erpCode").ToList();
foreach (var erpCodeJToken in erpCodeList)
{
var value = erpCodeJToken?.Value<string>();
var erpCodeParent = erpCodeJToken?.Parent.Parent;
var erpCodeProperty = erpCodeParent?.Path.Split(".").Last();
var result =
_dataService.GetLovFromErpCode(topRetailerSysName, erpCodeProperty, value);
if (result == null)//reset lov obj
{
if (erpCodeParent?.Parent is JProperty prop)
prop.Value = JObject.FromObject(new LovObject { ErpCode = value });
}
else//set lov obj
{
result.ErpCode = value;
if (erpCodeParent?.Parent is JProperty prop)
prop.Value = JObject.FromObject(result);
}
}
return JsonConvert.SerializeObject(doc);
}
catch (Exception e)
{
throw new Exception("ErpConvert.TranslateDocErpCodes() : " + e);
}
}
mb something like;
function processObject(jsonData) {
for (prop in jsonData) {
if (jsonData.hasOwnProperty(prop)) {
// We get our prop
if (prop === 'code') {
let codeValue = jsonData[prop]
doSomeAsync(codeValue)
.then(response => {
jsonData[prop] = response;
})
}
let curValue = jsonData[prop];
if (Array.isArray(curValue)) {
// Loop through the array, if array element is an object, call processObject recursively.
processArray(curValue);
} else if (typeof curValue === 'object') {
processObject(curValue);
}
}
}
}
I took the answer from Aravindh as a starting point and managed to reach what seems to be a complete solution.
I will share it here,
async function convertErpCodes(jsonData, orgName, parentPropertyName){
for (let prop in jsonData) {
if (jsonData.hasOwnProperty(prop)) {
if (prop === 'erpCode') {
const erpCodeValue = jsonData[prop]
const req = {"query": {"erpCode": erpCodeValue, "orgName": orgName, "typeSysName": parentPropertyName}};
const result = await viewLookupErpService.findOne(req);
if(result)
return result;
}
const curValue = jsonData[prop];
if (Array.isArray(curValue)) {
for(let i in curValue){
const res = await convertErpCodes(curValue[i], orgName, prop);
}
} else if (curValue && typeof curValue === 'object') {
const response = await convertErpCodes(curValue, orgName, prop);
if(response){
jsonData[prop] = response;
}
}
}
}
}
P.S.
I set up the values only if I get a response from the third party API ( this is the reason for the result and response logic in the recursion.
I'd use object-scan and lodash.set in combination
// const objectScan = require('object-scan');
// const lodash = require('lodash');
const stats = { cars: [{ name: 'X222', carType: { erpCode: 'skoda', value: null, typeSysName: 'carType' } }], model: { year: 1999, details: { erpCode: '112', value: null, typeSysName: 'details' } } };
const entries = objectScan(['**.erpCode'], { rtn: 'entry' })(stats);
console.log(entries);
// => [ [ [ 'model', 'details', 'erpCode' ], '112' ], [ [ 'cars', 0, 'carType', 'erpCode' ], 'skoda' ] ]
// where you would query the external api and place results in entries
entries[0][1] = 'foo';
entries[1][1] = 'bar';
entries.forEach(([k, v]) => lodash.set(stats, k, v));
console.log(stats);
// => { cars: [ { name: 'X222', carType: { erpCode: 'bar', value: null, typeSysName: 'carType' } } ], model: { year: 1999, details: { erpCode: 'foo', value: null, typeSysName: 'details' } } }
.as-console-wrapper {max-height: 100% !important; top: 0}
<script src="https://bundle.run/object-scan#13.8.0"></script>
<script src="https://bundle.run/lodash#4.17.20"></script>
Disclaimer: I'm the author of object-scan
I am in process of learning expressJS with NodeJS.
I am trying to insert multiple rows into mySQL table. Since the bulk insert query requires data like
[["a",1], ["b",2], ["c",3]]
How can I transform my array of objects to such form? Here is my JSON post data
[
{
"productID" : 1,
"stock": -3
},
{
"productID" : 1,
"stock": 5
}
]
How to tranform such JSON object into the multidimensional array?
[[1,-3],[1,5]]
Here is what I have tried so far.
let promises = []
req.body.map((n) => {
promises.push(new Promise(resolve => {
let { productID, stock } = n
let values = {
PRODUCT_ID: productID,
STOCK: stock
}
let sql = 'INSERT INTO product_stock_history SET ?'
db.connection.query(sql, values, (err, results) => {
if (err) {
console.log("Failed to add stocks record: " + err)
res.sendStatus(500)
return
} else {
res.send("Stock record has been added")
}
})
}))
})
The above code is working, but in the end I have error with the mySQL syntax which I believe something to do with the promise. I am not familiar with the promise :)
Error: Can't set headers after they are sent.
So what i want to achieve is just the mapping without Promise.
thanks
You could pass Object.values as parameter to map like this:
const input = [
{
"productID" : 1,
"stock": -3
},
{
"productID" : 1,
"stock": 5
}
]
const output = input.map(Object.values)
console.log(output)
Javascript:
$(document).ready(function() {
const ores = "../js/json/oreList.json";
const priceURL = "https://esi.tech.ccp.is/latest/markets/prices/?datasource=tranquility";
let oreArray = [];
let priceArray = [];
let total = 0;
// Retrieve list of ores
function getOres() {
$.getJSON(ores, function(ores) {
ores.forEach(function(ore) {
total++;
if (total === 48) {
getPrices();
}
oreArray.push(ore);
});
});
}
// Retrieve all items & prices via API
function getPrices() {
$.getJSON(priceURL, function(prices) {
prices.forEach(function(data) {
priceArray.push(data);
console.log(data);
});
});
}
getOres();
});
The first function creates an internal array from my .JSON file and the second function creates an internal array from the URL.
In the first array oreArray, an object looks like this:
{ id: 1234, name: "Title" }
In the second array priceArray, an object looks like this:
{ type_id: 1234, average_price: 56.34 }
My oreArray has 48 objects and unfortunately the priceArray has about 11,000 objects. I need to create a new array by comparing the two arrays and building new objects, where the ID's match. So for example objects in newArray would look like:
{ id: 1234, name: "Title", average_price: 56.34 }
Basically I'm having trouble figuring out the logic for:
For each object in oreArray, find the object with the same ID value in priceArray and append the new array with a new object using values from both arrays.
I would do it this way:
const ores = "../js/json/oreList.json",
priceURL = "https://esi.tech.ccp.is/latest/markets/prices/?datasource=tranquility";
let oreArray,
priceArray,
joinedArray = [];
function getOres() {
$.getJSON(ores, function(ores) {
oreArray = ores;
getPrices();
});
}
function getPrices() {
$.getJSON(priceURL, function(prices) {
priceArray = prices;
joinPrices();
});
}
function joinPrices() {
oreArray.forEach(function(ore) {
var matchingPrice = getMatchingPrice(ore);
if(matchingPrice !== false) {
joinedArray.push({
id: ore.id,
name: ore.name,
average_price: matchingPrice.average_price
});
}
});
}
function getMatchingPrice(ore) {
for(var i=0; i<priceArray.length; i++) {
if(priceArray[i].type_id === ore.id) {
return priceArray[i];
}
}
return false;
}
getOres();
I think that a good way to approach this problem is by changing the data structure of the average prices a little bit.
Instead of having them in an array, where each item has type_id and average_price field, you might want to consider using an object to store them, where the key is the type_id and the value is the average_price.
To be more concrete, you can replace:
prices.forEach(function(data) {
priceArray.push(data);
});
With:
const pricesMap = {};
prices.forEach(price => {
pricesMap[price.type_id] = price.average_price
});
And when looping on the oreArray, you can access each product's average_price by simply referring to pricesMap[ore.id]
You can check out this JSBin: http://jsbin.com/fogayaqexe/edit?js,console
You can use reduce to loop over each oreArr item and collect the data you need in the accumulator:
var oreArr=[
{ id: 1234, name: "Title" },
{ id: 2234, name: "2Title" },
]
var priceArr= [
{ type_id: 1234, average_price: 56.34 },
{ type_id: 2234, average_price: 256.34 },
{ type_id: 3234, average_price: 56.34 },
{ type_id: 4234, average_price: 56.34 },
]
var resArr = oreArr.reduce((ac,x) => {
var priceMatch = priceArr.find( z => z.type_id === x.id )
if(! priceMatch)
return ac //bail out if no priceMatch found
var res = Object.assign({}, x, priceMatch)
ac.push(res)
return ac
},[])
console.log(resArr)
other methods used:
arrayFind to check intersection
Object.assign to create the merged object to populate the accumulator
I suggest you to change your small json as object
eg : '{"1234":{"id": 1234, "name": "Title" }}';
var json = '{"1234":{"id": 1234, "name": "Title" }}';
oreArray = JSON.parse(json);
alert(oreArray['1234'].name); // oreArray[priceArraySingle.id].name
we can easily match priceArray id with oreArray.