I have a file with data in it that I am needing to parse and store in a DB. Below, is an example of 2 entries in the file. I'm not quite sure what the structure is (although it looks to be ndJSON). I am trying to parse the data in to a JSON object in order to store it in a DB, but cannot seem to figure it out. Here is what I have so far
var ndjson = {
"sequence-num": "0123456789",
"version": "N1.4",
"record-type": "R",
"session-id": "197-30760303",
"date": "2021-07-23 15:00:53",
"passport-header": { "alg": "ES256", "ppt": "test", "typ": "passport", "x5u": "https://cr.com" },
"passport-payload": { "attest": "A", "dest": { "tn": ["0123456789"] }, "iat": 0123456789, "orig": { "tn": "0123456789" }, "origid": "c699f78a-ebc6-11eb-bfd8-bec0bbc98888" },
"identity-header": "eyJhbGciOiJFUzI1NiIsInBwdCI6InNoYWtlbiIsInR5cCI6InBhc3Nwb3J0IiwieDV1IjoiaHR0cHM6Ly9jci5zYW5zYXkuY29tL1RvdWNodG9uZV82ODNBIn0.eyJhdHRlc3QiOiJCIiwiZGVzdCI6eyJ0biI6WyIxMjUeyJhdHRlc3QiOiJCIiwiZGVzdCI6eyJ0biI6WyIxMj;info=<https://google.com/>;alg=ES256;ppt=\"test\""
}
{
"sequence-num": "0123456788",
"version": "N1.4",
"record-type": "R",
"session-id": "214-30760304",
"date": "2021-07-23 15:00:53",
"passport-header": { "alg": "ES256", "ppt": "test", "typ": "passport", "x5u": "https://cr.com" },
"passport-payload": { "attest": "B", "dest": { "tn": ["0123456788"] }, "iat": 0123456788, "orig": { "tn": "0123456788" }, "origid": "c69d0588-ebc6-11eb-bfd8-bec0bbc98888" },
"identity-header": "eyJhbGciOiJFUzI1NiIsInBwdCI6InNoYWtlbiIsInR5cCI6InBhc3Nwb3J0IiwieDV1IjoiaHR0cHM6Ly9jci5zYW5zYXkuY29tL1RvdWNodG9uZV82ODNBIn0.eyJhdHRlc3QiOiJCIiwiZGVzdCI6eyJ0biI6WyIxMjUeyJhdHRlc3QiOiJCIiwiZGVzdCI6eyJ0biI6WyIxMj;info=<https://google.com/>;alg=ES256;ppt=\"test\""
};
let result = ndjson.split(',').map(s => JSON.parse(s));
console.log('The resulting array of items:');
console.log(result);
console.log('Each item at a time:');
for (o of result) {
console.log("item:", o);
}
When I run this, I get Uncaught SyntaxError: Unexpected token ':' error on line 12 at the 2nd node of "sequence-num": "0123456788",.
Any help is appreciated, thank you!
If you actually have ndJSON(newline-delimited JSON) then each line in the file is valid JSON, delimited by newlines. A simple file would look like this:
{"key1": "Value 1","key2": "Value 2","key3": "Value 3","key4": "Value 4"}
{"key1": "Value 5","key2": "Value 6","key3": "Value 7","key4": "Value 8"}
This differs from the formatted data you've posted here, and the difference is important since once you've formatted it, the valid JSON objects cannot simply be distinguished by the presence of newlines.
So, on the assumption that you do have valid ndJSON, in its original form, you can extract it by using split() on newLines and using JSON.parse() on the resulting array.
This snippet adds a little file handling to allow a file to be uploaded, but thereafter it uses split() and JSON.parse() to extract the data:
"use strict";
document.getElementsByTagName('form')[0].addEventListener('submit',function(e){
e.preventDefault();
const selectedFile = document.getElementById('inputFile').files[0];
let fr = new FileReader();
fr.onload = function(e){
let ndJSON = e.target.result; // ndJSON extracted here
let ndJSONLines = ndJSON.split('\n');
// Process JSON objects here
ndJSONLines.forEach(function(el){
let obj = JSON.parse(el);
Object.keys(obj).forEach(key=>{
console.log(`Key: ${key}, Value: ${obj[key]}`);
});
});
}
fr.readAsText(selectedFile)
});
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Parsing ndJSON</title>
</head>
<body>
<form method="post" enctype="multipart/form-data">
<input type="file" name="inputFile" id="inputFile">
<input type="submit">
</form>
</body>
</html>
Output, based on the sample file above:
Here is what I do
const end_point_url = 'https://ipfs.io/ipfs/bafkqap33ejwgs3tfgerduitomrvhg33oebtg64tnmf2c4lroej6qu6zcnruw4zjsei5ce5dinfzsa2ltebqsa43fmnxw4zbanruw4zjcpufa';
let json = await fetch(end_point_url).
then( resp => resp.text() ).
then( buf => { // NDJSON format ...
return buf.slice(0,-1).split('\n').map(JSON.parse);
}).
catch(console.error);
Related
I have a flask route where I am trying to pipe a list_object [{}] into a template. It works but the data comes out as a string on the front end. How can I convert the string back into a array_object so I can do stuff with it?
Here is the flask route:
app.py
#app.route('/vgmplayer')
def vgmplayer():
musicdata = music.query.order_by(func.random()).first()
musicimgdata = musicimages.query.filter_by(gamealbum = musicdata.gamealbum).order_by(func.random()).first()
if musicimgdata == None:
musicimgdata = "https://media.idownloadblog.com/wp-content/uploads/2018/03/Apple-Music-icon-003.jpg"
else:
musicimgdata = musicimgdata.gameart
musicobject = [{
"name": musicdata.gametrackname,
"path": musicdata.gamelink,
"img": musicimgdata,
"album": musicdata.gamealbum,
"artists": musicdata.artists,
"platform": musicdata.platform,
"year": musicdata.year,
"genre": musicdata.genre
}]
print(musicobject)
return render_template("musicplayer.html",rvgm = musicobject)
but when I get back the template:
musicplayer.html
<script type="text/javascript">
function grvgm(){
All_song = "{{ rvgm|safe }}"
return All_song
}
</script>
it comes in a string:
Example data:
All_song = grvgm()
"[{'name': '10 - M Stage Jungle B', 'path': 'https://vgmsite.com/soundtracks/zhuzhu-pets-quest-for-zhu-20…-nds-gamerip/ebulpznnyw/10%20-%20M%20Stage%20Jungle%20B.mp3', 'img': None, 'album': 'ZhuZhu Pets - Quest for Zhu', 'artists': None, 'platform': 'DS', 'year': '2011', 'genre': None}]"
I would need the list_dict to not have the qoutes at the end so that the javascript can treat it as an array.
EDIT:
I forgot to mention that I tried:
function grvgm(){
All_song = "{{ rvgm }}"
All_song = JSON.parse(All_song)
return All_song
}
SyntaxError: JSON.parse: expected property name or '}' at line 1 column 3 of the JSON data
Turns out the string was not a valid json format as #Andy pointed out. I had to use a different method. Instead of piping the list_dict into the template, I used a get request to get the list_dict into the front end so I can do stuff with it.
app.py
#app.route('/grvgm', methods=['GET'])
def grvgm():
musicdata = music.query.order_by(func.random()).first()
musicimgdata = musicimages.query.filter_by(gamealbum = musicdata.gamealbum).order_by(func.random()).first()
if musicimgdata == None:
musicimgdata = "https://media.idownloadblog.com/wp-content/uploads/2018/03/Apple-Music-icon-003.jpg"
else:
musicimgdata = musicimgdata.gameart
musicobject = [{
"name": musicdata.gametrackname,
"path": musicdata.gamelink,
"img": musicimgdata,
"album": musicdata.gamealbum,
"artists": musicdata.artists,
"platform": musicdata.platform,
"year": musicdata.year,
"genre": musicdata.genre
}]
print(musicobject)
musicobject = json.dumps(musicobject)
return musicobject
musicplayer.js
async function load_track(index_no){
fetch('/grvgm')
.then(response => response.json())
.then(All_song => {
console.log(All_song)
// stuff goes here
})
By using the JSON.parse() method, you can easily parse the string.
Example:
JSON.parse(All_song)
// returns an array
Right, so I am trying to wrap my head around editing (appending data) to a JSON file.
The file (users.json) looks like this:
{
"users": {
"id": "0123456789",
"name": "GeirAndersen"
}
}
Now I want to add users to this file, and retain the formatting, which is where I can't seem to get going. I have spent numerous hours now trying, reading, trying again... But no matter what, I can't get the result I want.
In my .js file, I get the data from the json file like this:
const fs = require('fs').promises;
let data = await fs.readFile('./test.json', 'utf-8');
let users = JSON.parse(data);
console.log(JSON.stringify(users.users, null, 2));
This console log shows the contents like it should:
{
"id": "0123456789",
"name": "GeirAndersen"
}
Just to test, I have defined a new user directly in the code, like this:
let newUser = {
"id": '852852852',
"name": 'GeirTrippleAlt'
};
console.log(JSON.stringify(newUser, null, 2));
This console log also shows the data like this:
{
"id": "852852852",
"name": "GeirTrippleAlt"
}
All nice and good this far, BUT now I want to join this last one to users.users and I just can't figure out how to do this correctly. I have tried so many version and iterations, I can't remember them all.
Last tried:
users.users += newUser;
users.users = JSON.parse(JSON.stringify(users.users, null, 2));
console.log(JSON.parse(JSON.stringify(users.users, null, 2)));
console.log(users.users);
Both those console logs the same thing:
[object Object][object Object]
What I want to achieve is: I want to end up with:
{
"users": {
"id": "0123456789",
"name": "GeirAndersen"
},
{
"id": "852852852",
"name": "GeirTrippleAlt"
}
}
When I get this far, I am going to write back to the .json file, but that part isn't an issue.
That's not really a valid data structure, as you're trying to add another object to an object without giving that value a key.
I think what you're really looking for is for 'users' to be an array of users.
{
"users": [
{
"id": "0123456789",
"name": "GeirAndersen"
},
{
"id": "852852852",
"name": "GeirTrippleAlt"
}
]
}
You can easily create an array in JS and the push() new items into your array. You JSON.stringify() that with no issue.
const myValue = {
users: []
};
const newUser = {
'id': '0123456789',
'name': "GeirAndersen'
};
myValue.users.push(newUser);
const strigified = JSON.stringify(myValue);
I'm working on a command that will automatically fetch a file from a link once a day and extract two of the elements in it and send that as a message in a channel.
My issue here is that I'm having issues actually getting the file downloaded. I've been trying several different functions to fetch the file but nothing has worked so far. I have attached one of the functions I've tried below.
async function getQuote () {
const url = "https://quotes.rest/qod?category=inspire";
const path = Path.resolve(__dirname, 'temp', 'qod.json')
const writer = fs.CreateWriteStream(path)
const response = await axios({
url,
method: 'GET',
responseType: 'stream'
})
response.data.pipe(writer)
getQuote();
return new Promise((resolve, reject) => {
writer.on('finish', resolve)
writer.on('error', reject)
})
}
fs.readFile('./temp/qod.json', 'utf8', function (err, data) {
if (err) throw err;
var obj = JSON.parse(data);
msg.channel.send(data);
})
The file I'm trying to work with here looks something like this:
{
"success": {
"total": 1
},
"contents": {
"quotes": [
{
"quote": "What you do speaks so loudly that I cannot hear what you say.",
"length": "61",
"author": "Ralph Waldo Emerson",
"tags": [
"action",
"inspire",
"leadership",
"management",
"tod"
],
"category": "inspire",
"language": "en",
"date": "2020-08-23",
"permalink": "https://theysaidso.com/quote/ralph-waldo-emerson-what-you-do-speaks-so-loudly-that-i-cannot-hear-what-you-say",
"id": "eZ0NtMPtGp8c5eQJOBfJmweF",
"background": "https://theysaidso.com/img/qod/qod-inspire.jpg",
"title": "Inspiring Quote of the day"
}
]
},
"baseurl": "https://theysaidso.com",
"copyright": {
"year": 2022,
"url": "https://theysaidso.com"
}
}
It wants to download as a json file, but when visiting the link, it is listed as a xml document.
How would I go about getting this downloaded and extracting two lines from it? If you're wondering, the two lines are the quote and author lines.
Thanks!
I copy your code and run my local machine and everythin fine.
Limitations are like mirages created by your own mind. When you realise that limitation do not exist, those around you will also feel it and allow you inside their space. - Stephen Richards
Looks like you are trying to write the result to a file and then read from the file which is not efficient. Here's a much simpler way of doing it.
async function getQuote() {
const url = "https://quotes.rest/qod?category=inspire";
const response = await axios(url);
const result = response.data;
/*
result =
{
"success": {
"total": 1
},
"contents": {
"quotes": [
{
"quote": "Limitations are like mirages created by your own mind. When you realise that limitation do not exist, those around you will also feel it and allow you inside their space. ",
"length": "171",
"author": "Stephen Richards",
"tags": [
"inspire",
"motivational",
"positive-thinking",
"self-empowerment",
"self-help",
"self-improvement",
"wealth",
"wealth-creation"
],
"category": "inspire",
"language": "en",
"date": "2020-08-24",
"permalink": "https://theysaidso.com/quote/stephen-richards-limitations-are-like-mirages-created-by-your-own-mind-when-you",
"id": "OLSVpLiSwrWplvCcFgPPiweF",
"background": "https://theysaidso.com/img/qod/qod-inspire.jpg",
"title": "Inspiring Quote of the day"
}
]
},
"baseurl": "https://theysaidso.com",
"copyright": {
"year": 2022,
"url": "https://theysaidso.com"
}
}
*/
//this is an array of quote objects
const quotes = result.contents.quotes;
//extracting first quote object from the array
const quoteObject = quotes[0];
//extracting quote text and author from quote object
const quote = quoteObject.quote;
const author = quoteObject.author;
//the >>> will make it look like a quote in discord.
console.log(`>>> ${quote}\n- ${author}`);
//send the formatted quote to the channel
msg.channel.send(`>>> ${quote}\n- ${author}`);
//if for some reason you want to save the result to a file
fs.writeFile(filePath, result, function(err) {
if (err) throw err;
console.log('Saved!');
});
}
getQuote();
<script src="https://cdn.jsdelivr.net/npm/axios/dist/axios.min.js"></script>
I would suggest simply reading the quote to an object, then creating a string using interpolation and send it on the discord channel:
async function getQuote () {
const url = "https://quotes.rest/qod?category=inspire";
console.log("getQuote: Reading quote...");
// Get the response as an object
const response = await axios({
url,
method: 'GET'
})
// Use destructuring to get the quote and author
let { quote, author } = response.data.contents.quotes[0];
// Format our quote
let data = `${quote} - ${author}`;
// Add a console.log for debugging purposes..
console.log("getQuote: Sending quote:", data);
// Send the quote on the channel
msg.channel.send(data);
}
Todays quote would then look like so:
Limitations are like mirages created by your own mind. When you realise that limitation do not exist, those around you will also feel it and allow you inside their space. - Stephen Richards
How can i extract text from the below json, As it contains \" it is giving me an undefined
{
"answers":[
{
"questions":[ ],
"answer":"{\"text\":\"I am Ellina. I can't believe you forgot my name\",\"speech\":\"I am Ellina. I can't believe you forgot my name\"}",
"score":100,
"id":106,
"source":"Editorial",
"metadata":[
],
"context":{
"isContextOnly":false,
"prompts":[
]
}
}
],
"debugInfo":null,
"activeLearningEnabled":false
}
i tried using console.log(Answer:
${JSON.stringify(res.data.answers[0].answer.text)}); and also
console.log(Answer: ${res.data.answers[0].answer.text});
The value of answer is a string.
It isn't an object so it doesn't have a text property (which is why it is undefined).
It appears to be JSON, so you can parse it:
const answer = JSON.parse(es.data.answers[0].answer);
const text = answer.text;
Note that having a JSON text where one of the values in it is a string representation of another JSON text is a good sign of really bad data format design.
Changing the API so it returns answer as an object instead of a JSON representation of an object would be a better approach.
You will have to do the following -
let parsed = JSON.parse(input.answers[0].answer);
where input is the json given by you. Also if you have a long list and you want the json to be parsed automatically then you can do something like this -
input.answers = input.answers.map((answer)=>{
answer.answer = JSON.parse(answer.answer);
return answer;
})
The above code will automatically turn your json string to a parsed JSON.
let input = {
"answers": [{
"questions": [],
"answer": "{\"text\":\"I am Ellina. I can't believe you forgot my name\",\"speech\":\"I am Ellina. I can't believe you forgot my name\"}",
"score": 100,
"id": 106,
"source": "Editorial",
"metadata": [
],
"context": {
"isContextOnly": false,
"prompts": [
]
}
}],
"debugInfo": null,
"activeLearningEnabled": false
}
console.log(input);
input.answers = input.answers.map((answer) => {
answer.answer = JSON.parse(answer.answer);
return answer;
});
console.log(input);
I have a CSV file and I want to parse it using PapaParse. How do I do this properly?
I have so far:
Papa.parse(fileInput, {
download: true,
complete: function(results) {
console.log(results.data);
console.log(results.errors);
}
});
However, is there a better way to do this? Is this the proper way to get errors? The documentation didn't emphasize download: true or anything so I was wondering if there are any experts on this subject here.
EDIT: Also, am I suppose to further parse the file with papacsv or do it in react. For instance, if I have multiple arrays in my data file which have a similar name reference. Should I initially somehow parse the file so it groups all those references together and how would I go about doing this?
For instance,
Date, Name , Win/Lose
I want to group all the winners together. How do I do that?
The method you are using of Papa parse, is for remote CSV.
download: true is for downloading the remote file.
By using Papa parse, this is the only way of getting errors, data, meta with parse result object.
//If(header:true)
var data = [
{
"date": "8/12/2018",
"name": "foo",
"win/loose": "win"
},
{
"date": "8/12/2018",
"name": "foo",
"win/loose": "loose"
},
{
"date": "8/12/2018",
"name": "foo1",
"win/loose": "win"
},
];
var winners = data.filter(d => d['win/loose'] == 'win');
console.log(winners);
//If you want to group winners and losers then:
var grouped = data.reduce(function(acc, co) {
var key = co['win/loose'];
if(!acc[key]) {
acc[key] = [];
}
acc[key].push(co);
return acc;
}, {});
console.log(grouped);
This'll give you separate array of winners from extracted data.