this is a very specific request, and for that i apologise, but i am at a loss for what to do..
for a javascript project i am working on i want to be able to parse javascript with python and i found this implementation`port of the original narcissus called pynarcissus:
https://github.com/jtolds/pynarcissus
the problem for me is the information is buried in the python class structure.. something with which i am only vaguely competent.. and i want the output to be in JSON
i have tried to mine the data out but each time the JSON is invalid
my question is how would you go about doing something like this? i'd appreciate any specifics because the project contains nested classes of disparate types creating what seems to be a wholly unique problem
here are my attempts:
i took the return value for parse() and created a function that descends through the class structure returning values based on their type: 'str', 'int', 'bool', 'NoneType', 'list', 'dict', 'main.Node', 'main.Tokenizer', 'main.Object'; but the returned object is missing some properties in the classes, ie 'type', while retaining others like 'type_', also tokenizer always contains the same values
i took the output of the str function that the program prints to stdout, removed the clear and copy functions and the tokenizer: [object Object], then tried to manually add in double quotes where necessary to make the output a valid JSON object.. a few problems here, first off ignoring the tokenizer object seems like i am missing out on vital information, and the other problem was that sometimes there is "value" : "{" and sometimes there is "value" : { .. }, after completing the work the JSON was invalid
assuming the issue lied in the "value" : { .. } issue i resolved to add a new function identical the the str function but instead of printing just %s values, it would print \"%s\" where necessary.. now, i could differentiate between "value": "{" and "value" : "{ .. }" but i would have to go through and manually remove the quotes around objects.. after doing so the JSON was invalid
i've tried every copy'pasta solution for python class to json from the stacks but the nested aspect of the classes along with the changing types add complexity.. some properties lack a .dict even when the type is "class 'dict'" so the one'method'fits'all lambda solutions fail
for posterity:
once i massaged the code of pynarcissus to print json i found that the process fails on nested functions.. hilariously exactly the reason i stepped away from my homebrew method
in another thread(i) #firstfire suggested esprima and pyesprima
so i looked into the esprima suggestions, and after a bit of 2to3`ing and some more work returning valid json i got it working and so far it fits my needs perfectly
check out the issue, pull request, and fork:
issues : https://github.com/int3/pyesprima/issues/1
pr : https://github.com/int3/pyesprima/pull/2
fork : https://github.com/ghissues/pyesprima
(i) http://www.linuxquestions.org/questi....php?p=5364199
the old answer::
closed the issue and added a pull request
https://github.com/jtolds/pynarcissus/issues/6
you can check out the fork if you got here by looking for a way to parse javascript to json in python 3
https://github.com/ghissues/pynarcissus
Related
What I did here is trying to replace JSON.stringify() with something else, to avoid RangeError. And I found the method below.
This is my code: (I use big-json module and this module implement json-stream-stringify for stringify)
const data = generateData();
const stringifyStream = json.createStringifyStream({
body: data
});
stringifyStream.on('data', function(str) {
console.log(str.toString());
})
I tried to print the str, it will print the string type data line by line, my question is: how to write the complete str to a file? I tried to add fs.writefileSync() after the console.log() and it will only write in the last line of str. Sorry I am new to stream and node.js. Hvae no idea how to fix it.
The other question is, can I find a way to re-format the output string just like JSON.stringify(data, null, 2) did?
FYI: The data is a really deep object. No circular, but may have repeated part. And I prefer to keep everything unchanged and not pruned.
Really appreciate any help here! Thank you for your time.
Have you tried separating JSON.stringify from fs.writeFileSync ? just to know wich one is actually having problems.
If it is JSON.stringify I suggest you to fragment the object in multiple ones or filter what you use. Whatever you can do to reduce the actual data. We don't know what de data really is so I cannot make assumptions on that but you never have those problems generally, if it's caused by the data being too big, you are doing things wrong!
If the problem is fs.writeFileSync and you have your JSON stringified then try using fs.createWriteStream to write the data.
I'm triying to find the records that includes "SO -" or "NS - SO" or "SO –" or "SWAT" on THE "RESUMEN" field from a CSV file to asigne a new category (in this cases would be "Call Center"). So, I used "indexOf" funtion witch worked so well.
The problem comes when I change the data source (It is a CSV too), this gave me next error on that step:
"Caused by: org.mozilla.javascript.EcmaError: TypeError: Cannot call method "indexOf" of null (script#2)"
The objective is to assign a category by identifying the words on the source file
My code
if (RESUMEN.indexOf("SO -")!=-1 || RESUMEN.indexOf("NS - SO")!=-1 || RESUMEN.indexOf("SO –" )!=-1 || RESUMEN.indexOf("SWAT")!=-1)
{
var RESULTADO = "Call Center"
}
else RESULTADO = ""
I expect to assigne call center category like I got with the first file (I did not change nothing)
regards!
You're overcomplicating the issue.
Before the answer, remember something, there are several steps, and combinations of steps, that achieve an incredible number of transformations to make usable patterns, the last resort IS User defined Java Expression.
Seems like what you want to achieve is a Value Mapping, thou the difference from a direct value map in your case, is that the row you're testing must contain "SO -", and the other cases, somewhere in the text.
With this simple filter, you can transform your data that contains those informations as you desire, and on the "FALSE" side, treat it for errors.
This will expand your transformation a bit, but when you need to change something it will be easier than with a single step with a lot of code.
As another answer pointed out, you can achieve the same result with different steps, you don't need the javascript step.
But, if you want to go that route, you should first convert null values into, e.g., empty strings.
Simply add this to the beginning of your javascript code:
if (!RESUMEN){ RESUMEN = ''}
That'll convert nulls to empty strings and then indexOf returns correctly.
The situation:
My sensor measures data, that I process in a NodeRED-function and afterwards parse into a JSON-object. The NodeRED-function allows me to write JavaScript-Code. The JSON-object gets send to a receiving module, written in C++, that works on the JSON with the JSON_spirit library. I can not change the receiving module.
The problem: The receiving app tries to get one value of the JSOn with the function value.get_float(). The sensors sometimes measure an exact 1.00. That gets passed to the JSON as {"value":1}. The receiving module terminates with the Error:
terminate called after throwing an instance of 'std::runtime_error'
what(): get_value< real > called on integer Value
Obviously, the function value.get_float() seems not to be able to change an 1 into a 1.0 and, as mentioned, I can not change the used function. So, I need to find a way to parse {"value":1.00} into the JSON.
What I have tried:
I tried in my NodeRED function value.toFixed(2) but this would return a string {"value":"1.00"}.
So, I tried to parse the string as a float again like this
value.toFixed(2);
value = parseFloat(value);
But this would lead for a 1.00 again to a JSON like this: {"value":1}.
I tried some tricks with rounding as well, but as soon as JavaScript can omit unnecessary decimals, it does. So, I havent found a solution yet.
Any ideas are welcome.
P.S.: This is my first time ever StackOverflow question so please do not be too harsh on me :)
Edit: I found the following workaround.
I use value.toFixed(2); in a first note to get {"value":"1.00"}. Later on, I use a regular expression on the string in a change-Node in NodeRED.
RegEx:
"Value":\"(\d+\.\d{2})\"
Replace with:
"Value":$1
My real case was a bit more complex than the example, so the regex was a little longer. But regex101 helped a lot.
Think this post was already been there:
Force float value when using JSON.stringify
So i think in Javascript there is no difference between 1 and 1.0
I am surprised that no one on StackOverflow asked this question before.
Looking through the JSON object documentation and a quick google search did not yield satisfactory results.
What's the advantage of it? How does it work?
Edit: To make it clear, take a look at this flatten/un-flatten example.
Fastest way to flatten / un-flatten nested JSON objects
Thank you.
There are many situations where you get JSON text that was automatically built by some library. Throughout the programming languages, there are many libraries that build JSON text (one example is here).
Whenever libraries add some additional object or array wrappings, you might want to get rid of them maybe because you send the JSON to the server and your code there crashes because it expects a primitive value instead of an object (or an array). Or, if your JSON is a server response, you don't want the resulting Javascript code having to differ between object/array or not object/array. In all these cases, flattening is helpful as it will save you time. You will have to implement lesser if/elses, and you can reliably expect your data structure to be as flat as possible.
The other approach to improve code for the scenario mentioned is to write the code in a maximal robust way so there is no way for it to crash by superfluous wrappings ever. So always expect some wrappers and get it's contents. Then, flattening is not needed.
You see, it depends on what is building the JSON and what is parsing it. The building may be out of your scope.
This leads also to data model questions. I've worked with XML code that needed to be parsed quiet a different way if there where 0 entries of some XY, or if there were >0 entries of some XY. Having a wrapper that is allowed to have 0 or more entries of some XY will make live easier. These are data model desicions.
In all cases where the JSON represents an object structure that I've combined manually, I expect it not to change. So flattening something I've designed in detail would be disturbing. Standard operations as far I've seen them do not need flattening (e.g. JSON.stringify(), json_encode() etc.)
Here's a simple scenario: In a web app you have an HTTP POST that is updating a complex relational object.
POST
update=1
&user.id=12345
&user.email=testmail#domain.tld
&user.profile.name=Mr. Test
&user.profile.age=42
&user.profile.friend.0.email=tom#domain.tld
&user.profile.friend.1.email=sally#domain.tld
&user.profile.friend.2.email=bob#domain.tld
&user.profile.skill.0.id=100
&user.profile.skill.0.name=javascript
&user.profile.skill.1.id=200
&user.profile.skill.1.name=piano
Everything is already in a flat structure, so why not have a simple one-to-one binding? If you had a list of constraints or security requirements that you needed to enforce you could validate them by searching directly on the sorted key list.
Flat structures are easier for people to understand and work with there's even some cross-over with database de-normalisation. It also allows for context specific security and constraints to be implemented in a readable, but more verbose way.
When showing a user's view in full you may want to hide the display of the primary key ID for the user's list of skills.
"user.profile.skill.#.id": { hidden: true, readonly: true }
But when looking directly at a skill (to possibly edit it as an administrator) you may want to see the ID.
"skill.id": { readonly: true }
If you were writing a user-centric/self-service type CMS application you'd get more users on board and able to contribute using a straightforward flat model (flat abstraction of the underlying nested relational model) than you would with just the nested model.
TLDR: Flat is easier to read than nested. While programmers can handle nested schemas, recursive parsing and processing; end-users and admins usually prefer that part abstracted away.
I realize this is a 5 year old question at this point, but I figured, I'd add my thoughts to it, in case someone runs into a similar use case and finds this useful.
One of the use cases why you would want to flatten a JSON object, is for dynamic template binding via Regular Expression (RegEx) string interpolation. Well wasn't that a mouthful 👀😅? It simply translates to "template filling a string without hardcoding".
Ok Imagine a scenario, you have a template string like this for an email:
Hello {{firstName}},
It is amazing you chose to join our site. We are happy to have you on board.
To get started, we would really love it if you can confirm your email address
by clicking on the link: {{confirm_url}}.
Welcome aboard
The Team!
Given the following JSON object in memory:
{
"user" : {
"prefix" : "Dr.",
"firstName" : "Awah",
"lastName" : "Teh",
"email" : "awah#superduperubercoolsite.com",
"address" : {
"street": "100 Main St",
"city" : "PleasantVille",
"state" : "NY",
"phone" : "+1-212-555-1212"
}
},
"meta" : {
"confirm_url" : "http://superduperubercoolsite.com/confirm/ABC123"
}
}
it seems super simple to do a Regular Expression replace like so (assuming our email template string was stored in a variable named template and the json object was stored in a variable called templateData:
template = template.replace(new RegExp('{{firstName}}', 'g'), templateData.user.firstName);
template = template.replace(new RegExp('{{confirm_url}}', 'g'), templateData.meta.confirm_url);
Easy right? --> Actually yes! How about this email had 10 templated fields, or you wanted to decouple the template from the code, by storing it in a separate system like SendGrid, where your cool head of marketing can access the template and make changes to the copy-language, without having to call someone from engineering to make changes to the code, test the code and redeploy to production (what a hassle).
This is exactly where flattening of the JSON comes save the day!
Now there are many ways to flatten JSON, I have attached a link to a codepen I wrote that has logic to flatten JSON (actually, I demonstrate two similar but different approaches in the methods flattenJSONIntoKVP and flattenJSONIntoRAW check 'em out!).
That said, there are other implementations out there, and it is worth remembering that the focus on this post is to discuss the WHY JSON flattening could be useful, not the HOW.
Moving on! Assume you flattened the JSON from above (using my implementation that results in key value pairs) to something like this:
[
{ "key": "user.prefix", "value": "Dr."},
{ "key": "user.firstName", "value": "Awah"},
{ "key": "user.lastName", "value": "Teh"},
{ "key": "user.email", "value": "awah#superduperubercoolsite.com"},
{ "key": "user.address.street", "value": "100 Main St"},
{ "key": "user.address.city", "value": "{PleasantVille"},
{ "key": "user.address.state", "value": "NY"},
{ "key": "user.address.phone", "value": "+1-212-555-1212"},
{ "key": "meta.confirm_url", "value": "http://superduperubercoolsite.com/confirm/ABC123"},
]
Now, my friend, you are cooking with GAS!
Why, cause now you can dynamically interpolate the template string with values from the JSON object without giving too much worry to the structure of the JSON (if it changes due to the application evolving, you don't have to also remember to come down here and change this interpolation code -- you simply just have to update the email template itself, which mind you, is on SendGrid [per this example]).
So how to do it you say?: Simple, iteratively. Let's assume that flattened from above was stored in a variable called flatJSON:
///Notice how I use Javascripts native string interpolation to create my RegExp
///Also note that I am replacing the dot (.) in my flattened JSON variable names with a double underscore (__), I only do this because my intended target is SendGrid, and I don't believe it likes dots in its template placeholders.
flatJSON.forEach(kvp=>template = template.replace(new RegExp(`{{${kvp.key.replace(/\./g, '__'}}}`, 'g'), kvp.value));
That's it, one line of code to replace possibly 10 or even hundreds or even thousands (ok.. maybe not thousands, but you get the point).
Ohh! almost forgot, we need to update our template string.
Notice how now, in our new templated string we can use a somewhat FQDN style variable to map back to our original JSON (Ideally if SendGrid supported dots in their template placeholders, this would look super sweet but alas, can't always win everything!😭.
Hello {{user__firstName}},
It is amazing you chose to join our site. We are happy to have you on board.
To get started, we would really love it if you can confirm your email address
by clicking on the link: {{meta__confirm_url}}.
Welcome aboard {{user__prefix}} {{user__lastName}}!
The Team!
Et Voila!
Just like that, we have accomplished some good here today; we have:
Answered the WHY of flattening JSON objects
We dibble-dabbled into the how, with the codepen example
And we even overviewed a use case where taking advantage of JSON flattening can help you write durable dynamic code, that evolves as your underlying object structures change -- and that doesn't require you to leverage the big bad ugly eval method (we can talk about big bad ugly eval on another post).
The JSON file which I am using is being interpreted differently than I would expect.
The external JSON file:
{"organizations":[{"member":{"source":"TCAN","target":"Resilient Toronto","category-source":"organization","category-target":"organization"}},{"member":{"source":"City of Toronto","target":"","category-source":"organization","category-target":""}},{"member":{"source":"Resilient Toronto","target":"City of Toronto","category-source":"organization","category-target":"organization"}},{"member":{"source":"Rita Bijons","target":"ZCO","category-source":"volunteer","category-target":"organization"}},{"member":{"source":"Rita Bijons","target":"Green 13","category-source":"volunteer","category-target":"community group"}},{"member":{"source":"Green 13","target":"TCAN","category-source":"community group","category-target":"organization"}},{"member":{"source":"ZCO","target":"TCAN","category-source":"organization","category-target":"organization"}}]}
I don't understand why the source field is being interpreted as an object while category-source is not. Is it just because of the name? and why is there a category field in the source object? And the name is undefined? Obviously I am missing some basic understanding. Is there somewhere this is documented?
I've just tested your JSON with PrettyPrint and it's formatted properly. So it appears that the problem is with the software/code you are using to display(format) the JSON data.
It looks like you are running your data through the force layout and it is conflicting with your property name "source". Specifically, the force.links call will change the data by adding "source" and "target" properties that look just like the structure you reference.
Check to see if you're passing your data into force.links at any point.