data cleansing javascript objects - javascript

I have a http server built in node.js and 'connect'. The web service currently parses JSON requests to an Object, does "stuff", and returns a synchronous response. The JSON is from an ecommerce cart and I end up with an Object that looks like (abridged version):
var myObj = {"request":{"industry":"social","transactionId":"gdfs23s","billing": {"addressLine1":"911 Fallen Street","addressLine2":"1 2"},"shipping":{"addressLine1":"1523 Willow Tree LAne","addressLine2":"unit 15"}}}
I want to clean up the data, performing tasks such as removing extra white space, normalizing postal abbreviation, ie street to st, etc.
I've written a series of regular expression that successfully achieve the cleansing/normalization.
However what I am unsure of is how to do this efficiently AND elegantly apply these cleanse processes to an Object in JS /Node.js. I will have scenarios where I want to cleanse request.billing.addressLine1, request.shipping.addressLine1 with the same regex pattern
I could of course do something like:
var addressCleaner= new RegExp("( str| street| strt)","g");
myObj.request.billing.addressLine1.replace(addressCleaner, "st");
myObj.request.shipping.addressLine1.replace(addressCleaner, "st");
But I dont feel this is very DRY and furthermore its not being done very "node"ishly.
Any suggestions or example approaches? I would like to avoid using a package like Mongoose etc. to do this as the type of normalizing i'm doing does not just consist of making sure a string is a string ,etc.
Thanks in advance.

So, I would suggest to have a hash with all normalizers, and seperately to have a list of properties,
which needs to be normalized. To have the idea here some code:
var Normalizers = {
// -> function
trim: function(str) { return str.trim(); },
// -> array [RegExp, StringToReplace]
street: [ /(str|street)/g, 'st']
//...
};
var properties = {
'request.billing.addressLine1': ['trim', 'street'],
// ..
};
obj_normalize(myObj, properties, Normalizers);
The code for obj_normalize/obj_getProprety/obj_setProperty I moved to the gist.

If your regex is applicable to every string found within the object you can simply recurse through the object and apply the regex to every string.
A general purpose object traversal function is very useful for something like this:
function object_traverse (name,obj,fn) {
obj = fn(name,obj);
if (obj instanceof Array) {
for (var n=0;n<obj.length;n++) {
obj[n] = object_traverse(n,obj[n],fn);
}
}
else if (typeof obj != "string" && typeof obj != "number") {
for (var n in obj) {
obj[n] = object_traverse(n,obj[n],fn);
}
}
return obj;
}
Now you can do this:
myObj = object_traverse('',myObj,function(name,obj){
if (typeof obj == "string" && name.match(/address/i)) {
obj = obj.replace(addressCleaner, "st");
}
return obj;
});

I'd have a model built from JSON files and serialize it as I see fit. This would avoid matching or searching for properties which couldn't possibly exist in the source. Some example:
function makeAddress(rawAddress) {
return { street: rawAddress["str"] ||
rawAddress["street"] ||
rawAddress["strt"],
. . . };
Being equipped with this function, say, then you have an array of "address" object, then converting them would be a matter of:
addresses.map(makeAddress);

Related

Convert Unity3D json to Javascript Object json format

My whole workflow:
Unity3D(encoding class data in json format) -> WebGL Build -> Javascript html -> Socket.IO
Those data type(int, byte[], string) should be readable in socket.io.
This is the function I used in Unity3D, in order to convert a class to json string.
JsonUtility.ToJson();
When I print in node.js, I got this and it couldn't understand the data type.
{"Type":0,"Str":"AAA","Byte":[0]}
If it's correct, I will get below result when I print in node.js.
{ Type: 0, Str: 'AAA', Byte: [ 0 ] }
Thus, I need a solution to convert them.
Could someone please tell me the easiest way to convert them in C#?
UPDATE: more info
Below C# script is working in Unity3D, and I want to convert this:
Application.ExternalEval(#"
if(typeof window.socketIO !== 'undefined')
{
window.socketIO.emit('" + e + #"', " + data + #");
}
");
https://docs.unity3d.com/Manual/webgl-interactingwithbrowserscripting.html
Because "Application.ExternalEval" is depreciated in latest Unity, I need to convert them into something like this:
mergeInto(LibraryManager.library, {
WebSocketEmitData: function (_e, _data) {
var e = Pointer_stringify(_e);
var data = Pointer_stringify(_data);
if(typeof window.socketIO !== 'undefined')
{
window.socketIO.emit(e, data);
}
}
});
Thanks!
I use Javascript/Unity/C# heavily on my current project, I strongly advice you to use JSON.NET For Unity it will emit a universal json format, that is similar to the one on the javascript JSON class.
Usage pretty similar to unity's primitive json utility.
JsonUtility.ToJson => JsonConvert.SerializeObject
JsonUtility.FromJson => JsonConvert.DeserializeObject<T>
That being said, let's convert a json string from the C# object to a javascript object.
// Convert your json string to a native javascript object.
var jsObject = JSON.parse(str);
console.log(jsObject.Type); // outputs 0
console.log(jsObject.Byte.length); // outputs 1
console.log(jsObject.Str); // outputs AAA
// Convert your native javascript object to another of certain type,
// will assume the desired javascript type is Player in a namespace Database
var player = Object.assign(new Database.Player(), jsObject);
Also instead of if(typeof window.socketIO !== 'undefined') use if(typeof window.socketIO) it is cleaner, checks for both null and undefined.
Finally I found the answers myself. I didn't notice that I can emit object instead of string message. and now it works!
WebSocketEmitData: function (_e, _data) {
var e = Pointer_stringify(_e);
var data = Pointer_stringify(_data);
var obj = JSON.parse(data);
if(typeof window.socketIO !== 'undefined')
{
window.socketIO.emit(e, obj);
}
}

Take an API response object and replace (map) all the keys

I'm adding some APIs to a JavaScript project to replace what used to be multi-field manual data entry with lookups. (E.g., we want to stop asking you 20 questions about a car, and instead just ask your VIN and autopopulate the other 19 answers from a VIN decoder.)
I'm spending more time than I'd like mapping data from the response schema into the existing internal schema of my app. Lots of the work requires a human touch to suss out synonyms, like:
internal.postal_code = api.zipCode;
Some times I find myself writing a really gnarly if to avoid a sometimes-null or missing object half way down a deep tree, like
if(api.a && api.a.b && api.a.b.c){
internal.z = api.a.b.c.d;
}
Is there a good library that would let me write a simple map and do all this work for me? A map might look like:
map = {
'zipCode' : 'postal_code',
'a.b.c.d' : 'z'
};
mapperTool( api, internal, map );
(Note the internal object is stitched together from several APIs and pre-existing tools, so adding or overwriting properties on internal is better than outputting a new object.)
Try this:
function mapperTool(source, desc, map) {
Object.keys(map).forEach(function(key) {
var value = key.split('.').reduce(function(obj, name) {
if (obj && obj[name]) {
return obj[name];
}
}, source);
desc[map[key]] = value;
});
}
var map = {
'zipCode' : 'postal_code',
'a.b.c.d' : 'z'
};
var internal = {};
mapperTool({a:{b:{c:{d: 10}}},zipCode:20}, internal, map);
document.body.innerHTML = '<pre>' + JSON.stringify(internal, true, 4) + '</pre>';

json nested lookup using string based key in javascript

With a javascript json object like this:
var data = {
blog : {
title: "my blog",
logo: "blah.jpg",
},
posts : [
{
title: "test post",
content: "<p>testing posts</p><br><p>some html</p>"
},
]
}
var lookup = "blog.title" //this gets generated from a template file
Now I know you can do something like, but these don't quite do what I need:
console.log(data['blog']); //works
console.log(data.blog.title); //works
console.log(data['blog']['title']); //works, but i dont know the depth of the lookup
But I need to be able to do something like the code below because I can't hardcode the structure, it gets generated and stored in lookup each time. Do I have to build this functionality using string cutting and recursion?? I really hope not
console.log(data['blog.title']); //does not work
console.log(data[lookup]); //does not work
EDIT....
Okay, possibly found a workaround. I don't know if this is safe or recommended practice, so comments on that would be great. Or alternative methods. So combining this with the code above.
var evaltest = "var asdf ="+JSON.stringify(data)+";\n"
evaltest += "asdf."+lookup
console.log(eval(evaltest)) //returns correctly "my blog" as expected
You could use dottie https://www.npmjs.org/package/dottie, which allows you to deep traverse an object using strings
var values = {
some: {
nested: {
key: 'foobar';
}
}
}
dottie.get(values, 'some.nested.key'); // returns 'foobar'
dottie.get(values, 'some.undefined.key'); // returns undefined
you could use:
data['blog']['title']
I've experimented with a couple ways of doing this including eval and using a dictionary lookup with switch(exp.length). This is the final version (comments stripped) I created:
var deepRead = function (data, expression) {
var exp = expression.split('.'), retVal;
do {
retVal = (retVal || data)[exp.shift()] || false;
} while (retVal !== false && exp.length);
return retVal || false;
};
//example usage
data = {
a1: { b1: "hello" },
a2: { b2: { c2: "world" } }
}
deepRead(data, "a1.b1") => "hello"
deepRead(data, "a2.b2.c2") => "world"
deepRead(data, "a1.b2") => false
deepRead(data, "a1.b2.c2.any.random.number.of.non-existant.properties") => false
Here's the Gist with full commenting: gist.github.com/jeff-mccoy/9700352. I use this to loop over several thousand items and have had no issues with deep-nested data. Also, I'm not wrapping in try/catch anymore due to the (small) performance hit: jsPerf.

Custom object to JSON then back to a custom object?

I've seen very similar questions to this, but I can't quite decide if they was answered clearly - maybe I'm being a bit dense, sorry.
I want to have the convenience (and clarity) of my own object, call it a CardboardBox(). It won't contain code, just data. I want to write this to a database and read it back later, but obviously, it is a type Object() when it's read back. All I can think of to find out what it used to be is:
Have a member variable type that I set to CARDBOARD_BOX
Instantiate a new CarbardBox() and use a function (in the box) to copy the properties of Object() to the new CardboardBox() object
Is there a better way of doing this? I'm pretty sure I can change the actual type.
function CardboardBox() {
this.type = "CARDBOARD_BOX"
this.name = "No set";
this.populate = new function(obj) {
// populate this object with obj properties
}
var box = new CarboardBox(); // CarboardBox
box.name = "My Box";
send = JSON.stringyfy(box);
.
.
.
obj = JSON.parse(send); // Object
if (obj.type == "CARDBOARD_BOX") {
savedBox = new CardboardBox();
savedBox.populate(obj);
}
Thanks in advance...
Steve
[edit] My test code.
function CardboardBox(n) {
this.name = n;
}
var box = new CardboardBox("My Box");
send = JSON.stringify(box); // JSON CarboardBox()
obj = JSON.parse(send, function fn(obj) { // Object() returned
log("OB: "+obj.type);
return obj.type === 'CardboardBox' ? new CardboardBox(obj) : CardboardBox;
});
console.log(obj);
Output is:
OB: undefined utils.js:40
OB: undefined utils.js:40
function CardboardBox(n) {
this.name = n;
}
One possible solution is the following:
function CardboardBox(n) {
if(typeof(n) == 'string') {
//build from name string
this.name = n;
} else {
//build from object
this.name = n.name;
}
//add in this object's "type" in a place
//that is unlikely to exist in other JSON strings
this.__type = 'CardboardBox';
}
var box = new CardboardBox("My Box");
send = JSON.stringify(box), // JSON CarboardBox()
obj = JSON.parse(send, function(key, val) {
//if this is an object, and is CardboardBox
if(typeof(val) === 'object' && val.__type === 'CardboardBox')
return new CardboardBox(val);
return val;
//or if your object is in a context (like window), and there are many of
//them that could be in there, you can do:
//
//if(typeof(val) === 'object' && context[val.__type])
// return new context[val.__type](val);
});
console.log(obj);
Basically store the object type in a place you know to look for later on when parsing the json. if you have multiple objects you can instantiate in a single scope the second parse method may be more appropriate. This also will account for objects in the JSON that are not CarboardBoxs.
Edit Here is a jsFiddle of this method in action.
Overall, you're correct: Javascript doesn't have any built-in way to serialize anything beyond plain objects, so going to and from JSON will not produce a particular class when you deserialize it. So you need to either work out serialization/deserialization yourself, or use a library that provides some support.
I personally like Backbone.js for this problem, as it handles serializing and deserializing quite well. You define a model class, which include a method to save its data to a server in a serialized form, and a method to deserialize it back to the model. The key design issue here is that deserializing is performed knowing the model you're deserializing to:
you either call myModel.fetch() to get data from the server based on the model id, or
you pass a bunch of new data to the model constructor: new Model(serializedData), or
you pass an array of data for multiple models to a collection that knows the model type: new ModelCollection(arrayOfSerializedData).
What Backbone doesn't do is deal with type-casting data of an unknown type. When I've dealt with this, I've usually done something similar to #Chad's response, but using an intermediary; you could see this as a proxy model, or as a factory:
var classes = {
CardboardBox: ...,
AluminumBox: ...
}
function Deserializer(json) {
// parse if you're actually dealing with a string
var data = JSON.parse(json),
// now look for some custom type flag - you'll need to set this yourself
type = data.type,
// class lookup, perhaps with a default
Cls = classes[type] || DefaultType;
return new Cls(data);
}
var obj = new Deserializer(send);
obj instanceof CardboardBox; // should work
This still relies on a custom flag to switch types, though - I'm not sure there's any way around this.

Check if a object is defined, best practice.

I have the following JSON response from a ajax-request.
var json = {
"response": {
"freeOfChargeProduct": {
"description": "Product",
"orderQty": 5,
"productName": "XYZ",
"qty": 6,
"details": {
"price": 55.5,
"instock": "true",
"focQuantity": 1
}
},
"orderLineId": 4788,
"totalOrderLinePrice": "741.36",
"totalOrderPrice": "1,314.92",
"totalQty": 17
};
The JSON dosen't always return a "freeOfChargeProduct" property. So if I want to get the "freeOfChargeProduct" price, then I have to do the following:
var getFreeOfChargeProductPrice = function() {
var r = json.response;
if (r && r.freeOfChargeProduct && r.freeOfChargeProduct.details) {
return r.freeOfChargeProduct.details.price;
}
return null;
};
No problems. But it's very annoying to check every property in the object, so I created a function that check if a property in a object is defined.
var getValue = function (str, context) {
var scope = context || window,
properties = str.split('.'), i;
for(i = 0; i < properties.length; i++) {
if (!scope[properties[i]]) {
return null;
}
scope = scope[properties[i]];
}
return scope;
};
var price = getValue('json.response.freeOfChargeProduct.details.price');
// Price is null if no such object exists.
Now to my question: Is this a good or bad way to check if a property exists in an object? Any better suggestions/methods?
EDIT:
I don't wan't to use the &&-operator. I am lazy and I'm looking for a reusable method to check if a object (or property of a object) is defined.
:) Thanks!
Use the guard pattern:
if (json.response && json.response.freeOfChargeProduct && json.response.freeOfChargeProduct.details) {
// you can safely access the price
}
This is how the guard pattern works.
if (a && a.b && a.b.c) { ... } else { ... }
The first check is "Does the property a exist?". If not, the else-branch gets executed. If yes, then the next check occurs, which is "Does object a contain the property b?". If no, the else-branch executes. If yes, the final check occurs: "Does the object a.b contain the property c?". If no, the else-branch executes. If yes (and only then), the if-branch executes.
Update: Why is it called "guard pattern"?
var value = a && b;
In this example, the member b (the right operand) is guarded by the && operator. Only if the member a (the left operand) is truthy ("worthy"), only then the member b is returned. If, however, the member a is falsy ("not worthy"), then it itself is returned.
BTW, members are falsy if they return these values: null, undefined, 0, "", false, NaN. Members are truthy in all other cases.
if(x && typeof x.y != 'undefined') {
...
}
// or better
function isDefined(x) {
var undefined;
return x !== undefined;
}
if(x && isDefined(x.y)) {
...
}
This will work for any data type in JavaScript, even a number that is zero. If you are checking for an object or string, just use x && x.y within the if statement, or if you already know that x is an object, if(x.y) ...
You could do something like this:
try{
var focp = json.response.freeOfChargeProduct
var text = "You get " + focp.qty + " of " +
focp.productName +
" for only $" + (focp.qty-focp.details.focQuantity)*focp.details.price +
", You save $" + focp.details.focQuantity*focp.details.price;
$("order_info").innerText = text;
} catch(e) {
// woops, handle error...
}
It would generate a message like this from the provided data in your question if the fields exists:
You get 6 of XYZ for only $277,5, You save $55.5
If the data is non-existing, you'll end up in the catch block. You could always just to a Try, Catch, Forget here if you can't come up with a way to handle the error (Maybe do a new AJAX request for the data?).
This is not a syntax issue as it is a design pattern issue.
Question A.
* Do you have control of the json server?
If the answer to this is no, which I assume, the situation will be all on the client.
Please read this:
http://martinfowler.com/eaaDev/PresentationModel.html
As the server is the source, in this case it will provide the model.
This pattern specifies an additional artifact: The presentation model (PM). In javascript i would suggest two artifacts, a additional for the convertor code.
According to this design pattern the PM is responsible for converting the model to the PM, and back again if necessary. In your case no conversion from PM to M will ever occur.
This means that a js object has a method or constructor that digest the model and translate itself, with the help of the convertor (below).
Doing this you will end up with a PM looking like this:
var OrderlinePM = {
"hasFreeOfCharge": false | true,
"freeOfCharge" : {...}
`enter code here`
this.getFreeOfCharge = function() {
...
}
this.fromModel = function(jsonEntry, convertor) {
//convert this with the convertor ;) to a for this specific view usable OrderlinePM
// also inwith
...
}
enter code here
"orderLineId":0,
"totalOrderLinePrice":"741.36",
"totalOrderPrice":"1,314.92",
"totalQty":17
};
function mySpecialFunctionPMConvertor {
this.fromModel = function() {
... //do strange stuff with the model and poulate a PM with it.
}
}
Ok, I give up trying to format code in this rich text editor :(
You can have several PM:s for diffrent tasks all depending on the same model object.
In addition this will make the converter object testable in something that could be automatically executed.... err ok maby manually, but anyway.
So the problem of the cumbersome reflection code is really not a problem. But cohesion is a issue, expessially in JavaScript.

Categories