I'm using LZstring to do a UTF-16 compression on a string and storing it in a database. Specifically, it was a JSON.stringify'd data stored in a cordova.sqllite db. When I retrieve the data from the db there are times when it causes SyntaxError: Unexpected EOF
I'm thinking that depending on the input, LZString's compressToUTF16 function produces an invalid character. The weird thing is, it was stored in the DB properly (no unexpected EOF errors when storing into the DB).
Storing a normal string or a compressToBase64'd string does not cause an error so I don't think it's a problem with the DB.
So apparently, LZString's UTF16 compression sometimes generates \u2028 and \u2029 characters which are treated by the parser as line breaks (see here for more details). I replaced all occurrences of both unicodes after compression and before storage, and after retrieval and before decompression to fix the problem.
before storage
var dbvalue = LZString.compressToUTF16(JSON.stringify(value));
dbvalue = dbvalue.replace(/\u2028/g, '\u32800').replace(/\u2029/g, '\u32801');
after retrieval
var utfdata = dbvalue.replace(/\u32800/g,'\u2028').replace(/\u32801/g,'\u2029');
var value = JSON.parse(LZString.decompressFromUTF16(utfdata));
EDITED
The replacement character is not used by LZString and takes up less space. See here for more details
Related
In NodeJS Backend, I send my data to client as:-
res.end(filex.replace("<userdata>", JSON.stringify({name:user.name, uid:user._id, profile:user.profile}) ))
//No error here and Object is stringified perfectly
//user is object returned in mongoDB's result
The JSON string looks like this:
{"name":"Rishavolva","uid":"5f3ce234fd83024334050872","profile":{"pic":{"small_link":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJyZXBsaWNhcyI6MiwidXJpcyI6W3siZGJfbmFtZSI6ImlmcmRiMDAxIiwidGFibGUiOiJGSUxFIiwiaWQiOjQ4fSx7ImRiX25hbWUiOiJpZnJkYjAwMiIsInRhYmxlIjoiRklMRSIsImlkIjo0OH1dLCJ1aWRfd2hpdGVsaXN0IjoiKiIsImlhdCI6MTU5ODE2MzMzNX0.9NkGnEumn4JW8IN0KFgxgN_6_4wN8qOgezNTyzz9osY","big_link":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJyZXBsaWNhcyI6MiwidXJpcyI6W3siZGJfbmFtZSI6ImlmcmRiMDAxIiwidGFibGUiOiJGSUxFIiwiaWQiOjQ3fSx7ImRiX25hbWUiOiJpZnJkYjAwMiIsInRhYmxlIjoiRklMRSIsImlkIjo0N31dLCJ1aWRfd2hpdGVsaXN0IjoiKiIsImlhdCI6MTU5ODE2MzMzNX0.yxQ1GrhLsWPn8Qwu42EfTDXqaYwFtrM6f_7cAH2eLRY"},"aboutme":"I am Rishav Bhowmik\r\nand this is navratna pulaow"}}
and that UID is just a mongodb's primary key as string, and other two base 64 strings are just JWT tokens.
Now, when this JSON string reaches the Browser, I parse it with simple:
JSON.parse(`<userdata>`)
//remember I used filex.replace("<userdata>", JSON.stringify...) in the server
For reference, my MongoDB Document here is:
Now when JSON.parse is executed on the JSON string it will look like this on final JS code.
JSON.parse(`{"name":"Rishavolva","uid":"5f3ce234fd83024334050872","profile":{"pic":{"small_link":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJyZXBsaWNhcyI6MiwidXJpcyI6W3siZGJfbmFtZSI6ImlmcmRiMDAxIiwidGFibGUiOiJGSUxFIiwiaWQiOjQ4fSx7ImRiX25hbWUiOiJpZnJkYjAwMiIsInRhYmxlIjoiRklMRSIsImlkIjo0OH1dLCJ1aWRfd2hpdGVsaXN0IjoiKiIsImlhdCI6MTU5ODE2MzMzNX0.9NkGnEumn4JW8IN0KFgxgN_6_4wN8qOgezNTyzz9osY","big_link":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJyZXBsaWNhcyI6MiwidXJpcyI6W3siZGJfbmFtZSI6ImlmcmRiMDAxIiwidGFibGUiOiJGSUxFIiwiaWQiOjQ3fSx7ImRiX25hbWUiOiJpZnJkYjAwMiIsInRhYmxlIjoiRklMRSIsImlkIjo0N31dLCJ1aWRfd2hpdGVsaXN0IjoiKiIsImlhdCI6MTU5ODE2MzMzNX0.yxQ1GrhLsWPn8Qwu42EfTDXqaYwFtrM6f_7cAH2eLRY"},"aboutme":"I am Rishav Bhowmik\r\nand this is navratna pulaow"}}`)
I get this error:
Uncaught SyntaxError: JSON.parse: bad control character in string literal at line 1 column 702 of the JSON data
the string at position 702 of the JSON string is \n
First of all, how can \n be a control character?
What should I do to resolve this?
Has this problem arrised due to MONGODB result?
\n is a control character signifying a new line. In JSON, those control characters (more specifically the \) must be escaped inside strings.
This will raise the error:
JSON.parse(`{"hello":"world\n"}`)
This wont:
JSON.parse(`{"hello":"world\\n"}`)
So one way would be to use something like replace to ensure your aboutme is properly escaped before JSON serialization. See: How to escape a JSON string containing newline characters using JavaScript?
Ok have done some experimentation and have a solution.
The Trick is to do JSON.stringify() twice,
Like,
html_text.replace('/*<whatever>*/', JSON.stringify( JSON.stringify(the_object) ) )
If suppose html_text has a line which is
<script>
const object_inbrowser = JSON.parse(/*<whatever>*/)
// no need to add qotes, `JSON.stringify` in the server will do that for you
</script>
I have the following data coming in from my server into my js code, from my server.
{"triggers": [{"message_type": "sms","recipients": "[\"+91xxxxxxxxx\",\"+91xxxxxxxxx\"]","message": "This is a test"}]}
My code parses the above json string in the following manner.
data = '{"triggers": [{"message_type": "sms","recipients": "[\"+91xxxxxxxx\",\"+91xxxxxxxx\"]","message": "This is a test"}]}'
parsed = JSON.parse(data);
This throws the following exception
Uncaught SyntaxError: Unexpected token + in JSON at position 54
at JSON.parse (<anonymous>)
at eval (eval at <anonymous> (entry.html:2), <anonymous>:1:6)
at entry.html:298
I did a little bit of further digging and found the source of the json string.
Here is where the string is coming in from my python code
data = {"recipients": "[\"+91xxxxxxxxx\",\"+91xxxxxxxx\"]"} # This data comes in from my database, and I can't do anything about what quotes are used.
javascript_supplied_data = json.dumps(data) #This data goes to the frontend via webhook
I tried putting the same data into a json view via this online viewer, and it didn't throw any error and displayed the data correctly.
What I can't understand is, I am doing a json.dumps in my python code, so the string response should be json parsable. So why does JSON.parse throw this error?
Is there something wrong with the json libraries at the python end or the javascript end, or am I too much of a noob?.
Please help me figure out what is causing this issue, and how to solve it.
NOTE: I don't have any control over the string that comes in from the server.
When you have valid JSON, but put it in a string literal, the escapes treated by the literal notation in JavaScript make the string different. The backslashes are interpreted as for escaping the next character.
So either you have to go through the string literal and double all the backslashes, or you can apply String.raw to the string literal as template string:
var data = String.raw`{"triggers": [{"message_type": "sms","recipients": "[\"+91xxxxxxxx\",\"+91xxxxxxxx\"]","message": "This is a test"}]}`;
var parsed = JSON.parse(data);
console.log(parsed);
Note however, that the JSON you posted at the start of your question is valid.
On the other hand, the error you get indicates that the \" just before the first + is interpreted as just ". This means the \ is not actually there when the string is received from the server. This is usually caused by a similar escaping problem at the server side, where the programmer intended to send the backslash to the client, but actually just escaped the " on the server, which resulted in only the " being sent to the client and not \".
You have to use single quotes here while escape sequence.
if we use double quotes for escape sequence here it will result in
"recipients": "["+91xxxxxxxx","+91xxxxxxxx"]"
double quotes inside double quotes which is why your code breaks.
data = '{"triggers": [{"message_type": "sms","recipients": "[\'+91xxxxxxxx\',\'+91xxxxxxxx\']","message": "This is a test"}]}';
parsed = JSON.parse(data);
console.log(parsed)
I am facing some issues with escaping of back slash, below is the code snippet I have tried. Issues is how to assign a variable with escaped slash to another variable.
var s = 'domain\\username';
var options = {
user : ''
};
options.user = s;
console.log(s); // Output : domain\username - CORRECT
console.log(options); // Output : { user: 'domain\\username' } - WRONG
Why when I am printing options object both slashes are coming?
I had feeling that I am doing something really/badly wrong here, which may be basics.
Update:
When I am using this object options the value is passing as it is (with double slashes), and I am using this with my SOAP services, and getting 401 error due to invalid user property value.
But when I tried the same with PHP code using same user value its giving proper response, in PHP also we are escaping the value with two slashes.
When you console.log() an object, it is first converted to string using util.inspect(). util.inspect() formats string property values as literals (much like if you were to JSON.stringify(s)) to more easily/accurately display strings (that may contain control characters such as \n). In doing so, it has to escape certain characters in strings so that they are valid Javascript strings, which is why you see the backslash escaped as it is in your code.
The output is correct.
When you set the variable, the escaped backslash is interpreted into a single codepoint.
However, options is an object which, when logged, appears as a JSON blob. The backslash is re-escaped at this point, as this is the only way the backslash can appear validly as a string value within the JSON output.
If you re-read the JSON output from console.log(options) into javascript (using JSON.parse() or similar) and then output the user key, only one backslash will show.
(Following question edit:)
It is possible that for your data to be accepted by the SOAP consuming service, the data needs to be explicitly escaped in-band. In this case, you will need to double-escape it when assigning the value:
var s = 'domain\\\\user'
To definitively determine whether you need to do this or not, I'd suggest you put a proxy between your working PHP app and the SOAP app, and inspect the traffic.
I want to pass an email address as a query string and as such I encode it just before sending it with this line of code:
var encoded = encodeURIComponent(email).replace('.' '%2E');
Apparently the period shouldn't matter but I keep getting "can't find module 'com' " if i run it that way (I'm coding on node and using express and using a res.render() call)
Don't really understand why in my case periods are causing issues but either way this is the exact error I get:
var encoded = encodeURIComponent(email).replace('.' '%2E');
^^^^^
SyntaxError: Unexpected string
Don't really understand why in my case periods are causing issues
It's not the presence of a period. It's the lack of a comma.
var encoded = encodeURIComponent(email).replace('.', '%2E');
// ^ this here
I'm using php's json_encode() to convert an array to json which then echo's it and is read from a javascript ajax request.
The problem is the echo'd text has unicode characters which the javascript json parse() function doesn't convert to.
Example array value is "2\u00000\u00001\u00000\u0000-\u00001\u00000\u0000-\u00000\u00001" which is "2010-10-01".
Json.parse() only gives me "2".
Anyone help me with this issue?
Example:
var resArray = JSON.parse(this.responseText);
for(var x=0; x < resArray.length; x++) {
var twt = resArray[x];
alert(twt.date);
break;
}
You have NUL characters (character code zero) in the string. It's actually "2_0_1_0_-_1_0_-_0_1", where _ represents the NUL characters.
The unicode character escape is actually part of the JSON standard, so the parser should handle that correctly. However, the result is still a string will NUL characters in it, so when you try to use the string in Javascript the behaviour will depend on what the browser does with the NUL characters.
You can try this in some different browsers:
alert('as\u0000df');
Internet Explorer will display only as
Firefox will display asdf but the NUL character doesn't display.
The best solution would be to remove the NUL characters before you convert the data to JSON.
To add to what Guffa said:
When you have alternating zero bytes, what has almost certainly happened is that you've read a UTF-16 data source without converting it to an ASCII-compatible encoding such as UTF-8. Whilst you can throw away the nulls, this will mangle the string if it contains any characters outside of ASCII range. (Not an issue for date strings of course, but it may affect any other strings you're reading from the same source.)
Check where your PHP code is reading the 2010-10-01 string from, and either convert it on the fly using eg iconv('utf-16le', 'utf-8', $string), or change the source to use a more reasonable encoding. If it's a text file, for example, save it in a text editor using ‘UTF-8 without BOM’, and not ‘Unicode’, which is a highly misleading name Windows text editors use to mean UTF-16LE.