We Keep Coding

How to fix an invalid random string to make it JSON valid

In Javascript, I need to "fix" a string, supposed to be JSON valid but may not be. The string has the following format (the unknown part is marked with "<INVALID_CHARS>"):
[
{ "key_1": "ok_data", "key_2": "something_valid <INVALID_CHARS>"},
{ "key_1": "ok_data", "key_2": "some_valid_value"}
]
"INVALID_CHARS" are chars which make the JSON.parse() function fail.
The errors are always localized on the "key_2" property of this array elements.
Note that these chars come from random binary data, and can thus be anything.
I would like to find the simplest solution, or at least one which is the least prone to errors.
I thought of replacing invalid characters, but there is also a problem with single backslash chars followed by a non special char, throwing an error too, or quote chars.
And I probably did not think of all the possible errors.
Thank you.

JSON is not allowed to contain arbitrary binary data; it must be a sequence of valid Unicode codepoints. (Usually these are transmitted in UTF-8 encoding, but regardless, arbitrary binary data is not possible.) So if you want to include arbitrary binary data you'll need to figure out how to unambiguously encode it for transmission. If you don't encode it in some way, then you won't be able to reliably distinguish a byte which happens to have the same code as " from the " which terminates the string.
There are a number of possible encodings you might use for which standard libraries exist in most languages. One of the most commonly used is base-64.

it's better to clarify the problem as seems you described wide range of the issues here. If you have problem with parsing structure above you just need to check the syntactic integrity of the structure. For example this structure parses well
let var1 = JSON.parse('[
{
"key_1":"ok_data",
"key_2":"something_valid <INVALID_CHARS>"
},
{
"key_1":"ok_data",
"key_2":"some_valid_value"
}
]');
In case if you need to replace <INVALID_CHARS> as binary data with json characters it's possible to encode <INVALID_CHARS> in base64 as it's the most reliable way. But I guess also problem not only to pack <INVALID_CHARS> to base64 and problem is also architectural and you need to prepare value of key_2 with valid part and invalid part. In this way, I would suggest separate (split) key_2 on two substrings separate by " " - "key_2": "something_valid <INVALID_CHARS>(can be omitted)".
Moreover, it's possible to use separate fields for string without error and a second for errors. Like this "key_2_1": "something_valid", "key_2_2":<INVALID_CHARS>
Another way is to look to using Multipart Form Data if it's possible, to transfer binary data

Convert invalid JSON (having equal sign instead of colon, no quotes etc) to valid JSON

I am getting data from Cardstream payment gateway android sdk (in native module) to react native but the data is not a valid JSON:
'{ __wafRequestID=2021-06-16T08:02:14Z|0e2314f32f|115.186.169.10|gk0GHP1i4V, action=SALE, addressCheckPref=not known,not checked,matched,not matched,partially matched, amount=14, amountRetained=0, avscv2CheckEnabled=Y, caEnabled=Y, cardCVVMandatory=Y, cardExpiryDate=0322, cardFlags=8323072, cardIssuer=UNKNOWN, cardIssuerCountry=United Kingdom, cardIssuerCountryCode=GBR, cardNumberMask=424242******4242, cardNumberValid=Y, cardScheme=Visa, cardSchemeCode=VC, cardType=Visa Credit, cardTypeCode=VC, cftEnabled=N, countryCode=826, currencyCode=826, currencyExponent=2, customerName=fgg, customerReceiptsRequired=Y, cv2CheckPref=not known,not checked,matched,not matched,partially matched, eReceiptsEnabled=N, merchantAlias=100001, merchantID=100001, merchantID2=100001, paymentMethod=card, postcodeCheckPref=not known,not checked,matched,not matched,partially matched, processMerchantID=100001, requestID=60c9b007225c7, requestMerchantID=100001, responseCode=65566, responseMessage=Disallowed cardnumber, responseStatus=2, riskCheckEnabled=Y, riskCheckPref=not known=continue,not checked=continue,approve=continue,decline=decline1,review=authonly,escalate=authonly, riskProcessorID=41, riskProcessorName=Kount, rtsEnabled=Y, scaExemption=lowvalue,trusted, state=finished, surchargeEnabled=Y, surchargeRequired=Y, threeDSCheckPref=authenticated, threeDSEnabled=N, timestamp=2021-06-16 09:02:16, transactionID=112446674, type=1, vcsResponseCode=0, vcsResponseMessage=Success, xref=21061609KX02TQ16YP35TTD}'
How to convert it to valid JSON so that I can parse it? I have tried few regex replacements but some values also have colons which is messing it up.

If you wanted to try an parse this with regex then this might work:
[{,] (?<key>[a-zA-Z0-9_]+)=(?<value>.*?(?=, [a-zA-Z0-9_]+=|\}$))
https://regex101.com/r/Fk7NvR/1
Just loop through the matches and access the captured groups named key and value respectively. The value for riskCheckPref seems suspect but I have no clue about their parsing rules.
An alternate idea would be to remove the outer curly braces, trim white space, split on , (comma space), and split again on the first =.

I converted response to JSON in android and then passed it to react-native instead of passing response to react-native (which gets passed in this invalid format) and then banging my head to convert it to JSON. My bad. Thanks all

JavaScript not decoding parameter

So from the textarea I take the shortcode %91samurai id="19"%93 it should be [samurai id="19"]:
var not_decoded_content = jQuery('[data-module_type="et_pb_text_forms_00132547"]')
.find('#et_pb_et_pb_text_form_content').html();
But when I try to decode the %91 and %93
self.content = decodeURI(not_decoded_content);
I get the error:
Uncaught URIError: URI malformed
How can i solve this problem?

The encodings are invalid. If you can't fix the whatever-system-produces-them to correctly produce %5B and %5D, then your only option is to do a replacement yourself: replace all %91 with character 91 which is '[', then replace all %93 with character 93 which is ']'.
Note that javascript String Replace as-is won't do "Replace all occurrences". If you need that, then create a loop (while it contains(...) do a replace), or search the internet for javascript replace all, you should find plenty results.
And a final note, I am used to using decodeURIComponent(...). If you can make the whatever-system-produces-them to correctly produce %5B and %5D, and you still get that error, then try using decodeURIComponent(...) instead of decodeURI(...).

The string you're trying to decode is not a URI. Use decodeURIComponent() instead.
UPDATE
Hmm, that's not actually the issue, the issues are the %91 and %93.
encodeURI('[]')
gives %5b%5d, it looks like whatever has encoded this string has used the decimal rather than hexadecimal value.
Decimal 91 = hex 5b
Decimal 93 = hex 5d
Trying again with the hex values
decodeURI('%5bsamurai id="19"%5d') == '[samurai id="19"]'

I know this is not the solution you want to see, but can you try using "%E2%80%98" for %91 and "%E2%80%9C" for %93 ?
The %91 and %93 are part of control characters which html does not like to decode (for reasons beyond me). Simply put, they're not your ordinary ASCII characters for HTML to play around with.

Extra Square brackets Get added when retrieve Json data

I am Trying to retrieve data from json in my code. Json data doesn't contain any brackets. It is just any array separated by commas(,). My Problem is that when I get data using $scope, an extra sqaure bracket get added outside my output. Demo of my code is
controller.js
$http.get("serverUrl")
.success(function(data) {
$scope.sour = data;
})
.error(function(err) {
console.log("Error in source: "+JSON.stringify(err));
});
html
<div>
{{sour}}
</div
expected json
data
error
[data]
I have tried old stack solutions but none of them worked out for me.
Please share if anyone know why this error being produced, As I have used this method a hundred times before but never faced this problem.Regards.

After trying found out a solution.
.toString().replace();
solved out my problem.

As per how JSON.stringify should work as specified here, the result that you explained is as expected.
Let separator be the result of concatenating the comma character,
the line feed character, and indent.
Let properties be a String
formed by concatenating all the element Strings of partial with each
adjacent pair of Strings separated with separator. The separator
String is not inserted either before the first String or after the
last String.
Let final be the result of concatenating "[", the line
feed character, indent, properties, the line feed character,
stepback, and "]"
Which will result in a valid JSON String. The format you are requesting is confusing and is probably an invalid JSON. You can try checking the JSON using some JSON validator online. (Notice that [1,2,3,4] is valid where as 1,2,3,4 is invalid)
You are probably having a service that is expecting a malformed JSON, it will be better to fix that part instead of duck taping your JSON.
Meanwhile, the only thing you are explaining is the format you are getting and the format that is working. You haven't still specified where exactly the error happens? Is it error from HTTP request? Is it error thrown in Javascript? What is causing the error? What error messages are you getting?

Why doesn't my attempt to escape quotation marks in JSON work?

I am parsing a JSON-string with the JQuery.parseJSON function, as I have done lot's of times in my code. On this particular case, though, I get: Uncaught SyntaxError: Unexpected token R. The only upper case R that exists, in my JSON-formatted String, comes right after an escaped quotation mark, ... \"R ... like this. It seems like too much of a coincidence to be caused by anything other than this, but as far as I can tell, I have completely followed the proper syntax as described on json.org.
EDIT:
I've tried to manually remove the occurrances of \" in a hardcoded test, and the string formats perfectly into a proper Javascript object. In other words, my \" is definitely the problem here...
var myObject = $.parseJSON(myString);
EDIT 2:
the problematic area of my String is here displayed, both in working, and not working condition. first the problematic one:
{"lineID":33,"boxID":10,"title":"My text with the \"Ruining Part\""}
Then the working one:
{"lineID":33,"boxID":10,"title":"My text with the Ruining Part"}
Finally how i format my javabean object into JSON string.
String jsonObjectAsString = new Gson().toJson(myJavaBeanObject);

You probably need to escape the backslash in your string, if it is hardcoded, so that the final string that gets parsed has a single backslash followed by a double quote. Otherwise, the browser thinks you are trying to escape a double quote in your string, which does nothing.
So change your string to:
...\\"R...