UnparseableJsonResponse for one specific, valid JSON - javascript

I have a standard Dialogflow agent, using javascript/node.js webhooks. It works perfectly well in most cases. I have recently encountered a problem which has me at a complete loss. I am currently saving some JSON-objects in conv.data to minimize the external API-calls my webhook have to make. For one specific JSON-object, fetched from an external API using node-fetch, the response I send from my side looks perfectly ordinary. I use firebase and the firebase logs do not show any error messages or any sign that there might be a problem. But I get this error in the Google Actions console:
UnparseableJsonResponse API Version 2: Failed to parse JSON response string with 'INVALID_ARGUMENT' error: "Parsing terminated before end of input. 8,\\"3\\":12},\\"w ^".
And in the stackdriver logs, the received response does not start with the usual
Received response from agent with body: HTTP/1.1 200 OK Server: ... etc
Instead it starts in the middle of the external API-JSON-file
Received response from agent with body: 8,\\"3\\":12},\\"winPercentage\\":1392}}}}, ... etc
This does not happen the first time the agent responds after fetching the JSON from the external API. The second time the agent responds after fetching the JSON, everything crashes regardless of whether the information from the JSON is used by that second call, regardless of anything at all except if the JSON file is overwritten between first and second call. If the file is overwritten the program runs perfectly. So the problem is likely part of storing and/or parsing this specific JSON file. Unfortunately the API I use in this application is not a public one and due to NDAs I cannot give any access to that JSON, so I understand that it is probably impossible for you to help me. I will however give as much information about the JSON as I can, and hope for the best:
It is valid according to https://codebeautify.org/jsonvalidator and jsonlint.com
It is structured the exact same way as other JSON files from the same API which do not crash the application
It is slightly larger that other JSON files from the same API. It has around 340 000 characters, others are around 280-300 000.
All JSONs, this as well as those that work, is from a Swedish company, therefore unusual characters like å, ä and ö are likely present.
The error message is always the same, except the start of the response is in different places in the JSON file. "8,\\"3\\":12}, ...", "ostPosition\\":2 ...", "3804,\\"startPoints\\":2960 ..." are some examples.
I am extremely grateful for any and all help I might receive, even if it's just what questions I need to ask, or where I might try troubleshooting next.

I suspect the problem is that the JSON you're trying to save is larger than the buffer size they allocate for conv.data, although I can't find any documentation to say there is some specific limit.
I'd check to see where the strings you're seeing in the error header are located in the JSON and try to keep it well under that limit.

Related

Why does Instagram API return two different user_id values for the same user?

Nutshell: PHP implementation gets correct user_id, React.js implementation gets incorrect user_id. Why!?
I implemented the Instagram Basic Display API OAuth flow in PHP and in Javascript under React.
The flow under PHP works fine - I get a code, I POST it using curl to /oauth/access_token, I receive a client_id and an access_token, all fine, subsequent Graph API requests work just fine.
Then I moved to rebuild it in Javascript. Here, when I get a code, and I POST it using fetch to /oauth/access_token, with the same client_id and client_secret, and the same authenticating user, essentially the same request, just a different redirect URL (obviously), I receive a token and a different user_id, one that does not work (it gets a 400 error from Graph API).
Oddly enough, the bad user_id is consistently smaller by 1 than the one that I get with PHP: the good one ends with 93, the bad one ends with 92. If I do the unspeakable and just manually increase the wrong user_id I received, I obtain a valid user_id for which the token works!
So why is Instagram sending me a wrong user_id, and how can I help it?
I finally found the cause, moments ago, after writing the question and earlier wasting 5 hours on analyzing whatever I could think of.
The user_id is sent as a number larger than Number.MAX_SAFE_INTEGER, and thus JSON.parse fails to parse it correctly. Instagram should be encoding the user_id field as a string. This API endpoint thus needs to be parsed manually. I "fixed" mine by filtering the results as such:
text = text.replace(/"user_id":\s*(\d+)/,'"user_id":"$1"'). Still, encoding such large numbers as "numbers" in JSON is a horrible practice and Instagram should fix their API instead.

Jmeter logging for API URL+Request+ResponseCode+ResponseBody in csv

I like to save the following parameters in a CSV file in single line for HTTP REST call in Jmeter.
Endpoint URL, request body, request header, response code, response body . could someone help with JS code or groovy code. Thanks in advance!
I don't think you will be able to save request and response "in single line" because:
for sure it will have line breaks
most probably it will have delimiters (i.e. CSV stands for "comma-separated values" and if your response will contain a comma - it will create another "column")
If you're going to use JSR223 Test Elements for this it can be done, but the correct work is guaranteed only for a one thread (virtual user), if you have more than 1 user concurrently writing the data into the same file you'll face the race condition resulting in data corruption or loss
So I would recommend considering using Flexible File Writer which provides full freedom to choose what to store, where to store and it "flushes" the metrics periodically in a thread-safe manner.

Chrome extension rejected: is it forbidden to fetch any json data from a backend?

so I made a Chrome extension whose whole purpose is fetching certain data from the backend and process it to do stuff on a certain domain that the user visits.
I'm trying to have it published but it's getting rejected, and this is what they told me:
Your item was found to have requested/fetched one or more external scripts. An example of one such instance in your item was backend URL in background.js. Please remove all external fetches including Json type.
(This is actually the last of 3 emails they sent me, they just added a few more words in this part I quoted with every email... Since they send only one per day, it's very frustrating...)
I use jQuery.ajax in my background script, and after searching with google I found out that by default it tries to process json requests as jsonp requests (I'm not 100% sure though...), so I've set the jsonp property to false in every ajax call in my code. My extension still got rejected today, and they didn't send another email, so I'm just gonna guess they really did mean that I need to remove that call that fetches json from my backend.
Here's an example of an ajax call in my code:
$.ajax({
url: backendUrl + '/theendpoint',
data: {
paramName: 'paramValue'
},
dataType: 'json',
cache: false,
jsonp: false
})
I'm pretty sure that I'm supposed to be allowed to do it. I've also searched to make sure, and other people do it too. So, what could actually be wrong?
I know it's hard by seeing hardly any code, but there's too much of it and the problem is just here in the ajax calls. And I can't post here the content of my manifest file.
I did add my backend to the permissions in the manifest. Do I have to add it to the content_security_policy too, even though I'm just fetching json from it, and not scripts?
Thanks for any help.
Edit: side question: is it mandatory to provide a physical adress and a link to a privacy policy in my developer account? If yes, could that be the reason why the extension keeps getting rejected? (Last time it got rejected, they didn't even send me an email)
(I'm not sure if I should post this as an answer, but)
Today I tried insisting again saying that json isn't a script and that I was supposed to be able to fetch it from my backend. I don't know if it was a coincidence but right after sending the email, I received another one saying this:
Thank you for reaching out to us.
Upon a subsequent review, we’ve reinstated your item and it will be available in the Chrome Web Store within 30 minutes.
Thank you for your cooperation,
Chrome Web Store team
I also must add that I did use this support form to ask for help. Maybe that's what actually did something.
Moral of the story: if your extension is getting mistakenly rejected, keep insisting and explain what you did and why it's valid...
Now, I've only gotta understand why it got immediately taken down from the store...
Edit: My extension was also taken down by mistake, they reinstated it after I used the support form to ask for the reason. So yeah, use that support form, it actually gets things done.

Does d3.json() support authentication? If not, what other JavaScript options are available for JSON retrieval?

I am developing an application that needs to gather information from GitHub, so I began looking at their API. My initial thought was to use the d3.json() function to get the data (because it's simple and has done good things for me in the past), but there doesn't appear to be a way for me to authenticate with that function. For example, $ curl -u "username" https://api.github.com is given as an example of basic authentication (from GitHub API--obviously they use curl for their examples).
So, is there a way to do authentication with the d3.json() function? And if not, what are my other options to get the JSON with JavaScript?
Thanks!
Edit:
I'm experimenting now with using jQuery's getJSON method as shown here, because I started getting the error "XMLHttpRequest cannot load url Origin url is not allowed by Access-Control-Allow-Origin." Of course, the switch doesn't help with the ability to authenticate, but I can at least get the public data (which is most).
Edit #2:
Has anyone experimented with michael/github or fitzgen/github-api? I'm going to start looking into those.
If you have a PHP script which echos JSON, you can do all authentication server-side. Note that you can also send a GET request to your PHP script, so the way you call your script can be dynamic.

Unable to consistently load large json response using jquery.ajax

Recently I started facing issue with one of my scripts that loads a json response from the server. I am using jquery.ajax() to make an ajax call. The code snippet is so -
var request = $.ajax({
url: "script.jsp",
type: "POST",
dataType: "json",
success: function(response) {
console.log(response);
},
error: function(response, error) {
console.log(response, error);
}
});
As I mentioned this script worked as recent as yesterday. I have not made any changes to either the server-side code or the front-end code. The json response is a bit large ~1 MB in size. But I validated the json output using -
python -mjson.tool < output.json
It prints out properly. Curious thing is FF & Chrome handle it differently.
In FF, I open firebug and see the ajax request being made. I see that the request is served in around 300ms but the loading wheel next to the link in the console is still animating for aorund 20 seconds. And after that the json response is properly processed and the result can be seen on the page. In IE also similar behavior, proper processing of the json after 20 seconds.
In Chrome, nothing happens for around 20 seconds after which I see an error in the console saying either "error": undefined or Failed to load resource. Alternatively it also prints the below stacktrace -
POST script.jsp
f.support.ajax.f.ajaxTransport.sendjquery.min.js:4
f.extend.ajaxjquery.min.js:4
DataTableWidget.extend._fetchBuildingBlockItemsPermissionBBItemsWidget.js:91
(anonymous function)PermissionBBItemsWidget.js:83
e.extend.eachjquery.min.js:2
DataTableWidget.extend._loadDataPermissionBBItemsWidget.js:82
DataTableWidget.extend.showPermissionBBItemsWidget.js:15
(anonymous function)permission-building-blocks.html:451
xLAB.min.js:5
ULAB.min.js:5
jLAB.min.js:5
ILAB.min.js:5
eLAB.min.js:5
a.onload.a.onreadystatechange
I dont understand this weird behavior in different browsers.
So in essence I made sure that -
Server side code returns the response fast. I put in some debug statements and saw the server log. None of the response form server take more than 500ms.
Made sure than json is properly validated. The fact that after 20 secs it is processed in both IE & FF without any issues is a proof of that. In addition to use of python's json.tool.
Set the dataType to json
So any pointers on the issue will be a great help. Thanks.
UPDATE
One more curious thing I noticed. While the request is being processed and I hit the refresh button even within 3 seconds of the original request, the process immediately completes. As in I see changes in the view of course fraction of a second later the page is wiped out due to refresh event.
UPDATE 2
I have noticed that after I slice up my big response by alphabets. The issue of looong response happens in only certain responses. I ran this split long response files through http://jsonformatter.curiousconcept.com/#jsonformatter and although it immediately returns saying that the josn is valid, it takes 20+ seconds to actually pretty print the response. I think the problem is happening due to certain characters like \u0026 so with this added info, how to resolve the problem? Here is the snipet of the problematic json.
I figured out the problem. The issue was with how the server-side code provided the client-side code JavaScript with the json string.
I was using a legacy method in our code base which actually rendered the already json string in a jsp page before passing it to the client. This was somehow screwing up the response. Also response type was text/html because of this.
As soon as I switched the response to be of actual application/json MIME type stream, everything was fine.
does your JSON have line feeds? Do you have Firebug or something line that open? The browser may be loading the JSON just fine, it might be your debugging tools crapping out. I Recall having issues before with inspecting certain JSON strings with certain tools just because it didn't have line feeds (or line feeds as understood in Windows). I'm wondering if the fact your logging the whole thing to the console doesn't have something to do with it.

Categories