We Keep Coding

UnparseableJsonResponse for one specific, valid JSON

I have a standard Dialogflow agent, using javascript/node.js webhooks. It works perfectly well in most cases. I have recently encountered a problem which has me at a complete loss. I am currently saving some JSON-objects in conv.data to minimize the external API-calls my webhook have to make. For one specific JSON-object, fetched from an external API using node-fetch, the response I send from my side looks perfectly ordinary. I use firebase and the firebase logs do not show any error messages or any sign that there might be a problem. But I get this error in the Google Actions console:
UnparseableJsonResponse API Version 2: Failed to parse JSON response string with 'INVALID_ARGUMENT' error: "Parsing terminated before end of input. 8,\\"3\\":12},\\"w ^".
And in the stackdriver logs, the received response does not start with the usual
Received response from agent with body: HTTP/1.1 200 OK Server: ... etc
Instead it starts in the middle of the external API-JSON-file
Received response from agent with body: 8,\\"3\\":12},\\"winPercentage\\":1392}}}}, ... etc
This does not happen the first time the agent responds after fetching the JSON from the external API. The second time the agent responds after fetching the JSON, everything crashes regardless of whether the information from the JSON is used by that second call, regardless of anything at all except if the JSON file is overwritten between first and second call. If the file is overwritten the program runs perfectly. So the problem is likely part of storing and/or parsing this specific JSON file. Unfortunately the API I use in this application is not a public one and due to NDAs I cannot give any access to that JSON, so I understand that it is probably impossible for you to help me. I will however give as much information about the JSON as I can, and hope for the best:
It is valid according to https://codebeautify.org/jsonvalidator and jsonlint.com
It is structured the exact same way as other JSON files from the same API which do not crash the application
It is slightly larger that other JSON files from the same API. It has around 340 000 characters, others are around 280-300 000.
All JSONs, this as well as those that work, is from a Swedish company, therefore unusual characters like å, ä and ö are likely present.
The error message is always the same, except the start of the response is in different places in the JSON file. "8,\\"3\\":12}, ...", "ostPosition\\":2 ...", "3804,\\"startPoints\\":2960 ..." are some examples.
I am extremely grateful for any and all help I might receive, even if it's just what questions I need to ask, or where I might try troubleshooting next.

I suspect the problem is that the JSON you're trying to save is larger than the buffer size they allocate for conv.data, although I can't find any documentation to say there is some specific limit.
I'd check to see where the strings you're seeing in the error header are located in the JSON and try to keep it well under that limit.

Chrome extension rejected: is it forbidden to fetch any json data from a backend?

so I made a Chrome extension whose whole purpose is fetching certain data from the backend and process it to do stuff on a certain domain that the user visits.
I'm trying to have it published but it's getting rejected, and this is what they told me:
Your item was found to have requested/fetched one or more external scripts. An example of one such instance in your item was backend URL in background.js. Please remove all external fetches including Json type.
(This is actually the last of 3 emails they sent me, they just added a few more words in this part I quoted with every email... Since they send only one per day, it's very frustrating...)
I use jQuery.ajax in my background script, and after searching with google I found out that by default it tries to process json requests as jsonp requests (I'm not 100% sure though...), so I've set the jsonp property to false in every ajax call in my code. My extension still got rejected today, and they didn't send another email, so I'm just gonna guess they really did mean that I need to remove that call that fetches json from my backend.
Here's an example of an ajax call in my code:
$.ajax({
url: backendUrl + '/theendpoint',
data: {
paramName: 'paramValue'
},
dataType: 'json',
cache: false,
jsonp: false
})
I'm pretty sure that I'm supposed to be allowed to do it. I've also searched to make sure, and other people do it too. So, what could actually be wrong?
I know it's hard by seeing hardly any code, but there's too much of it and the problem is just here in the ajax calls. And I can't post here the content of my manifest file.
I did add my backend to the permissions in the manifest. Do I have to add it to the content_security_policy too, even though I'm just fetching json from it, and not scripts?
Thanks for any help.
Edit: side question: is it mandatory to provide a physical adress and a link to a privacy policy in my developer account? If yes, could that be the reason why the extension keeps getting rejected? (Last time it got rejected, they didn't even send me an email)

(I'm not sure if I should post this as an answer, but)
Today I tried insisting again saying that json isn't a script and that I was supposed to be able to fetch it from my backend. I don't know if it was a coincidence but right after sending the email, I received another one saying this:
Thank you for reaching out to us.
Upon a subsequent review, we’ve reinstated your item and it will be available in the Chrome Web Store within 30 minutes.
Thank you for your cooperation,
Chrome Web Store team
I also must add that I did use this support form to ask for help. Maybe that's what actually did something.
Moral of the story: if your extension is getting mistakenly rejected, keep insisting and explain what you did and why it's valid...
Now, I've only gotta understand why it got immediately taken down from the store...
Edit: My extension was also taken down by mistake, they reinstated it after I used the support form to ask for the reason. So yeah, use that support form, it actually gets things done.

jquery upload - accessing responses with chunked fileupload

I am using jquery upload for chunked uploads. Creating a new file and first chunk in my php back-end goes fine, but for the rest chunks I need a file-ID to tell my php script where to append file data.
So my problem is I don't know a way to read the response (ie. file-id) from ajax with jquery upload and edit the post data, in this case add the file id to the requests after the first chunk. The responses seem to behave mysterically.
I console log in my progress-callback console.log(data) and I get a buch of stuff. I can log ie. console.log(data._progress.field) but cannot do this for data._response.file-id as console says undefined. But when i console log the whole data I can clearly see a _response -object inside data. but when I specifically try to log that I get Object { } so it seems to be empty, but when i click it I see jqXHR-object and result-object that has the file-id I want to put to post requests for the rest chunks.
TL,DR:
I can console.log whole data in callback and see the result ie. file-id, but when I try to log data.result.file-id, console says "undefined".
Some fields I can log but some I can't Ie. data._progress.total logs just fine but data._result doesn't.
I need a field from response and change the post parameters for the next requests of chunked file upload.
What on earth I am missing here?
EDIT:
I have found a back-end solution to my problem and I didn't need jQuery/ajax request response things I tried to use. This was proprietary works so sorry I can't post my code.

JSON post using Python requests

I am using python 2.7 on windows 7 with requests requests module.
I am trying to find the source of some data from a website that uses AJAX/javascript to drive events.
Here is my python post:
result = s.post(url, headers = {'referer': my_referer})
The output from this post looks like this:
{"ADATA":{"COUNT":0.0,"AITEM":[]},"WM":0.0,"CM":0.0,"PC":"","PW":"","NC":14,"RR":false,"RTIME":{"RITEM":[],"COUNT":0.0},"WC":1.0,"CC":1.0,"RW":false,"RA":false,"RC":true}
"AITEM":[] should be populated with all the data I'm after but as you see its not. Also RC: should be false. Or at least it is when I go through my browser
Note that I get "RA":true and "RC":false if I use get instead of post request. I dont know why.
Here is the corresponding post in the javascript server side:
$.getJSON (url, function(data){UpdateStuff(data);});
Disclaimer: I have been programming for about a week now.
This JSON post is in a function that receives no parameters and returns nothing as far as I can tell. data is not referenced in the function prior to the post. I dont really understand what function(data) is. UpdateStuff(data) is another function with a bunch of code that takes data and returns nothing. The code reveals some stuff about the data structure with references such as:
if (data.RA){resetAL();} else {
if (data.RR ){objAR.attr('ref','Y');}
if (data.RC ){objAC.attr('ref','Y');}
if (data.RW ){objAW.attr('ref','Y');}
Which I guess I am failing this logic gate since my data is different than what I see in a successful browser request.
and references to data.ADATA.COUNT and data.ADATA.AITEM[i] etc
I've been told this would be much simpler to write in JS but I have about 70% of this program done in python already and I've never used JS.
Any help is greatly appreciated.

Ajax read HTML after fully processed

I am trying to load the HTML content of a webpage outside of my domain, which I can do just fine using functionality provided by this jQuery plugin: http://www.ajax-cross-origin.com/. However, when I print out the HTML there are pieces missing, which I assume is because the ajax request gets the HTML before the page is fully loaded. When I say "pieces missing," I mean that some tags that should have innerHTML in fact have none. Here's my code:
$.ajax({
crossOrigin: true,
url: "http://siriusxm.com/bpm",
success: function(data) {
console.log(data);
},
timeout: 5000
});
The crossOrigin attribute is from the plugin I mentioned. I get the same behavior with and without the timeout (and strangely, it doesn't seem as though the timeout is doing anything at all--when I check the console, it logs data pretty much immediately).
Is there a way to wait until the page is fully loaded before getting the content? For what it's worth, this is all part of a chrome extension I'm developing, so if there's anything else code-wise you might need just ask.
Thanks!

So according to your comments, the information you're looking for is just the Now Playing artist and Song, which you won't be able to get by loading just the source of the main page.
To find the data you're looking for just open up your Chrome DevTools, go to the network tab, and Refresh to see all requests on the page.
It looks like this is the request you want, you just need to update the timestamp every minute:
http://www.siriusxm.com/metadata/pdt/en-us/json/channels/thebeat/timestamp/08-12-03:48:00
Just parse that json and grab what you need. Of course they can always change the location or format of the file, but for right now that's what it is.

If the console log is showing all of the data you're looking for then the ajax call should be fine.
Any code in the success callback will be ran after the ajax call, so just use JQuery in the success callback function to insert data into the html. All I see there now is the console.log(data) unless you've removed some code.
The timeout just gives the ajax call a set amount of time to complete before it "times out", in other words it tells it to stop waiting after the set amount of time.