We Keep Coding

UnparseableJsonResponse for one specific, valid JSON

I have a standard Dialogflow agent, using javascript/node.js webhooks. It works perfectly well in most cases. I have recently encountered a problem which has me at a complete loss. I am currently saving some JSON-objects in conv.data to minimize the external API-calls my webhook have to make. For one specific JSON-object, fetched from an external API using node-fetch, the response I send from my side looks perfectly ordinary. I use firebase and the firebase logs do not show any error messages or any sign that there might be a problem. But I get this error in the Google Actions console:
UnparseableJsonResponse API Version 2: Failed to parse JSON response string with 'INVALID_ARGUMENT' error: "Parsing terminated before end of input. 8,\\"3\\":12},\\"w ^".
And in the stackdriver logs, the received response does not start with the usual
Received response from agent with body: HTTP/1.1 200 OK Server: ... etc
Instead it starts in the middle of the external API-JSON-file
Received response from agent with body: 8,\\"3\\":12},\\"winPercentage\\":1392}}}}, ... etc
This does not happen the first time the agent responds after fetching the JSON from the external API. The second time the agent responds after fetching the JSON, everything crashes regardless of whether the information from the JSON is used by that second call, regardless of anything at all except if the JSON file is overwritten between first and second call. If the file is overwritten the program runs perfectly. So the problem is likely part of storing and/or parsing this specific JSON file. Unfortunately the API I use in this application is not a public one and due to NDAs I cannot give any access to that JSON, so I understand that it is probably impossible for you to help me. I will however give as much information about the JSON as I can, and hope for the best:
It is valid according to https://codebeautify.org/jsonvalidator and jsonlint.com
It is structured the exact same way as other JSON files from the same API which do not crash the application
It is slightly larger that other JSON files from the same API. It has around 340 000 characters, others are around 280-300 000.
All JSONs, this as well as those that work, is from a Swedish company, therefore unusual characters like å, ä and ö are likely present.
The error message is always the same, except the start of the response is in different places in the JSON file. "8,\\"3\\":12}, ...", "ostPosition\\":2 ...", "3804,\\"startPoints\\":2960 ..." are some examples.
I am extremely grateful for any and all help I might receive, even if it's just what questions I need to ask, or where I might try troubleshooting next.

I suspect the problem is that the JSON you're trying to save is larger than the buffer size they allocate for conv.data, although I can't find any documentation to say there is some specific limit.
I'd check to see where the strings you're seeing in the error header are located in the JSON and try to keep it well under that limit.

create file in Ajax javascript

I was wondering if there was any way to create something like a txt file in Ajax. I'm using this to save logs created by the user on my site, this is the link if its helpful: site. If this cannot be done with ajax can it be done with cookies? Or would the log be to big for cookies? Thanks(sorry if this is a duplicate, some I looked around and everything was about loading/reading from a file).

AJAX is just making asynchronus HTTP calls on the browser.
So, can you create a file just making an ajax call? No. You would need to setup somekind of script on the server that will handle the file creation and writing and call that script with AJAX.

The short answer is - no. You can not do this DIRECTLY with AJAX.
The longer answer is - what you are looking for should be done using a database and SQL. Not that PHP couldn't do it directly - but a good database is better able to handle this problem.
AJAX is mainly used to just send and receive messages, files, etc... It isn't for storing anything. With that being said, the steps would be to have a small PHP script that just reads and writes to a database and make that database be a log file. So your SQL database (I'd just call it log) would have two entries. These are the date the log entry went into it and the entry itself. You could also throw in last_accessed if you want or other fields you think you might need. Your PHP script, in its simplest form, would open the database and, depending upon whether you send it a "GET" or "PUT" command it would read from or write to the log file. If it writes to the log file, it sends back an ACK or NAK (meaning everything is ok or the program had a problem). If it is reading from the log file, you supply how many lines you want to read and the PHP script either sends back the lines or a NAK again to signify something went wrong. You can also replace the simple NAK with the actual error if you want but then your customers are seeing the error. Better to have just a generic "There has been a problem, we have contacted our system people and the site should be back up shortly" kind of message. Save the actual errors for you to see via e-mail.
That's it. You should be able to write this in about ten to fifteen minutes and have your log file readily available to you in no time.

AJAX calls - where does logic go?

This might not even be an AngularJS question and could just be an AJAX question. I'm new to the "developer" side of the frontend so bear with me.
When making an AJAX call to fetch JSON data, where does the logic behind what data is returned and viewed fall? In my mind, there would be a couple of possibilities and I want to understand which is the proper choice and why.
Let's use an example of searching and playing a Youtube video.
The logic could fall to the backend (controller), where the JSON is rendered based on some logic to give you a JSON file with exactly the right data. i.e. you search "cat videos" and when making an AJAX call, the JSON file you pull has been rendered to be only cat videos.
The opposite end would be that the Angular controller has the logic. This would imply that all data is called (cat videos along with everything else... music videos, funny videos, tutorials, and so on) and then sorted through on the client side. This, to me anyway, would be more inefficient / slow for the client, so doesn't seem to make sense. I suppose still might do some filtering of the data on the client side though. So, maybe a search for "cat videos" wouldn't return ALL videos, but definitely all cat videos and any filtering based on, say, # of views, video length, and so on would be done on the client side (vs. calling the database again for a "new" set of videos).
Not sure if this is accurate, but could you have logic in your factory to return only a portion of the data? However, I believe the entire JSON file would need to be rendered, but only portions would be returned. I guess depending on where the JSON file renders (i.e. backend or frontend) this could be similar to either option #1 or #2.
Or maybe I'm misunderstanding things entirely and the way this works is entirely different!
I'm basically looking to figure out how the scenarios of 1. user searches a term and results are shown, 2. user clicks a search result and now more detailed data of the result is on it's own page. And how this ends up working out. I'm looking for help with AngularJS, but I think this ultimately an AJAX question (single page app or not) more than anything.

There's a few critical concepts you may be confused about.
First. JSON is not a file, it's a format, more simply, a type of string. It's really good for collapsing arrays and storing address-value pairs, so a lot of data flies around in that format. Strictly speaking, they are JSON objects, but they're a lot like strings and arrays. It looks like this, if I remember correctly:
{ "name" : "john doe", "pet" : "dog", "hobby" : "parasailing" }
Second, AJAX is a request to the server, made from the client (the browser) after the original page has loaded. That is, you type in 'youtube.com' and the youtube server receives the request and sends a big pile of HTML back to your browser.
You watch your video, make a rating, and the browser doesn't reload the page but instead sends a separate request back to the youtube server with your rating. There's a parameter in the request that says "send it to ratingspage.php". This request is AJAX.
Now, the logic happens (server-side). ratingspage.php receives your request. It contacts the databases, updates or fails or whatever, and sends back a response to your browser. This response may be in JSON format.
Finally, your browser parses that response and updates the DOM (HTML document) as appropriate.
At this point, it's worth noting that if the logic happened on the client-side (browser), the user could see it - this is a security problem! So, sensitive operations should be carried out on the server side, where you can test and sanitize the request data.
In summary:
AJAX is separate from the initial load event.
Information sent is gathered from the client browser
Logic happens server-side
Logic can use whatever language the server understands (PHP, Java, Ruby, etc.)
Information is returned to the browser
Information sent and received may use JSON format
Everything client-side happens in Javascript
Here's a bare-bones ajax request (done in Javascript) with comments. This has no exception handling, state checking, or anything so don't use it! But it gives you the basic idea.
// Make a new request
var req = new XMLHttpRequest(); }
// Requests will have various states depending on whether they're processing,
// finished, error, etc. We'll assume everything went OK.
// We need to establish a handler before the request
// is sent so it knows what to do.
req.onreadystatechange = function() {
// Here's what the server sent back to the browser
alert(req.responseText);
}
// Using the GET method, set up some parameters
req.open("GET", "somelogicpage.php?blah=blee&bloo=bar", true);
// Send the request
req.send(null);
Server-side, somelogicpage.php may look like:
<?php
if ($_GET['blah'] != 'blee']) {
// This is the response text!
echo "Sorry, you need to blee when you blah.";
}
else {
// (or this)
echo "I'm ecstatic to report nothing is wrong!";
}
?>
Your alert(req.responseText) from the handler function in the previous Javascript will say whatever the PHP has dumped out.
So yes, you can use whatever portion of the request you like, and return whatever you like. Javascript kicks bleep.

Javascript (client-side): Database query and Data handling

I have built a web application that accepts a Member_ID # from the client (javascript). The Member_ID is stored as a var in my javascript… now I need to make queries with it…first query to get member information (name) and then join tables to gather information for health plans that the member is elligible under … and so on.
As far as I can remember the concept is… send a async request to the server and wait for a response, once the response is received, store it and then parse it to extract useful information.
The end goal is to use the 'useful' information to plot graphs using HTML5 Canvas.
I need some direction with how to make the query? because when I read this forum, it is recommended that client-side query is 'bad' for a plethora of reasons.
Since most of my stuff is happening in the client side... few things in C# asp.net...how do I proceed?
It is also important to note that the web application should be accessible via the internet. outside the local network.
Does it make sense to Call a Web Service from the Client Side Using the AJAX Extension Toolkit??

You must handle it server side. Think about the ajax request as a simple POST or GET in the format of ?member_id=123&time=321 pointed at your handler file.
In your handler file you can construct your query from the request variables, execute it, and give a response by printing to the screen in either JSON or XML format.
Take a look here:
http://api.jquery.com/jQuery.ajax/
http://api.jquery.com/jQuery.getJSON/

Dealing with incremental server response in AJAX (in JavaScript)

I have AJAX app which updates page based on server response. The command that AJAX server response is based on takes long time to generate full response, but it sends partial information as soon as it is calculated. This partial response / partial info is send in "burst", and time and size of each burst is unpredictable. CGI script (in Perl) that streams command output to web browser (to AJAX request) has autoflush turned on.
The server response is based on output of external command. While 'time cmd > /dev/null" gives around 10.0 seconds on average, 'time cmd | head > /dev/null' gives less than 0.1 seconds (for example data). All data is result of single call to this external command.
The situation looks like the following (ASCII-art diagram follows):
client | | server
--------- ---------
request -\
\
\
\
\->
/- response
/ .
/ .
/ /- .
<-/ / .
/ .
/ /- [end]
<-/ /
/
/
<-/
I have a few questions about this problem.
Note: server side is done as CGI script in Perl, and I would prefer to see (also) solution without using JavaScript library / framework like jQuery.
The output of command used by server side of AJAX app is line based. Each group of lines, beginning with one defined kind of line, and ending with other kind of line, consist of independend and unchangeable data. Should I just stream response from a command as 'text/plain' and do processing in JavaScript on client side, or should I pre-process data on server, and send whole chunks of data as JSON using 'application/json' mimetype?
It might happen that large chunk of data send at once by server is followed soon by another chunk of data. How to deal with situation when onreadystatechange handler is invoked while previous invocation didn't finished work? Should I use global variable as semaphore, or pass state variable as handler parameter (well, use xhr.onreadystatechange = function() { handleRequest(xhr, state) })?
Should I use 'text/plain' or 'application/json', or perhaps 'multipart/x0mixed-replace' for this? Note: this app should work in (alomst) any browser.
How to deal with web browser (JavaScript engines) which invoke onReadyStateChange only after receiving complete response (so I don't see xhr.readyState == 3 i.e. partial response more than once)? Well, beside using some JavaScript framework.
How to deal with incomplete responses (which in this situation mean incomplete lines).
Should I send end of response marker, or rely on counter to check if we received all data, or can I simply rely on detecting xhr.readyState == 4?
Even partial response would be helpful.

I think Comet is part of what you need for your solution. You can additionally (if I got that right) checkout the Bayeux Protocol which was implemented by the Dojo Foundation.
The whole thing is still very new (though some of it might be possible with the first HTML5 implementations).
Besides that you would probably have to implement the polling approach. The other question is, how much data the client side JavaScript interpreter can handle. Is there any possibility for you to somehow "page" your data so that you won't have the problem of request handling still processing while aother response comes in already?

I think the client side could be designed to process data in chunks, sending repeated AJAX requests until all the data had been supplied. This assumes that each chunk could be sent in a timely manner (without client-side timeout problems) even if the entire response was large; and this design is probably simpler than developing checks for specific partial-response statuses which might vary from browser to browser.
In terms of whether you do processing server-side or client-side, that would depend on e.g. how many concurrent clients you would need to service, and whether you could use caching for any of the responses; in some scenarios where there are lots of clients, it's better to parcel out some of the processing load to them (as long as they can handle it, of course).