I am calling a REST API in my project for creating some records.
Everything is working fine but I got a challenge, the JSON request body is too big (having thousands of keys and values).
Now I want to compress the request body. I tried it using JavaScript
var reqJSON = { ... } // too big JSON object
var compressedJSON = JSON.stringify(reqJSON, null, 0); // converting JSON to String (compression)
Now I am sending the string in the request body and converting this string into JSON at the server-side.
I am curious, is it the correct way of JSON compression? If yes how can I check the difference in the request body size?
Thanks for your time.
That isn't compression at all.
var reqJSON = { ... } // too big JSON object
That will give you a JavaScript object, not JSON. Possibly your Ajax library will convert it to JSON if you pass it. There's no way for us to know. If the data is to get to the server then it will need serializing to some format that can be sent over the wire so something must be converting it before the HTTP request is made.
var compressedJSON = JSON.stringify(reqJSON, null, 0); // converting JSON to String (compression)
That will give you JSON. There's no compression involved though.
If you want to compress it then you'd need to look for a library that can do actual compression.
You can use gzip for compress json its working fine
Related
I cannot send data of <textarea> to node js, Node js don't see the my sent data.
For Fetch data to Node Js
continueBtn.addEventListener("click", async () => {
console.log(myMsg.value)
console.log(typeof(myMsg.value))
const req = await fetch("/sendmsg", {method: "POST",body: myMsg.value}) ;
console.log("Fetched")
})
For get data in Node js
const {userMessage} = Object.keys(req.body)
You are passing a string to body. Since you aren't overriding it, it will be given a text/plain content-type.
On the server you expect req.body to contain an object.
You haven't shown us what, if any, body parsing middlewares you have configured in your server side code, but none of the common ones will convert something that is text/plain into an object.
You should:
Make sure that you have a body parsing middleware configured
Encode the data you are passing to body (e.g. with URLSearchParams) so it has name=value pairs instead of being a plain string.
If you end up passing a string (e.g. if you pass a string of JSON instead of a URLSearchParams object) then you'll also need to set a Content-Type request header that matches your data format.
"const {userMessage} = Object.keys(req.body)"
I don't see why you'd be expecting to have a userMessage key in your body object with the code you pasted. It's possible the data is being passed properly, but you are trying to access the wrong key. I would try logging in your server code to debug the response format as a first step:
console.log(JSON.stringify(req.body))
Let me know if that gets you enough help to move forward, and I can circle back here if not.
I have a webhook.
data0=[{"senderId":"smsdia","requestId":"******383233353233","report":[{"date":"2017-12-02 17:00:41","number":"12345","status":"1","desc":"DELIVERED"}],"userId":"119385","campaignName":"viaSOCKET"}]
I receive the above data in a POST request to my server.
Content-Type: application/x-www-form-urlencoded
How do I parse it?
I know that if it is a list: data1=['sree','kanth'] I can parse it with request.POST.getlist('data1[]')
But I don't know how to parse when it is a list containing dict.
edit1
Moreover, I get len(data1) is 2. But len(data0) is 0.
edit2
using request.lib:
https://requestb.in/13df2891?inspect
This appears to be JSON sent inside a form field. You can use the json library to parse it:
data = json.loads(request.POST['data'])
For example, I have a .JSON file that has the following:
[{"honda": "accord", "color": "red"},{"ford": "focus", "color": "black"}]
What would be the javascript code to push another object {"nissan": "sentra", "color": "green"} into this .json array to make the .json file look like
[{"honda": "accord", "color": "red"},{"ford": "focus", "color": "black"},{"nissan": "sentra", "color": "green"}]
The reason I'm asking is I am finding a lot of information online on how to pull data from a .json file using AJAX but not writing new data to the .json file using AJAX to update the .json file with additional data.
Any help would be appreciated!
You have to be clear on what you mean by "JSON".
Some people use the term JSON incorrectly to refer to a plain old JavaScript object, such as [{a: 1}]. This one happens to be an array. If you want to add a new element to the array, just push it, as in
var arr = [{a: 1}];
arr.push({b: 2});
< [{a: 1}, {b: 2}]
The word JSON may also be used to refer to a string which is encoded in JSON format:
var json = '[{"a": 1}]';
Note the (single) quotation marks indicating that this is a string. If you have such a string that you obtained from somewhere, you need to first parse it into a JavaScript object, using JSON.parse:
var obj = JSON.parse(json);
Now you can manipulate the object any way you want, including push as shown above. If you then want to put it back into a JSON string, then you use JSON.stringify:
var new_json = JSON.stringify(obj.push({b: 2}));
'[{"a": 1}, {"b": 1}]'
JSON is also used as a common way to format data for transmission of data to and from a server, where it can be saved (persisted). This is where ajax comes in. Ajax is used both to obtain data, often in JSON format, from a server, and/or to send data in JSON format up to to the server. If you received a response from an ajax request which is JSON format, you may need to JSON.parse it as described above. Then you can manipulate the object, put it back into JSON format with JSON.stringify, and use another ajax call to send the data to the server for storage or other manipulation.
You use the term "JSON file". Normally, the word "file" is used to refer to a physical file on some device (not a string you are dealing with in your code, or a JavaScript object). The browser has no access to physical files on your machine. It cannot read or write them. Actually, the browser does not even really have the notion of a "file". Thus, you cannot just read or write some JSON file on your local machine. If you are sending JSON to and from a server, then of course, the server might be storing the JSON as a file, but more likely the server would be constructing the JSON based on some ajax request, based on data it retrieves from a database, or decoding the JSON in some ajax request, and then storing the relevant data back into its database.
Do you really have a "JSON file", and if so, where does it exist and where did you get it from? Do you have a JSON-format string, that you need to parse, mainpulate, and turn back into a new JSON-format string? Do you need to get JSON from the server, and modify it and then send it back to the server? Or is your "JSON file" actually just a JavaScript object, that you simply need to manipulate with normal JavaScript logic?
JSON can be written into local storage using the JSON.stringify to serialize a JS object. You cannot write to a JSON file using only JS. Only cookies or local storage
var obj = {"nissan": "sentra", "color": "green"};
localStorage.setItem('myStorage', JSON.stringify(obj));
And to retrieve the object later
var obj = JSON.parse(localStorage.getItem('myStorage'));
Unfortunatelly, today (September 2018) you can not find cross-browser solution for client side file writing.
For example: in some browser like a Chrome we have today this possibility and we can write with FileSystemFileEntry.createWriter() with client side call, but according to the docu:
This feature is obsolete. Although it may still work in some browsers, its use is discouraged since it could be removed at any time. Try to avoid using it.
For IE (but not MS Edge) we could use ActiveX too, but this is only for this client.
If you want update your JSON file cross-browser you have to use server and client side together.
The client side script
On client side you can make a request to the server and then you have to read the response from server. Or you could read a file with FileReader too. For the cross-browser writing to the file you have to have some server (see below on server part).
var xhr = new XMLHttpRequest(),
jsonArr,
method = "GET",
jsonRequestURL = "SOME_PATH/jsonFile/";
xhr.open(method, jsonRequestURL, true);
xhr.onreadystatechange = function()
{
if(xhr.readyState == 4 && xhr.status == 200)
{
// we convert your JSON into JavaScript object
jsonArr = JSON.parse(xhr.responseText);
// we add new value:
jsonArr.push({"nissan": "sentra", "color": "green"});
// we send with new request the updated JSON file to the server:
xhr.open("POST", jsonRequestURL, true);
xhr.setRequestHeader("Content-Type", "application/x-www-form-urlencoded");
// if you want to handle the POST response write (in this case you do not need it):
// xhr.onreadystatechange = function(){ /* handle POST response */ };
xhr.send("jsonTxt="+JSON.stringify(jsonArr));
// but on this place you have to have a server for write updated JSON to the file
}
};
xhr.send(null);
Server side scripts
You can use a lot of different servers, but I would like to write about PHP and Node.js servers.
By using searching machine you could find "free PHP Web Hosting*" or "free Node.js Web Hosting". For PHP server I would recommend 000webhost.com and for Node.js I would recommend to see and to read this list.
PHP server side script solution
The PHP script for reading and writing from JSON file:
<?php
// This PHP script must be in "SOME_PATH/jsonFile/index.php"
$file = 'jsonFile.txt';
if($_SERVER['REQUEST_METHOD'] === 'POST')
// or if(!empty($_POST))
{
file_put_contents($file, $_POST["jsonTxt"]);
//may be some error handeling if you want
}
else if($_SERVER['REQUEST_METHOD'] === 'GET')
// or else if(!empty($_GET))
{
echo file_get_contents($file);
//may be some error handeling if you want
}
?>
Node.js server side script solution
I think that Node.js is a little bit complex for beginner. This is not normal JavaScript like in browser. Before you start with Node.js I would recommend to read one from two books:
Learning Node: Moving to the Server-Side
Node.js Web Development: Server-side development
The Node.js script for reading and writing from JSON file:
var http = require("http"),
fs = require("fs"),
port = 8080,
pathToJSONFile = '/SOME_PATH/jsonFile.txt';
http.createServer(function(request, response)
{
if(request.method == 'GET')
{
response.writeHead(200, {"Content-Type": "application/json"});
response.write(fs.readFile(pathToJSONFile, 'utf8'));
response.end();
}
else if(request.method == 'POST')
{
var body = [];
request.on('data', function(chunk)
{
body.push(chunk);
});
request.on('end', function()
{
body = Buffer.concat(body).toString();
var myJSONdata = body.split("=")[1];
fs.writeFileSync(pathToJSONFile, myJSONdata); //default: 'utf8'
});
}
}).listen(port);
Related links for Node.js:
How to Develop Web Application using pure Node.js (HTTP GET and POST, HTTP Server) (detailed video tutorial)
Anatomy of an HTTP Transaction
How to handle POST request in Node.js
How do you extract POST data in Node.js?
I'm developing a REST API with Play 2 and I'm wondering how to implement file upload functionality.
I've read the official Play documentation but it just provides a multipart/form-data example, while my backend does not provide any form... it just consists of a REST API to be invoked by a JavaScript client or whatever else.
That said, what's the correct way to implement such an API? Should I implement a PartHandler and then still use the mutipartFormData parser? How should I pass the file content to the API? Is there any exhaustive example on this topic?
Any help would be really appreciated.
You should look into BodyParsers: http://www.playframework.com/documentation/2.2.x/ScalaBodyParsers
What you are trying to do is not especially complicated, especially if you are only handling smaller files that would fit in memory. After all uploading a file is just about sending the file as a body of a POST or something like that. It is not any different from receiving some XML or JSON in a request.
Hope this helps
import org.apache.http.entity.mime._
import java.io.File
import org.apache.http.entity.mime.content._
import java.io.ByteArrayOutputStream
import play.api.libs.ws.WS
val contents ="contents string"
val file = File.createTempFile("sample", ".txt")
val bw = new java.io.BufferedWriter(new java.io.FileWriter(file)
bw.write(new_contents);
bw.close();
builder.addPart("file", new FileBody(file, org.apache.http.entity.ContentType.create("text/plain"), "sample"))
builder.setMode(HttpMultipartMode.BROWSER_COMPATIBLE);
val entity = builder.build
val outputstream = new ByteArrayOutputStream
entity.writeTo(outputstream)
val header = (entity.getContentType.getName -> entity.getContentType.getValue)
val response = WS.url("/post/file").withHeaders(header).post(outputstream.toByteArray())
To pass your contents, depending on your client side, you can encode the contents to Base64 at client side to pass the contents as Json (You can use Json body parser). Then on the server side you can decode the contents using a Base64 decoder (e.g. Apache Commons) to get the byte array. It will be as simple as
Base64.decodeBase64(YourEncodedFileContent)
When you have the byte array you can simply write it on disk or save it into database etc. We are using this approach in production and it works fine however we only handle small file uploads.
OK, thank you all for your suggestions... here below is how I solved my issue:
object Files extends Controller {
def upload = SecuredAction[Files.type]("upload").async(parse.multipartFormData(partHandler)) { implicit request =>
future { request.body.files.head.ref match {
case Some((data, fileName, contentType)) => Ok(success(Json.obj("fileName" -> fileName)))
case _ => BadRequest
}}.recover { case e =>
InternalServerError(error(errorProcessingRequest(e.getMessage)))
}
}
...
private def partHandler = {
parse.Multipart.handleFilePart {
case parse.Multipart.FileInfo(partName, fileName, contentType) =>
Iteratee.fold[Array[Byte], ByteArrayOutputStream](
new ByteArrayOutputStream
) { (outputStream, data) =>
outputStream.write(data)
outputStream
}.map { outputStream =>
outputStream.close()
Some(outputStream.toByteArray, fileName, contentType.get)
}
}
}
}
I hope it helps.
while my backend does not provide any form... it just consists of a REST API to be invoked by a JavaScript client
Then your backend is not a REST API. You should follow the HATEOAS principle, so you should respond with links and forms along with data to every GET request. You don't have to send back HTML, you can describe these things with hypermedia json or xml media types, for example with JSON-LD, HAL+JSON, ATOM+XML, etc... So you have to describe your upload form in your preferred hypermedia, and let the REST client to turn that description into a real HTML file upload form (if the client is HTML). After that you can send a multipart/form-data as usual (REST is media type agnostic, so you can send data in any media type you want, not just in a JSON format). Check the AJAX file upload techniques for further detail...
I'm creating an android app which takes in some json data, is there a way to set up a directory such as;
http://......./jsons/*.json
Alternatively, a way to add into a json file called a.json, and extend its number of containing array data, pretty much add more data into the .json file this increase its size.
It could be by PHP or Javascript.
Look into Parsing JSON, you can use the JSON.parse() function, in addition, I'm not sure about getting all your JSON files from a directory call, maybe someone else will explain that.
var data ='{"name":"Ray Wlison",
"position":"Staff Author",
"courses":[
"JavaScript & Ajax",
"Buildinf Facebook Apps"]}';
var info = JSON.parse(data);
//var infostoring = JSON.stringify(info);
One way to add to a json file is to parse it, add to it, then save it again. This might not be optimal if you have large amounts of data but in that case you'll probably want a proper database anyway (like mongo).
Using PHP:
$json_data = json_decode(file_get_contents('a.json'));
array_push($json_data, 'some value');
file_put_contents('a.json', json_encode($json_data));