JSON seems to have a lot of benefits over formData for sending data to server. Some of them include sending nested data without having to manually stringfy, or making possible a simple code like below to work:
data() {
return {
form: {},
}
},
methods: {
submit() {
this.form = await this.$axios.$post('/images', this.form)
},
No matter how 'form' object is structured, i could easily send it and manage the JSON in the server. The problem with this approach seems to arrive when we need to send some uploaded files together. The most common approach seems to be sending the files as base64, but this is a bad practice since it makes file sizes bigger. Is there anyway so we can send a file appended to this JSON body without converting it to base64 or the only way would be to use formData method? Something like multipart-formdata but with JSON?
No, the json content-type cant have a file appended to it.
For APIs its better to separately do a file upload, then use the path to the file (or the disk and file name) for relating the resource to the file.
If you have to do a single request then it must be in "formData" form.
Update :
Another way is to encode the file in base64 format from the client, then decode it in the back-end (potential quality loss, have not tried this, just a suggestion)
Related
I have been developing a web app where the user can upload a PDF file, and then later retrieve it and view it. What I have been doing to achieve this is having the PDF uploaded to a PostgreSQL database as bytea datatype (the column is called "attachment"), and then having nodeJS offer up this data to be fetched back and turned back into a PDF to view.
However I have been struggling to convert the data back into a valid PDF. This is the method I am using so far.
var file = new Blob([res[i].attachment], { type: 'application/pdf' });
var fileURL = URL.createObjectURL(file);
document.getElementById("pdf_box").data = fileURL;
The #pdf_box identifier refers to an object element in a HTML file which is used to display the PDF (this has been shown to work when I provide the file location of a dummy PDF file to the data attribute of this element).
The res[i].attachment is also shown to provide valid buffer data in JSON format, with an example provided below:
"attachment":{"type":"Buffer","data":[91,111,98,106,101,99,116,32,70,105,108,101,93]}
When I load the created fileURL into the data attribute of #pdf_box however, I get an error indicating along the lines of an invalid PDF file. My research so far appears to indicate this may be because the data is coming in as buffer whereas it needs to be in byte form, but I haven't found much that helps show me how this may be achieved or if there is a way to convert between the forms with the data I have access to? I have also seen occasional reference to a method called pdf.create(), but I cannot find documentation on this and I assume it must belong to a third-party library for JS.
If you know of any information that can help me with understanding what my problem is and what to search to figure out a solution, it would all be greatly appreciated, thank you.
To all reading this question, this method does not seem possible and would likely be incredibly inefficient to implement. If you send a string to nodeJS larger than the MTU of your link to this server (anywhere between 68 bytes and >1500 bytes depending on every component of your network) the packet will be silently dropped with no attempts to resolve or throw an error. What you must do is take the approach of using "multipart/form-data" encoding to send any files to the server (more on this found here).
It should also be mentioned that uploading a file to the database is not recommended in any capacity (due to databases being quite inefficient storage for large amounts of data) and what should be done is to upload a reference to the file path or URL to find the file at. Then on the client-side this file could be rendered as such when you have retrieved the file location...
<object id="pdf" data="../files/test.pdf"></object>
To change the data attribute, the following can be done...
document.getElementById("pdf").data = "../files/test2.pdf";
I am sending FormData() to a cloud function with something like this in the JavaScript frontend.
//Frontend Angular
const formData = mew FormData();
formData.append('file1', zipFile1);
formData.append('file2', zipFile2);
formData.append('name', 'MyFiles');
this.http.post(urlAPI, formData) //angularJS
//Cloud Function
function main(param) {
console.log(param);
}
The console log on the param shows the content-type is multipart/form-data is in the header.
There is also a property labeled as __ow_body. This value is a very long string of characters and digits. I am unsure what this is but I am assuming it is the files I am sending in stream/serialized format.
{
__ow_body: 'hf381fh891hv831h93n19384v938v892vn98vn2890vn29n9vn9892vn948vn2893vn2985hv98...'
}
I wanted to confirm if this is the stream data, and if so, how can I parse this?
I need to send this file, which is a zip file containing images, to a API I am using. In this documents API, the examples show sending a file in the local file system such as
someApiFunction('./myImgs.zip').then(...);
The problem lies in me sending my zip file over http network protocol and the format is very different I think from the example of reading a file in the local file system/machine. How can I deserialize/ or parse my file so that it can be recognized as a zip file containing images inside?
I tried using fs.createReadStream but it doesn't seem to be parsing it. It will just make a more structured object, but within that object, I don't see my formData and its keys such as file1 file2 and name.
Do I need to write this file some where first?
Even if it is less probable, you can try to change a bit the way of how you are posting the formData. I have found a really nice documentation about how to use formData with Angular. As it is stated there you can try to post is using something like this :
this.httpClient.post<any>(this.SERVER_URL, formData).subscribe(
(res) => console.log(res),
(err) => console.log(err)
);
This is less probable to work, but you may give it a shot.
The thing which is certain is that because of the way Cloud Functions pre-processes some requests, you can expect that some libraries not to work and I think this is the exact situation for you.
As it is stated in the Cloud Functions documentation , you need to use rawBody property of the request and 'the bus-boy' library in order to process multipart/form-data.
Here is one of the code example for handling this kind of requests.
I can send a file to the server using axios and the FormData api, like that :
persist(avatar){
let data = new FormData();
data.append('avatar', avatar);
axios.post(`/api/users/${this.user.name}/avatar`, data)
.then(() => flash('Avatar uploaded!'));
}
The avatar param, passed to persist(), is a file object from a form input of type "file".
I can then grab the file on the server side.
Is it possible to do it without using FormData ? That is, to simulate the FormData work ? Basically, I'm trying to understand the extra work done by the FormData api. Maybe it's not possible using axios, and I should do that with plain XMLHttpRequest.
Of course, simply sending the file object won't work :
axios.post(`/api/users/${this.user.name}/avatar`, {avatar: avatar})
On the server side, the avatar object will be empty. I can send metadata like avatar.name, but not the whole object.
Yes, it is possible to do the encoding manually on the client. Writing a form data encoder yourself may be useful if you really want to learn every little detail about how forms work. However, for most applications, I would not recommend it. The FormData API should be used in production.
You will need to refer to RFC 7578 for how to implement the encoder.
This specification defines the multipart/form-data media type, which can be used by a wide variety of applications and transported by a wide variety of protocols as a way of returning a set of values as the result of a user filling out a form.
More resources:
Meaning of multipart/form-data
Source code for the form-data library on npm
I have a GraphQL server, hosted on express. I want to return images to the client by sending back nodejs buffer objects. How can i config graphql server, to return bytes, instead of json? I don't wish to do this through base64, as the image are large in size.
You have to return JSON, but there's still a way. We're using GraphQL to return images stored in Blob fields in a legacy sql database. We're using sequelize, graphql-sequelize, and graphql-js.
We've defined the Blob fields to be String types in our graphql schema and so they come through in the json reply just fine.
Then we convert to a buffer before delivering, like
const imgBuffer = new Buffer.from(imgObj.data, 'ascii');
The only problem is we're now having trouble saving image data back to the database via the graphql interface. Our mutation function gives us a syntax error when it finds certain bad unicode characters in the strings, like \U0000 and whatnot (so I found your question looking for a solution to that).
There's a way, but it's complicated, and very manual, and I'm only going to give you an overview of what I've done in ApolloServer, but I think it should be enough.
First, you need to use the "Accept" header in your request to send a binary mime type, and send a matching "Content-Type" in your response. This is nessisary to be efficient, but not nessisary to work, as you'll see (with EJSON).
To serialize and deserialize respecting the headers you may need to write an express middleware, and you'll need to handle base64 encoding with a {$data: "..."} encapsulating object (as EJSON does) or just (strangely) returning null, if someone makes a request for binary data using "application/json" for their "accept" header. You'll also want to choose what binary formats that you'll support. I only use 1: "application/x-msgpack", but I hear that "application/cbor" is becoming more popular. You can use a library for EJSON, MessagePack, and CBOR to do your serialization, so this isn't as hard as it sounds.
I would then strongly recommend using the #defer on any images. See this post for more information on #defer: https://www.apollographql.com/blog/introducing-defer-in-apollo-server-f6797c4e9d6e/
I've done it. It wasn't easy, and it would be better if ApolloServer worked this way "out of the box".
It's better to send a hashed & temporary link to download it
The URL can be hashed to be non-accessible by other users.
Backend can expire the link or remove the binary file on the static server.
There might be an answer to your question by using the node module found here.
I want to upload a binary file using json.
I choose Json because with the file I would also like to send additional information.
I am going to do this by -
Select a file in the file input tag.
Use the HTML5 File Reader Api to read a file first.
Convert the file content into base64.
Add the base64 content to a JS object in a data uri format.
Convert the JS object to json and post it to the server.
I wonder if this is the only legitimate way to achieve my goal? Also, if there is a plugin already available somewhere which give me this ability?
No, this is not the only way - one of the other ways is just to submit a form with a file in it. Such form uses multipart/form-data content type.
See W3C documentation on the subject:
The content type "application/x-www-form-urlencoded" is inefficient for sending large quantities of binary data or text containing non-ASCII characters.
The content type "multipart/form-data" should be used for submitting forms that contain files, non-ASCII data, and binary data.
So, there is no need to reinvent the wheel - browsers already support sending the files along with additional information, in a simple way. You just create a form where the user can enter data and select files, then all of them are sent to the server with multipart/form-data content type, and your web framework should be able to understand that it deals with both files and textual data.