We Keep Coding

Nodejs - Fetch file from url and send content to client

For some reason, I don't want to share the URL public (sercet url),
My workflow is as below:
Client send API request to my server:
Api: mywebsite.com/api/image_abc.jpg
I have Nodejs express server to fetch the file from the url:
Eg: sercret_url.com/image_abc.jpg
And then from response image content from Nodejs, I send back the image content to the client and display as image_abc.jpg
I looked around on stackoverflow, but just got an answer from reading file from disk and send to client. What I want is just redirect the image content to client, not saving file to the disk.
Thank you.

Assuming you want to return the contents of a file from a certain URL to the client as buffer here's the solution I suggest
Get the file using axios and return the buffer to you client
const axios = require('axios');
let URL='some-valid-url'
const response = await axios.get(
URL,
{ responseType: 'arraybuffer' }
);
const buffer = Buffer.from(response.data, 'utf-8');
res.status(200).send(buffer);
In case you want to save it to your server you can use fs as follows to write the file to the folder of your choice
fs.writeFile(fileName, buffer, (err) => {
if(!err) console.log('Data written');
});

Node.js Express send huge data to client vanilla JS

In my application I read huge data of images, and send the whole data to the client:
const imagesPaths = await getFolderImagesRecursive(req.body.rootPath);
const dataToReturn = await Promise.all(imagesPaths.map((imagePath) => new Promise(async (resolve, reject) => {
try {
const imageB64 = await fs.readFile(imagePath, 'base64');
return resolve({
filename: imagePath,
imageData: imageB64,
});
} catch {
return reject();
}
})));
return res.status(200).send({
success: true,
message: 'Successfully retreived folder images data',
data: dataToReturn,
});
Here is the client side:
const getFolderImages = (rootPath) => {
return fetch('api/getFolderImages', {
method: 'POST',
headers: { 'Content-type': 'application/json' },
body: JSON.stringify({ rootPath }),
});
};
const getFolderImagesServerResponse = await getFolderImages(rootPath);
const getFolderImagesServerData = await getFolderImagesServerResponse.json();
When I do send the data I get failure due to the huge data. Sending the data just with res.send(<data>) is impossible. So, then, how can I bypass this limitation - and how should I accept the data in the client side with the new process?

The answer to your problem requires some read :
Link to the solution
One thing you probably haven’t taken full advantage of before is that webserver’s http response is a stream by default.
They just make it easier for you to pass in synchron data, which is parsed to chunks under the hood and sent as HTTP packages.
We are talking about huge files here; naturally, we don’t want them to be stored in any memory, at least not the whole blob. The excellent solution for this dilemma is a stream.
We create a readstream with the help of the built-in node package ‘fs,’ then pass it to the stream compatible response.send parameter.
const readStream = fs.createReadStream('example.png');
return response.headers({
'Content-Type': 'image/png',
'Content-Disposition': 'attachment; filename="example.png"',
}).send(readStream);
I used Fastify webserver here, but it should work similarly with Koa or Express.
There are two more configurations here: naming the header ‘Content-Type’ and ‘Content-Disposition.’
The first one indicates the type of blob we are sending chunk-by-chunk, so the frontend will automatically give the extension to it.
The latter tells the browser that we are sending an attachment, not something renderable, like an HTML page or a script. This will trigger the browser’s download functionality, which is widely supported. The filename parameter is the download name of the content.
Here we are; we accomplished minimal memory stress, minimal coding, and minimal error opportunities.
One thing we haven’t mentioned yet is authentication.
For the fact, that the frontend won’t send an Ajax request, we can’t expect auth JWT header to be present on the request.
Here we will take the good old cookie auth approach. Cookies are set automatically on every request header that matches the criteria, based on the cookie options. More info about this in the frontend implementation part.
By default, cookies arrive as semicolon separated key-value pairs, in a single string. In order to ease out the parsing part, we will use Fastify’s Cookieparser plugin.
await fastifyServer.register(cookieParser);
Later in the handler method, we simply get the cookie that we are interested in and compare it to the expected value. Here I used only strings as auth-tokens; this should be replaced with some sort of hashing and comparing algorithm.
const cookies = request.cookies;
if (cookies['auth'] !== 'authenticated') {
throw new APIError(400, 'Unauthorized');
}
That’s it. We have authentication on top of the file streaming endpoint, and everything is ready to be connected by the frontend.

POST image to django rest API always returns 'No file was submitted'

I am trying to get this working now for days -.-
Using a simple NodeJS express server, I want to upload an image to a Django instance through Post request, but I just can't figure out, how to prepare the request and embed the file.
Later I would like to post the image, created from a canvas on the client side,
but for testing I was trying to just upload an existing image from the nodeJS server.
app.post('/images', function(req, res) {
const filename = "Download.png"; // existing local file on server
// using formData to create a multipart/form-data content-type
let formData = new FormData();
let buffer = fs.readFileSync(filename);
formData.append("data", buffer); // appending the file a buffer, alternatively could read as utf-8 string and append as text
formData.append('name', 'var name here'); // someone told me, I need to specify a name
const config = {
headers: { 'content-type': 'multipart/form-data' }
}
axios.post("http://django:8000/images/", formData, config)
.then(response => {
console.log("success!"); // never happens :(
})
.catch(error => {
console.log(error.response.data); // no file was submitted
});
});
What am I doing wrong or did I just miss something?
EDIT
I just found a nice snippet with a slighlty other approach on the npm form-data page, on the very bottom (npmjs.com/package/form-data):
const filename = "Download.png"; // existing local file on server
let formData = new FormData();
let stream = fs.createReadStream(filename);
formData.append('data', stream)
let formHeaders = formData.getHeaders()
axios.post('http://django:8000/images/', formData, {
headers: {
...formHeaders,
},
})
.then(response => {
console.log("success!"); // never happens :(
})
.catch(error => {
console.log(error.response.data); // no file was submitted
});
sadly, this doesn't change anything :( I still receive only Bad Request: No file was submitted
I don't really have much Django code just a basic setup using the rest_framework with an image model:
class Image(models.Model):
data = models.ImageField(upload_to='images/')
def __str__(self):
return "Image Resource"
which are also registered in the admin.py,
a serializer:
from .models import Image
class ImageSerializer(serializers.HyperlinkedModelSerializer):
class Meta:
model = Image
fields = ('id', 'data')
using automatic URL routing.
I wrote a simple test script and put the same image on the django server, to verify that image uploads works, and it does:
import requests
url = "http://127.0.0.1:8000/images/"
file = {'data': open('Download.png', 'rb')}
response = requests.post(url, files=file)
print(response.status_code) # 201

I had a similar problem: I used the same Django endpoint to upload a file using axios 1) from the client side and 2) from the server side. From the client side it worked without any problem, but from the server side, the request body was always empty.
My solution was to use the following code:
const fileBuffer = await readFile(file.filepath)
const formData = new FormData()
formData.append('file', fileBuffer, file.originalFilename)
const response = await fetch(
urlJoin(BACKEND_URL),
{
method: 'POST',
body: formData,
headers: {
...formData.getHeaders(),
},
}
)
A few relevant references that I found useful:
This blog post, even though it seems the author manages to send form data from the server side using axios, I did not manage to reproduce it on my case.
This issue report in the axio repository, where one comment suggests to use fetch.

In your node.js express server instead of adding the image to the form data, try directly sending the stream in the API post.
const filename = "Download.png"; // existing local file on server
//let formData = new FormData();
let stream = fs.createReadStream(filename);
//formData.append('data', stream)
let formHeaders = formData.getHeaders()
axios.post('http://django:8000/images/', stream, {
headers: {
...formHeaders,
},
})
.then(response => {
console.log("success!"); // never happens :(
})
.catch(error => {
console.log(error.response.data); // no file was submitted
});

I still didn't manage to get this working with axios so I tried another package for sending files as post requests, namely unirest, which worked out of the box for me.
Also it is smaller, requires less code and does everything I needed:
const filename = "Download.png"; // existing local file on server
unirest
.post(url)
.attach('data', filename) // reads directly from local file
//.attach('data', fs.createReadStream(filename)) // creates a read stream
//.attach('data', fs.readFileSync(filename)) // 400 - The submitted data was not a file. Check the encoding type on the form. -> maybe check encoding?
.then(function (response) {
console.log(response.body) // 201
})
.catch((error) => console.log(error.response.data));
If I have some spare time in the future I may look into what was wrong with my axios implementation or someone does know a solution pls let me know :D

How do I Display Images in React which are downloaded as Files & saved in a Folder on the Server with the Filepath stored in my Database?

Short Setup, Problem & Code Summary
Setup
React.js, Node.js/Express.js & PostgreSQL.
I scrape an ImageUrl of a Book, download and store it in a File in an Image Folder with Node.js and store the Filepath in my postgreSQL Database AND I would like to fetch and display the Image in an <img /> tag on my React frontend.
Problem
The Image I Fetch from my Database through my API doesn't get displayed, eventhough I can console.log the Image Object in the devtool-console and even make a succesful Postman request with the Image-Object as JSON response.
Furthermore I can get the image even be displayed if I go to localhost:3001/image-193x278.png.
My <img /> Tag has even the right Img src attribute displayed in the devtools (images\image-193x278.png)
In the devtool-console I get the following error:
GEThttp://localhost:3002/images/image-193x278.png
[HTTP/1.1 404 Not Found 0ms]
(Note: My Frontend runs on localhost 3002 while my server & frontend are running concurrently with a proxy on 3001, but that isn't the causing Problem here)
Code
React Code
const [image, setImage] = useState([]);
useEffect(() => {
async function fetchData() {
const response = await fetch('http://localhost:3001/test', {
method: 'GET',
headers: {
Accept: 'application/json',
'Content-Type': 'application/json',
}
})
const data = await response.json();
console.log(data);
setImage(data);
}
fetchData();
}, []);
I tried to get it displayed in two different ways:
<img className={css(styles.Image)} src={image.test_images} alt="" />
{image.map((img) =>
<img className={css(styles.Image)} src={img.test_images} alt="" />)}
Server-Side
app.get('/test', async (req, res) => {
try {
const image = await testdb.query('SELECT * FROM test_table');
res.json(image.rows);
} catch (err) {
console.error(err);
}
})
I serve an Express Static File
app.use(express.static('images'));
I even tried two additional varients just in case
app.get('/test', express.static(__dirname + '../images'));
app.use(express.static('../images'));
Still with no succes.
I hope i could make myself understood as good as possible and I would be extremely thankful for every help thrown out my way.

The problem is with your static folder. The static files will be available at localhost:3002/:filename.
If you want to make it work under /images path, then you can add the path to the static declaration like
app.use('/images', express.static('images'))
Then you can hit the localhost:3002/images/:filename.
You can find the documentation here: https://expressjs.com/en/starter/static-files.html

Node server to send image to cloud to be hosted

I send an image file to my node server via my react app -
I want to host these images on google cloud or similar so they have an accessible URL.
I have tried using cloudinary and google cloud but to no avail thus far!
My react-side code (shortened):
imageFile = this.state.files[0])
const formData = new FormData()
formData.append('file', imageFile);
sendImage(formData)
sendImage(image) {
axios.post("https://137a6167.ngrok.io/image-upload", image, {
})
.then(res => { // then print response status
console.log(res.statusText)
})
}
The file is successfully sent to my server and consoled:
app.post('/image-upload', (req, res) => {
console.log('consoling the req.body!!!!' + JSON.stringify(req.body))
})
THE CONSOLE: consoling the req.body!!!!{"1":"[object File]"}
I did try use this following cloudinary method, yet it threw errors:
cloudinary.config({
cloud_name: process.env.CLOUD_NAME,
api_key: process.env.API_KEY,
api_secret: process.env.API_SECRET
})
app.use(formData.parse())
app.post('/image-upload', (req, res) => {
const values = Object.values(req.files)
const promises = values.map(image => cloudinary.uploader.upload(image.path))
Promise
.all(promises)
.then(results => res.json(results))
})
this gave me the error that an unhandled error in the promise wasnt handled and i got a bit lost with where to go beyond that!
I looked at google cloud storage too but couldn't get it working! Any advice?
What I really want to do is return back to my react app the URL of the hosted image - so it can be stored in DB for the user!
If you can help at all that would be greatly appreciated, thank you.

There are couple of things you need to fix on the front end before you try to upload to any cloud.
First you need to set 'Content-Type': 'multipart/form-data' header in axios to send the file data properly.
Check this thread for more details: How do I set multipart in axios with react?
Then on the express side you need multer or some other similar library to receive the data. You can't access it from req.body. multer adds req.files for example.
https://github.com/expressjs/multer
Try there steps and then post the exact error message you are receiving from google cloud.