We Keep Coding

Fetch API stream doesn't work on server but does locally

As I explained in a previous question I have a FastAPI endpoint that returns a StreamingResponse response that is then consumed by a React application using fetch().body.getReader() API.
The problem I'm facing appears when I open my React application, select the image (s) using Uppy and send it to my FastAPI endpoint, locally it works just fine and the images are returned as a stream response:
But when I deploy my application on Heroku or Render the rendered response is all broken:
Adding more context to my previous question I'm rendering the stream using an async generator:
async function* submit({data}) {
const formData = new FormData()
data?.current?.files?.successful.map(image =>
formData.append('images', image.data)
)
formData.append('language', data.current.language.code)
try {
const response = await fetch(
'endpoint',
{
method: 'POST',
body: formData
}
)
const reader = response.body.getReader()
while (true) {
const { value, done } = await reader.read()
if (done) break
const base64 = `data:${response.headers.get(
'content-type'
)};base64,${btoa(String.fromCharCode(...new Uint8Array(value)))}`
yield base64
}
} catch (error) {
// ...
}
}
That is called when the "Gallery" component in the screenshot is rendered:
const [images, setImages] = useState([])
useEffect(() => {
;(async () => {
for await (const image of await translateSubmit({
data
})) {
setImages(previous => [...previous, image])
}
})()
// eslint-disable-next-line
}, [])
I was expecting to get the same result in the server when the application is deployed, but it just doesn't work as it should and I'm not sure how to approach the problem. Any tips?

How to upload a file into Firebase Storage from a callable https cloud function

I have been trying to upload a file to Firebase storage using a callable firebase cloud function.
All i am doing is fetching an image from an URL using axios and trying to upload to storage.
The problem i am facing is, I don't know how to save the response from axios and upload it to storage.
First , how to save the received file in the temp directory that os.tmpdir() creates.
Then how to upload it into storage.
Here i am receiving the data as arraybuffer and then converting it to Blob and trying to upload it.
Here is my code. I have been missing a major part i think.
If there is a better way, please recommend me. Ive been looking through a lot of documentation, and landed up with no clear solution. Please guide. Thanks in advance.
const bucket = admin.storage().bucket();
const path = require('path');
const os = require('os');
const fs = require('fs');
module.exports = functions.https.onCall((data, context) => {
try {
return new Promise((resolve, reject) => {
const {
imageFiles,
companyPIN,
projectId
} = data;
const filename = imageFiles[0].replace(/^.*[\\\/]/, '');
const filePath = `ProjectPlans/${companyPIN}/${projectId}/images/${filename}`; // Path i am trying to upload in FIrebase storage
const tempFilePath = path.join(os.tmpdir(), filename);
const metadata = {
contentType: 'application/image'
};
axios
.get(imageFiles[0], { // URL for the image
responseType: 'arraybuffer',
headers: {
accept: 'application/image'
}
})
.then(response => {
console.log(response);
const blobObj = new Blob([response.data], {
type: 'application/image'
});
return blobObj;
})
.then(async blobObj => {
return bucket.upload(blobObj, {
destination: tempFilePath // Here i am wrong.. How to set the path of downloaded blob file
});
}).then(buffer => {
resolve({ result: 'success' });
})
.catch(ex => {
console.error(ex);
});
});
} catch (error) {
// unknown: 500 Internal Server Error
throw new functions.https.HttpsError('unknown', 'Unknown error occurred. Contact the administrator.');
}
});

I'd take a slightly different approach and avoid using the local filesystem at all, since its just tmpfs and will cost you memory that your function is using anyway to hold the buffer/blob, so its simpler to just avoid it and write directly from that buffer to GCS using the save method on the GCS file object.
Here's an example. I've simplified out a lot of your setup, and I am using an http function instead of a callable. Likewise, I'm using a public stackoverflow image and not your original urls. In any case, you should be able to use this template to modify back to what you need (e.g. change the prototype and remove the http response and replace it with the return value you need):
const functions = require('firebase-functions');
const axios = require('axios');
const admin = require('firebase-admin');
admin.initializeApp();
exports.doIt = functions.https.onRequest((request, response) => {
const bucket = admin.storage().bucket();
const IMAGE_URL = 'https://cdn.sstatic.net/Sites/stackoverflow/company/img/logos/so/so-logo.svg';
const MIME_TYPE = 'image/svg+xml';
return axios.get(IMAGE_URL, { // URL for the image
responseType: 'arraybuffer',
headers: {
accept: MIME_TYPE
}
}).then(response => {
console.log(response); // only to show we got the data for debugging
const destinationFile = bucket.file('my-stackoverflow-logo.svg');
return destinationFile.save(response.data).then(() => { // note: defaults to resumable upload
return destinationFile.setMetadata({ contentType: MIME_TYPE });
});
}).then(() => { response.send('ok'); })
.catch((err) => { console.log(err); })
});
As a commenter noted, in the above example the axios request itself makes an external network access, and you will need to be on the Blaze or Flame plan for that. However, that alone doesn't appear to be your current problem.
Likewise, this also defaults to using a resumable upload, which the documentation does not recommend when you are doing large numbers of small (<10MB files) as there is some overhead.
You asked how this might be used to download multiple files. Here is one approach. First, lets assume you have a function that returns a promise that downloads a single file given its filename (I've abridged this from the above but its basically identical except for the change of INPUT_URL to filename -- note that it does not return a final result such as response.send(), and there's sort of an implicit assumption all the files are the same MIME_TYPE):
function downloadOneFile(filename) {
const bucket = admin.storage().bucket();
const MIME_TYPE = 'image/svg+xml';
return axios.get(filename, ...)
.then(response => {
const destinationFile = ...
});
}
Then, you just need to iteratively build a promise chain from the list of files. Lets say they are in imageUrls. Once built, return the entire chain:
let finalPromise = Promise.resolve();
imageUrls.forEach((item) => { finalPromise = finalPromise.then(() => downloadOneFile(item)); });
// if needed, add a final .then() section for the actual function result
return finalPromise.catch((err) => { console.log(err) });
Note that you could also build an array of the promises and pass them to Promise.all() -- that would likely be faster as you would get some parallelism, but I wouldn't recommend that unless you are very sure all of the data will fit inside the memory of your function at once. Even with this approach, you need to make sure the downloads can all complete within your function's timeout.

Uploading a video to Gfycat using node.js returns `NotFoundo` from the Gfycat API

I'm building a node app that uses Gfycat's API to upload a local MP4 to it and give me the URL of the converted gif. Here's what I have so far:
const fs = require("fs");
const axios = require("axios");
async function main() {
const res = await axios({
method: "POST",
url: "https://api.gfycat.com/v1/gfycats",
data: { title: "test" }
});
const name = res.data.gfyname;
console.log(`The key name is: ${name}`)
const stream = fs.createReadStream("./Video.mp4");
const { size } = fs.statSync("./Video.mp4");
stream.on("error", console.warn);
const sendResult = await axios({
method: "PUT",
url: `https://filedrop.gfycat.com/${name}`,
data: stream,
headers: {
"Content-Type": "video/mp4",
"Content-Length": size
}
});
// Check the status of the encoding every n seconds until it says complete. When complete, return the gfyname.
await waitTillPosted(name);
return gfyname = checkResult.data.gfyname;
}
// Basic GET API request to check the encoding status of the gif
async function isPosted(name) {
var checkResult = await axios({
method: "GET",
url: `https://api.gfycat.com/v1/gfycats/fetch/status/${name}`,
});
console.log(`Current status: ${checkResult.data.task}`)
if (checkResult.data.task == "complete") {
return true;
}
throw new Error('The gif is still encoding.');
}
// Generic sleep function
function sleep(ms) {
console.log(`Sleeping for ${ms} ms.`)
return new Promise(resolve => setTimeout(resolve, ms));
}
// If the gif is not done encoding, sleep 30s
async function waitTillPosted(name) {
return isPosted(name).catch(() => sleep(30e3).then(waitTillPosted));
}
// ======================================================
main()
.then(name => {
console.log(`https://gfycat.com/${gfyname}`);
})
.catch(err => {
console.error(err);
});
According to the API, you can check the status of the conversion task with GET https://api.gfycat.com/v1/gfycats/fetch/status/gfyname.
When I run this app, the first GET request it makes reports encoding, as expected. I'm making a GET request every 30s until eventually, I expect it to say complete, where I'll return the Gfycat URL. However I'm get a NotFoundo instead.
I took one of the key names I got in my testing and checked the status with a GET request separately, and I got a proper complete response with the details of the gif. It works and the gif exists at the URL.
Why am I getting a NotFoundo when I do the same check as part of my main function?

It may be a bug with gfycat. I often get it during rush-hours when I think their servers are overloaded.
My solution: Start all over a couple of times after waiting a short time-out. That's what "works" for me.
Also you may want to try a different video. Maybe they can't decode your file.

How do I pass a react DOM state's information to backend(NodeJS) when a button is invoked?

I'm using his logic on the frontend, but I'm having some trouble actually receiving that data on the backend. I'm using the Sails.js framework. Any suggestions?
handleSubmit = () => {
// Gathering together the data you want to send to API
const payload = {
subject: this.state.subject,
message: this.state.message,
};
this.handleAjaxRequest(payload);
};
// Method to send data to the backend
// Making the req -I'm using Axios here.
handleAjaxRequest = (payload) => {
let request = axios({
method: 'post',
url: '/api/',
data: payload,
headers: 'Content-Type: application/json'
});
// Do stuff with the response from your backend.
request.then(response => {
console.debug(response.data);
})
.catch(error => {
console.error(error);
})
};
I used to do this using Express and didn't have these problems.
Any help, method, a suggestion is more than welcome :)
Please forgive my ignorance, I'm just here to learn.

Okay, so the first thing I had to do is generate a new restful API using the command sails generate api data. In the package.json file I set up a proxy that includes the backends endpoint, like this "proxy": "http://localhost:1337" - I mean, you don't need to do this, but if you don't then you have to include this URL part on every request. Because it doesn't change, it's pretty convenient to do so.
On the frontend, I made a function sendData() that takes the necessary data from my previous component (depending on what the user selected) and send that data using axios to the backend -->
sendData = () => {
const { relYear } = this.props.history.location.state.dev;
const { relMonth } = this.props.history.location.state.dev;
const selectedMonth = moment().month(relMonth).format("MM");
const finalSelect = parseInt(relYear + selectedMonth, 10);
axios.post('/data', { 'selectedDate' : finalSelect })
.then(res => console.log('Data send'))
.catch(err => console.error(err));
}
On the backend I fetched the data, did the calculations and send back the result to the frontend of my app. -->
getApiData = () => {
let apiData = [];
axios.get('/data')
.then(res => {
let first = Object.values(res.data.pop()).shift(); // Getting the relevant 'selectedDate'
apiData.push(first);
}).catch(err => console.error(err));
return apiData;
}

how to load an image from url into buffer in nodejs

I am new to nodejs and am trying to set up a server where i get the exif information from an image. My images are on S3 so I want to be able to just pass in the s3 url as a parameter and grab the image from it.
I am u using the ExifImage project below to get the exif info and according to their documentation:
"Instead of providing a filename of an image in your filesystem you can also pass a Buffer to ExifImage."
How can I load an image to the buffer in node from a url so I can pass it to the ExifImage function
ExifImage Project:
https://github.com/gomfunkel/node-exif
Thanks for your help!

Try setting up request like this:
var request = require('request').defaults({ encoding: null });
request.get(s3Url, function (err, res, body) {
//process exif here
});
Setting encoding to null will cause request to output a buffer instead of a string.

Use the axios:
const response = await axios.get(url, { responseType: 'arraybuffer' })
const buffer = Buffer.from(response.data, "utf-8")

import fetch from "node-fetch";
let fimg = await fetch(image.src)
let fimgb = Buffer.from(await fimg.arrayBuffer())

I was able to solve this only after reading that encoding: null is required and providing it as an parameter to request.
This will download the image from url and produce a buffer with the image data.
Using the request library -
const request = require('request');
let url = 'http://website.com/image.png';
request({ url, encoding: null }, (err, resp, buffer) => {
// Use the buffer
// buffer contains the image data
// typeof buffer === 'object'
});
Note: omitting the encoding: null will result in an unusable string and not in a buffer. Buffer.from won't work correctly too.
This was tested with Node 8

Use the request library.
request('<s3imageurl>', function(err, response, buffer) {
// Do something
});
Also, node-image-headers might be of interest to you. It sounds like it takes a stream, so it might not even have to download the full image from S3 in order to process the headers.
Updated with correct callback signature.

Here's a solution that uses the native https library.
import { get } from "https";
function urlToBuffer(url: string): Promise<Buffer> {
return new Promise((resolve, reject) => {
const data: Uint8Array[] = [];
get(url, (res) => {
res
.on("data", (chunk: Uint8Array) => {
data.push(chunk);
})
.on("end", () => {
resolve(Buffer.concat(data));
})
.on("error", (err) => {
reject(err);
});
});
});
}
const imageUrl = "https://i.imgur.com/8k7e1Hm.png";
const imageBuffer = await urlToBuffer(imageUrl);
Feel free to delete the types if you're looking for javascript.
I prefer this approach because it doesn't rely on 3rd party libraries or the deprecated request library.

request is deprecated and should be avoided if possible.
Good alternatives include got (only for node.js) and axios (which also support browsers).
Example of got:
npm install got
Using the async/await syntax:
const got = require('got');
const url = 'https://www.google.com/images/branding/googlelogo/2x/googlelogo_color_272x92dp.png';
(async () => {
try {
const response = await got(url, { responseType: 'buffer' });
const buffer = response.body;
} catch (error) {
console.log(error.body);
}
})();

you can do it that way
import axios from "axios";
function getFileContentById(
download_url: string
): Promise < Buffer > {
const response = await axios.get(download_url, {
responseType: "arraybuffer",
});
return Buffer.from(response.data, "base64");
}