We Keep Coding

How can I Read/Write Images with Node.js?

I'm pretty new to JS and I'm working on a Discord bot that uses Discord.js that uses COCO-SSD JS and requires me to Read/Write files from and to my PC.
I know this is probably not the best idea but I'll cross that bridge when I get to it.
Right now, I need a way to Read and write Images to and from my PC.
NOTE: The files need to be Written to my computer from a URL. However if there's any way to circumvent having to downlead the images on my pc I would appreciate some help with that as well.
I'm using "fs", "https", and "fetch".
The problem with my method is that the pixels that I'm receiving from the images are NULL and so I cant do much with them.
Here's my current code:
Sorry for the horrible formatting, english is not my first language and it's 2 AM here.
const fetchUrl = require("fetch").fetchUrl;
const https = require('https');
function saveImageToDisk(url,path)
{
var fullUrl = url;
var localPath = fs.createWriteStream(path);
var request = https.get(fullUrl,function(response) { console.log(response)
response.pipe(localPath)
});
}
saveImageToDisk("https://post.greatist.com/wp-content/uploads/sites/3/2020/02/322868_1100-1100x628.jpg","./images/" + Date.now + ".png");
const img = fs.readFile("./images/" + Date.now + ".png", function(response){console.log(response)});

I know how to achieve something like this using the module node-fetch, so I will provide an example below. I don't know what exactly you need this functionality for in your discord bot, but the below code will be able to properly save an image url to a specified path.
const fetch = require("node-fetch");
const fs = require("fs");
function saveImageToDisk(url, path)
{
fetch(url)
.then(res => res.buffer())
.then(buffer => fs.writeFileSync(path, buffer));
}
I've tested the above example code and it works perfectly. As for the rest of the code, I do have a few fixes:
const filepath = "./images/" + Date.now() + ".png";
const imageurl = "https://post.greatist.com/wp-content/uploads/sites/3/2020/02/322868_1100-1100x628.jpg";
saveImageToDisk(imageurl, filepath);
const img = fs.readFile(filepath, (err, response) =>{
if (err) throw err;
console.log(response);
});
Specifically, the main issues were:
Date.now is a function (hence it should be Date.now())
You should only create the filepath string once instead of doing so twice (for accuracy and simplicity)
The callback of fs.readFile actually has its error object as the first parameter (so you were actually logging an error, or lack thereof, to the console instead of the response/file itself).
It is entirely possible that these issues alone were causing your code to not work, and that your original saveImageToDisk() was working fine. Just be aware that these issues were definitely causing problems with how you were reading the file, regardless.

My bot pulls images from a local directory, how can I change this to a folder hosted on the web?

I have followed a tutorial to produce a Twitter bot using node.js, github and Heroku. Everything works great, the bot pulls a random image from a folder at timed intervals and tweets the image.
I'm trying to change the process so that instead of pulling images from a local folder (called 'images'), it pulls them from a web hosted folder. For example, rather than get the images from the local /images folder, I'd like it to pull the image from http://mysite/images. I have tried changing what I think are the relevant bits of code below, but am having no luck. Could anybody offer some advice please?
The whole code is below, but for reference, the bits I have tried changing are:
var image_path = path.join(__dirname, '/images/' +
random_from_array(images))
and
fs.readdir(__dirname + '/images', function(err, files) {
In both cases above I tried changing the /images folder to http://mysite/images but it doesn't work. I get an error stating that no such folder can be found. I have tried changing/deleting the __dirname part too but to no avail.
Any help appreciated!
Full code below:
const http = require('http');
const port=process.env.PORT || 3000
const server = http.createServer((req, res) => {
res.statusCode = 200;
res.setHeader('Content-Type', 'text/html');
res.end('<h1>Hello World</h1>');
});
server.listen(port,() => {
console.log(`Server running at port `+port);
});
var Twit = require('twit')
var fs = require('fs'),
path = require('path'),
Twit = require('twit'),
config = require(path.join(__dirname, 'config.js'));
var T = new Twit(config);
function random_from_array(images){
return images[Math.floor(Math.random() * images.length)];
}
function upload_random_image(images){
console.log('Opening an image...');
var image_path = path.join(__dirname, '/images/' +
random_from_array(images)),
b64content = fs.readFileSync(image_path, { encoding: 'base64' });
console.log('Uploading an image...');
T.post('media/upload', { media_data: b64content }, function (err, data,
response) {
if (err){
console.log('ERROR:');
console.log(err);
}
else{
console.log('Image uploaded!');
console.log('Now tweeting it...');
T.post('statuses/update', {
/* You can include text with your image as well. */
// status: 'New picture!',
/* Or you can pick random text from an array. */
status: random_from_array([
'New picture!',
'Check this out!'
]),
media_ids: new Array(data.media_id_string)
},
function(err, data, response) {
if (err){
console.log('ERROR:');
console.log(err);
}
else{
console.log('Posted an image!');
}
}
);
}
});
}
fs.readdir(__dirname + '/images', function(err, files) {
if (err){
console.log(err);
}
else{
var images = [];
files.forEach(function(f) {
images.push(f);
});
/*
You have two options here. Either you will keep your bot running, and
upload images using setInterval (see below; 10000 means '10 milliseconds',
or 10 seconds), --
*/
setInterval(function(){
upload_random_image(images);
}, 30000);
/*
Or you could use cron (code.tutsplus.com/tutorials/scheduling-tasks-
with-cron-jobs--net-8800), in which case you just need:
*/
// upload_random_image(images);
}
});

Well, my first answer to a question about building a twitter bot would probably be: "Don't do that!" (Because the world doesn't need more twitter bots.) But, putting that aside...
Your code is using the "fs" library, which is exactly what you needed for grabbing stuff from the local file system. That was fine. But now you want to grab stuff from web servers, which "fs" is not going to be able to do. Instead, you need a library that gives you the ability to make an HTTP or HTTPS request across the web and bring you back some data. There are different libraries that do this. Looks like you are already bringing in the "http" library, so I think you are on the right track there, but you seem to be using it to set up a server, and I don't think that's what you want. Rather, you need to use http as a client, and replace your fs.readFileSync() calls with the appropriate calls from the http library (if that's the one you choose to use) to pull in the data you want from whatever server has the data.
Hope that helps. And I hope your twitter bot is going to be a good little bot, not an evil one!

Getting variables From a config file to the browser

I have a rather complex setup which requires the web browser local storage has the computer's name populated in order for the application to work properly. In order to do this I read from a configuration file:
kiosk-name: Mort
I read the config file when I start my node.js web server:
var filesys = require('fs');
var os = require('os');
filesys.readFile(project_path + '/kiosk.cfg', 'utf8', function(err, data) {
var kioskname;
if (err) {
//console.log(err);
kioskname = os.hostname();
} else {
var configArray = data.split(':');
if('' != configArray[1]) {
kioskname = configArray[1];
} else {
kioskname = os.hostname();
}
}
});
All of this works as designed, using the computer's os.hostname() as a default when the config file is not populated.
The client side features a base page (index.html) which loads a default page (default.html) into an iframe. Based on a websocket messaging system the default page gets replaced by another page from a remote IP. In an older version of the system (prior to implementing a config file) we were able to set the local storage element with the following code:
var win = document.getElementsByTagName('iframe')[0].contentWindow;
win.postMessage(JSON.stringify({key: 'kiosk-name', data: kioskName}), "*");
We identify the iframe when the websocket message is received and then send a post message containing a JSON string to set the local storage element. In this case kioskName is a variable containing a hard-coded value.
The Problem
Now that we wish to read values from a config file we need a way to pass kioskname out to the client-side JavaScript so we can set the local storage element in the iframe.
I attempted putting the file reading function in an export wrapper:
(function(exports){
// file reading code here
return kioskname;
})(typeof exports === 'undefined' ? this['kioskname']={} : exports);
I got an error:
Uncaught ReferenceError: require is not defined
Placing a static value in the export function (with out the require's allows the export function to work properly, but doesn't allow me to read the config file which requires both the os and fs modules.
How do I get the value returned from the config file to a place where I can use it on the client-side to set a local storage element?

This is a creative solution which may not be suitable for every case as it involves utilizing a websocket between the Node.js web server and the client.
Websocket setup to send to client (assumes webserver at 'node_server':
var io = require('socket.io').listen(node_server); // 'attaches' socket.io to this web server
io.sockets.on('connection', function (socket) {
socket.emit('message', 'socket.io connected'); // output a connection message
// receive JSON message and send to the websocket
socket.on('message', function (data) {
var address = node_server.address();
var client = dgram.createSocket("udp4");
var message = new Buffer(data);
// out of the airlock!
client.send(message, 0, message.length, address.port, address.address, function(err, bytes) {
client.close();
});
});
});
Read the config file, then parse and send a message to the socket (done on the server-side):
filesys.readFile(project_path + '/kiosk.cfg', 'utf8', function(err, data) {
var kioskname;
if (err) {
//console.log(err);
kioskname = os.hostname();
} else {
var configArray = data.split(':');
if('' != configArray[1]) {
kioskname = configArray[1];
} else {
kioskname = os.hostname();
}
}
// create JSON string for transmission
KioskName = JSON.stringify({'config':{'kiosk-name': kioskname}});
var send_KioskName = setInterval(function(){ // could be a setTimeout for a one time send
io.sockets.emit('message', KioskName.toString()); // send config file data to browser via socket
}, 30000);
});
NOTE this can be expanded to send multiple pieces of data via JSON to the client should the need arise. A couple of small edits are all that is needed to setup a more detailed JSON object.
Receive the socket message on the client side (this code is loaded by the client), then parse. The resulting object is added to the namespace for this application, making the object available to multiple scripts when required.
CAUTION: You should only use this methodology for objects which do not interfere with objects you may create or destroy in your scripts along the way.
// make sure a string is JSON before using
function isJSON(str) {
try {
JSON.parse(str);
return true;
} catch (e) {
return false;
}
}
// set up object 'array's
var kioskname = {};
// connect a socket to listen for incoming messages from the Big Giant Head
var socket = io();
socket.on('message', function (data) {
if(isJSON(data)) {
// parse the json
var json = $.parseJSON(data);
// determine how to use this JSON object, multiple objects may be sent
if('config' == Object.keys(json)[0]) {
/*
* send config data where needed - future proofed, just add cases
* and namespaced objects where required
*/
kioskname['name'] = json.config['kiosk-name'];
}
}
});
// attach objects to namespace
window.KIOSK.kioskname = kioskname;
Now we can use the object to set local storage. In our case we post a message to the app's server and it responds with localStorage.setItem():
Post the message:
var currentFrame = document.getElementsByTagName('iframe')[0].contentWindow;
currentFrame.postMessage(JSON.stringify({key: 'user-name', data: KIOSK.kioskname.name}), "*");
By opening a socket and using the JSON string passed through the socket to populate a namespaced object we are able to use server-side information from a configuration file in our application's client.

Grab file extension from URL, in order to return correct Content-Type, using RegEx - Good practice?

I have 2 questions:
No1 :
I'm trying to come up with a RegEx that will return the file extension found at the end of the URL :
e.g.
Assuming that a file extension in a URL can only appear in this format :
.../.../..../..../filename.fileextension.
I'm getting the file extension as follows (please excuse my C++ inspired syntax, it's just much easier/cleaner for me to read code like that):
var fileExtension;
if (document.location.href.match(/.*\.(.*)$/) && document.location.href.match(/.*\.(.*)$/)[1])
{
fileExtension = document.location.href.match(/.*\.(.*)$/)[1];
}
What I hope the above RegEx means is match everything after that last dot in the URL.
Is that the case or is there a case where the above RegEx will fail? And if so, what would the correct RegEx be?
No2 :
The reason why I want to do the above is because I want to set up a my first Node.js server and I'm trying to make it as efficient as possible. Whenever a request is received for a specific file, I want to loop through an array of all possible file extensions, find the right one, and return the appropriate content-type in the head of my response. For this reason, I have created a JSON file (content-types.json) containing all possible file extensions (646 to be exact..), along with the relevant content types. It looks something like this :
{
"3dm" : "x-world/x-3dmf",
"3dmf" : "x-world/x-3dmf",
"a" : "application/octet-stream",
"aab" : "application/x-authorware-bin",
"aam" : "application/x-authorware-map",
......etc.
}
So using the above JSON file, I'm creating my server.js like this :
var http = require('http');
var fs = require('fs');
var contentTypes = JSON.parse(fs.readFileSync("content-types.json"));
var host = "127.0.01";
var port = 1825;
var server = http.createServer(function (request, response)
{
fs.readFile("." + request.url, function (error, data)
{
if (error)
{
response.writeHead(404, {"Content-type" : "text/plain"});
response.end("something went wrong");
}
else if (request.url.match(/.*\.(.*)$/) && request.url.match(/.*\.(.*)$/)[1])
{
var extension = request.url.match(/.*\.(.*)$/)[1];
var contentType;
var found = false;
// now I'll loop through the content-types object and look for a match
// if a match is found then
// found = true;
// contentType = contentTypes[extension];
if (!found)
{
response.writeHead(404, {"Content-type" : "text/plain"});
response.end("wrong file extension");
}
else
{
response.writeHead(200, {"Content-type" : contentType});
response.end(data);
}
}
else
{
response.writeHead(404, {"Content-type" : "text/plain"});
response.end("file extension not recognised");
}
});
});
server.listen(port, host, function ()
{
console.log("listening " + host + ":" + port);
});
Do you think the above is efficient?
I'm simply trying to avoid doing something like "if extension is js then use this content type, if it's css then this..etc."
However, it's true that's I'm reading/looping through a JSON (for every request) containing 646 extensions, so I don't know if that's wise either, from a performance perspective.
I also want to clarify that I don't want to use Express.js in this scenario (although an example would be appreciated). I'm trying to keep things as low-level as possible, because I'm a noob and I want to understand how stuff works first.
Also, as you can tell I haven't written the proper loop that will look for the right content-type because basically I don't know how to loop through an object. Should I use an array of objects instead? But in that case, I wouldn't be able to save that as an external file that I would simply read right? It has to be a JSON file I believe. So how do I loop through that contentTypes object in the above scenario in order to grab the appropriate contentType based on a given extension-key?
Thank you in advance for any help! :)

Just put the mimes into a "associative array" (actually just a object with named properties being the extensions) and then use the extension as the key/index
mimes = {
"js":"text/javascript",
"css":"text/css",
"json":"application/json"
};
var ext = request.url.split(".").pop();
//Check if the extension mime association exists, if not default to text/html
if( typeof(mimes[ext]) != "undefined" )
mime = mimes[ext];
else
mime = "text/html";
if you need to load the mimes from a file you could just save the mimes in a seperate json file and ajax it like:
mimes.json
{
"js":"text/javascript",
"css":"text/css",
"json":"application/json"
}

What if the URL is something like http://www.example.com/mywebsite/mypage? In that case, both the regex in the question and Patrick Evans' answer would give com/mywebsite/mypage as the file extension. Do you care about this case? If so, you'll probably ave to treat it separately, e.g. if the output of the first step contains a / then change it to html.

Nodejs: How return different content types with same response (text and image)?

I'm trying to learn nodejs and I thought the best way to do this would be to try doing some stuff without using express or any other non-core modules. I'm stuck on trying to get some text and an image to be delivered simultaneously. The code I'm trying is:
var http = require('http');
var fs = require('fs');
var server = http.createServer(function(request,response) {
fs.readFile('my_pic.jpg', function(error, file) {
response.writeHead(200, {'content-type':'text/html'});
response.write('<p>hi there!</p>');
response.writeHead(200, {'content-type':'image/jpg'});
response.write(file, 'image');
response.end();
});
});
var port = process.env.PORT || 8080;
server.listen(port, function() {
console.log('Listening on port ' + port);
});
So ideally what should be delivered is:
<html>
<body>
<p>hi there</p>
<img src='my_pic.jpg'/>
</body>
</html>
But instead, nothing is appearing.
I tried putting a response.end() after writing 'hi there', which displays the text, and after that I tried swapping the places of the text and picture (headers included), which displayed the picture, but I can't figure out how to get both to display simultaneously, like on a real web page.
Can you explain how to go about putting in different types of content onto the same webpage - do they need to be in different responses? Something I came across on another question:
nodejs - How to read and output jpg image?
Something like that looks like the solution, but I can't figure out how to apply this to my situation (I don't have a lot of experience in the server side of things.)
Many thanks
EDIT: got it working from user568109's reply:
var http = require('http');
var fs = require('fs');
var server = http.createServer(function(request,response) {
fs.readFile('my_pic.jpg', function(error, file) {
var imagedata = new Buffer(file).toString('base64');
response.writeHead(200, {'content-type':'text/html'});
response.write("hi there!<img src='data:my_pic.jpg;base64,"+imagedata+"'/>");
response.end();
});
});
var port = process.env.PORT || 8080;
server.listen(port, function() {
console.log('Listening on port' + port);
});
This seems like a lot of work in embedding images though - if I wanted to put more than one image in, I'd have to just keep nesting those filestream callbacks?
I also tried it this way but it doesn't work:
var http = require('http');
var fs = require('fs');
var server = http.createServer(function(request,response) {
response.writeHead(200, {'content-type':'text/html'});
response.write("hi there!<img src='my_pic.jpg'/>");
response.end();
});
var port = process.env.PORT || 8080;
server.listen(port, function() {
console.log('Listening on port' + port);
});
I wasn't sure if this was what people meant by the line
response.write('<img src="my_pic.jpg"/>')
, because the server doesn't seem to be making another request on my behalf to go fetch the image? It's just showing a broken icon image.

If you do response.write('<img src="my_pic.jpg"/>'); as mentioned above the image file would be sent only when browser sends GET for the image. It would become multi-part request.
Or you can do it like this. It is possible to send your image in binary form in HTML. Use :
<img src="data:image/gif;base64,imagedata">
where imagedata is a base64 encoding of gif image. So do this in node.js :
//Write text in response.
content = get-image-file-contents; //store image into content
imagedata = new Buffer(content).toString('base64'); //encode to base64
response.write('<img src="data:image/gif;base64,'+imagedata+'">');//send image
response.end();
Check here for correct image conversion NodeJS base64 image encoding/decoding not quite working
This sends one response which sends both text and image. Only one header is required for response response.writeHead(200, {'content-type':'text/html'});

You can only write one value to a given header, so the second header is overwriting the first. Two solutions are
write out a url to the image in the html - will be slightly slower for the user (needing an extra http request to fetch the image), but depending on the use case this is normally acceptable and very simple to implement
convert the image to a data-uri string and include this in the html as the source of the image. More complicated to implement than the first approach ( I don't know of any libraries for doing the conversion in node) and with negligible benefits.