I'm building a Meteor app that communicates with a desktop client via HTTP requests with https://github.com/crazytoad/meteor-collectionapi
The desktop client generates images at irregular time intervals, and I want the Meteor site to only display the most recently generated image (ideally in real time). My initial idea was to use a PUT request to a singleton collection with the base64 imagedata, but I don't know how to turn that data into an image in the web browser. Note: the images are all pretty small (much less than 1 MB) so using gridFS should be unnecessary.
I realize this idea could be completely wrong, so if I'm completely on the wrong track, please suggest a better course of action.
You'll need to write a middleware to serve your images with proper MIME type. Example:
WebApp.connectHandlers.stack.splice (0, 0, {
route: '/imageserver',
handle: function(req, res, next) {
// Assuming the path is /imageserver/:id, here you get the :id
var iid = req.url.split('/')[1];
var item = Images.findOne(iid);
if(!item) {
// Image not found
res.writeHead(404);
res.end('File not found');
return;
}
// Image found
res.writeHead(200, {
'Content-Type': item.type,
});
res.write(new Buffer(item.data, 'base64'));
res.end();
},
});
Related
I have followed a tutorial to produce a Twitter bot using node.js, github and Heroku. Everything works great, the bot pulls a random image from a folder at timed intervals and tweets the image.
I'm trying to change the process so that instead of pulling images from a local folder (called 'images'), it pulls them from a web hosted folder. For example, rather than get the images from the local /images folder, I'd like it to pull the image from http://mysite/images. I have tried changing what I think are the relevant bits of code below, but am having no luck. Could anybody offer some advice please?
The whole code is below, but for reference, the bits I have tried changing are:
var image_path = path.join(__dirname, '/images/' +
random_from_array(images))
and
fs.readdir(__dirname + '/images', function(err, files) {
In both cases above I tried changing the /images folder to http://mysite/images but it doesn't work. I get an error stating that no such folder can be found. I have tried changing/deleting the __dirname part too but to no avail.
Any help appreciated!
Full code below:
const http = require('http');
const port=process.env.PORT || 3000
const server = http.createServer((req, res) => {
res.statusCode = 200;
res.setHeader('Content-Type', 'text/html');
res.end('<h1>Hello World</h1>');
});
server.listen(port,() => {
console.log(`Server running at port `+port);
});
var Twit = require('twit')
var fs = require('fs'),
path = require('path'),
Twit = require('twit'),
config = require(path.join(__dirname, 'config.js'));
var T = new Twit(config);
function random_from_array(images){
return images[Math.floor(Math.random() * images.length)];
}
function upload_random_image(images){
console.log('Opening an image...');
var image_path = path.join(__dirname, '/images/' +
random_from_array(images)),
b64content = fs.readFileSync(image_path, { encoding: 'base64' });
console.log('Uploading an image...');
T.post('media/upload', { media_data: b64content }, function (err, data,
response) {
if (err){
console.log('ERROR:');
console.log(err);
}
else{
console.log('Image uploaded!');
console.log('Now tweeting it...');
T.post('statuses/update', {
/* You can include text with your image as well. */
// status: 'New picture!',
/* Or you can pick random text from an array. */
status: random_from_array([
'New picture!',
'Check this out!'
]),
media_ids: new Array(data.media_id_string)
},
function(err, data, response) {
if (err){
console.log('ERROR:');
console.log(err);
}
else{
console.log('Posted an image!');
}
}
);
}
});
}
fs.readdir(__dirname + '/images', function(err, files) {
if (err){
console.log(err);
}
else{
var images = [];
files.forEach(function(f) {
images.push(f);
});
/*
You have two options here. Either you will keep your bot running, and
upload images using setInterval (see below; 10000 means '10 milliseconds',
or 10 seconds), --
*/
setInterval(function(){
upload_random_image(images);
}, 30000);
/*
Or you could use cron (code.tutsplus.com/tutorials/scheduling-tasks-
with-cron-jobs--net-8800), in which case you just need:
*/
// upload_random_image(images);
}
});
Well, my first answer to a question about building a twitter bot would probably be: "Don't do that!" (Because the world doesn't need more twitter bots.) But, putting that aside...
Your code is using the "fs" library, which is exactly what you needed for grabbing stuff from the local file system. That was fine. But now you want to grab stuff from web servers, which "fs" is not going to be able to do. Instead, you need a library that gives you the ability to make an HTTP or HTTPS request across the web and bring you back some data. There are different libraries that do this. Looks like you are already bringing in the "http" library, so I think you are on the right track there, but you seem to be using it to set up a server, and I don't think that's what you want. Rather, you need to use http as a client, and replace your fs.readFileSync() calls with the appropriate calls from the http library (if that's the one you choose to use) to pull in the data you want from whatever server has the data.
Hope that helps. And I hope your twitter bot is going to be a good little bot, not an evil one!
I just started working with the Microsoft Azure Storage SDK for NodeJS (https://github.com/Azure/azure-storage-node) and already successfully uploaded my first pdf files to the cloud storage.
However, now I started looking at the documentation, in order to download my files as a node_buffer (so I dont have to use fs.createWriteStream), however the documentation is not giving any examples of how this works. The only thing they are writing is "There are also several ways to download files. For example, getFileToStream downloads the file to a stream:", but then they only show one example, which is using the fs.createWriteStream, which I dont want to use.
I was also not able to find anything on Google that really helped me, so I was wondering if anybody has experience with doing this and could share a code sample with me?
The getFileToStream function need a writable stream as param. If you want all the data wrote to a Buffer instead of a file, you just need to create a custom writable stream.
const { Writable } = require('stream');
let bufferArray = [];
const myWriteStream = new Writable({
write(chunk, encoding, callback) {
bufferArray.push(...chunk)
callback();
}
});
myWriteStream.on('finish', function () {
// all the data is stored inside this dataBuffer
let dataBuffer = Buffer.from(bufferArray);
})
then pass myWriteStream to getFileToStream function
fileService.getFileToStream('taskshare', 'taskdirectory', 'taskfile', myWriteStream, function(error, result, response) {
if (!error) {
// file retrieved
}
});
I would like to resize and compress images after users upload them to my site. It seems like there are quite a few options for resizing images with node (e.g. https://github.com/lovell/sharp), but I would like to compress the images as well in order to save disk space and allow for faster serving times. Is there a library or something else that makes this possible?
Here is a simplified version of my current (functioning) route as it stands today:
var router = require('express').Router();
var bucket = require('../modules/google-storage-bucket');
var Multer = require('multer');
var multer = Multer({
storage: Multer.memoryStorage(),
limits: {
fileSize: 5 * 1024 * 1024 // no larger than 5mb
}
});
// Process the file upload and upload to Google Cloud Storage.
router.post('/', multer.single('file'), (req, res) => {
// rejecting if no file is uploaded
if (!req.file) {
res.status(400).send('No file uploaded.');
return;
}
// Create a new blob in the bucket and upload the file data.
var blob = bucket.file('fileName');
var blobStream = blob.createWriteStream();
blobStream.on('error', (err) => {
console.log('error', err);
res.send(500);
});
blobStream.on('finish', () => {
console.log('everything worked!')
});
blobStream.end(req.file.buffer);
});
module.exports = router;
Sharp has API for adjusting image compression. For example:
https://sharp.pixelplumbing.com/api-output#jpeg
Lists the options for JPEG write. You can adjust a range of knobs on the JPEG compressor to tune it for the performance/compression/quality tradeoff you want.
You can improve compression by up to about 30% in the best case, which honestly feels a bit marginal to me. It's unlikely to make a significant difference to your page size or your load times.
In terms of storage savings, the best solution is not to store images at all --- sharp is fast enough that you can simply generate all of your images on demand (most dynamic image resizing web proxies use sharp as the engine). That's a change that really will save you money and make your pages look better.
From my experience I would recommend imagemin. I've used it as a Gulp plugin, so you could 100% use in your project. But also you have to download the third-party modules: imagemin-pngquant and imagemin-jpegtran.
Hope you appreciate it :)
I am trying to take audio recorded by one client and send it to other connected clients in realtime. The objective being a sort of "broadcast". I have read many explanations to help guide me, with no luck.
Currently I have the audio being written to file like so:
var fileWriter = new wav.FileWriter(outFile, {
channels: 1,
sampleRate: 44100,
bitDepth: 16
});
client.on('stream', function(stream, meta) {
stream.pipe(fileWriter);
stream.on('end', function() {
fileWriter.end();
console.log('wrote to file ' + outFile);
});
});
});
As you can see, I'm currently using Binaryjs to send the audio data to the server, at which point I pipe the stream to the FileWriter
I then tried to read the file and pipe it to the response
app.get('/audio', function(req, res) {
fs.createReadStream(__dirname + '/demo.wav').pipe(res);
})
As I'm sure you've already noticed, this doesn't work. I thought that (maybe) while the file is being constructed, it would playback all updated content added to the file as well. This didn't happen, it played up to the point a client requested the file and then ended.
I am unsure of how to pass the stream data in real time to the clients requesting it. As a result of being completely new to nodejs I am not sure of the methods and terms used for this procedure and have been unable to find any direct working examples.
I'm looking to create static text files based upon the content of a supplied object, which can then be downloaded by the user. Here's what I was planning on doing:
When the user hits 'export' the application calls a Meteor.method() which, in turn, parses and writes the file to the public directory using typical Node methods.
Once the file is created, in the callback from Meteor.method() I provide a link to the generated file. For example, 'public/userId/file.txt'. The user can then choose to download the file at that link.
I then use Meteor's Connect modele (which it uses internally) to route any requests to the above URL to the file itself. I could do some permissions checking based on the userId and the logged in state of the user.
The problem: When static files are generated in public, the web page automatically reloads each time. I thought that it might make more sense to use something like Express to generate a REST endpoint, which could deal with creating the files. But then I'm not sure how to deal with permissions if I don't have access to the Meteor session data.
Any ideas on the best strategy here?
In version 0.6.6.3 0.7.x - 1.3.x you can do the following:
To write
var fs = Npm.require('fs');
var filePath = process.env.PWD + '/.uploads_dir_on_server/' + fileName;
fs.writeFileSync(filePath, data, 'binary');
To serve
In vanilla meteor app
var fs = Npm.require('fs');
WebApp.connectHandlers.use(function(req, res, next) {
var re = /^\/uploads_url_prefix\/(.*)$/.exec(req.url);
if (re !== null) { // Only handle URLs that start with /uploads_url_prefix/*
var filePath = process.env.PWD + '/.uploads_dir_on_server/' + re[1];
var data = fs.readFileSync(filePath);
res.writeHead(200, {
'Content-Type': 'image'
});
res.write(data);
res.end();
} else { // Other urls will have default behaviors
next();
}
});
When using iron:router
This should be a server side route (ex: defined in a file in /server/ folder)
Edit (2016-May-9)
var fs = Npm.require('fs');
Router.route('uploads', {
name: 'uploads',
path: /^\/uploads_url_prefix\/(.*)$/,
where: 'server',
action: function() {
var filePath = process.env.PWD + '/.uploads_dir_on_server/' + this.params[0];
var data = fs.readFileSync(filePath);
this.response.writeHead(200, {
'Content-Type': 'image'
});
this.response.write(data);
this.response.end();
}
});
Outdated format:
Router.map(function() {
this.route('serverFile', {
...// same as object above
}
});
Notes
process.env.PWD will give you the project root
if you plan to put files inside your project
don't use the public or private meteor folders
use dot folders (eg. hidden folders ex: .uploads)
Not respecting these two will cause local meteor to restart on every upload, unless you run your meteor app with: meteor run --production
I've used this approach for a simple image upload & serve (based on dario's version)
Should you wish for more complex file management please consider CollectionFS
The symlink hack will no longer work in Meteor (from 0.6.5). Instead I suggest creating a package with similar code to the following:
packge.js
Package.describe({
summary: "Application file server."
});
Npm.depends({
connect: "2.7.10"
});
Package.on_use(function(api) {
api.use(['webapp', 'routepolicy'], 'server');
api.add_files([
'app-file-server.js',
], 'server');
});
app-file-server.js
var connect = Npm.require('connect');
RoutePolicy.declare('/my-uploaded-content', 'network');
// Listen to incoming http requests
WebApp.connectHandlers
.use('/my-uploaded-content', connect.static(process.env['APP_DYN_CONTENT_DIR']));
I was stuck at the exact same problem, where i need the users to upload files in contrast to your server generated files. I solved it sort of by creating an "uploads" folder as sibling to the "client public server" on the same folder level. and then i created a simbolic link to the '.meteor/local/build/static' folder like
ln -s ../../../../uploads .meteor/local/build/static/
but with nodejs filesystem api at server start time
Meteor.startup(function () {
var fs = Npm.require('fs');
fs.symlinkSync('../../../../uploads', '.meteor/local/build/static/uploads');
};
in your case you may have a folder like "generatedFiles" instead of my "uploads" folder
you need to do this every time the server starts up cuz these folders are generated every time the server starts up e.g. a file changes in your implementation.
Another option is to use a server side route to generate the content and send it to the user's browser for download. For example, the following will look up a user by ID and return it as JSON. The end user is prompted to save the response to a file with the name specified in the Content-Disposition header. Other headers, such as Expires, could be added to the response as well. If the user does not exist, a 404 is returned.
Router.route("userJson", {
where: "server",
path: "/user-json/:userId",
action: function() {
var user = Meteor.users.findOne({ _id: this.params.userId });
if (!user) {
this.response.writeHead(404);
this.response.end("User not found");
return;
}
this.response.writeHead(200, {
"Content-Type": "application/json",
"Content-Disposition": "attachment; filename=user-" + user._id + ".json"
});
this.response.end(JSON.stringify(user));
}
});
This method has one big downside, however. Server side routes do not provide an easy way to get the currently logged in user. See this issue on GitHub.