I hava a node.js app on heroku thath checks for emails and then if the email match certian cliteria it would reply.
Because heroku restarts the dyno every so often I made a file that saves the ids of the emails (a small array) I've already cheaked (so it doesn't reply twice to the same email), but silly me, heroku restarts that file too, so no changes I made will be save.
Is there a way to save the file changes the app makes to it?
If you know of a better way of to do what I want to?
Heroku enforces this behavior (the local file deletion stuff) because it is a best practice. Writing files locally doesn't scale well, and can lead to some odd edge-case behaviors when you have multiple processes on the same VM all performing file I/O operations.
What you should use instead is either a database, a cache (like Redis), or even just write your file directly to a file storage service like Amazon S3.
I realize it sounds annoying that you have to do these extra things even for a simple use case like what you're doing here -- but Heroku's platform is geared around enforcing best practices to help people build scalable, reliable software.
If you're looking for a way to do stuff like this without the extra hassle, you might want to consider just purchasing a small VPS server from another company where you can have direct control over processes, disk, etc.
Related
Background:
I am building a node.js-based Web app that needs to make use of various fonts. But it only needs to do so in the backend since the results will be delivered as an image. Consequently, the client/browser does not need access to the fonts at all in my case.
Question:
I will try to formulate the question as little subjective as possible:
What are the typical options to provide a node.js backend with a large collection of fonts?
The options I came up with so far are:
Does one install these hundreds or thousands of fonts in the operating system of the (in my case: Ubuntu) server?
Does one somehow serve the fonts from a cloud storage such as S3 or (online) database such as a Mongo DB server?
Does one use a local file system to store the fonts and retrieve them?
...other options
I am currently leaning towards Option 1 because this is the way a layman like me does it on a local machine.
Without starting a discussion here, where could I find resources discussing the (dis-)advantages of the different options?
EDIT:
Thank you for all the responses.
Thanks to these, I noticed that I need to clarify something. I need the fonts to be used in SVG processing libraries such as p5.js, paper.js, raphael.js. So I need to make the fonts available to these libraries that are run on node.js.
The key to your question is
hundreds or thousands of fonts
Until I took that in there is no real difference between your methods. But if that number is correct (kind of mind-boggling though) I would:
not install them in the OS. What happens if you move servers without an image? Or move OS?
Local File system would be a sane way of doing it, though you would need to keep track manually of all the file names and paths for your code.
MongoDB - store file names+paths in the collection..and store the actual fonts in your system.
In advent of moving servers you would have to pick up the directory where all the actual files are stored and the DB where you hold the file name+paths.
If you want you can place it all in a MongoDB but then that file would also be huge, I assume - that is up to you.
Choice #3 is probably what I would do in such a case.
If you have a decent enough server setup (e.g. a VPS or some other VM solution where you control what's installed) then another option you might want to consider is to do this job "out of node". For instance, in one of my projects where I need to build 175+ as-perfect-as-can-be maths statements, I offload that work to XeLaTeX instead:
I run a node script that takes the input text and builds a small but complete .tex file
I then tell node to call "xelatex theFileIJustMade.tex" which yields a pdf
I then tell node to call "pdfcrop" on that pdf, to remove the margins
I then tell node to call "pdf2svg", which is a free and amazingly effective utility
Then as a final step mostly to conserve space and bandwidth, I use "svgo" which is a nodejs based svg optimizer that can run either as normal script code, or as CLI utility.
(some more details on that here, with concrete code here)
Of course, depending on how responsive a system you need, you can do entirely without steps 3 and 5. There is a limit to how fast we can run, but as a server-side task there should never be the expectation of real-time responsiveness.
This is a good example of remembering that your server runs inside a larger operating system that might also offer tools that can do the job. While you're using Node, and the obvious choice is a Node solution, Node is also a general purpose programming language and can call anything else through spawn and exec, much like python, php, java, C#, etc. As such, it's sometimes worth thinking about whether there might be another tool that is even better suited for your needs, especially when you're thinking about doing a highly specialized job like typesetting a string to SVG.
In this case, LaTeX was specifically created to typeset text from the command line, and XeLaTeX was created to do that with full Unicode awareness and clean, easy access to fonts both from file and from the system, with full OpenType feature control, so would certainly qualify as just as worthwhile a candidate as any node-specific solution might be.
As for the tools used: XeLaTeX and pdfcrop come with TeX Live (installed using whatever package manager your OS uses, or through MiKTeX on Windows, but I suspect your server doesn't run on windows) pdf2svg is freely available on github, and svgo is available from npm)
I have a simple audio sequencer application using javascript and HTML5. The goal of the application is to let users make a beat and save it so other people can open it up, add on to it, then re save. The issue I'm having is the save/open feature. How Do I add this? Or at least where can I start looking fro this solution?
If you are asking for an only-JS solution to this:
Opening Files
You can use the FileAPI to open the files from your users local machine, as long as its your users who are selecting the files manually(you are not digging into the file system yourself).
Here is nice demo for this functionality: http://html5demos.com/file-api
Saving Files
However, saving files was a W3C recommendation that got ditched so your only choice here is to allow your users to download the file. See this relevant question I made some time ago
If I were doing this for a class or prototype I might use PUT or POST requests to save the beat sequence in json form to a web server. When it's time to save your work, take your beat sequence data, convert from a js (I presume) object to a json string and PUT it to an appropriate url on your server. Give it a name and use the name in the url path, like "myserver.com/beats/jpbeat1.json".
When you want to retrieve that beat, issue an HTTP get to the url you saved. You can, of course also maintain and rewrite a directory page in json or html, or you can just navigate the directory offered by your web server. Most developers will think this crude, I imagine, but in a race for time, crude can be good if it meets the requirements.
Choose a web server, the easiest one for you. I might use Amazon S3. Costs money, but available to the internet, has authentication, security features, and supports http GET, PUT of static files. There are many other site hosting services.
For a local install, I might use Node.js. Also very easy to set up to serve static content. For example, the Express framework has static content service and easy examples how to set it up if you are comfortable with Javascript. With Node, you can make the site a bit fancier, by writing server side code to maintain an index, for example. Any real world implementation would require at least some server side code.
There are many other choices. Most developers would want a database, but you don't have to have one if simple files are ok for your class project. It all depends on your requirements.
I hope those ideas are helpful.
I have designed a meteor.js application and it works great on localhost and even when deployed to the internet. Now I want create a sign-up site that will spin up new instances of the application for each client who signs up on the back-end. Assuming a meteor.js application and python or javascript for the sign-up site, what high level steps need to be taken to implement this?
I am looking for a more correct and complete answer that takes the form of my poorly imagined version of this:
Use something like node or python to call a shell script that may or may not run as sudo
That script might create a new folder to hold instance specific stuff (like client files, config, and or that instances database).
The script or python code would deploy an instance of the application to that folder and on a specific port
Python might add configuration information to a tool like Pound to forward a subdomain to a port
Other things....!?
I don't really understand the high level steps that need to be taken here so if someone could provide those steps and maybe even some useful tools or tutorials for doing so I'd be extremely grateful.
I have a similar situation to you but ended up solving it in a completely different way. It is now available as a Meteor smart package:
https://github.com/mizzao/meteor-partitioner
The problem we share is that we wanted to write a meteor app as if only one client (or group of clients, in my case) exists, but that it needs to handle multiple sets of clients without them knowing about each other. I am doing the following:
Assume the Meteor app is programmed for just a single instance
Using a smart package, hook the collections on server (and possibly client) so that all operations are 'scoped' only to the instance of the user that is calling them. One way to do this is to automatically attach an 'instance' or 'group' field to each document that is being added.
Doing this correctly requires a lot of knowledge about the internals of Meteor, which I've been learning. However, this approach is a lot cleaner and less resource-intensive than trying to deploy multiple meteor apps at once. It means that you can still code the app as if only one client exists, instead of explicitly doing so for multiple clients. Additionally, it allows you to share resources between the instances that can be shared (i.e. static assets, shared state, etc.)
For more details and discussions, see:
https://groups.google.com/forum/#!topic/meteor-talk/8u2LVk8si_s
https://github.com/matb33/meteor-collection-hooks (the collection-hooks package; read issues for additional discussions)
Let me remark first that I think spinning up multiple instances of the same app is a bad design choice. If it is a stop gap measure, here's what I would suggest:
Create an archive that can be readily deployed. (Bundle the app, reinstall fibers if necessary, rezip). Deploy (unzip) the archive to a new folder when a new instance is created using a script.
Create a template of an init script and use forever or daemonize or jesus etc to start the site on reboot and keep the sites up during normal operation. See Meteor deploying to a VM by installing meteor or How does one start a node.js server as a daemon process? for examples. when a new instance is deployed populate the template with new values (i.e. port number, database name, folder). Copy the filled out template to init.d and link to the runlevel. Alternatively, create one script in init.d that executes other scripts to bring up the site.
Each instance should be listening to its own port, so you'll need a reverse proxy. AFAIK, Apache and Nginx require restarts when you change the configuration, so you'll probably want to look at Hipache https://github.com/dotcloud/hipache. Hipache uses redis to store the configuration information. Adding the new instance requires to add a key to redis. There is an experimental port of Hipache that brings the functionality to Nginx https://github.com/samalba/hipache-nginx
What about DNS updates? Once you create a new instance, do you need to add a new record to your DNS configuration?
I don't really have an answer to your question... but I just want to remind you of another potential problem that you may run into cuz I see you mentioned python, in other words, you may be running another web app on Apache/Nginx, etc... the problem is Meteor is not very friendly when it comes to co-exist with another http server, the project I'm working on was troubled by this issue and we had to move it to a stand alone server after days of hassle with guys from Meteor... I did not work on the issue, so I'm not able to give you more details, but I just looked up online and found something similar: https://serverfault.com/questions/424693/serving-meteor-on-main-domain-and-apache-on-subdomain-independently.
Just something to keep in mind...
I have a simple offline html5/javascript single-html-file web application that I store in my dropbox. It's a sort of time tracking tool I wrote, and it saves the application data to local storage. Since its for my own use, I like the convenience of an offline app.
But I have several computers, and I've been trying to come up with any sort of hacky way to synchronize this app's data (which is currently using local storage) between my various machines.
It seems that chrome allows synchronization of data, but only for chrome extensions. I also thought I could perhaps have the web page automatically save/load its data from a file in a dropbox folder, but there doesn't appear to be a way to automatically sync with a specific file without user prompting.
I suppose the "obvious" solution is to put the page on a server and store the data in a database. But suppose I don't want a solution which requires me to maintain apps on a server - is there another way, however hacky, to cobble together synchronization?
I even looked for a while to see if there was a vendor offering a web database service - where I could, say, post/get a blob of json on demand, and then somehow have my offline app sync with this service, but the same-origin policy seems to invalidate that plan (and besides I couldn't find such a service).
Is there a tricky/sneaky solution to this problem using chrome, or google drive, or dropbox, or some other tool I'm not aware of? Or am I stuck setting up my own server?
I have been working on a Project that basically gives you versioned localStorage with support for conflict resolution if the same resource ends up being edited by two different clients. At this point there are no drivers for server or client (they are async in-memory at the moment for testing purposes) but there is a lot of code and abstraction to make writing your own drivers really easy... I was even thinking of doing a dropbox/google docs driver myself, except I want DynamoDB/MongoDB and Lawnchair done first.
The code is not dependent on jQuery or any other libraries and there's a pretty full features (though ugly) demo for it as are well.
Anyway the URL is https://github.com/forbesmyester/SyncIt
Apparently, I have exactly the same issue and invetigated it thoroghly. The best choice would be remoteStorage, if you could manage to make it work. It allows to use 3rd party server for data storage or run your own instance.
In my .NET web applications I usually have a Scripts folder that contains all of my JavaScript files - jQuery mostly these days, with the occasional JavaScript library of some sort or another.
I'm running vulnerability scans against one of my websites through a scanner called Nexpose, and it informed me that the Scripts folder is open to the world - meaning unauthenticated users can download the JavaScript files contained in the folder and that this is a critical vulnerability. According to Nexpose, the Scripts folder should be restricted to only allow authenticated users to access it. Which leads me to my first question.
How do I restrict the Scripts folder to only authenticated users? I tried placing a web.config file into the Scripts folder and denying access to all unauthenticated users that way, but it didn't work. I was able to determine this myself but going to my website's login page, but not logging in, and then typing https://mywebsite/scripts/menubar.js and sure enough it allowed me to download the menubar.js file.
Second question - Why is this considered a vulnerability? I've tried to reason my way through the possibilities here, but I've failed to come up with much at all. Is it a vulnerability simply because Joe the l33t h4x0r could figure out the various libraries that I'm using and then maybe use known exploits against them?
Update
Overwhelmingly the answer seems to be that in no way should a vulnerability exist just because a .js file can be opened and read on the client's browser. The only vulnerability that might exist would be if the developer were using the .js file in an insecure fashion of some sort (which I'm not).
Logically, you wouldn't want to actually disallow access to the actual files because then you couldn't use them in your webpage. The webserver makes no distinction between a browser requesting a file as part of the process of rendering a webpage versus someone just manually downloading the file.
As a result, the answer to your first question is: you can't and wouldn't want to. If you don't want users to access take it out of the web folder. If it's required to render your site, then you want anyone to have access to it so your site can render properly.
As to why it's considered a vulnerabiliy, who's saying it is? I can go pull any JavaScript Facebook uses right now. Or, more to the point, I could go to Bank of America or Chase's website and start looking through their JavaScript. If I had an account, I could even take a look at the JavaScript used once the user is logged in.
The only thing that you might need to worry about is the same thing you always need to worry about: exposing details that shouldn't be exposed. I'm not sure why you would, but it obviously wouldn't be a good idea to put your database password in a JavaScript file, for example. Other than things like that, there's nothing to worry about.
In most cases it's not a vulnerability. Consider all of the massive public sites with anonymous traffic and/or where it's very easy to become an authenticated user (Google, eBay, Amazon, etc.) These sites also have some of the most elaborate scripts.
The more subtle thing to watch out for are other files which you DO want protected. For example, if users must login to your site and purchase a document, video, image, etc. before viewing it, it certainly should not be in a publicly accessible folder.
Yes. You should keep most of your processing to be done server-side, as most (if not all) client-side scripts can be edited, and such. Most sites use Javascript, so simply using it isn't dangerous, you just have to be careful about what you do with it.
Also, to answer your first question, don't protect them if unauthenticated users need them too.
Sounds like some security suite has an itchy trigger finger. The only two problems I could see is that you could end up loaning your server out as a CDN if someone chooses to point to your jQuery or your -insert library name here- OR (now this is a real security risk) if you are also serving any kind of dynamic .js files out of there that could pose a potential threat. The only other thing I can think of is if you have your "custom" app js in the mix with all the libraries someone could potentially discover your endpoints (web services and such) and try and see if they're secure... but that's it! nothing more... (unless you did something really dumb like hard code a password or something in there... lol)
So the attack isn't that people can edit the script the attack is getting the web server to arbitrarily write to a directory. What you need to do is make sure they are read-only files. chmod 400 or windows read. In terms of Defense in Depth (DiD) you want to make sure the web server is a non-privileged user that cannot log into the system. Further what needs to occur is that you are doing all data sanitization on the server, irrespective of what you are doing on client side, because you do not control the client side. Typically this involves making sure that you cleanse all data coming from the web as well as the database before it gets served. One of my favorite things to do is insert arbitrary javascript into the database and watch it do things in the UI because the development team assumed everything would be fine since they already cleaned it once.
I can provide more details around securing the system if it is warranted.