I have made an application that is receiving location data from a few web clients on regular intervals. I made a quick implementation using couchdb, but as couchdb creates a new revision for each update and the data is quite frequently updated, it consumed a lot of disk space whereas the historic data was of little significance. I looked into MongoDB instead, but as I was thinking how I could make the MongoDB implementation, I had another idea:
The global object is in the process scope, so it can be used to share data between sessions. Persistence beyond session is not required, so I dropped the database completely and stored all the data in the global object (and persisted some data for user convenience in the HTML5 localStorage using javascript). The complexity of the backend was greatly reduced, and the solution felt somewhat elegant, but I still feel like I need to take a shower...
So to my question: Are there any obvious pitfalls with this solution that I haven't thought about?
Congrats you have rediscovered memcache. (I did it twice)
If you need to store this data then you actually should to save it to db, because server app restart will erase all data from RAM. So actually is better to use memcache and asynchronous writing to db.
Related
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
Can't pass mysqli connection in session in php
Many of us have written PHP applications that require databases; mostly MySQL, but I have often used very small MS Access databases for people less technically capable so they can download an tweak them/save backups/etc. on their own (whether this is correct or not, I have no idea).
What I notice, is a lot of time is spent connecting and running some of the same queries. Because of this I had an interesting thought: Storing the connection and possible result sets that are mostly static in a $_SESSION variable to reduce the burden as the user navigates the site.
Obviously doing so requires a lot of consideration. Things like closing the connection when the session is destroyed is just the start.
My question boils down to: Is this really something possible? And if so, what things should I be aware of (besides session fixation, as it is its own problem that applies to all sessions)?
You can't store database connections or result sets in the session, since those are resources, and:
Some types of data can not be serialized thus stored in sessions. It includes resource variables or objects with circular references (i.e. objects which passes a reference to itself to another object).
http://php.net/manual/en/intro.session.php
You can extract a result set into a normal array and store that in the session like any other variable. That would be a fairly typical use case for sessions anyway. Just be careful not to store too much data in the session, as that can become more taxing than fetching it from the database.
Even if you could do this (resource vs. data), this is a bad idea. You'll wind up with lots of concurrent open connections, which will blow your max connections very quickly... especially if its lifecycle is expanded beyond sub 100ms (depending on your queries) to 20 minutes or more. With open connections, something like MySQL also won't be able to reset its memory allocations properly, and the whole system sort of goes to hell. In short, this is not what DBs are for unless the only consumer of your code will be a single user.
As an alternative, I'd highly recommend caching technologies which are designed specifically to reduce database load and obviate connection times. Using something like, at its simplest, memcached will dramatically improve performance all the way around, and you'll be able to specify exactly how many system resources go into the cache -- while letting the database do its job of getting data when it needs to.
You can check permanent connections for the connection part.
http://php.net/manual/en/function.mysql-pconnect.php
You should those in some config file for better use. Sessions are for specific sessions and not for global.
I just started programming with node.js and before this I was using PHP and I used to store data on sql database. However, I am confused now. Should I use sql to save data or Json. I would need to save data and show them on the web page later on Or feed those data to draw graphs and charts.
When storing data in a Node.js application (or with any application really), you have three main options...
Memory
You can simply store the data as a variable in the application, this
will most likely be fastest to access.
You may start to encounter problems if you are storing lots of data
as you will use up a large proportion of your machine's memory.
All data will be lost when the script restarts, so the data should be
able to be generated again on startup.
If you use a process manager like PM2 (which you'll want to do if
hosting on a multi-core machine, see https://www.digitalocean.com/community/tutorials/how-to-serve-flask-applications-with-uwsgi-and-nginx-on-ubuntu-16-04), you won't be able to store data across multiple requests as processes will not share a common memory.
I/O
You could also encode data as JSON and store it in a file on your machine.
I can't think of a case where this would be better than using a DB, I/O is usually just used for static files like images.
The obvious drawback of this is that it will probably be slower than database calls but you'd also have to watch out for I/O concurrency issues if you're using miultiple processes.
However, if you really want to store JSON, consider a non-relational DB like Mongo DB instead.
Database
This would be my preferred method for almost every case (apart from static files or very short-lived data, as I mentioned above).
A DB also has helpful ways of storing and finding data - you'll need to choose between a relational or non-relational DB, take a look at https://www.mongodb.com/scale/relational-vs-non-relational-database for some of the differences.
Again, the main thing to be wary of is concurrency when using multiple processes (usually only a problem when updating or inserting records).
I hope that covers your question - I may have missed something but I'm sure others will let me know if I have. Node.js is a great choice for web servers as it is asynchronous so can naturally handle multiple connections at once.
I don't know much about databases, I've been asking a few questions about them lately to get a better understanding but I'm still a bit confused about what does and doesn't need one.
I'm making a simple application using HTML/CSS/JavaScript, it has a few quizzes and "tutorials" targeted towards children. I don't want the next tutorial/quiz to be unlocked until the previous one is completed.
So for that would I need a database so that it "saves" when one is completed? I don't need to save scores or anything like that, they just get to move on once they get a passing score.
Any other requirements such as saving to a profile or needing to persist between sessions (e.g. changing of device)?
Browsers have localStorage APIs now which allow you to save a lot of the data (and keep it for a set duration of time). There are also good'ol'fashioned cookies which allow you save pieces of information as well.
Keep in mind that both of the above mandate the user use the same browser and allow these mechanisms. Obviously using "private"/"incognito" browsing would also affect saving status.
It's up to what you feel the requirements are.
EDIT Just saw your mention of a mobile app. If you're planning on allowing the experience to transcend devices, you'll need a database. otherwise, you'll be relying heavily on if they use cross-device sync (like Chrome and Firefox do with bookmarks, passwords, etc.)
If you don't mind that people can do a "view source" on the webpage or use every browsers' developer tools to find out the answers or move on to the next tutorial or quiz, then you can use cookies to store the user's status. Or you can use the preferable Web Storage API.
You might want to look at Firebase. Using just simple JavaScript on the web browser, you can have users with logins (or just allow them to login via Facebook or other services) very easily. And then you can store and retrieve data very easily as well, like quizzes, tutorials and results. This way nobody can see the answers even if they're adept at analyzing the webpage.
When you don't use database, before any check, you have to load all data in your static page.
So My sloution: store students situation in a cookie. On each page check cookie status and then use Jquery remove() to remove (Client-side) those parts of page that he/she can not access.
EDIT
This wont work when JavaScript is disabled.
There seems to be a lot of ideas but no clarifying on the database subject.
TL;DR is: No.
Now for the specifics. A database is nothing more than a way to store information. While traditional "SQL" databases (it is pronounced "Sequel" as in "My Sequel" for MySQL) have concepts of tables, where you define columns with items to store and saves each row with its value, much like an Excel file, some databases like Redis store key-value pairs and others lide MongoDB store JavaScript Objects.
You can store information in the source code (As Variables possibly) or in a file. A database is a way to organize that information.
With that said, in your case, you probably need a backend or an API. An API is basically a means of communication with a server through AJAX (JavaScript in the browser asks for stuff). That would be your way to retrieve information from the server as needed, so that users wouldn't see the answers before they answer.
With that out of the way, there are some options. FireBase (As noted on other answer) and AppBase are easy ways to integrate this concept with little effort. But they tie you and your information to their system, and they are mostly targeting more resource intensive apps.
Since you are using JS and seem to be enjoying your learning experience, I would suggest you consider suing NodeJS and defining the data as either a JSON file or a variable in JS. You keep working on your problem but add options and get to learn some stuff.
If you decide to integrate a database and possibly do some neat stuff, you have most of the groundwork done already.
If NodeJS picks your interest, Mean.IO and KrakenJS are, in my opinion, the best places to start, though they may both seem overkill in your specific case.
Do consider though: A database is just a small possible piece in a puzzle, and it's mostly a horrible way to name some of the software that tries to organize your information. Consider first if you need to organize information, and what and how do you need to organize, then start thinking if databases are the best way to organize it.
I am planning to use Qualtrics REST API, in order to get the data collected from a survey. Can i still retain Meteor's reactivity directly thru the rest api or should I save the data from the rest API into MongoDB to enable for real time updates within the app?
Any advice and further reading will be great.
This will sound like a noob question probably but I am just starting off with Meteor and JS as server side code and never used a web api before.
It entirely depends on what you do with the data it returns. Assuming you're either polling periodically or the API has some kind of push service (I've never heard of it before, so I have no idea), you would need to store the data it returns in a reactive data source: probably a Collection or Session variable, depending on how much persistence is required. Any Meteor templates that access these structures have reactivity built in, as documented here.
Obviously, you will probably need to be polling the API at an appropriately regular interval for this set up to work though. Take a look at Meteor.setInterval, or the meteor-cron package, which is probably preferable.
I would like to see a decent example of a mobile web app using the Sencha framework with a client side DB accessed with SQLite. I'm currently digesting JqTouch and kinda get the binding method used there from reading Jonathon Stark's "iPhone apps" book, but cant find any examples of accessing Senchas features ie listed elements with SQLite. The DB will be small; 30 records, with about 5 fields, mostly numeric, a few of them calculated. All the math is done in javascript and I have that part working (in dash code). I need to add, delete, and edit the records.
Any pointers or examples would be very much appreciated. I'm an old dog trying to learn new tricks. Thanks
Sencha is client-side Javascript, so your application actually runs on top of Safari. That means you can forget about accessing (or installing) your own SQLite database from within the browser sandbox.
Having said that, you want to learn some new tricks, so why dont you read up on localStorage and DOM Storage. Basically the HTML5 specification allows for offline database storage based on SQLite (imagine relational database cookies). There is 1 per domain and they can be up to 5MB in size. I believe the iPhone supports this as well.
Here are some links: Introduction some API Information and a nice little blog entry by a chap called Ben Lister
Your client side code (i.e. Sencha/Javascript) would not access the SQLLite database. It will either need to read JSON or XML from the server. You'll need server side code to read the data from the database and format it in a way that your Sencha data readers will understand.
What are you using server side? If it's PHP you should look into MDB2
I had very good experience integrating Lawnchair library with Sencha Touch. Take a look at their guide, it's very easy.
Looks like there is a SQLite proxy available for sencha 2 now. http://market.sencha.com/addon/sqliteproxy-
Check out this thread on the Sencha Forums - it's a user created proxy for SQLite which I've successfully used to put data into a SQLite DB. The proxy comes with an example, but I might try and make a slightly more complicated one at some point.
Sencha's local storage doesn't take advantage of SQLite via the JavaScript API in the browser, but does use local key:value storage and has it's own way of referencing data to make it pseudo relational. This is still part of the WebDB spec, which is probably still SQLite under the hood if I had to guess. It's more persistent than a cookie or session, regardless.
You can also receive XML/JSON from a server over JSONP or Ajax if you're on the same domain, create a model to handle that data as well and bind it to a local store so that your data is available offline.