I recently followed a tutorial to create a node.js server connecting to orchestrate.io database. The problem is I now want to point the server at a mongodb hosted on mongolab - currently I am declaring a variable:
var db = require('orchestrate')(APIKEY);
which allows me to retrieve data using something like:
db.get('collection', key)
.then(function(result){
console.log(result.body);
});
My question is - Is there any way I can switch the value of 'db' to point at a mongolab database without changing the structure of the get request?
I work at Orchestrate and we do not believe in data lock-in. I hope you'll reconsider using our service, but here's some advice if you choose to leave...
It sounds like your code is fairly minimal, so you may be best off recreating your Node server with another tutorial specific to Mongo.
That said, if you are using simple key-value storage, it should be as easy as rewriting the db.get Orchestrate lines to be db.find functions from MongoDB. If you've loaded a lot of data you could export it from Orchestrate, then import into Mongo (either manually, or using another tool).
If you're using some advanced, built-in Orchestrate features, such as full-text search, relation graphing, time-series data, and geographic look-ups, it may take some more effort (and MongoDB experience) to switch. If you'd like these features in a highly scalable database-as-a-service that you don't have to maintain, you know where to find us.
Related
In aws dynamo db we cannot store more than 400KB data in a single record [Reference].
Based on suggestions online I can either compress the data before storing or upload part of it to aws s3 bucket which I am fine by
But my application (javascript/express server plus many js lambdas/microservices) is too large and adding the above logic which require a heavy re-write and extensive testing. Currently there is an immediate requirement from a big client that demands >400KB storage in db, so is there any alternative way to solve the problem that doesn't make me change my existing code to fetch the record from db.
I was thinking more in these lines:
My backend makes a dynamo db call to fetch the record as its doing now (we use a mix of vogels and aws-sdk to make db calls) -> The call is intercepted by a lambda (or something else) which handles the necessary compression/decompression/s3 with dynamodb and returns the data to the backend.
Is the above approach possible to do and if yes then how can i go about implementing it? Or if you have a better way, please do tell.
PS. Going forward I will definitely re-write my codebase to take care of this, what I am asking for is an immediate stopgap solution.
Split the data into multiple items. You’ll have to change a little client code but hopefully you have a data access layer so it’s just a small change in one place. If you don’t have a DAL, from now on always have a DAL. :)
For the payload of a big item, use the regular item as the manifest which can point at the segmented items. Then batch get items those segmented items.
This assumes compression alone isn’t always sufficient. If it is, do that.
I have a local container created with Docker with MongoDB & an express node server
What is the recommended way to populate it with new data?
1) Use the cli
2) Using Mongoose
3) Use a GUI such as Compass
Thanks!
I don't know if there's a 'correct' way to do this, but I've run a couple of 'seeds' files for my projects:
https://github.com/rmgreenstreet/surfShop/blob/master/seeds.js
https://github.com/rmgreenstreet/yelpcamp/blob/master/seeds.js
https://github.com/rmgreenstreet/custom-forms/blob/master/seeds.js
I almost wish there was some kind of niche field/need/position for being able to generate fake data!
The point is that you'll need to set and understand the structure of your data and essentially go through a bunch of nested loops for any connected/dependent data types.
Now if you're working with a SQL database, I'm totally clueless. That's next on my "things to learn" once I feel more comfortable with Javascript/NoSQL.
This would possibly depend on the usecase here,
Answer would be:
MONGOOSE : If you are planning to deploy an application using express. As mongoose goes hand in hand with express. (https://medium.com/#SigniorGratiano/mongoose-and-express-68994fcfdeff) As in many stacks like MEAN, MERN.
GUI like Compass: When you have to visualise the data or do ONE TIME OPERATIONS.
I would like to know what do you think about the following task. I want to write data from JSON object in a database. I would like to separate the SQL logic with the business logic.
I read t'hi strategy has not good performance, when the file js contain a lot of queries.
Which approach is the best practice in your opinion? Can you provide a little example?
Your performance question is definitely a 'race your horses' scenario (i.e. test it and see). But in general, if you're going to do this I'd simply export an object with all your named queries like so:
module.exports = {
getAllUsers: "SELECT username, email, displayName FROM users;",
/* other queries */
}
Your calling code can then just require that file and get what it needs:
const queries = require('./db/queries');
queries.getAllUsers // <-- this is now that string
Performance should be about as good as it gets, since your require cache will ensure the file is only read once, and a key-based lookup in JS is pretty quick, even with a thousand or two entries.
I think is always a good practice to separate DB code from business code, and from API code if it exists.
Creating these different layers, you get different advantages:
Testing every layer separately (with unit tests), mocking other layers. With this you can detect errors very fast when you make changes in your code.
You can change very easy your DB connector, or even your database, without impacting your business code (e.g. MySQL by MongoDB)
You can change your API or add a new one without changing your business code (e.g. REST API by/and GraphQL)
If you want to see a project with this layers, we published recently a simple project that allow you to create a collaborative newsletter. You can check backend part, which has db folder, domain folder and api folder. Those are the 3 layers I was talking about:
Colaborative newsletter
Hope it helps you
In my application I receive json data in a post request that I store as raw json data in a table. I use postgresql (9.5) and node.js .
In this example, the data is an array of about 10 quiz questions experienced by a user, that looks like this:
[{"QuestionId":1, "score":1, "answerList":["1"], "startTime":"2015-12-14T11:26:54.505Z", "clickNb":1, "endTime":"2015-12-14T11:26:57.226Z"},
{"QuestionId":2, "score":1, "answerList":["3", "2"], "startTime":"2015-12-14T11:27:54.505Z", "clickNb":1, "endTime":"2015-12-14T11:27:57.226Z"}]
I need to store (temporarily or permanently) several indicators computed by aggregating data from this json at quizz level, as I need these indicators to perform other procedures in my database.
As of now I was computing the indicators using javascript functions at the time of handling the post request and inserting the values in my table alongside the raw json data. I'm wondering if it wouldn't be more performant to have the calculation performed by a stored trigger function in my postgresql db (knowing that the sql function would need to retrieve the data from inside the json raw data).
I have read other posts on this topic, but it was asked many years ago and not with node.js, so I thought people might have some new insight on the pros and cons of using sql stored procedures vs server-side javascript functions.
edit: I should probably have mentioned that most of my application's logic already mostly lies in postgresql stored procedures and views.
Generally, I would not use that approach due to the risk of getting the triggers out of sync with the code. In general, the single responsibility principle should be the guide: DB to store data and code to manipulate it. Unless you have a really pressing business need to break this pattern, I'd advise against it.
Do you have a migration that will recreate the triggers if you wipe the DB and start from scratch? Will you or a coworker not realise they are there at a later point when reading the app code and wonder what is going on? If there is a standardised way to manage the triggers where the configuration will be stored as code with the rest of your app, then maybe not a problem. If not, be wary. A small performance gain may well not be worth the potential for lost developer time and shipping bugs.
Currently working somewhere that has gone all-in on SQL functions.. We have over a thousand.. I'd strongly advise against it.
Having logic split between Javascript and SQL is a real pain when debugging issues especially if, like me, you are much more familiar with JS.
The functions are at least all tracked in source control and get updated/created in the DB as part of the deployment process but this means you have 2 places to look at when trying to follow the code.
I fully agree with the other answer, single responsibility principle, DB for storage, server/app for logic.
I know memcache and redis are used when caching needs to be there for more than one servers.
I'm creating a node application which will run on single server only and uses mysql as db, and i need to hash around 100,000 keys and each key will contain json string of 200 in length, so that i dont have to call mysql for reads.
If i use memcache or redis i will use a callback to get my data, but if i use javascript hash i can get the data synchronously, but will it affect the application somehow, like high usage of memory. Which one i should be using for a application like this?
I know memcache and redis are used when caching needs to be there for more than one servers.
Not necessarily, for instance Facebook puts a memcache instance in front of each of their mysql servers. You can use Redis/Memcache for fast computation (e.g. real-time analytics) without having a whole cluster.
and i need to hash around 100,000 keys and each key will contain json string of 200 in length, so that i dont have to call mysql for reads.
It seems like premature optimization to mee, if MySQL have enough RAM (the dataset live in memory) you don't have to worry about performance, that's just 100 keys.
If i use memcache or redis i will use a callback to get my data
If really depends on what language you use (Ruby and Python offers synchronous Redis clients) and what type of paradygm is used (event-loop, thread pool...)
but if i use javascript hash i can get the data synchronously
To be more precise, that's just because you are using node_redis and not because you are using a javascript "hash" (an object in fact).
but will it affect the application somehow, like high usage of memory
It depends if you are loading all keys in your process or not, if you are using a Redis Hash, you will be able to only query the field you want and not the whole field each time.
Which one i should be using for a application like this?
The best thing to keep in mind is to lower the number of application you have to maintain in your stack while still using the right tool for the right job. Here MySQL could be enough but if you really want to use Redis or MemCached, I would go for Redis. It will offers simirarly the same features as memcached with the same performances will allowing you to use its other data-structures in the future without needing another application in your stack.
Moreover, if you put all your data in a Redis HASH, you will be able to retrieve a field (hget) or a group of fields (hmget) or all fields (hgetall) with just one call.
Finally, regarding recent statistics and Redis ecosystem (GUI, hosting, librairies, ...), Redis seems to be way more future proof than Memcached if you really want to go that way.
Disclaimer: I am the founder of Redsmin, an online developer oriented service for administrating and monitoring Redis.
It depends- you could even opt for memcached over mysql :). For simple operation such as only -readonly just storing it within your javascript code (I believe as dictionary objects) is enough. But be sure that you have enough RAM :) .