I have a trouble with mongoDB and Mongoose. I need to store data in mongoDB, because I use it for saving settings. Settings must load when app initializes. The trouble is that database does not always return data. Here are the screenshots to make everything clear:
'Setup' array is returned.
'Setup' array is not returned.
'Setup' array returned 'undefined', so script can't run.
1-3 steps are done randomly!
My question is: how to connect to DB right? Is there any problems with the DB connection? (OS - Linux mint 19)
I tried several ways to connect to this DB:
Setup.find({}).exec()
Setup.findById('...').exec()
etc.
mongoose.connect('mongodb://localhost:27017/config', { useNewUrlParser: true });
const setupSchema = new Schema({
eula: Boolean,
lang: String,
styles: Number
});
mongoose.set('debug', true);
setupSchema.set('collection', 'setup');
const Setup = mongoose.model('Setup', setupSchema);
var eula, lang, styles;
Setup.findById('5ccfaf5a0c3c1612d4e2c905', function(err, setting){
if (err) {
console.log('Setup Init error');
console.log(err);
} else {
console.log('Setup Contents');
if (setting !== null || setting !== undefined){
console.log(setting[0]);
// eula = setting[0].eula;
// lang = setting[0].lang;
// styles = setting[0].styles;
} else {
console.log('Setting is null');
}
}
});
I expect that the data will ALWAYS be returned.
Actual output is shown on screenshots (see above).
UPD: I've now reached this state (on the screenshot below), but still it is not good...
UPD2: I've found the solution! I tried to use MongoDB.Client, not mongoose.
I guess findById function will return an object and not an array.That's why you get this error.
TRY THIS:
Setup.findById('5ccfaf5a0c3c1612d4e2c905', function(err, setting){
if (err) {
console.log('Setup Init error');
console.log(err);
} else {
console.log('Setup Contents');
if (setting){
console.log(setting);
// eula = setting.eula;
// lang = setting.lang;
// styles = setting.styles;
} else {
console.log('Setting is null');
}
}
});
UPDATED CODE:
Setup.findOne({id :'5ccfaf5a0c3c1612d4e2c905'}, function(err, setting){
if (err) {
console.log('Setup Init error');
console.log(err);
} else {
console.log('Setup Contents');
if (setting){
console.log(setting);
// eula = setting.eula;
// lang = setting.lang;
// styles = setting.styles;
} else {
console.log('Setting is null');
}
}
});
Related
I need to load and interpret Parquet files from an S3 bucket using node.js. I've already tried parquetjs-lite and other npm libraries I could find, but none of them seems to interpret date-time fields correctly. So I'm trying to AWS's own SDK instead, in the believe that is should be able to deserialize its own Parquet format correctly -- the objects were originally written from SageMaker.
The way to go about it, apparently, is to use the JS version of
https://docs.aws.amazon.com/AmazonS3/latest/API/API_SelectObjectContent.html
but the documentation for that is horrifically out of date (it's referring to the 2006 API, https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#selectObjectContent-property). Likewise, the example they show in their blog post doesn't work either (data.Payload is neither a ReadableStream not iterable).
I've already tried the response in
Javascript - Read parquet data (with snappy compression) from AWS s3 bucket. Neither of them work: the first uses
node-parquet, which doesn't currently compile, and the second uses parquetjs-lite (which doesn't work, see above).
So my question is, how is SelectObjectContent supposed to work nowadays, i.e., using aws-sdk v3?
import { S3Client, ListBucketsCommand, GetObjectCommand,
SelectObjectContentCommand } from "#aws-sdk/client-s3";
const REGION = "us-west-2";
const s3Client = new S3Client({ region: REGION });
const params = {
Bucket: "my-bucket-name",
Key: "mykey",
ExpressionType: 'SQL',
Expression: 'SELECT created_at FROM S3Object',
InputSerialization: {
Parquet: {}
},
OutputSerialization: {
CSV: {}
}
};
const run = async () => {
try {
const data = await s3Client.send(new SelectObjectContentCommand(params));
console.log("Success", data);
const events = data.Payload;
const eventStream = data.Payload;
// Read events as they are available
eventStream.on('data', (event) => { // <--- This fails
if (event.Records) {
// event.Records.Payload is a buffer containing
// a single record, partial records, or multiple records
process.stdout.write(event.Records.Payload.toString());
} else if (event.Stats) {
console.log(`Processed ${event.Stats.Details.BytesProcessed} bytes`);
} else if (event.End) {
console.log('SelectObjectContent completed');
}
});
// Handle errors encountered during the API call
eventStream.on('error', (err) => {
switch (err.name) {
// Check against specific error codes that need custom handling
}
});
eventStream.on('end', () => {
// Finished receiving events from S3
});
} catch (err) {
console.log("Error", err);
}
};
run();
The console.log shows data.Payload as:
Payload: {
[Symbol(Symbol.asyncIterator)]: [AsyncGeneratorFunction: [Symbol.asyncIterator]]
}
what should I do with that?
I was stuck on this exact same issue for quite some time. It looks like the best option now is to append a promise() to it.
So far, I've made progress using the following (sorry, this is incomplete but should at least enable you to read data):
try {
const s3Data = await s3.selectObjectContent(params3).promise();
// using 'any' here temporarily, but will need to address type issues
const events: any = s3Data.Payload;
for await (const event of events) {
try {
if(event?.Records) {
if (event?.Records?.Payload) {
const record = decodeURIComponent(event.Records.Payload.toString().replace(/\+|\t/g, ' '));
records.push(record);
} else {
console.log('skipped event, payload: ', event?.Records?.Payload);
}
}
else if (event.Stats) {
console.log(`Processed ${event.Stats.Details.BytesProcessed} bytes`);
} else if (event.End) {
console.log('SelectObjectContent completed');
}
}
catch (err) {
if (err instanceof TypeError) {
console.log('error in events: ', err);
throw err;
}
}
}
}
catch (err) {
console.log('error fetching data: ', err);
throw err;
}
console.log("final records: ", records);
return records;
}
I'm using This Node.js module nano
Why I Can't Update my Document? I will Want to Make crazy: true and then False again.
This is My Code:
var nano = require('nano')('http://localhost:5984');
// clean up the database we created previously
nano.db.destroy('alice', function() {
// create a new database
nano.db.create('alice', function() {
// specify the database we are going to use
var alice = nano.use('alice');
// and insert a document in it
alice.insert({ crazy: true }, 'rabbit', function(err, body, header) {
if (err) {
console.log('[alice.insert] ', err.message);
return;
}
console.log('you have inserted the rabbit.')
console.log(body);
});
});
});
Nano doesn’t come with an update method by default. That is why we need to define a custom method that would do it for us. Declare the following near the top of your app.js file, right after your database connection code.
test_db.update = function(obj, key, callback){
var db = this;
db.get(key, function (error, existing){
if(!error) obj._rev = existing._rev;
db.insert(obj, key, callback);
});
}
You can then use the update method in your code:
// and update a document in it
alice.update({ crazy: false }, 'rabbit', function(err, body, header) {
if (err) {
console.log('[alice.insert] ', err.message);
return;
}
console.log('you have updated the rabbit.')
console.log(body);
});
});
});
I am new to rethinkdb.
When I try out the sample code in https://github.com/rethinkdb/rethinkdb-example-nodejs/tree/master/todo-angular-express
function create(req, res, next) {
var todo = req.body;
todo.createdAt = r.now(); // Set the field `createdAt` to the current time
r.table('todos').insert(todo, {returnVals: true}).run(req._rdbConn, function(error, result) {
if (error) {
handleError(res, error)
}
else if (result.inserted !== 1) {
handleError(res, new Error("Document was not inserted."))
}
else {
res.send(JSON.stringify(result.new_val));
}
next();
});
}
I got the following error:
500 Internal Server Error
{"error":"return_vals renamed to return_changes in:\nr.table(\"todos\").insert({title: r.json(\"\"abcde\"\"), completed: r.json(\"false\"), createdAt: r.now()}, {returnVals: true})\n
And then I tried out the sample code in http://rethinkdb.com/docs/examples/node-todo/
function create(req, res, next) {
var todo = req.body; // req.body was created by `bodyParser`
todo.createdAt = r.now(); // Set the field `createdAt` to the current time
r.table('todos').insert(todo, {returnChanges: true}).run(req._rdbConn, function(error, result) {
if (error) {
handleError(res, error)
}
else if (result.inserted !== 1) {
handleError(res, new Error("Document was not inserted."))
}
else {
res.send(JSON.stringify(result.changes[0].new_val));
}
next();
});
}
I got the following error:
500 Internal Server Error
{"error":"Unrecognized optional argument returnChanges. in:\nr.table(\"todos\").insert({title: r.json(\"\"abcde\"\"), completed: r.json(\"false\"), createdAt: r.now()}, {returnChanges: true})\n "}
It seems that rethinkdb have changed returnVals to return_changes / returnChanges, and the argument of insert().
And I have the problem fixed when I used return_changes.
What is the right way to work on insert in latest version?
Do rethinkdb always changes its syntax?
this is indeed a bug in the example code. I've opened https://github.com/rethinkdb/rethinkdb-example-nodejs/issues/3 so we can fix it.
Your second problem with returnChanges not being recognized might come from using an old RethinkDB node driver. Have you tried updating the driver? http://rethinkdb.com/docs/install-drivers/javascript/
I've configured a FB graph call that would retrieve data from the API, however I'm having trouble inserting it into MongoDb. Right now if I run Photos.find().count(); in the browser it shows that there are photos, however if I run db.Photos.find().count(); in MongoDb it shows nothing. Also, if I run db.users.find(); in MongoDb it returns results from the FB user account, so MongoDb is talking to the API to some extent.
Any thoughts on what might be causing the issue?
Here is my code:
Client:
Template.test.events({
'click #btn-user-data': function(e) {
Meteor.call('getUserData', function(err, data) {
if(err) console.error(err);
});
}
});
Template.facebookphoto.helpers({
pictures: function () {
return Photos.find();
}
});
Server:
function Facebook(accessToken) {
this.fb = Meteor.require('fbgraph');
this.accessToken = accessToken;
this.fb.setAccessToken(this.accessToken);
this.options = {
timeout: 3000,
pool: {maxSockets: Infinity},
headers: {connection: "keep-alive"}
}
this.fb.setOptions(this.options);
}
Facebook.prototype.query = function(query, method) {
var self = this;
var method = (typeof method === 'undefined') ? 'get' : method;
var data = Meteor.sync(function(done) {
self.fb[method](query, function(err, res) {
done(null, res);
});
});
return data.result;
}
Facebook.prototype.getUserData = function() {
return this.query('me/photos');
}
Meteor.methods({
getUserData: function() {
var fb = new Facebook(Meteor.user().services.facebook.accessToken);
var data = fb.getUserData();
_.forEach(data.data, function(photo) {
if(Photos.findOne({id: photo.id})) return;
Photos.insert(photo, function(err) {
if(err) console.error(err);
});
});
}
});
Collection:
Photos = new Meteor.Collection('picture');
Thanks in advance!
Instead of db.Photos.find().count();, try db.picture.find().count();
Photos is just the name you gave to the JavaScript variable. The actual name of the collection in MongoDB is whatever you use when you initialized the Collection - in this case, picture.
Im using connect-domain and connect-redis. Below code checks for redis cache in Redis database.
function redis_get(key, req, res) {
var redisClient = redis.createClient();
redisClient.get(redisKey, function (err, data) {
if (err) {
console.log("Error in RedisDB");
}
else if (data == null) {
// Calling external function
}
else {
// Calling external function
}
redisClient.quit(); // Not working
});
}
When cache is not avaiable Im calling external function. I want redis connection to be closed once the cache check has been done.
redisClient.quit() // Not working
Any help on this will be really helpful.
Thanks
Below code is working fine without any problem.So check your status reply in the quit method if you get status as 'OK' means that method is working fine.
var redis=require('redis');
var redisClient = redis.createClient();
redisClient.get('name', function (err, data) {
if (err) {
console.log("Error in RedisDB");
}
else if (data == null) {
console.log('null');
}
else {
console.log(data);
}
redisClient.quit(redis.print);
});