We Keep Coding

Firebase - getting the latest document

I have a node server running where I want to listen to a collection update and get the added data.
My solution was using db.collection("posts").onSnapshot to listen to a update and to get the latest I would order by date.
db.collection("posts").onSnapshot(async () => {
const newPost = await db
.collection("posts")
.orderBy("date", "desc")
.limit(1)
.get()
.data();
console.log(newPost);
});
but .data() is not a function so I don't know how to retrive the data. I did a little debbuging and couldn't find any keys in the object that would give me the data from the post.
This is what it returns without .data()
QuerySnapshot$1 {
_delegate:
QuerySnapshot {
_firestore:
FirebaseFirestore$1 {
_persistenceKey: '[DEFAULT]',
_settings: [FirestoreSettings],
_settingsFrozen: true,
_app: [FirebaseAppImpl],
_databaseId: [DatabaseId],
_credentials: [FirebaseCredentialsProvider],
_queue: [AsyncQueue],
_firestoreClient: [FirestoreClient] },
_userDataWriter: UserDataWriter { firestore: [Firestore] },
_snapshot:
ViewSnapshot {
query: [QueryImpl],
docs: [DocumentSet],
oldDocs: [DocumentSet],
docChanges: [Array],
mutatedKeys: [SortedSet],
fromCache: false,
syncStateChanged: true,
excludesMetadataChanges: false },
metadata:
SnapshotMetadata { hasPendingWrites: false, fromCache: false },
query:
Query {
_converter: null,
_query: [QueryImpl],
type: 'query',
firestore: [FirebaseFirestore$1] } },
_firestore:
Firestore {
_delegate:
FirebaseFirestore$1 {
_persistenceKey: '[DEFAULT]',
_settings: [FirestoreSettings],
_settingsFrozen: true,
_app: [FirebaseAppImpl],
_databaseId: [DatabaseId],
_credentials: [FirebaseCredentialsProvider],
_queue: [AsyncQueue],
_firestoreClient: [FirestoreClient] },
_persistenceProvider: IndexedDbPersistenceProvider {},
INTERNAL: { delete: [Function: delete] },
_appCompat:
FirebaseAppImpl {
firebase_: [Object],
isDeleted_: false,
name_: '[DEFAULT]',
automaticDataCollectionEnabled_: false,
options_: [Object],
container: [ComponentContainer] } } }

Your code will have to first await the result of the get(), then reach into the returned QuerySnapshot to find the document data. Note that a QuerySnapshot contains zero or more documents, and you will need to use its API to find out if there are any documents returned. Even if you think it will just return 1 document, you still need to reach into the result set to find that one document.
const newPost = await db
.collection("posts")
.orderBy("date", "desc")
.limit(1)
.get();
// newPost is a QuerySnapshot
if (newPost.size > 0) {
const data = newPost.docs[0].data();
// do what you want with the document data
}
else {
// figure out what you want to do if no documents were queried
}

NodeJS - How to get response of URL?

I have an url adress. When I go that adress in browser it shows a json file. I need to get that json file in NodeJs.
How can I do that?
Edit: This is my current code:
const got = require('got');
var url = "https://www.instagram.com/p/CFcyO54Hc7k/?__a=1";
got(url).json().then(result => {
console.log(result);
}).catch(err => {
console.log(err);
});
And This is the result I get:
Promise { <pending> }
Hint: hit control+c anytime to enter REPL.
RequestError: Unexpected token < in JSON at position 0 in "https://www.instagram.com/accounts/login/?next=/p/CFcyO54Hc7k/%3F__a%3D1"
at Object.parseBody [as default] (/home/runner/VengefulUltimateSdk/node_modules/got/dist/source/as-promise/parse-body.js:22:15)
at /home/runner/VengefulUltimateSdk/node_modules/got/dist/source/as-promise/index.js:157:40
at JSON.parse (<anonymous>)
at parseJson (/home/runner/VengefulUltimateSdk/node_modules/got/dist/source/index.js:118:35)
at Object.parseBody [as default] (/home/runner/VengefulUltimateSdk/node_modules/got/dist/source/as-promise/parse-body.js:11:48)
at /home/runner/VengefulUltimateSdk/node_modules/got/dist/source/as-promise/index.js:157:40
at processTicksAndRejections (internal/process/task_queues.js:97:5) {
name: 'ParseError',
code: undefined,
timings: {
start: 1601274389096,
socket: 1601274389096,
lookup: 1601274389106,
connect: 1601274389117,
secureConnect: 1601274389128,
upload: 1601274389128,
response: 1601274389467,
end: 1601274389511,
error: undefined,
abort: undefined,
phases: {
wait: 0,
dns: 10,
tcp: 11,
tls: 11,
request: 0,
firstByte: 339,
download: 44,
total: 415
}
}
}
My node version is 12.16.1

If you want to get it as json you can use the fetch api (https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) then in the promises you can read the json as an object in your js code.
fetch('http://server/file.json').then(resp=>resp.json()).then(data=>{do something with data});

You can use axios in NodeJS.
axios.get('https://someapiendpoint.domain.com/v1/api')
.then((response) => {
console.log(response.data);
});

You can either parse the JSON yourself or you can get a library for making an Http request that parses the JSON automatically for you. There are lots of libraries to choose from here. My favorite is got() and it will let you parse the JSON with a simple .json() added to it so the resolved value you get back is a Javascript object.
const got = require('got');
got("https://www.instagram.com/p/CFcyO54Hc7k/?__a=1").json().then(result => {
console.log(result);
}).catch(err => {
console.log(err);
});
When I run the above code with your instagram URL, I get this Javascript object back:
{
graphql: {
shortcode_media: {
__typename: 'GraphSidecar',
id: '2404017227651993316',
shortcode: 'CFcyO54Hc7k',
dimensions: [Object],
gating_info: null,
fact_check_overall_rating: null,
fact_check_information: null,
sensitivity_friction_info: null,
media_overlay_info: null,
media_preview: null,
display_url: 'https://scontent-sjc3-1.cdninstagram.com/v/t51.2885-15/e35/119980944_890957481429986_8757958615208227911_n.jpg?_nc_ht=scontent-sjc3-1.cdninstagram.com&_nc_cat=102&_nc_ohc=w7zBW_FdRNgAX9BYknl&_nc_tp=18&oh=11424efc118b55180e4d3302d75181ca&oe=5F99EB23',
display_resources: [Array],
is_video: false,
tracking_token: 'eyJ2ZXJzaW9uIjo1LCJwYXlsb2FkIjp7ImlzX2FuYWx5dGljc190cmFja2VkIjp0cnVlLCJ1dWlkIjoiZGIyMzdkMzgzYWVhNDY5YmI0ZWVkZWI4YzlhZDY0YWUyNDA0MDE3MjI3NjUxOTkzMzE2In0sInNpZ25hdHVyZSI6IiJ9',
edge_media_to_tagged_user: [Object],
edge_media_to_caption: [Object],
caption_is_edited: false,
has_ranked_comments: false,
edge_media_to_parent_comment: [Object],
edge_media_to_hoisted_comment: [Object],
edge_media_preview_comment: [Object],
comments_disabled: false,
commenting_disabled_for_viewer: false,
taken_at_timestamp: 1600801207,
edge_media_preview_like: [Object],
edge_media_to_sponsor_user: [Object],
location: null,
viewer_has_liked: false,
viewer_has_saved: false,
viewer_has_saved_to_collection: false,
viewer_in_photo_of_you: false,
viewer_can_reshare: true,
owner: [Object],
is_ad: false,
edge_web_media_to_related_media: [Object],
edge_sidecar_to_children: [Object],
edge_related_profiles: [Object]
}
}
}
If you'd rather modify your existing code, then you will probably need to use JSON.parse(). You will have to add your existing code to your question for us to help with that.

Create with Include Sequelize

recently I discovered this on the sequelize documentation where you can create using include. Now I trying to do it on my program but only creates the records of the "parent" model and not for the children.
This is my model and my controller.
var MainMenu = sequelize.define('MainMenu', {
Name: {
type: DataTypes.STRING(50)
},
Day: {
type: DataTypes.DATE
},
RecordStatus:{
type: DataTypes.BOOLEAN,
defaultValue: true
},
DeletedAt: {
type: DataTypes.DATE
}
},
{
associate: function(models){
models.MainMenu.hasMany(models.MainMeal, {as: 'Menu'});
}
}
);
exports.createIn = (req, res) => {
let Menu = {
Name: 'MenuTest',
MainMeal: [{
Type: 'Breakfast',
Name: 'MealTest1'
}, {
Type: 'Lunch',
Name: 'MealTest2'
}]
};
db.MainMenu.create(Menu, {
include: [{
model: db.MainMeal,
as: 'Menu'
}]
})
.then( mainmenu => {
if (!mainmenu) {
return res.send('users/signup', {
errors: 'Error al registrar el mainmenu.'
});
} else {
return res.jsonp(mainmenu);
}
})
.catch( err => {
console.log(err);
return res.status(400)
.send({
message: errorHandler.getErrorMessage(err)
});
});
};
On my case it only creates the MainMenu record and not the MainMeal records. What am I doing wrong?

Change your menu object, and include Menu array and not MainMeal
You have to give the aliased name in the object
let mainMenu = {
Name: 'MenuTest',
Menu: [{
Type: 'Breakfast',
Name: 'MealTest1'
}, {
Type: 'Lunch',
Name: 'MealTest2'
}]
};
Now,
db.MainMenu.create(mainMenu, {
include: [{
model: db.MainMeal,
as: 'Menu'
}]
})
.then( mainmenu => {
if (!mainmenu) {
return res.send('users/signup', {
errors: 'Error al registrar el mainmenu.'
});
} else {
return res.jsonp(mainmenu);
}
})
.catch( err => {
console.log(err);
return res.status(400)
.send({
message: errorHandler.getErrorMessage(err)
});
});

The main thing is of course the naming of Menu should be within the data passed to .create() itself, along with the arguments presented there and if you really need to specify the alias "twice", which you do not. But there are some other things to be aware of.
I'd personally prefer storing the association as it's own export and including that within the statement. This generally becomes a bit clearer when you understand the usage of that association later.
I would also strongly encourage that when you are "writing" things across multiple tables, then you implement transactions to ensure all related items are actually created and not left orphaned should any errors arise.
As a brief listing based on the example:
const Sequelize = require('sequelize');
const sequelize = new Sequelize('sqlite:menu.db',{ logging: console.log });
const MainMeal = sequelize.define('MainMeal', {
Type: { type: Sequelize.STRING(50) },
Name: { type: Sequelize.STRING(50) }
});
const MainMenu = sequelize.define('MainMenu', {
Name: { type: Sequelize.STRING(50) }
});
MainMenu.Meals = MainMenu.hasMany(MainMeal, { as: 'Menu' });
(async function() {
try {
await sequelize.authenticate();
await MainMeal.sync({ force: true });
await MainMenu.sync({ force: true });
let result = await sequelize.transaction(transaction =>
MainMenu.create({
Name: 'MenuTest',
Menu: [
{ Type: 'Breakfast', Name: 'MealTest1' },
{ Type: 'Lunch', Name: 'MealTest2' }
]
},{
include: MainMenu.Meals,
transaction
})
);
} catch(e) {
console.error(e);
} finally {
process.exit();
}
})();
Which would output something like:
Executing (default): SELECT 1+1 AS result
Executing (default): DROP TABLE IF EXISTS `MainMeals`;
Executing (default): CREATE TABLE IF NOT EXISTS `MainMeals` (`id` INTEGER PRIMARY KEY AUTOINCREMENT, `Type` VARCHAR(50), `Name` VARCHAR(50), `createdAt` DATETIME NOT NULL, `updatedAt` DATETIME NOT NULL, `MainMenuId` INTEGER REFERENCES `MainMenus` (`id`) ON DELETE
SET NULL ON UPDATE CASCADE);
Executing (default): PRAGMA INDEX_LIST(`MainMeals`)
Executing (default): DROP TABLE IF EXISTS `MainMenus`;
Executing (default): CREATE TABLE IF NOT EXISTS `MainMenus` (`id` INTEGER PRIMARY KEY AUTOINCREMENT, `Name` VARCHAR(50), `createdAt` DATETIME NOT NULL, `updatedAt` DATETIME NOT NULL);
Executing (default): PRAGMA INDEX_LIST(`MainMenus`)
Executing (3d645847-56ca-435a-b786-6be62a05e8d5): BEGIN DEFERRED TRANSACTION;
Executing (3d645847-56ca-435a-b786-6be62a05e8d5): INSERT INTO `MainMenus` (`id`,`Name`,`createdAt`,`updatedAt`) VALUES (NULL,'MenuTest','2018-04-14 08:08:17.132 +00:00','2018-04-14 08:08:17.132 +00:00');
Executing (3d645847-56ca-435a-b786-6be62a05e8d5): INSERT INTO `MainMeals` (`id`,`Type`,`Name`,`createdAt`,`updatedAt`,`MainMenuId`)
VALUES (NULL,'Breakfast','MealTest1','2018-04-14 08:08:17.152 +00:00','2018-04-14 08:08:17.152 +00:00',1);
Executing (3d645847-56ca-435a-b786-6be62a05e8d5): INSERT INTO `MainMeals` (`id`,`Type`,`Name`,`createdAt`,`updatedAt`,`MainMenuId`)
VALUES (NULL,'Lunch','MealTest2','2018-04-14 08:08:17.153 +00:00','2018-04-14 08:08:17.153 +00:00',1);
Executing (3d645847-56ca-435a-b786-6be62a05e8d5): COMMIT;
The important part there being the transaction BEGIN and COMMIT wrapping all of those INSERT statements as data is created. Even without the transaction implemented, you still see both items being created along with the related "parent". But the point of the argument is this is where you "should" be implementing transactions.
Also note that the "aliased" Menu as used in the data creation and for subsequent access, is not actually "required" to be included within the .create() method on the include option. It's "optional" and is already defined under the .hasMany() arguments, so you don't really need to do it again.
Even if you did, then that part would still be the "association" as used with the model argument:
{
include: {
model: MainMenu.Meals,
as: 'Menu'
},
transaction
}
So that's not to be confused with the original name of the model for the "table" which is referenced, which also might be another point of confusion.

Koa-session getting reset after appending object to it?

I have a controller which looks up a character, and then does some stuff with it, the controller looks like:
router.post('/profile/characters', async ctx => {
try {
ctx.type = 'json';
let req = ctx.request;
if (!('charname' in req.body) || !('charserver' in req.body)) {
return res.json({
'success': false,
error: 'You are missing either the character name, or server'
});
}
let foundChar = await new Promise((res, rej) => {
bnet.wow.character.aggregate({
origin: 'us',
realm: req.body.charserver,
name: req.body.charname,
fields: ['items', 'talents']
}, (err, charData) => {
if (err) {
console.log(err);
return rej(err);
}
return res(charData);
});
});
if ('status' in foundChar) {
if (foundChar.status === 'nok') {
return ctx.body = {
'success': false,
error: 'There was an error looking up your character, please ensure its a US character, and has been logged into recently'
};
}
}
foundChar.items.level = foundChar.level;
foundChar.items.class = foundChar.class;
foundChar.items.thumbnail = foundChar.thumbnail;
foundChar.items.name = foundChar.name;
let {
items, talents
} = foundChar;
let specF = talents.find(x => x.selected) || {};
let charData = {
items, specF
};
if ('legs' in items || 'hands' in items || 'shoulder' in items) {
return ctx.body = {
success: false,
error: 'To verify it is your own character, please remove your (Shoulders, Hands, and Pants) from your character and try again.'
};
}
ctx.session.foundChar = foundChar; // This line here
console.log(ctx.session);
ctx.body = {
success: true,
charData
};
} catch (err) {
console.log(err);
ctx.status = err.status || 500;
ctx.body = {
message: err.message
};
}
});
When it processes ctx.session.foundChar = foundChar it seems to reset my session for some reason, and logging the session shows {} instead of
{
authenticated: true,
userid: 1
...
}
But if I change ctx.session.foundChar = "Hello"; < Works just fine.
I don't know if there is a data limit or something to the session or what as this wasn't an issue with express-session but I'm trying to convert it all over to Koa, anyways not sure why my session is getting reset.
Example of what foundChar looks like
{ userid: 1,
username: 'Blah',
authenticated: true,
userLevel: 5,
hasMainCharacter: true,
foundChar:
{ lastModified: 1453702285000,
name: 'Blah',
realm: 'Mal\'Ganis',
battlegroup: 'Vindication',
class: 4,
race: 5,
gender: 0,
level: 100,
achievementPoints: 6335,
thumbnail: 'internal-record-3684/9/119507209-avatar.jpg',
calcClass: 'c',
faction: 1,
items:
{ averageItemLevel: 714,
averageItemLevelEquipped: 573,
head: [Object],
neck: [Object],
back: [Object],
chest: [Object],
wrist: [Object],
waist: [Object],
feet: [Object],
finger1: [Object],
finger2: [Object],
trinket1: [Object],
trinket2: [Object],
mainHand: [Object],
offHand: [Object],
level: 100,
class: 4,
thumbnail: 'internal-record-3684/9/119507209-avatar.jpg',
name: 'Blah' },
talents: [ [Object], [Object] ],
totalHonorableKills: 258 } }
So this logs properly, but then after refreshing the page im no longer authenticated and ctx.session is {}

Problem
Your problem is, because of koajs/session usage which is
Simple cookie-based session middleware for Koa.
Which means when ctx.session is being serialized into json and stored in cookie after each request and is being deserialized before each request.
Unfortunately cookie has limited size and when you try to store big object into it with ctx.session.foundChar = foundChar it exceeds maximum cookie size and results currupted session cookie.
For same reason ctx.session.foundChar = "Hello" works, because json size does not exceed max cookie size.
Solution
Use db based storage for session, good choice for it could be koa-session-storage.
Look session storage layer for configuration options
The store configuration option specifies where the session data is
stored. If omitted or set to "cookie" then session data will be
stored in the cookie itself.
If you wish to store session data elsewhere (e.g. in Mongo, Redis,
etc.) then you must set this to an object which exposes the following
API:
load(sid) - load session data for given session id * sid -
{String} session identifier. * returns a Promise, Thunk or generator which returns a JSON string of the session object data.
save(sid, data) - save session data for given session id *
sid - {String} session identifier. * data - _{String} session
data converted to JSON string. * returns a Promise, Thunk or
generator which returns once data is saved.
remove(sid) - remove session data for given session id *
sid - {String} session identifier. * returns a Promise, Thunk
or generator which returns once removal is complete.
The following storage layers are currently available:
MongoDB -
koa-session-mongo

node.js: Mongodb db.collection.find() not working while collection.insert works

I'm using node.js/express and I have a Mongodb to store some sets of data. On a webpage the user can enter, edit and delete data (which all works fine). For example, to add data I have the following code:
router.post('/addset', function(req,res) {
var db = req.db;
var collection = db.get('paramlist');
collection.insert(req.body, function(err, result){
res.send(
(err === null) ? { msg: '' } : { msg: err }
);
});
});
In my app.js file I include the lines
// Database
var mongo = require('mongodb');
var monk = require('monk');
var db = monk('localhost:27017/paramSet1');
as well as
app.use(function(req,res,next){
req.db = db;
next();
});
to make the database accessible in the rest of the code (following this tutorial: http://cwbuecheler.com/web/tutorials/2013/node-express-mongo/ , I'm a beginner with these things).
So all this works fine. My problem is the following: I would like to add a test if a dataset with the same name is already in the database and give a message to the user. Following this answer How to query MongoDB to test if an item exists? I tried using collection.find.limit(1).size() but I get the error
undefined is not a function
I tried the following things. In the cost above (router.post) I tried adding after the line var collection...
var findValue = collection.find({name:req.body.name});
If i then do console.log(findValue), I get a huge output JSON. I then tried console.log(findValue.next()) but I get the same error (undefined is not a function). I also tried
collection.find({name:req.body.name}).limit(1)
as well as
collection.find({name:req.body.name}).limit(1).size()
but also get this error. So in summary, collection.insert, collection.update and collection.remove all work, but find() does not. On the other hand, when I enter the mongo shell, the command works fine.
I would be grateful for any hints and ideas.
Edit:
The output to console.log(findValue) is:
{ col:
{ manager:
{ driver: [Object],
helper: [Object],
collections: [Object],
options: [Object],
_events: {} },
driver:
{ _construct_args: [],
_native: [Object],
_emitter: [Object],
_state: 2,
_connect_args: [Object] },
helper: { toObjectID: [Function], isObjectID: [Function], id: [Object] },
name: 'paramlist',
col:
{ _construct_args: [],
_native: [Object],
_emitter: [Object],
_state: 2,
_skin_db: [Object],
_collection_args: [Object],
id: [Object],
emitter: [Object] },
options: {} },
type: 'find',
opts: { fields: {}, safe: true },
domain: null,
_events: { error: [Function], success: [Function] },
_maxListeners: undefined,
emitted: {},
ended: false,
success: [Function],
error: [Function],
complete: [Function],
resolve: [Function],
fulfill: [Function],
reject: [Function],
query: { name: 'TestSet1' } }

find returns a cursor, not the matching documents themselves. But a better fit for your case would be to use findOne:
collection.findOne({name:req.body.name}, function(err, doc) {
if (doc) {
// A doc with the same name already exists
}
});

If you're using the method on that website http://cwbuecheler.com/web/tutorials/2013/node-express-mongo/
you have to change your code
collection.find({name:req.body.name}).limit(1)
and use it like this
collection.find({name:req.body.name},{limit:1})
and if you want to add more options
do like this
collection.find({name:req.body.name},{limit:1,project:{a:1},skip:1,max:10,maxScan:10,maxTimeMS:1000,min:100})
You can find everything here http://mongodb.github.io/node-mongodb-native/2.0/api/Cursor.html

you can use like that
app.get('/', (req, res) => {
db.collection('quotes').find().toArray((err, result) => {
console.log(result);
})
})

The first thing that looks wrong is your syntax is incorrect for find. You need to call it as a function. Try:
collection.find({name:req.body.name}).limit(1).size();

Probably you might have missed to connect a Database.
Try adding below code before executing
mongodb.connect(dbURI).then((result)=>console.log('connected to DB')).catch((err)=>console.log('connected to DB'));