We Keep Coding

puppeteer page.$$(selector).length is 2 but page.$$(selector)[1] is undefined

I'm trying to run the following selector through puppeteer page.$$(".paging:not(.ng-hide) button.paging-button")
This returns an empty array what appears to be an array of two elementHandles but when I access pagingButton[1] I get undefined
, but when I go to the page and run document.querySelectorAll(".paging:not(.ng-hide) button.paging-button") I get two elements (which is what I want) It's my understanding that puppeteer uses document.querySelectorAll() under the hood but I don't know why my selector is broken in puppeteer. Could it be my use of :not()? My query won't work without that selector since there's n amount of elements that match my selector without it but only 2 that don't have the .ng-hide class
Here's my code
async function advancePage(page)
{
const pagingButton = await page.$$(PAGING_BUTTON)[1];
console.log(pagingButton)
let adHocs = await getCalcsOnPage(page);
const isButtonDisabled = await page.evaluate(el=>el.disabled,pagingButton)
if(isButtonDisabled)
{return adHocs}
else{
console.log(chalk.bgBlue("ADVANCING PAGE"))
await pagingButton.click();
const nextPageCalcs = await advancePage()
adHocs.concat(nextPageCalcs)
return adHocs
}
}
and I have my selectors declared in another file
export const PAGING_BUTTON = ".paging:not(.ng-hide) button.paging-button"
this is what I get when I log the output of await page.$$(PAGING_BUTTON)
[
ElementHandle {
_disposed: false,
_context: ExecutionContext {
_client: [CDPSession],
_world: [DOMWorld],
_contextId: 3,
_contextName: ''
},
_client: CDPSession {
eventsMap: [Map],
emitter: [Object],
_callbacks: Map(0) {},
_connection: [Connection],
_targetType: 'page',
_sessionId: '23E3541B1A29459485B04919F5C5C74A'
},
_remoteObject: {
type: 'object',
subtype: 'node',
className: 'HTMLButtonElement',
description: 'button.paging-button',
objectId: '144722812805955845.3.5'
},
_frame: Frame {
_url: 'https://invawspivis01.dtinet.net/PIVision/#/Displays/3734/VIC%20-%20Site%20Update%20Dashboard',
_detached: false,
_loaderId: '9F2810160C0B1BFDD2882993D0524F3B',
_lifecycleEvents: [Set],
_frameManager: [FrameManager],
_parentFrame: null,
_id: '1B585848F903D3A22CDE31DD75022A81',
_childFrames: Set(0) {},
_client: [CDPSession],
_mainWorld: [DOMWorld],
_secondaryWorld: [DOMWorld],
_name: undefined
},
_page: Page {
eventsMap: Map(0) {},
emitter: [Object],
_closed: false,
_timeoutSettings: [TimeoutSettings],
_pageBindings: Map(0) {},
_javascriptEnabled: true,
_workers: Map(0) {},
_fileChooserInterceptors: Set(0) {},
_userDragInterceptionEnabled: false,
_handlerMap: [WeakMap],
_client: [CDPSession],
_target: [Target],
_keyboard: [Keyboard],
_mouse: [Mouse],
_touchscreen: [Touchscreen],
_accessibility: [Accessibility],
_frameManager: [FrameManager],
_emulationManager: [EmulationManager],
_tracing: [Tracing],
_coverage: [Coverage],
_screenshotTaskQueue: [TaskQueue],
_viewport: [Object]
},
_frameManager: FrameManager {
eventsMap: [Map],
emitter: [Object],
_frames: [Map],
_contextIdToContext: [Map],
_isolatedWorlds: [Set],
_client: [CDPSession],
_page: [Page],
_networkManager: [NetworkManager],
_timeoutSettings: [TimeoutSettings],
_mainFrame: [Frame]
}
},
ElementHandle {
_disposed: false,
_context: ExecutionContext {
_client: [CDPSession],
_world: [DOMWorld],
_contextId: 3,
_contextName: ''
},
_client: CDPSession {
eventsMap: [Map],
emitter: [Object],
_callbacks: Map(0) {},
_connection: [Connection],
_targetType: 'page',
_sessionId: '23E3541B1A29459485B04919F5C5C74A'
},
_remoteObject: {
type: 'object',
subtype: 'node',
className: 'HTMLButtonElement',
description: 'button.paging-button',
objectId: '144722812805955845.3.6'
},
_frame: Frame {
_url: 'https://invawspivis01.dtinet.net/PIVision/#/Displays/3734/VIC%20-%20Site%20Update%20Dashboard',
_detached: false,
_loaderId: '9F2810160C0B1BFDD2882993D0524F3B',
_lifecycleEvents: [Set],
_frameManager: [FrameManager],
_parentFrame: null,
_id: '1B585848F903D3A22CDE31DD75022A81',
_childFrames: Set(0) {},
_client: [CDPSession],
_mainWorld: [DOMWorld],
_secondaryWorld: [DOMWorld],
_name: undefined
},
_page: Page {
eventsMap: Map(0) {},
emitter: [Object],
_closed: false,
_timeoutSettings: [TimeoutSettings],
_pageBindings: Map(0) {},
_javascriptEnabled: true,
_workers: Map(0) {},
_fileChooserInterceptors: Set(0) {},
_userDragInterceptionEnabled: false,
_handlerMap: [WeakMap],
_client: [CDPSession],
_target: [Target],
_keyboard: [Keyboard],
_mouse: [Mouse],
_touchscreen: [Touchscreen],
_accessibility: [Accessibility],
_frameManager: [FrameManager],
_emulationManager: [EmulationManager],
_tracing: [Tracing],
_coverage: [Coverage],
_screenshotTaskQueue: [TaskQueue],
_viewport: [Object]
},
_frameManager: FrameManager {
eventsMap: [Map],
emitter: [Object],
_frames: [Map],
_contextIdToContext: [Map],
_isolatedWorlds: [Set],
_client: [CDPSession],
_page: [Page],
_networkManager: [NetworkManager],
_timeoutSettings: [TimeoutSettings],
_mainFrame: [Frame]
}
}
]
As you can see I have what appear to be two elementHandles, but somehow await page.$$(PAGING_BUTTON)[1] returns undefined?

mongoDB: getMongo() on a db instance returns error

I'm trying to do a sql transaction. for this I need to start a session which a possible only with a mongo dB connection instance.
In my app, I can reach only the DB instance. and I need to be able to get from it the mongodb connection instance. As I understand, getMongo() function should do exactly that.
The problem is the getMongo() is not acting as expected.
when i run this following code:
const db = await this.dbClient;
console.log(db,"db");
console.log(db.getMongo(),"getMongo");
const session = db.getMongo().startSession();
I get this following console.log and errors:
Db {
_events: [Object: null prototype] {},
_eventsCount: 0,
_maxListeners: undefined,
s: {
dbCache: {},
children: [],
topology: NativeTopology {
_events: [Object: null prototype],
_eventsCount: 35,
_maxListeners: Infinity,
s: [Object],
[Symbol(kCapture)]: false,
[Symbol(waitQueue)]: [Denque]
},
options: {
retryWrites: true,
readPreference: [ReadPreference],
promiseLibrary: [Function: Promise]
},
logger: Logger { className: 'Db' },
bson: BSON {},
readPreference: ReadPreference { mode: 'primary', tags: undefined },
bufferMaxEntries: -1,
parentDb: null,
pkFactory: undefined,
nativeParser: undefined,
promiseLibrary: [Function: Promise],
noListener: false,
readConcern: undefined,
writeConcern: undefined,
namespace: MongoDBNamespace { db: 'MyDB', collection: undefined }
},
serverConfig: [Getter],
bufferMaxEntries: [Getter],
databaseName: [Getter],
[Symbol(kCapture)]: false
} db
{"Error msg: TypeError: db.getMongo is not a
function\n...","level":"error"}
{"session is not defined....","level":"error"}
Anybody knows what it the problem?
My mongo version is 4.2.6.
Many thanks

Why I am seeing extra fields when printing the model object of mongoose in console . I am using node JS with mongoose. How can i hide these attribute

const mongoose = require('mongoose');
mongoose.connect('mongodb://localhost/playgroundDB')
.then(()=>console.log('Connected to Mongo DB....'))
.catch(( err)=>console.log('Could not connect to MongoDb', err));
const courseSchema = new mongoose.Schema({
name : String,
author : String,
tags :[String],
date:{ type: Date, default:Date.now},
isPublished: Boolean
});
const Course= mongoose.model('Course', courseSchema);
const course = new Course({
name : "xxxx",
author : 'rahul123',
tags :['Frontend', 'Backend'] ,
isPublished : true
});
course.save().then(res=>console.log(res)).catch(err=> console.log(err));
When I am executing above code . My data is getting saved in db and when i am printing it using (res=>console.log(res)), I am seeing extra fields in my console.
below is my output in console
model {
'$__': InternalCache {
strictMode: true,
selected: undefined,
shardval: undefined,
saveError: undefined,
validationError: undefined,
adhocPaths: undefined,
removing: undefined,
inserting: true,
version: undefined,
getters: {},
_id: 5dea6a7256bf9212c81361a9,
populate: undefined,
populated: undefined,
wasPopulated: false,
scope: undefined,
activePaths: StateMachine {
paths: {},
states: [Object],
stateNames: [Array],
forEach: [Function],
map: [Function]
},
pathsToScopes: {},
ownerDocument: undefined,
fullPath: undefined,
emitter: EventEmitter {
_events: [Object: null prototype] {},
_eventsCount: 0,
_maxListeners: 0
},
'$options': true
},
isNew: false,
errors: undefined,
_doc: {
tags: [
'Frontend',
'Backend',
toBSON: [Function: toBSON],
_atomics: {},
_parent: [Circular],
_cast: [Function: _cast],
_markModified: [Function: _markModified],
_registerAtomic: [Function: _registerAtomic],
'$__getAtomics': [Function: $__getAtomics],
hasAtomics: [Function: hasAtomics],
_mapCast: [Function: _mapCast],
push: [Function: push],
nonAtomicPush: [Function: nonAtomicPush],
'$pop': [Function: $pop],
pop: [Function: pop],
'$shift': [Function: $shift],
shift: [Function: shift],
pull: [Function: pull],
splice: [Function: splice],
unshift: [Function: unshift],
sort: [Function: sort],
addToSet: [Function: addToSet],
set: [Function: set],
toObject: [Function: toObject],
inspect: [Function: inspect],
indexOf: [Function: indexOf],
remove: [Function: pull],
_path: 'tags',
isMongooseArray: true,
validators: [],
_schema: [SchemaArray]
],
date: 2019-12-06T14:49:22.524Z,
_id: 5dea6a7256bf9212c81361a9,
name: 'xxxx',
author: 'rahul123',
isPublished: true,
__v: 0
}
}
As you can see in last section , I am seeing my saved data in last section.
but why i am seeing these many extra attributes. I have tested the same code with Mongo DB version 3.6.16 and 4.2 .
How can I get rid of these fields.

I had the same issue. The clean way to fix this is to add toObject() in the console statement as below:
(res=>console.log(res.toObject()))
This should remove the extra unwanted fields.

Try:
console.log(JSON.stringify(res));

Can't retrieve data from mongoDB with Node.js

I'm pretty new in Node.js and I have this project. Basically I have some data sits in a mongoDB collection("data") and I'm trying to get that data and display it on the browser.
Here is what I've got so far;
var MongoClient = require('mongodb').MongoClient
,format = require('util').format;
var sys = require ("sys");
my_http = require("http");
my_http.createServer(function(request, response){
sys.puts("Touched !!");
response.writeHeader(200, {"Content-Type": "text/plain"});
response.write(extractData()).toString();
response.end();
}).listen(8080);
sys.puts("Server is running on 8080"); // Server kicks in...np
function extractData(){
MongoClient.connect('mongodb://127.0.0.1:27017/mongoDB', function(err, db){
if (err){
console.log("Can't Connect to DB !!");
}else {
console.log("Connected to DB !!"); // connects to DB, np
db.data.find({}, function(err, data){ // .find is the problem
if (err || !data) console.log("No Data Found");
else data.forEach(function (data){
console.log(data);
});
}).toArray();
}
});
}
And after I run "node server.js" and refresh the already open localhost:8080, I get this;
Server is running on 8080
Touched !!
Touched !!
Connected to DB !!
d:\Projects\SCRIPTS\mdp.scripts.testing-tools\jsFinderWmongoDB\node_modules\mongodb\lib\mongodb\mongo_client.js:475
throw err
^
TypeError: Cannot call method 'find' of undefined
at d:\Projects\SCRIPTS\mdp.scripts.testing-tools\jsFinderWmongoDB\server.js:21:21
at d:\Projects\SCRIPTS\mdp.scripts.testing-tools\jsFinderWmongoDB\node_modules\mongodb\lib\mongodb\mongo_client.js:4
72:11
at process._tickCallback (node.js:415:13)
Don't understand why there is a problem with .find() and of course can't display any data...
Any ideas?
Edit:
Well, we are certainly getting somewhere. I've made some changes.
Current code:
function extractData(){
MongoClient.connect('mongodb://127.0.0.1:27017/mongoDB', function(err, db){
if (err){
console.log("Can't Connect to DB !!");
}else {
sys.puts("Connected to DB !!"); // connects to DB, np
db.collection('data').find({}, function(err, data){
if (err || !data) console.log("No Data Found");
//else db.collection('data').forEach(function (data){
// console.log(data);
//});
});//.toArray();
}
});
}
Browser response is "undefined"
I'm guessing "extractData" function is NOT returning something legit. Therefore the collection set "data" is returning empty.
And yes I've checked one more time, I have data in the dataset.

You need to set a collection to use before you can do anything with it.
var collection = db.collection('data');
collection.find({},function(err,data){
console.log(data);
});
Would be how you do it.
Update ** This was how I did my first mongoDB stuff using express
Might help you.
var mongo = require('mongodb');
var monk = require('monk');
var db = monk('localhost:27017/nodetest1');
app.get('/userlist', function(req, res) {
var db = req.db;
var collection = db.get('usercollection');
collection.find({},{},function(e,docs){
res.render('userlist', {
"userlist" : docs
});
});
});

I get the details about the stored data when do console.log(data);
Here is a glimpse of it
Server is running on 8080
Touched !!
Connected to DB !!
{ db:
{ domain: null,
_events: {},
_maxListeners: 10,
databaseName: 'mongoDB',
serverConfig:
{ domain: null,
_events: {},
_maxListeners: 10,
auth: [Getter],
_callBackStore: [Object],
_commandsStore: [Object],
_dbStore: [Object],
host: '127.0.0.1',
port: 27017,
options: [Object],
internalMaster: true,
connected: true,
poolSize: 5,
disableDriverBSONSizeCheck: false,
_used: true,
replicasetInstance: null,
emitOpen: false,
ssl: false,
sslValidate: false,
sslCA: null,
sslCert: undefined,
sslKey: undefined,
sslPass: undefined,
serverCapabilities: [Object],
name: '127.0.0.1:27017',
socketOptions: [Object],
logger: [Object],
eventHandlers: [Object],
_serverState: 'connected',
_state: [Object],
recordQueryStats: false,
socketTimeoutMS: [Getter/Setter],
_readPreference: [Object],
db: [Circular],
dbInstances: [Object],
connectionPool: [Object],
isMasterDoc: [Object] },
options:
{ read_preference_tags: null,
read_preference: 'primary',
url: 'mongodb://127.0.0.1:27017/mongoDB',
native_parser: true,
readPreference: [Object],
safe: false,
w: 1 },
_applicationClosed: false,
slaveOk: false,
bufferMaxEntries: -1,
native_parser: true,
bsonLib:
{ BSON: [Object],
Long: [Object],
ObjectID: [Object],
DBRef: [Object],
Code: [Object],
Timestamp: [Object],
Binary: [Object],
Double: [Object],
MaxKey: [Object],
MinKey: [Object],
Symbol: [Object] },
bson: { promoteLongs: true },
bson_deserializer:
{ Code: [Object],
Symbol: [Object],
BSON: [Object],
DBRef: [Object],
Binary: [Object],
ObjectID: [Object],
Long: [Object],
Timestamp: [Object],
Double: [Object],
MinKey: [Object],
MaxKey: [Object],
promoteLongs: true },
bson_serializer:
{ Code: [Object],
Symbol: [Object],
BSON: [Object],
DBRef: [Object],
Binary: [Object],
ObjectID: [Object],
Long: [Object],
Timestamp: [Object],
Double: [Object],
MinKey: [Object],
MaxKey: [Object],
promoteLongs: true },
_state: 'connected',
pkFactory:
{ [Function: ObjectID]
index: 16043018,
createPk: [Function: createPk],
createFromTime: [Function: createFromTime],
createFromHexString: [Function: createFromHexString],
isValid: [Function: isValid],
ObjectID: [Circular],
ObjectId: [Circular] },
forceServerObjectId: false,
safe: false,
notReplied: {},
isInitializing: true,
openCalled: true,
commands: [],
logger: { error: [Function], log: [Function], debug: [Function] },
tag: 1425061857066,
eventHandlers:
{ error: [],
parseError: [],
poolReady: [],
message: [],
close: [] },
serializeFunctions: false,
raw: false,
recordQueryStats: false,
retryMiliSeconds: 1000,
numberOfRetries: 60,
readPreference: { _type: 'ReadPreference', mode: 'primary', tags: undefined } },
collection:

long poll with node.js and express: how to cache res object into redis or other cache framework

I try to build a chat room webpage with node.js/express/redis on the server side.
Following this snips of code : A Message Wall With Long Poll Properties in Node.JS and Express , I succeed to make one node server running correctly. In this example, the res objects are saved in a list, no any transformation is needed.
But I want to run the node app with pm2 cluster mode(-i 4), so I have to save res object into some place that shared between 4 nodes.
I already used redis in my node project, for the express.session. So I want to cache res into redis.
But the problem occured: when I try to sting-lized res object with JSON.stringify(res), I got :
TypeError: Converting circular structure to JSON
My problem is:
how could I save one res object for later use across node cluster, with redis or something else.
Appreciate.
I use util.inspect to print out my container of res (chatroom_id:res) object:
{ '0': null,
'1390640136999':
{ domain: null,
_events:
{ finish: [Object],
header: [Function],
close: [Function: logRequest] },
_maxListeners: 10,
output: [],
outputEncodings: [],
writable: true,
_last: false,
chunkedEncoding: false,
shouldKeepAlive: false,
useChunkedEncodingByDefault: false,
sendDate: true,
_headerSent: false,
_header: '',
_hasBody: true,
_trailer: '',
finished: false,
_hangupClose: false,
socket:
{ _connecting: false,
_handle: [Object],
_readableState: [Object],
readable: true,
domain: null,
_events: [Object],
_maxListeners: 10,
_writableState: [Object],
writable: true,
allowHalfOpen: true,
onend: [Function],
destroyed: false,
errorEmitted: false,
bytesRead: 894,
_bytesDispatched: 0,
_pendingData: null,
_pendingEncoding: '',
server: [Object],
_idleTimeout: 120000,
_idleNext: [Object],
_idlePrev: [Object],
_idleStart: 1390640145289,
parser: [Object],
ondata: [Function],
_paused: false,
_httpMessage: [Circular],
_peername: [Object] },
connection:
{ _connecting: false,
_handle: [Object],
_readableState: [Object],
readable: true,
domain: null,
_events: [Object],
_maxListeners: 10,
_writableState: [Object],
writable: true,
allowHalfOpen: true,
onend: [Function],
destroyed: false,
errorEmitted: false,
bytesRead: 894,
_bytesDispatched: 0,
_pendingData: null,
_pendingEncoding: '',
server: [Object],
_idleTimeout: 120000,
_idleNext: [Object],
_idlePrev: [Object],
_idleStart: 1390640145289,
parser: [Object],
ondata: [Function],
_paused: false,
_httpMessage: [Circular],
_peername: [Object] },
_headers: { 'x-powered-by': 'Express' },
_headerNames: { 'x-powered-by': 'X-Powered-By' },
req:
{ _readableState: [Object],
readable: true,
domain: null,
_events: {},
_maxListeners: 10,
socket: [Object],
connection: [Object],
httpVersion: '1.0',
complete: true,
headers: [Object],
trailers: {},
_pendings: [],
_pendingIndex: 0,
url: '/robot/chat/query/99/1390640136999/270125/',
method: 'GET',
statusCode: null,
client: [Object],
_consuming: false,
_dumped: false,
httpVersionMajor: 1,
httpVersionMinor: 0,
upgrade: false,
originalUrl: '/robot/chat/query/99/1390640136999/270125/',
_parsedUrl: [Object],
query: {},
res: [Circular],
next: [Function: next],
secret: undefined,
cookies: [Object],
signedCookies: {},
sessionStore: [Object],
sessionID: '4PACUldyCHhT8NgdGY1yz9Pk',
session: [Object],
_startTime: Sat Jan 25 2014 16:55:45 GMT+0800 (CST),
_remoteAddress: '127.0.0.1',
body: {},
originalMethod: 'GET',
_route_index: 2,
route: [Object],
params: [Object] },
locals: [Function: locals],
end: [Function],
student_id: '99',
channel_id: '1390640136999',
last_msg_id: '270125' } }
There are three [Circular].
My pseudocode:
/*
* query from http client
* url: /chat/query/:student_id/:channel_id/:last_msg_id/
*/
exports.query = function(req, res){
// if find some new msg
// return them as json, immediately
// else
// set participator info into res object
// read the res_list from redis
// put this res into res_list
// write back res_list into redis
};
/*
* notification from other web server: one new msg been created
* url: /chat/notify/:new_msg_id/
*/
exports.notify = function(req, res){
// get new_msg from database by id
// read the res_list from redis
// for old_res in res_list
// if this old_res is releated with the new_msg (participator)
// old_res.sent(json_content)
// remove this old_res from res_list
// write back res_list into redis
};
how could i implement those 'read-and-write-back' part?

The short answer: it's not possible to put the res object into a cache and obtain it again from another process, in any meaningful way.
If you think about how long poll works, each HTTP client maintains an open connection to the server, waiting (a long time) for something to be sent back. In other words, when you come to push out new messages, you're just sending data down an already open connection. If you cache your res into redis, what would happen to the connection?
It may not matter though, as long as you have a way to pass the content between your 4 backend processes, they can each update their own set of res connections. New connections would still be load balanced.