We Keep Coding

Cloud function fails after changing Runtime to Node.js 10

I am trying to take firebase backup through a cloud function. The function was running completely fine when I was using Runtime: Node.js 8. However, since it is going to be deprecated soon, I now have to use Node.js 10. My fcloud function now fails with the below error:
Error: function execution failed. Details:
Cannot read property 'charCodeAt' of undefined
Detailed error log:
2020-05-27 11:01:21.820 IST
firestore_export
8kxlp9s867dy
TypeError: Cannot read property 'charCodeAt' of undefined at peg$parsetemplate (/workspace/node_modules/google-gax/build/src/pathTemplateParser.js:304:17) at Object.peg$parse [as parse] (/workspace/node_modules/google-gax/build/src/pathTemplateParser.js:633:18) at new PathTemplate (/workspace/node_modules/google-gax/build/src/pathTemplate.js:55:54) at segments.forEach.segment (/workspace/node_modules/google-gax/build/src/pathTemplate.js:120:29) at Array.forEach (<anonymous>) at PathTemplate.render (/workspace/node_modules/google-gax/build/src/pathTemplate.js:114:23) at FirestoreAdminClient.databasePath (/workspace/node_modules/#google-cloud/firestore/build/src/v1/firestore_admin_client.js:904:57) at exports.scheduledFirestoreBackup (/workspace/index.js:6:31) at Promise.resolve.then (/layers/google.nodejs.functions-framework/functions-framework/node_modules/#google-cloud/functions-framework/build/src/invoker.js:330:28) at process._tickCallback (internal/process/next_tick.js:68:7)
Expand all | Collapse all
{
insertId: "000000-f688386c-8f2b-4146-aaf7-1fe67c656fa2"
labels: {…}
logName: "projects/firestore-249705/logs/cloudfunctions.googleapis.com%2Fcloud-functions"
receiveTimestamp: "2020-05-27T05:31:31.171084310Z"
resource: {…}
severity: "ERROR"
textPayload: "TypeError: Cannot read property 'charCodeAt' of undefined
at peg$parsetemplate (/workspace/node_modules/google-gax/build/src/pathTemplateParser.js:304:17)
at Object.peg$parse [as parse] (/workspace/node_modules/google-gax/build/src/pathTemplateParser.js:633:18)
at new PathTemplate (/workspace/node_modules/google-gax/build/src/pathTemplate.js:55:54)
at segments.forEach.segment (/workspace/node_modules/google-gax/build/src/pathTemplate.js:120:29)
at Array.forEach (<anonymous>)
at PathTemplate.render (/workspace/node_modules/google-gax/build/src/pathTemplate.js:114:23)
at FirestoreAdminClient.databasePath (/workspace/node_modules/#google-cloud/firestore/build/src/v1/firestore_admin_client.js:904:57)
at exports.scheduledFirestoreBackup (/workspace/index.js:6:31)
at Promise.resolve.then (/layers/google.nodejs.functions-framework/functions-framework/node_modules/#google-cloud/functions-framework/build/src/invoker.js:330:28)
at process._tickCallback (internal/process/next_tick.js:68:7)"
timestamp: "2020-05-27T05:31:21.820Z"
trace: "projects/firestore-249705/traces/74e27700d135763bc4e7892ebb1a2333"
}
My index.js is as below:
const firestore = require('#google-cloud/firestore');
const client = new firestore.v1.FirestoreAdminClient();
// Replace BUCKET_NAME
const bucket = 'gs://gcp_firestore_ae2/firestore_export'
exports.scheduledFirestoreBackup = (event, context) => {
const databaseName = client.databasePath(
process.env.GCLOUD_PROJECT,
'(default)'
);
return client
.exportDocuments({
name: databaseName,
outputUriPrefix: bucket,
// Leave collectionIds empty to export all collections
// or define a list of collection IDs:
// collectionIds: ['users', 'posts']
collectionIds: ['most_valuable_items','nric_img','pay_slip','pic_of_ofc_entrance'],
})
.then(responses => {
const response = responses[0];
console.log(`Operation Name: ${response['name']}`);
return response;
})
.catch(err => {
console.error(err);
});
};

I had the same issue. I fixed it by changing:
process.env.GCLOUD_PROJECT
to my actual Project ID (e.g. "my_app_37274")

I had the same issue, It seems like in previous versions we don't need to GCLOUD_PROJECT in environment variable i.e it automatically detects it, but from node 10 onwards we need to pass this explicitly.
That's how I resolved it.
So instead of hardcoding it, try passing GCLOUD_PROJECT in cloud functions environment variables.
Note: GCLOUD_PROJECT is project id, not the project name.

How to send request with bulk? Elasticsearch

I have an object with data, and i want to send this data to elasticsearch container
for(let key in params)
{
bulk.push(JSON.stringify({
index: {
_id: params[ key ][ 'id' ],
_type: 'id',
_index: 'geo'
}
}));
bulk.push(JSON.stringify(params[key]));
}
let bulks = bulk.join("\n") + "\n";
I made request
let cat = request(
{
'method' : 'PUT',
'uri' : 'http://dev4.int10h.net:40024/_bulk',
'body' : bulks ,
'json' : true,
'headers':
[
'Content-Type: application/x-ndjson'
],
'agent' : false
}
);
but has error
Unhandled rejection StatusCodeError: 400 - {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The bulk request must be terminated by a newline [\n]"}],"type":"illegal_argument_exception","reason":"The bulk request must be terminated by a newline [\n]"},"status":400}
at new StatusCodeError (/usr/lib/node_modules/request-promise/node_modules/request-promise-core/lib/errors.js:32:15)
at Request.plumbing.callback (/usr/lib/node_modules/request-promise/node_modules/request-promise-core/lib/plumbing.js:104:33)
at Request.RP$callback [as _callback] (/usr/lib/node_modules/request-promise/node_modules/request-promise-core/lib/plumbing.js:46:31)
at Request.self.callback (/usr/lib/node_modules/request/request.js:185:22)
at Request.emit (events.js:182:13)
at Request. (/usr/lib/node_modules/request/request.js:1161:10)
at Request.emit (events.js:182:13)
at IncomingMessage. (/usr/lib/node_modules/request/request.js:1083:12)
at Object.onceWrapper (events.js:273:13)
at IncomingMessage.emit (events.js:187:15)
at endReadableNT (_stream_readable.js:1098:12)
at process.internalTickCallback (internal/process/next_tick.js:72:19)
How to properly send bulk?
bulks type string

Not sure if this helps you but here I found similar problem with _bulk and JSON.stringify.
The answer is:
It looks like the meta characters in your payload does not get
translated into newlines. If you instead used the
elasticsearch.js client, it would handle this for you.

MongoDB oplog stream Node.js

I have a local instance of MongoDB, started with --master, and I can see in the start log that I have a replica set oplog (file: /data/db/local.1) created. And it appears local.1 is updated every time I perform and insert().
I am trying to tail / stream the oplog with the following code...
MongoDB.connect('mongodb://localhost:27017/test',(e,db) => {
if ( e ) { return console.log('Connect MongoDB Error',e); }
console.log('Connected MongoDB');
Server.DB = db;
db.collection('oplog.rs').find({}).toArray((e,d) => {
console.log('oplog.rs');
console.log(e);
console.log(d);
})
var updates = db.collection('oplog.rs').find({},{
tailable: true,
awaitData: true,
noCursorTimeout: true,
oplogReplay: true,
numberOfRetries: Number.MAX_VALUE
}).stream();
updates.on('data',(data) => { console.log('data',data); });
updates.on('error',(e) => { console.log('error',e); });
updates.on('end',(end) => { console.log('end',end); });
});
The Code logs the following on start...
oplog.rs
null
[]
However there is no output from stream, I set numberOfRetries to a lower value I get the error..
error { MongoError: No more documents in tailed cursor
at Function.MongoError.create (/Users/peter/node_modules/mongodb-core/lib/error.js:31:11)
at nextFunction (/Users/peter/node_modules/mongodb-core/lib/cursor.js:637:50)
at /Users/peter/node_modules/mongodb-core/lib/cursor.js:588:7
at queryCallback (/Users/peter/node_modules/mongodb-core/lib/cursor.js:211:18)
at Callbacks.emit (/Users/peter/node_modules/mongodb-core/lib/topologies/server.js:116:3)
at Connection.messageHandler (/Users/peter/node_modules/mongodb-core/lib/topologies/server.js:282:23)
at Socket.<anonymous> (/Users/peter/node_modules/mongodb-core/lib/connection/connection.js:273:22)
at emitOne (events.js:96:13)
at Socket.emit (events.js:189:7)
at readableAddChunk (_stream_readable.js:176:18)
name: 'MongoError',
message: 'No more documents in tailed cursor',
tailable: true,
awaitData: true }
end undefined

oplog.rs is used in replica sets. Starting mongodb with --master you are using long-deprecated master-slave replication, which uses local.oplog.$main database, so there is nothings in oplog.rs.
You need to start mongod with --replSet to benefit from replica set oplog. Read more how to deploy and configure replica sets.

Banned words filter (tmi.js)

So I've been trying to make this work for about 3 hours now, and I' almost giving up.
My current code is:
var tmi = require('tmi.js');
var linksDisallowed = ["http://", "https://", ".dk"];
var options = {
options: {
debug: true
},
connection: {
cluster: "aws",
reconnect: true
},
identity: {
username: "botstormen",
password: "**CENSORED FOR SECURITY REASONS**"
},
channels: ["dunkstormen"]
};
var client = new tmi.client(options);
client.connect();
client.on('connected', function(adress, port) {
client.action("dunkstormen", "joinede chatten og er klar til at hjælpe! :3");
});
client.on('chat', function(channel, user, message, self) {
if(message === "!social" || message === "!Social") {
client.action("dunkstormen", "Twitter: twitter.com/dunkstormen Facebook: facebook.com/dunkstormen");
}
});
client.on('chat', function(channel, user, message, self) {
for (var i = 0; i < 3; i++) {
if(message.indexOf(linksDisallowed[i]) >= 0) {
client.say("Hello, " + user);
}
}
});
But whenever I type one of the word in the array linksDisallowed into the chat the bot just instantly crashes with the following error:
C:\Users\Benjamin Jørgensen\Desktop\botstormen\node_modules\tmi.js\lib\commands.js:207
if (message.toLowerCase().startsWith("/me ") || message.toLowerCase().startsWith("\\me ")) {
^
TypeError: Cannot read property 'toLowerCase' of undefined
at client.say (C:\Users\Benjamin Jørgensen\Desktop\botstormen\node_modules\tmi.js\lib\commands.js:207:20)
at client.<anonymous> (C:\Users\Benjamin Jørgensen\Desktop\botstormen\app.js:37:11)
at client.EventEmitter.emit (C:\Users\Benjamin Jørgensen\Desktop\botstormen\node_modules\tmi.js\lib\events.js:99:50)
at client.handleMessage (C:\Users\Benjamin Jørgensen\Desktop\botstormen\node_modules\tmi.js\lib\client.js:792:34)
at C:\Users\Benjamin Jørgensen\Desktop\botstormen\node_modules\tmi.js\lib\client.js:919:18
at Array.forEach (native)
at client._onMessage (C:\Users\Benjamin Jørgensen\Desktop\botstormen\node_modules\tmi.js\lib\client.js:917:11)
at WebSocket.onMessage (C:\Users\Benjamin Jørgensen\Desktop\botstormen\node_modules\tmi.js\node_modules\ws\lib\WebSocket.js:442:14)
at emitTwo (events.js:87:13)
at WebSocket.emit (events.js:172:7)

tmi.js command say takes 2 arguments, you are passing only one.
https://docs.tmijs.org/v0.0.29/Commands.html#say
Change client.say("Hello, " + user); to client.say(channel,"Hello, " + user);
EDIT to help about comment:
According to docs, https://docs.tmijs.org/v0.0.29/Events.html#chat the chat event returns a user object. There is the description of a user object just below with a notice about it can change... But you can certainly do this for now: replace user by user.username .
If it does not work, inspect the user object to find the right property you should use.

That would mean your app.js isn't sending a message into the tmi.js service

Sequelize TypeError: Cannot read property '1' of null

Normally this type of error would not be a problem but i simply cannot understand where this is happening:
Here is my setup:
router.route('/api/academyModule')
.post(function (req, res) {
req.body.academyModule.module_id = req.body.academyModule.module.id;
req.body.academyModule.module_module_type_id = req.body.academyModule.module.module_type.id;
var am = AcademyModule.build(req.body.academyModule);
am.add(req.body.academyModule.requirements, req.body.academyModule, function (success) {
res.json({id: this[null]});
},
function (err) {
res.status(err).send(err);
});
if(req.body.teams != null)
{
req.body.teams.forEach(function(y)
{
var atm = academy_team_has_academy_module.build({academy_team_id: y.id, academy_id: y.academy_id, academy_module_module_id: req.body.academyModule.module_id })
atm.add(function(success)
{
}, function(err)
{
res.status(err).send(err);
});
});
}
})
For this i have the following model:
academy_team_has_academy_module = sequelize.define('academy_team_has_academy_module', {
academy_team_id: {
type: DataTypes.INTEGER,
primaryKey: true,
autoIncrement: false
},
academy_id: DataTypes.INTEGER,
academy_module_module_id: DataTypes.INTEGER
}, {
freezeTableName: true,
instanceMethods: {
add: function (onSuccess, onError) {
academy_team_has_academy_module.build(this.dataValues)
.save().ok(onSuccess).error(onError);
}
}
});
i know for sure this happens in this model and not AcademyModule because when i remove this code it runs without any issues. So in my console i get the following print out:
Executing (default): INSERT INTO `requirements` (`id`,`value`,`requirement_type_id`) VALUES (DEFAULT,5,'2');
Executing (default): INSERT INTO `academy_team_has_academy_module` (`academy_team_id`,`academy_id`,`academy_module_module_id`) VALUES (1,3,11);
Executing (default): INSERT INTO `academy_module` (`academy_id`,`module_id`,`module_module_type_id`,`sort_number`,`requirements_id`) VALUES ('3',11,4,4,40);
And just right after i get:
/var/www/learningbankapi/src/node_modules/mysql/lib/protocol/Parser.js:82
throw err;
^
TypeError: Cannot read property '1' of null
at module.exports.Query.formatError (/var/www/learningbankapi/src/node_modules/sequelize/lib/dialects/mysql/query.js:155:23)
at Query.module.exports.Query.run [as _callback] (/var/www/learningbankapi/src/node_modules/sequelize/lib/dialects/mysql/query.js:38:23)
at Query.Sequence.end (/var/www/learningbankapi/src/node_modules/mysql/lib/protocol/sequences/Sequence.js:96:24)
at Query.ErrorPacket (/var/www/learningbankapi/src/node_modules/mysql/lib/protocol/sequences/Query.js:93:8)
at Protocol._parsePacket (/var/www/learningbankapi/src/node_modules/mysql/lib/protocol/Protocol.js:271:23)
at Parser.write (/var/www/learningbankapi/src/node_modules/mysql/lib/protocol/Parser.js:77:12)
at Protocol.write (/var/www/learningbankapi/src/node_modules/mysql/lib/protocol/Protocol.js:39:16)
at Socket.Connection.connect (/var/www/learningbankapi/src/node_modules/mysql/lib/Connection.js:82:28)
at Socket.EventEmitter.emit (events.js:95:17)
at Socket.stream.pause.paused (_stream_readable.js:746:14)
at Socket.EventEmitter.emit (events.js:92:17)
at emitReadable_ (_stream_readable.js:408:10)
at emitReadable (_stream_readable.js:404:5)
at readableAddChunk (_stream_readable.js:165:9)
at Socket.Readable.push (_stream_readable.js:127:10)
at TCP.onread (net.js:526:21)
Ive debugged the whole thing and i can't seem to find any undefined variables. What might have gone wrong here?

Your problem is a ForeignKeyContraintError. Sequelize is trying to throw a new error but get's an unexpected error.message. If your configuration is fine, this may be an bug of sequelize.
Maybe deactivating foreign contraint checks (if possible) will solve your problem.
See: https://github.com/sequelize/sequelize/blob/master/lib/dialects/mysql/query.js#L153