We Keep Coding

Using Transactions and batched writes within a Cloud Function

Transactions and batched writes can be used to write multiple documents by means of an atomic operation.
Documentation says that Using the Cloud Firestore client libraries, you can group multiple operations into a single transaction.
I cannot understand what is the meaning of client libraries here and if it's correct to use Transactions and batched writes within a Cloud Function.
Example given: suppose in the database I have 3 elements (which doc IDs are A, B, C). Now I need to insert 3 more elements (which doc IDs are C, D, E). The Cloud Function should add just the latest ones and send a Push Notification to the user telling him that 2 new documents are available.
The doc ID could be the same but since I need to calculate how many documents are new (the ones that will be inserted) I need a way to read the doc ID first and check for its existence. Hence, I'm wondering if Transactions fit Cloud Functions or not.
Also, each transaction or batch of writes can write to a maximum of 500 documents. Is there any other way to overcome this limit within a Cloud Function?

Firestore Transaction behaviour is different between the Clients SDKs (JS SDK, iOS SDK, Android SDK , ...) and the Admin SDK (a set of server libraries), which is the SDK we use in a Cloud Function. More explanations on the differences here in the documentation.
Because of the type of data contention used in the Admin SDK you can, with the getAll() method, retrieve multiple documents from Firestore and hold a pessimistic lock on all returned documents.
So this is exactly the method you need to call in your transaction: you use getAll() for fetching documents C, D & E and you detect that only C is existing so you know that you need to only add D and E.
Concretely, it could be something along the following lines:
const db = admin.firestore();
exports.lorenzoFunction = functions
.region('europe-west1')
.firestore
.document('tempo/{docId}') //Just a way to trigger the test Cloud Function!!
.onCreate(async (snap, context) => {
const c = db.doc('coltest/C');
const d = db.doc('coltest/D');
const e = db.doc('coltest/E');
const docRefsArray = [c, d, e]
return db.runTransaction(transaction => {
return transaction.getAll(...docRefsArray).then(snapsArray => {
let counter = 0;
snapsArray.forEach(snap => {
if (!snap.exists) {
counter++;
transaction.set(snap.ref, { foo: "bar" });
} else {
console.log(snap.id + " exists")
}
});
console.log(counter);
return;
});
});
});
To test it: Create one of the C, D or E doc in the coltest collection, then create a doc in the tempo collection (Just a simple way to trigger this test Cloud Function): the CF is triggered. Then look at the coltest collection: the two missing docs were created; and look a the CF log: counter = 2.
Also, each transaction or batch of writes can write to a maximum of
500 documents. Is there any other way to overcome this limit within a
Cloud Function?
AFAIK the answer is no.

There used to also be a one second delay required as well between 500 record chunks. I wrote this a couple of years ago. The script below reads the CSV file line by line, creating and setting a new batch object for each line. A counter creates a new batch write per 500 objects and finally asynch/await is used to rate limit the writes to 1 per second. Last, we notify the user of the write progress with console logging. I had published an article on this here >> https://hightekk.com/articles/firebase-admin-sdk-bulk-import
NOTE: In my case I am reading a huge flat text file (a manufacturers part number catalog) for import. You can use this as a working template though and modify to suit your data source. Also, you may need to increase the memory allocated to node for this to run:
node --max_old_space_size=8000 app.js
The script looks like:
var admin = require("firebase-admin");
var serviceAccount = require("./your-firebase-project-service-account-key.json");
var fs = require('fs');
var csvFile = "./my-huge-file.csv"
var parse = require('csv-parse');
require('should');
admin.initializeApp({
credential: admin.credential.cert(serviceAccount),
databaseURL: "https://your-project.firebaseio.com"
});
var firestore = admin.firestore();
var thisRef;
var obj = {};
var counter = 0;
var commitCounter = 0;
var batches = [];
batches[commitCounter] = firestore.batch();
fs.createReadStream(csvFile).pipe(
parse({delimiter: '|',relax_column_count:true,quote: ''})
).on('data', function(csvrow) {
if(counter <= 498){
if(csvrow[1]){
obj.family = csvrow[1];
}
if(csvrow[2]){
obj.series = csvrow[2];
}
if(csvrow[3]){
obj.sku = csvrow[3];
}
if(csvrow[4]){
obj.description = csvrow[4];
}
if(csvrow[6]){
obj.price = csvrow[6];
}
thisRef = firestore.collection("your-collection-name").doc();
batches[commitCounter].set(thisRef, obj);
counter = counter + 1;
} else {
counter = 0;
commitCounter = commitCounter + 1;
batches[commitCounter] = firestore.batch();
}
}).on('end',function() {
writeToDb(batches);
});
function oneSecond() {
return new Promise(resolve => {
setTimeout(() => {
resolve('resolved');
}, 1010);
});
}
async function writeToDb(arr) {
console.log("beginning write");
for (var i = 0; i < arr.length; i++) {
await oneSecond();
arr[i].commit().then(function () {
console.log("wrote batch " + i);
});
}
console.log("done.");
}

How do I Collect User IDs + Retrieve Corresponding Tokens + Send a Push Notification Via Firebase Cloud Function (JS)

The Problem:
I have been unable to use Firebase (Google) Cloud Functions to collect and utilize device tokens for the cloud messaging feature.
Context:
I am a self-taught android-Java developer and have no JavaScript experience. Despite that, I believe I have code that should work and am not sure what the problem is. To my understanding, it could be one of three things:
Somehow my Firebase Realtime Database references are being called incorrectly and I am not retrieving data as expected.
I may need to use Promises to wait for all calls to be made before proceeding, however I don't really understand how I would incorporate that into the code I have.
I may be using multiple return statements incorrectly (which I am also fuzzy on).
My error message on the Firebase Realtime Database console is as follows:
#firebase/database: FIREBASE WARNING: Exception was thrown by user callback. Error: Registration token(s) provided to sendToDevice() must be a non-empty string or a non-empty array.
at FirebaseMessagingError.FirebaseError [as constructor] (/srv/node_modules/firebase-admin/lib/utils/error.js:42:28)
at FirebaseMessagingError.PrefixedFirebaseError [as constructor] (/srv/node_modules/firebase-admin/lib/utils/error.js:88:28)
at new FirebaseMessagingError (/srv/node_modules/firebase-admin/lib/utils/error.js:254:16)
at Messaging.validateRegistrationTokensType (/srv/node_modules/firebase-admin/lib/messaging/messaging.js:729:19)
at Messaging.sendToDevice (/srv/node_modules/firebase-admin/lib/messaging/messaging.js:328:14)
at admin.database.ref.once.snapshot (/srv/index.js:84:12)
at onceCallback (/srv/node_modules/#firebase/database/dist/index.node.cjs.js:4933:51)
at /srv/node_modules/#firebase/database/dist/index.node.cjs.js:4549:22
at exceptionGuard (/srv/node_modules/#firebase/database/dist/index.node.cjs.js:698:9)
at EventList.raise (/srv/node_modules/#firebase/database/dist/index.node.cjs.js:9684:17)
The above indicates I am not retrieving data either at all or by the time the return is called. My JavaScript function code is:
'use strict';
const admin = require('firebase-admin');
admin.initializeApp();
exports.pushNotification = functions.database.ref('/Chat Messages/{chatId}/{pushID}').onCreate((snapshot, context) => {
const valueObject = snapshot.after.val();
return admin.database().ref(`/Chat Basics/${valueObject.chatKey}/Chat Users`).once('value', statusSnapshot => {
var index = 0;
var totalkeys = statusSnapshot.numChildren();
var msgIDs = [];
statusSnapshot.forEach(msg=>{
msgIDs.push(msg.key.toString());
if(index === totalkeys - 1){
const payload = {
notification : {
title: valueObject.userName,
body: valueObject.message,
sound: "default"
}
}
sendNotificationPayload(valueObject.uid, payload);
}
index++;
});
});
});
function sendNotificationPayload(uid, payload){
admin.database()
.ref(`/User Token Data/${uid}`)
.once('value', snapshot=> {
var tokens = [];
//if(!snapshot.exists())return;
snapshot.forEach(item =>{
tokens.push(item.val())
});
admin.messaging()
.sendToDevice(tokens, payload)
.then(res => {
return console.log('Notification sent')
})
.catch(err => {
return console.log('Error in sending notification = '+err)
});
});
}
This code is mostly inspired by what was said to be a working example here from another Stack Overflow question here. I have successfully tested sending a notification to a single device by manually copying a device token into my function, so the function does run to completion. My Java code seems to be irrelevant to the problem, so I have not added it (please ask in the comments if you would like it added for further context).
What I Have Tried:
I have tried implementing promises into my code, but I don't think I was doing it properly. My main reference for this was here. I have also looked at the documentation for literally everything related to this topic, however my knowledge of JS is not sufficient to really apply barebones examples to my code.
My Firebase Realtime Database Nodes:
#1: Loop through chat members to collect user IDs:
"Chat Basics" : {
"1607801501690_TQY41wIfArhHDxEisyupZxwyHya2" : {
"Chat Users" : {
"JXrclZuu1aOwEpCe6KW8vSDea9h2" : true,
"TQY41wIfArhHDxEisyupZxwyHya2" : true
},
#2: Collect user tokens from collected IDs (ignore that tokens are matching):
"User Token Data" : {
"JXrclZuu1aOwEpCe6KW8vSDea9h2" : "duDR3KH3i3I:APA91bH_LCeslZlqL8akYw-LrM9Dv__nx4nU1TquCS0j6bGF1tlIARcheREuNdX1FheC92eelatBC8LO4t6gt8liRdFHV-NDuNLa13oHYxKgl3JBPPlrMo5rB5XhH7viTo4vfYOMftRi",
"TQY41wIfArhHDxEisyupZxwyHya2" : "duDR3KH3i3I:APA91bH_LCeslZlqL8akYw-LrM9Dv__nx4nU1TquCS0j6bGF1tlIARcheREuNdX1FheC92eelatBC8LO4t6gt8liRdFHV-NDuNLa13oHYxKgl3JBPPlrMo5rB5XhH7viTo4vfYOMftRi"
}
Conclusion:
Concrete examples would be much appreciated, especially since I am crunching right now. Thanks for your time and help!
Update:
After some more testing, it looks like the problem is definitely due to my lack of understanding of promises in two areas. Firstly, only one user is collected before the final return is called. Secondly, the final return is called before the 2nd forEach() loop can store snapshot data to an array.
For this code then, how may I modify (or rebuild) it so that it collects all keys before proceeding to retrieve token data from all keys - ultimately before returning the notification?

Just as with every question I post, I managed to figure out how to do it (tentatively) a few hours later. Below is a full example of how to send a notification to chat users based on a message sent (although it does not yet exclude the sender) to a given chat. The order of operations are as such:
User message is saved and triggers event. Relevant data the message contains are:
username, chat key, message
These are retrieved, with (username + message) as the (title + body) of the
notification respectively, and the chat key is used for user id reference.
Loop through chat user keys + collect.
Loop through array of chat user keys to collect array of device tokens.
Send notification when complete.
The code:
//Use firebase functions:log to see log
exports.pushNotification = functions.database.ref('/Chat Messages/{chatId}/{pushId}').onWrite((change, context) => {
const valueObject = change.after.val();
return admin.database().ref(`/Chat Basics/${valueObject.chatKey}/Chat Users`).once('value', statusSnapshot => {
var index = 0;
var totalkeys = statusSnapshot.numChildren();
var msgIDs = [];
statusSnapshot.forEach(msg=>{
msgIDs.push(msg.key.toString());
if(index === totalkeys - 1){
const payload = {
notification : {
title: valueObject.userName,
body: valueObject.message,
sound: "default"
}
}
let promises = [];
var tokens = [];
for(let i=0; i < msgIDs.length; i++){
let userId = msgIDs[i];
let promise = admin.database().ref(`/User Token Data/${userId}`).once('value', snapshot=> {
tokens.push(snapshot.val());
})
promises.push(promise);
}
return Promise.all(promises).then(() => {
return admin.messaging().sendToDevice(tokens, payload);
});
}
index++;
return false;
});
});
});

Firebase Auth: Edit UID

Is it possible to to change a user's UID in Firebase programmatically? There can't seem to be a way to do so manually within Firebase's console.

TL;DR: If you need to specify the UID, you'll need to create a new user with that UID.
You can't directly change the UID, but I was able to hack something together using the firebase admin API (docs)
My use case was that I needed to change a user's email address. I tried update email with "Update a User", but this actually ended up changing the UID under the hood. In my app, the UID is tied to so much stuff, that I'd have to do a huge architecture change, so this wasn't an option.
The general way I did this with the API was:
Pull Down a user using admin.auth().getUserByEmail
Delete the user with admin.auth().deleteUser
Create a new user with admin.auth().createUser, using relevant data from the getUserByEmail call above, replacing the email address with the new email.
"reset password" in the firebase admin console (I think there's a way to do this programmatically too)
User gets an email to reset their password and they have a new account with their old UID.
Unlike admin.auth().updateUser, createUser actually lets you specify a UID.

Building on the answer by RoccoB, the below is a complete set of instructions for changing a user's UID:
Create a new folder, and run npm init with default values.
Run npm install firebase-admin.
Create a NodeJS script file (eg. UpdateUserUID.js), with this code:
let admin = require("firebase-admin");
// config
let email = "XXX";
let serviceAccountData = require("XXX.json");
let adminConfig = {
credential: admin.credential.cert(serviceAccountData),
databaseURL: "https://XXX.firebaseio.com",
};
let newUserOverrides = {
uid: "XXX",
};
Start();
async function Start() {
console.log("Initializing firebase. databaseURL:", adminConfig.databaseURL);
admin.initializeApp(adminConfig);
console.log("Starting update for user with email:", email);
let oldUser = await admin.auth().getUserByEmail(email);
console.log("Old user found:", oldUser);
await admin.auth().deleteUser(oldUser.uid);
console.log("Old user deleted.");
let dataToTransfer_keys = ["disabled", "displayName", "email", "emailVerified", "phoneNumber", "photoURL", "uid"];
let newUserData = {};
for (let key of dataToTransfer_keys) {
newUserData[key] = oldUser[key];
}
Object.assign(newUserData, newUserOverrides);
console.log("New user data ready: ", newUserData);
let newUser = await admin.auth().createUser(newUserData);
console.log("New user created: ", newUser);
}
Replace email and adminConfig.databaseURL with the correct values.
Replace newUserOverrides.uid with the desired new uid. (you can change some other fields too)
Generate/download a private key for your project's Firebase Admin service account: https://firebase.google.com/docs/admin/setup (can skip to the "Initialize the SDK" section)
Update the serviceAccountData variable's import to point to the key json-file from the previous step.
Run node ./UpdateUserUID.js.
If applicable (I didn't seem to need it), use the "reset password" option in the Firebase Admin Console, to have a password-reset email sent to the user, apparently completing the account update. (Perhaps I didn't need this step since I don't use the accounts/authentications for anything besides sign-in on my website...)

The UID of a user is controlled by the identity provider that creates that user. This means that you can't change the UID for any of the built-in providers.
But you can control the UID if you create a custom identity provider. Note that this is quite a bit more involved than changing something in the Firebase console. It requires you to write code that runs in a secure/trusted environment, such as a server you control, or Cloud Functions.

You can't, since is the main tree node of possibles more entries inside it, you can get it, modify and then put it inside the same UID (or create a new one) but you can have things inside, for example take this.
You create your main UID which will hold user data (name, phone, email etc) lets say the structure is this:
-9GJ02kdj2GKS55kg
-Name:
-Phone:
-Email:
so, you can get the main user UID 9GJ02kdj2GKS55kg with mAuth.getCurrentUser().getUid(); and then change it and set a new value inside 9GJ02kdj2GKS55kg, this new value should be the same UID you got but changed, and then inside your main UID you can still have the same structure
-9GJ02kdj2GKS55kg
-6GL02kZj2GKS55kN (this is your changed UID)
-Name:
-Phone:
-Email:
or you can get that changed UID and make a new child, and that will be your parent node with custom UID for the data.

Piggybacking on #Vinrynx's post.
I recently created a migration tool where I am migrating collections from 1 Firebase Project to another and it required that after I insert users to "users" collection I also create an authentication record with the same doc.id
Variables in the functions below:
outCollData : Data that I am inserting for the user (contains the email inside it)
sourceDBApp : output of the admin.initializeApp({/*service-account.json file location for source firebase project */});
destDBApp : output of the admin.initializeApp({/*service-account.json file location for destination firebase project */});
async function updateUsersUID(
outCollData: any,
sourceDBApp: admin.app.App | undefined,
destDBApp: admin.app.App | undefined
) {
if (destDBApp === undefined) return;
const admin = destDBApp;
const email = outCollData.personali.email ? outCollData.personali.email : "";
console.log("Email is ", email);
if (email === "" || email === undefined) return;
console.log("Inside updateUsersUID");
let newUserOverrides = {
uid: outCollData._id,
};
let oldUser: any;
try {
console.log("Starting update for user with email:", email);
oldUser = await admin.auth().getUserByEmail(email!);
//console.log("Old user found:", oldUser);
if (oldUser.uid === outCollData._id) {
console.log(
"User " +
email +
" already exists in the destination DB with UID " +
outCollData._id
);
return;
}
await admin.auth().deleteUser(oldUser.uid);
console.log("Old user deleted.");
} catch (e) {
console.log("User not found in destination DB ", email);
console.log("Copying the user data from source DB");
oldUser = await sourceDBApp?.auth().getUserByEmail(email);
}
let dataToTransfer_keys = [
"disabled",
"displayName",
"email",
"emailVerified",
"phoneNumber",
"photoURL",
"uid",
"providerData",
];
let newUserData: any = {};
for (let key of dataToTransfer_keys) {
newUserData[key] = oldUser[key];
}
Object.assign(newUserData, newUserOverrides);
//console.log("New user data ready: ", newUserData);
let newUser = await admin.auth().createUser(newUserData);
console.log("New user created ");
}

how to get token transaction list by address using web3 js

I am using web3.js. I want token transaction list (Not transaction List) by address. I already used the getBlock function but its only for particular block. I have no block list and I want the list by address only.
How can I get the token transaction list?

I've implemented this with the web3-eth and web3-utils 1.0 betas using getPastEvents.
The standardAbi for ERC20 tokens I retrieved from this repo
import Eth from "web3-eth";
import Utils from "web3-utils";
async function getERC20TransactionsByAddress({
tokenContractAddress,
tokenDecimals,
address,
fromBlock
}) {
// initialize the ethereum client
const eth = new Eth(
Eth.givenProvider || "ws://some.local-or-remote.node:8546"
);
const currentBlockNumber = await eth.getBlockNumber();
// if no block to start looking from is provided, look at tx from the last day
// 86400s in a day / eth block time 10s ~ 8640 blocks a day
if (!fromBlock) fromBlock = currentBlockNumber - 8640;
const contract = new eth.Contract(standardAbi, tokenContractAddress);
const transferEvents = await contract.getPastEvents("Transfer", {
fromBlock,
filter: {
isError: 0,
txreceipt_status: 1
},
topics: [
Utils.sha3("Transfer(address,address,uint256)"),
null,
Utils.padLeft(address, 64)
]
});
return transferEvents
.sort((evOne, evTwo) => evOne.blockNumber - evTwo.blockNumber)
.map(({ blockNumber, transactionHash, returnValues }) => {
return {
transactionHash,
confirmations: currentBlockNumber - blockNumber,
amount: returnValues._value * Math.pow(10, -tokenDecimals)
};
});
}
I haven't tested this code as it is slightly modified from the one I have and it can definitely be optimized, but I hope it helps.
I’m filtering by topics affecting the Transfer event, targeting the address supplied in the params.

Writing and Reading JSON files in javascript

So I am doing a project right now requiring the storage of user preferences with JSON. I have searched for a decent amount of time now but can find no solution.For example sake There are three variables user, permissions, serverid . I figured this would work.
tempObject = {
user: []
};
tempObject.user.push({perm:permissions, server:serverid});
Then i would stringify and turn into a JSON. However the output came out like this:
{user[{perm:4, server:883}]}
This was my desperate attempt at grouping the perm and server variables under the indivisuals UserID so further down in the code i can fetch the permissions of each userID. But as you can see it didnt print the user variable, just changed it to an array and took user as a litteral string.
tl;dr
In short i need help being able to have a JSON file be written to where it stores the perm and serverID under the UserID.

Make user an object. Change this:
user: []
for this:
user: {}
and then set the keys like this:
user.perm = 4;
user.server = 883;

For security reasons, client-side JavaScript is not permitted to write to the disk. This sounds like you need a database.
You could leverage localStorage, or perhaps a cookie as an alternate to a database.
I think you should change the users array to an object; that way could key by userID.
for example:
var data = {
users: {}
};
const userID = 1234; // or could be a string like 'john_doe'
const userPermissions = { perm: 4, server: 883 };
// set the user's permissions
data.users[userID] = userPermissions;
// fetching user's permissions
const userData = data.users[userID];
console.log('User ' + userID +' has perm = ' + userData.perm + ' and server = ' + userData.server);
Now saving and loading of this data using local storage is easy:
function saveData() {
localStorage.setItem('UserData', JSON.stringify(data));
}
function loadData() {
data = JSON.parse(localStorage.getItem('UserData'));
}