I have some code like so:
export async function handleRefresh() {
if (!existsSync('postr.toml')) fail('not a postr directory');
const posts = expandGlob('posts/*');
for await (const post of posts) {
if (!post.isDirectory) {
console.warn('warning: non-folder found in posts directory');
continue;
}
let {parsedFrontMatter, contents} = extractFrontMatter(await read(post.path + '/post.md'));
const adapters = parsedFrontMatter.adapters ?? [];
if (!parsedFrontMatter) {
fail('no frontmatter for ' + post.path);
continue;
}
if (!Array.isArray(adapters)) {
fail('adapters is not an array');
continue;
}
if (isValidFrontMatter(parsedFrontMatter)) {
fail('frontmatter is not valid');
continue;
}
adapters.forEach(async (adapter: string) => {
const adapterPlugins = parseToml(await read('postr.toml')).adapterPlugins ?? {};
if (!isObject(adapterPlugins)) {
fail('adapterPlugins in the configuration is not an object');
return;
}
const adapterPath = adapterPlugins[adapter];
if (!adapterPath) {
console.warn('warn: an adapter was set' +
'but the corresponding plugin was not configured in `postr.toml`. Skipping');
return;
}
if (!('path' in <any>adapterPath)) {
fail(`adapter ${adapter} does not have a path`);
return;
}
import((<any>adapterPath).path)
.then(async module => {
const action = getActionForPost(parsedFrontMatter);
if (module[action]) {
await module[action](contents, parsedFrontMatter, (<any>adapterPath).config, {
updateFrontMatter(newData: {[x: string]: any}) {
parsedFrontMatter = Object.assign(parsedFrontMatter, newData);
},
mapID(remote: string | number) {
addMapping(parsedFrontMatter.id as string, remote.toString(), adapter);
}
})
} else {
console.warn(`Adapter ${adapter} does not support action \`${action}\``);
return;
}
writeFinalContents(parsedFrontMatter, contents, post.path)
})
.catch(error => fail(`could not run adapter because of ${error.name}: ${error.message}`));
});
}
}
Huge function.
There are a lot of these necessary if checks. 3/4 of the function is if checks, you could say. I want some advice on how I could refactor these statements.
As you can see the checks are not always the same, there are some different checks going on there.
EDIT: I've added real code.
I have a few functional components with a few common variables and functions. Please see below.
const Customer = () => {
const [isReleased, setIsReleased] = useState(false)
const release = () => {
setIsReleased(true)
}
}
const Order = () => {
const [isReleased, setIsReleased] = useState(false)
const release = () => {
setIsReleased(true)
}
}
As you can see from the above code fragment, the release() function has common logic. It accesses component's variables/functions.
Is there a way to move this release() function to a common file and import it from each component?
Please note that the release() method should be able to access scoped variables and functions of the caller.
Update
Below is the actual content of the release() function. I have put this. to denote that it refers to the variables/functions in the caller.
const release = () => {
if (action === "new") {
history.push(`/customers/new`)
} else if (action === "save") {
(async () => {
try {
if (this.dataMode === "new") {
this.setMessage()
this.setFormStatus("updating")
let _res = await this.customer_api_create(this.formData)
if ((_res.status === 200) && (_res.data.status === "success")) {
this.setFormStatus()
history.replace({ pathname: `/customers/${_res.data.data[0].id}` })
}
} else if (this.dataMode === "edit") {
this.setFormStatus("updating")
this.setMessage()
let _res = await this.customer_api_update(this.formData)
if ((_res.status === 200) && (_res.data.status === "success")) {
this.setFormStatus()
this.setMessage({ type: "info", text: "Saved" })
this.setFormData(_res.data.data[0])
}
}
} catch (e) {
this.openMessageBox({
prompt: e.response.data.message,
type: this.constants.app.MessageBoxType.MB_Error,
buttons: this.constants.app.MessageBoxButton.MB_Ok,
show: true,
setResult: () => console.log(this.constants.app.MessageBoxResult.MB_Ok)
})
this.setFormStatus("updating_error")
}
})();
} else if (action === "del") {
this.openMessageBox({
prompt: `{ "": ["Are you sure you want to delete context customer?"] }`,
type: this.constants.app.MessageBoxType.MB_Warning,
buttons: this.constants.app.MessageBoxButton.MB_YesNo,
show: true,
setResult: (val) => {
this.openMessageBox({ show: false })
this.setShouldRecordDeleted(val)
}
})
}
}
You can declare the function release in a different file and pass the setIsReleased method as an argument
import {release} from 'release';
const Customer = () => {
const [isReleased, setIsReleased] = useState(false)
release(setIsReleased)
}
And in your release.js file
export const release = (setIsReleased) => {
setIsReleased(true)
}
Maybe callback solve the problem, how about this?
anotherfile.js
const release = (param, setter) => {
if (true){
setter(true)
}
}
yourfile.js
import {release} from "anotherfile"
const Customer = () => {
const [isReleased, setIsReleased] = useState(false)
release([parameter],setIsReleased)
}
Hi i want to return the promise that gets resolved when the first poll finishes...
I have a start_polling method and i want to modify it such that it returns a promise that gets resolved when the items_promise gets resolved...i am not sure how to do it.
What i have tried?
i have created a promise in start_polling method that gets resolved when the item_promise_finish() is executed in then method of items_promise.
Once this promise in start_polling is resolved then i call handle_location method.
It works fine but sometimes i get error cannot read property then of undefined.
How can i fix it.
below is the code,
componentDidUpdate(prevProps, prevState) {
const {state, props} = this;
const current_id = props.item && props.item.id;
const prev_id = prevProps.item && prevProps.item.id;
if (current_id !== prev_id) {
this.stop_polling();
this.setState(this.get_initial_state(), () => {
this.start_polling();
});
} else {
const prev_poll = this.should_poll(prevProps, prevState);
const next_poll = this.should_poll(props, state);
if (prev_poll !== next_poll) {
if (next_poll) {
this.start_polling();
} else {
this.stop_polling();
}
}
if (this.props.location.search) {
this.start_polling().then(() => {
this.handle_location();
})
}
start_polling = () => {
if (this.should_poll(this.props, this.state)) {
this.poll();
return new Promise((resolve) => {
this.item_promise_finish=resolve;
});
}
};
poll = () => {
let still_polling = true;
if (this.cancel_poll_request) {
this.cancel_poll_request();
}
});
const items_promise = client.get_items(this.props.item.id,
cancel_poll_promise);
items_promise.then((request) => {
const next_items = [];
let something_changed = false;
for (const new_item of request.response) {
const old_item = this.state.items.find(
old_item =>
old_item.id === new_item.id);
if (old_item) {
next_items.push(old_item);
} else {
something_changed = true;
next_items.push(new_item);
}
}
if (something_changed) {
const next_state = {items: next_items};
this.setState(next_state);
}
}).then(() => {
if (still_polling) {
this.poll_timeout = setTimeout(this.poll, 100s);
}
this.item_promise_finish();
}).catch(this.props.notifications.request_error);
};
Could someone let me know where i am going wrong or what to be done to fix this. thanks.
I'm trying to update a field timestamp with the Firestore admin timestamp in a collection with more than 500 docs.
const batch = db.batch();
const serverTimestamp = admin.firestore.FieldValue.serverTimestamp();
db
.collection('My Collection')
.get()
.then((docs) => {
serverTimestamp,
}, {
merge: true,
})
.then(() => res.send('All docs updated'))
.catch(console.error);
This throws an error
{ Error: 3 INVALID_ARGUMENT: cannot write more than 500 entities in a single call
at Object.exports.createStatusError (C:\Users\Growthfile\Desktop\cf-test\functions\node_modules\grpc\src\common.js:87:15)
at Object.onReceiveStatus (C:\Users\Growthfile\Desktop\cf-test\functions\node_modules\grpc\src\client_interceptors.js:1188:28)
at InterceptingListener._callNext (C:\Users\Growthfile\Desktop\cf-test\functions\node_modules\grpc\src\client_interceptors.js:564:42)
at InterceptingListener.onReceiveStatus (C:\Users\Growthfile\Desktop\cf-test\functions\node_modules\grpc\src\client_interceptors.js:614:8)
at callback (C:\Users\Growthfile\Desktop\cf-test\functions\node_modules\grpc\src\client_interceptors.js:841:24)
code: 3,
metadata: Metadata { _internal_repr: {} },
details: 'cannot write more than 500 entities in a single call' }
Is there a way that I can write a recursive method which creates a batch object updating a batch of 500 docs one by one until all the docs are updated.
From the docs I know that delete operation is possible with the recursive approach as mentioned here:
https://firebase.google.com/docs/firestore/manage-data/delete-data#collections
But, for updating, I'm not sure how to end the execution since the docs are not being deleted.
I also ran into the problem to update more than 500 documents inside a Firestore collection. And i would like to share how i solved this problem.
I use cloud functions to update my collection inside Firestore but this should also work on client side code.
The solution counts every operation which is made to the batch and after the limit is reached a new batch is created and pushed to the batchArray.
After all updates are completed the code loops through the batchArray and commits every batch which is inside the array.
It is important to count every operation set(), update(), delete() which is made to the batch because they all count to the 500 operation limit.
const documentSnapshotArray = await firestore.collection('my-collection').get();
const batchArray = [];
batchArray.push(firestore.batch());
let operationCounter = 0;
let batchIndex = 0;
documentSnapshotArray.forEach(documentSnapshot => {
const documentData = documentSnapshot.data();
// update document data here...
batchArray[batchIndex].update(documentSnapshot.ref, documentData);
operationCounter++;
if (operationCounter === 499) {
batchArray.push(firestore.batch());
batchIndex++;
operationCounter = 0;
}
});
batchArray.forEach(async batch => await batch.commit());
return;
I liked this simple solution:
const users = await db.collection('users').get()
const batches = _.chunk(users.docs, 500).map(userDocs => {
const batch = db.batch()
userDocs.forEach(doc => {
batch.set(doc.ref, { field: 'myNewValue' }, { merge: true })
})
return batch.commit()
})
await Promise.all(batches)
Just remember to add import * as _ from "lodash" at the top. Based on this answer.
You can use default BulkWriter. This method used 500/50/5 rule.
Example:
let bulkWriter = firestore.bulkWriter();
bulkWriter.create(documentRef, {foo: 'bar'});
bulkWriter.update(documentRef2, {foo: 'bar'});
bulkWriter.delete(documentRef3);
await close().then(() => {
console.log('Executed all writes');
});
As mentioned above, #Sebastian's answer is good and I upvoted that too. Although faced an issue while updating 25000+ documents in one go.
The tweak to logic is as below.
console.log(`Updating documents...`);
let collectionRef = db.collection('cities');
try {
let batch = db.batch();
const documentSnapshotArray = await collectionRef.get();
const records = documentSnapshotArray.docs;
const index = documentSnapshotArray.size;
console.log(`TOTAL SIZE=====${index}`);
for (let i=0; i < index; i++) {
const docRef = records[i].ref;
// YOUR UPDATES
batch.update(docRef, {isDeleted: false});
if ((i + 1) % 499 === 0) {
await batch.commit();
batch = db.batch();
}
}
// For committing final batch
if (!(index % 499) == 0) {
await batch.commit();
}
console.log('write completed');
} catch (error) {
console.error(`updateWorkers() errored out : ${error.stack}`);
reject(error);
}
Explanations given on previous comments already explain the issue.
I'm sharing the final code that I built and worked for me, since I needed something that worked in a more decoupled manner, instead of the way that most of the solutions presented above do.
import { FireDb } from "#services/firebase"; // = firebase.firestore();
type TDocRef = FirebaseFirestore.DocumentReference;
type TDocData = FirebaseFirestore.DocumentData;
let fireBatches = [FireDb.batch()];
let batchSizes = [0];
let batchIdxToUse = 0;
export default class FirebaseUtil {
static addBatchOperation(
operation: "create",
ref: TDocRef,
data: TDocData
): void;
static addBatchOperation(
operation: "update",
ref: TDocRef,
data: TDocData,
precondition?: FirebaseFirestore.Precondition
): void;
static addBatchOperation(
operation: "set",
ref: TDocRef,
data: TDocData,
setOpts?: FirebaseFirestore.SetOptions
): void;
static addBatchOperation(
operation: "create" | "update" | "set",
ref: TDocRef,
data: TDocData,
opts?: FirebaseFirestore.Precondition | FirebaseFirestore.SetOptions
): void {
// Lines below make sure we stay below the limit of 500 writes per
// batch
if (batchSizes[batchIdxToUse] === 500) {
fireBatches.push(FireDb.batch());
batchSizes.push(0);
batchIdxToUse++;
}
batchSizes[batchIdxToUse]++;
const batchArgs: [TDocRef, TDocData] = [ref, data];
if (opts) batchArgs.push(opts);
switch (operation) {
// Specific case for "set" is required because of some weird TS
// glitch that doesn't allow me to use the arg "operation" to
// call the function
case "set":
fireBatches[batchIdxToUse].set(...batchArgs);
break;
default:
fireBatches[batchIdxToUse][operation](...batchArgs);
break;
}
}
public static async runBatchOperations() {
// The lines below clear the globally available batches so we
// don't run them twice if we call this function more than once
const currentBatches = [...fireBatches];
fireBatches = [FireDb.batch()];
batchSizes = [0];
batchIdxToUse = 0;
await Promise.all(currentBatches.map((batch) => batch.commit()));
}
}
Based on all the above answers, I put together the following pieces of code that one can put into a module in JavaScript back-end and front-end to easily use Firestore batch writes, without worrying about the 500 writes limit.
Back-end (Node.js)
// The Firebase Admin SDK to access Firestore.
const admin = require("firebase-admin");
admin.initializeApp();
// Firestore does not accept more than 500 writes in a transaction or batch write.
const MAX_TRANSACTION_WRITES = 499;
const isFirestoreDeadlineError = (err) => {
console.log({ err });
const errString = err.toString();
return (
errString.includes("Error: 13 INTERNAL: Received RST_STREAM") ||
errString.includes("Error: 4 DEADLINE_EXCEEDED: Deadline exceeded")
);
};
const db = admin.firestore();
// How many transactions/batchWrites out of 500 so far.
// I wrote the following functions to easily use batchWrites wthout worrying about the 500 limit.
let writeCounts = 0;
let batchIndex = 0;
let batchArray = [db.batch()];
// Commit and reset batchWrites and the counter.
const makeCommitBatch = async () => {
console.log("makeCommitBatch");
await Promise.all(batchArray.map((bch) => bch.commit()));
};
// Commit the batchWrite; if you got a Firestore Deadline Error try again every 4 seconds until it gets resolved.
const commitBatch = async () => {
try {
await makeCommitBatch();
} catch (err) {
console.log({ err });
if (isFirestoreDeadlineError(err)) {
const theInterval = setInterval(async () => {
try {
await makeCommitBatch();
clearInterval(theInterval);
} catch (err) {
console.log({ err });
if (!isFirestoreDeadlineError(err)) {
clearInterval(theInterval);
throw err;
}
}
}, 4000);
}
}
};
// If the batchWrite exeeds 499 possible writes, commit and rest the batch object and the counter.
const checkRestartBatchWriteCounts = () => {
writeCounts += 1;
if (writeCounts >= MAX_TRANSACTION_WRITES) {
batchIndex++;
batchArray.push(db.batch());
writeCounts = 0;
}
};
const batchSet = (docRef, docData) => {
batchArray[batchIndex].set(docRef, docData);
checkRestartBatchWriteCounts();
};
const batchUpdate = (docRef, docData) => {
batchArray[batchIndex].update(docRef, docData);
checkRestartBatchWriteCounts();
};
const batchDelete = (docRef) => {
batchArray[batchIndex].delete(docRef);
checkRestartBatchWriteCounts();
};
module.exports = {
admin,
db,
MAX_TRANSACTION_WRITES,
checkRestartBatchWriteCounts,
commitBatch,
isFirestoreDeadlineError,
batchSet,
batchUpdate,
batchDelete,
};
Front-end
// Firestore does not accept more than 500 writes in a transaction or batch write.
const MAX_TRANSACTION_WRITES = 499;
const isFirestoreDeadlineError = (err) => {
return (
err.message.includes("DEADLINE_EXCEEDED") ||
err.message.includes("Received RST_STREAM")
);
};
class Firebase {
constructor(fireConfig, instanceName) {
let app = fbApp;
if (instanceName) {
app = app.initializeApp(fireConfig, instanceName);
} else {
app.initializeApp(fireConfig);
}
this.name = app.name;
this.db = app.firestore();
this.firestore = app.firestore;
// How many transactions/batchWrites out of 500 so far.
// I wrote the following functions to easily use batchWrites wthout worrying about the 500 limit.
this.writeCounts = 0;
this.batch = this.db.batch();
this.isCommitting = false;
}
async makeCommitBatch() {
console.log("makeCommitBatch");
if (!this.isCommitting) {
this.isCommitting = true;
await this.batch.commit();
this.writeCounts = 0;
this.batch = this.db.batch();
this.isCommitting = false;
} else {
const batchWaitInterval = setInterval(async () => {
if (!this.isCommitting) {
this.isCommitting = true;
await this.batch.commit();
this.writeCounts = 0;
this.batch = this.db.batch();
this.isCommitting = false;
clearInterval(batchWaitInterval);
}
}, 400);
}
}
async commitBatch() {
try {
await this.makeCommitBatch();
} catch (err) {
console.log({ err });
if (isFirestoreDeadlineError(err)) {
const theInterval = setInterval(async () => {
try {
await this.makeCommitBatch();
clearInterval(theInterval);
} catch (err) {
console.log({ err });
if (!isFirestoreDeadlineError(err)) {
clearInterval(theInterval);
throw err;
}
}
}, 4000);
}
}
}
async checkRestartBatchWriteCounts() {
this.writeCounts += 1;
if (this.writeCounts >= MAX_TRANSACTION_WRITES) {
await this.commitBatch();
}
}
async batchSet(docRef, docData) {
if (!this.isCommitting) {
this.batch.set(docRef, docData);
await this.checkRestartBatchWriteCounts();
} else {
const batchWaitInterval = setInterval(async () => {
if (!this.isCommitting) {
this.batch.set(docRef, docData);
await this.checkRestartBatchWriteCounts();
clearInterval(batchWaitInterval);
}
}, 400);
}
}
async batchUpdate(docRef, docData) {
if (!this.isCommitting) {
this.batch.update(docRef, docData);
await this.checkRestartBatchWriteCounts();
} else {
const batchWaitInterval = setInterval(async () => {
if (!this.isCommitting) {
this.batch.update(docRef, docData);
await this.checkRestartBatchWriteCounts();
clearInterval(batchWaitInterval);
}
}, 400);
}
}
async batchDelete(docRef) {
if (!this.isCommitting) {
this.batch.delete(docRef);
await this.checkRestartBatchWriteCounts();
} else {
const batchWaitInterval = setInterval(async () => {
if (!this.isCommitting) {
this.batch.delete(docRef);
await this.checkRestartBatchWriteCounts();
clearInterval(batchWaitInterval);
}
}, 400);
}
}
}
No citations or documentation, this code i invented by myself and for me it worked and looks clean, and simple for read and usage. If some one like it, then can use it too.
Better make autotest becose code use private var _ops wich can be changed after packages upgrade. Forexample in old versions its can be _mutations
async function commitBatch(batch) {
const MAX_OPERATIONS_PER_COMMIT = 500;
while (batch._ops.length > MAX_OPERATIONS_PER_COMMIT) {
const batchPart = admin.firestore().batch();
batchPart._ops = batch._ops.splice(0, MAX_OPERATIONS_PER_COMMIT - 1);
await batchPart.commit();
}
await batch.commit();
}
Usage:
const batch = admin.firestore().batch();
batch.delete(someRef);
batch.update(someRef);
...
await commitBatch(batch);
Simple solution
Just fire twice ?
my array is "resultsFinal"
I fire batch once with a limit of 490 , and second with a limit of the lenght of the array ( results.lenght)
Works fine for me :)
How you check it ?
You go to firebase and delete your collection , firebase say you have delete XXX docs , same as the lenght of your array ? Ok so you are good to go
async function quickstart(results) {
// we get results in parameter for get the data inside quickstart function
const resultsFinal = results;
// console.log(resultsFinal.length);
let batch = firestore.batch();
// limit of firebase is 500 requests per transaction/batch/send
for (i = 0; i < 490; i++) {
const doc = firestore.collection('testMore490').doc();
const object = resultsFinal[i];
batch.set(doc, object);
}
await batch.commit();
// const batchTwo = firestore.batch();
batch = firestore.batch();
for (i = 491; i < 776; i++) {
const objectPartTwo = resultsFinal[i];
const doc = firestore.collection('testMore490').doc();
batch.set(doc, objectPartTwo);
}
await batch.commit();
}
I am working on a Resource Manager and some jobs must be atomic, because there may be race condition.
I see a solution is provided in a similar post
Javascript semaphore / test-and-set / lock?
which says implement a basic integer semaphore, just add the variable into the DOM and lock/unlock it and make sure your functions keep checking it, else timeout them
Can someone help me how can I implement it for below code ? and How can I add a variable to DOM ?
In the code below retrievePartners methods returns partners based on what user asked in capabilities object, and retrievePartners method may have been called same time from another user, asking for same partners (browsers.) So this method should be atomic, means should deal only with one capabilities at a same time.
async retrievePartners (capabilities) {
const appropriatePartners = { chrome: [], firefox: [], safari: [], ie: [] }
const partners = await this.getAllPartners()
allTypeNumber = 0
// first check if there is available appropriate Partners
Object.keys(capabilities.type).forEach(key => {
let typeNumber = parseInt(capabilities.type[key])
allTypeNumber = allTypeNumber + typeNumber
for (let i = 0; i < typeNumber; i++) {
partners.forEach((partner, i) => {
if (
key === partner.value.type &&
partner.value.isAvailable &&
appropriatePartners[key].length < typeNumber
) {
appropriatePartners[key].push(partner)
}
})
if (appropriatePartners[key].length < typeNumber) {
throw new Error(
'Sorry there are no appropriate Partners for this session'
)
}
}
})
if (allTypeNumber === 0) {
throw new Error('Please mention at least 1 type of browser !')
} else {
Object.keys(appropriatePartners).forEach(key => {
appropriatePartners[key].forEach(partner => {
this.instructorPeer.set('/partners/' + partner.id + '/states/', {
isAvailable: false
})
})
})
return appropriatePartners
}
}
getAllPartners method
async getAllPartners (capabilities) {
const partners = []
const paths = await this.instructorPeer.get({
path: { startsWith: '/partners/' }
})
paths.forEach((path, i) => {
if (path.fetchOnly) {
let obj = {}
obj.value = path.value
obj.id = path.path.split('/partners/')[1]
obj.value.isAvailable = paths[i + 1].value.isAvailable
partners.push(obj)
}
})
return partners
}
Here is how I call the method
async function startTest () {
const capabilities = {
type: {
chrome: 1
}
}
test('Example test', async t => {
try {
session = await workout.createSession(capabilities)
const partners = session.partners
const partner = partners.chrome[0]
...