I am performing pagination inside and Observable stream.
The pagination is implemented with a cursor and a total count using recursion.
I am able to emit the every page using the following code observer.next(searches);, by the way I would like to use just observable and no promises but I cannot express recursion using RxJs operators.
Any suggestions?
const search = id =>
new Observable(observer => { recursePages(id, observer) })
const recursePages = (id, observer, processed, searchAfter) => {
httpService.post(
"http://service.com/search",
{
size: 50,
...searchAfter ? { search_after: searchAfter } : null,
id,
})
.toPromise() // httpService.post returns an Observable<AxiosResponse>
.then(res => {
const body = res.data;
const searches = body.data.hits.map(search => ({ data: search.data, cursor: search.id }));
observer.next(searches);
const totalProcessed = processed + searches.length;
if (totalProcessed < body.data.total) {
return recursePages(id, observer, totalProcessed, searches[searches.length - 1].cursor);
}
observer.complete();
})
}
// General Observer
incomingMessages.pipe(
flatMap(msg => search(JSON.parse(msg.content.toString()))),
concatAll(),
).subscribe(console.log),
these methods will recursively gather all the pages and emit them in an array. the pages can then be streamed with from as shown:
// break this out to clean up functions
const performSearch = (id, searchAfter?) => {
return httpService.post(
"http://service.com/search",
{
size: 50,
...searchAfter ? { search_after: searchAfter } : null,
id,
});
}
// main recursion
const _search = (id, processed, searchAfter?) => {
return performSearch(id, searchAfter).pipe( // get page
switchMap(res => {
const body = res.data;
const searches = body.data.hits.map(search => ({ data: search.data, cursor: search.id }));
const totalProcessed = processed + searches.length;
if (totalProcessed < body.total) {
// if not done, recurse and get next page
return _search(id, totalProcessed, searches[searches.length - 1].cursor).pipe(
// attach recursed pages
map(nextPages => [searches].concat(nextPages)
);
}
// if we're done just return the page
return of([searches]);
})
)
}
// entry point
// switch into from to emit pages one by one
const search = id => _search(id, 0).pipe(switchMap(pages => from(pages))
if what you really need is all of the pages to emit one by one before they're all fetched, for instance so you can show page 1 as soon as it's available rather than wait on page 2+, then that can be done with some tweaking. let me know.
EDIT: this method will emit one by one
const _search = (id, processed, searchAfter?) => {
return performSearch(id, searchAfter).pipe( // get page
switchMap(res => {
const body = res.data;
const searches = body.data.hits.map(search => ({ data: search.data, cursor: search.id }));
const totalProcessed = processed + searches.length;
if (totalProcessed < body.total) {
// if not done, concat current page with recursive call for next page
return concat(
of(searches),
_search(id, totalProcessed, searches[searches.length - 1].cursor)
);
}
// if we're done just return the page
return of(searches);
})
)
}
const search = id => _search(id, 0)
you end up with an observable structure like:
concat(
post$(page1),
concat(
post$(page2),
concat(
post$(page3),
post$(page4)
)
)
)
and since nested concat() operations reduce to a flattened structure, this structure would reduce to:
concat(post$(page1), post$(page2), post$(page3), post$(page4))
which is what you're after and the requests run sequentially.
it also seems like expand might do the trick as per #NickL 's comment, soemthing like:
search = (id) => {
let totalProcessed = 0;
return performSearch(id).pipe(
expand(res => {
const body = res.data;
const searches = body.data.hits.map(search => ({ data: search.data, cursor: search.id }));
totalProcessed += searches.length;
if (totalProcessed < body.data.total) {
// not done, keep expanding
return performSearch(id, searches[searches.length - 1].cursor);
}
return EMPTY; // break with EMPTY
})
)
}
though I've never used expand before and this is based off some very limited testing of it, but I am pretty certain this works.
both of these methods could use the reduce (or scan) operator to gather results if you ever wanted:
search(id).pipe(reduce((all, page) => all.concat(page), []))
This is my used solution combining the expand and reduce operator
searchUsers(cursor?: string) {
return from(
this.slackService.app.client.users.list({
token: this.configService.get('SLACK_BOT_TOKEN'),
limit: 1,
...(cursor && { cursor }),
}),
);
}
Usage
.......
this.searchUsers()
.pipe(
expand((res) => {
if (!!res.response_metadata.next_cursor) {
return this.searchUsers(res.response_metadata.next_cursor);
}
return EMPTY;
}),
reduce((acc, val) => {
return [...acc, ...val.members];
}, []),
)
.subscribe((users) => {
console.log(JSON.stringify(users));
});
....
Related
How can I build a function which gets some data asynchronously then uses that data to get more asynchronous data?
I am using Dexie.js (indexedDB wrapper) to store data about a direct message. One thing I store in the object is the user id which I'm going to be sending messages to. To build a better UI I'm also getting some information about that user such as the profile picture, username, and display name which is stored on a remote rdbms. To build a complete link component in need data from both databases (local indexedDB and remote rdbms).
My solution returns an empty array. It is being computed when logging it in Google Chrome and I do see my data. However because this is not being computed at render time the array is always empty and therefor I can't iterate over it to build a component.
const [conversations, setConversations] = useState<IConversation[]>()
const [receivers, setReceivers] = useState<Profile[]>()
useEffect(() => {
messagesDatabase.conversations.toArray().then(result => {
setConversations(result)
})
}, [])
useEffect(() => {
if (conversations) {
const getReceivers = async () => {
let receivers: Profile[] = []
await conversations.forEach(async (element) => {
const receiver = await getProfileById(element.conversationWith, token)
// the above await is a javascript fetch call to my backend that returns json about the user values I mentioned
receivers.push(receiver)
})
return receivers
}
getReceivers().then(receivers => {
setReceivers(receivers)
})
}
}, [conversations])
/*
The below log logs an array with a length of 0; receivers.length -> 0
but when clicking the log in Chrome I see:
[
0: {
avatarURL: "https://lh3.googleusercontent.com/..."
displayName: "Cool guy"
userId: "1234"
username: "cool_guy"
}
1: ...
]
*/
console.log(receivers)
My plan is to then iterate over this array using map
{
receivers && conversations
? receivers.map((element, index) => {
return <ChatLink
path={conversations[index].path}
lastMessage={conversations[index].last_message}
displayName={element.displayName}
username={element.username}
avatarURL={element.avatarURL}
key={index}
/>
})
: null
}
How can I write this to not return a empty array?
Here's a SO question related to what I'm experiencing here
I believe your issue is related to you second useEffect hook when you attempt to do the following:
const getReceivers = async () => {
let receivers: Profile[] = []
await conversations.forEach(async (element) => {
const receiver = await getProfileById(element.conversationWith, token)
receivers.push(receiver)
})
return receivers
}
getReceivers().then(receivers => {
setReceivers(receivers)
})
}
Unfortunately, this won't work because async/await doesn't work with forEach. You either need to use for...of or Promise.all() to properly iterate through all conversations, call your API, and then set the state once it's all done.
Here's is a solution using Promise.all():
function App() {
const [conversations, setConversations] = useState<IConversation[]>([]);
const [receivers, setReceivers] = useState<Profile[]>([]);
useEffect(() => {
messagesDatabase.conversations.toArray().then(result => {
setConversations(result);
});
}, []);
useEffect(() => {
if (conversations.length === 0) {
return;
}
async function getReceivers() {
const receivers: Profile[] = await Promise.all(
conversations.map(conversation =>
getProfileById(element.conversationWith, token)
)
);
setReceivers(receivers);
}
getReceivers()
}, [conversations]);
// NOTE: You don't have to do the `receivers && conversations`
// check, and since both are arrays, you should check whether
// `receivers.length !== 0` and `conversations.length !== 0`
// if you want to render something conditionally, but since your
// initial `receivers` state is an empty array, you could just
// render that instead and you won't be seeing anything until
// that array is populated with some data after all fetching is
// done, however, for a better UX, you should probably indicate
// that things are loading and show something rather than returning
// an empty array or null
return receivers.map((receiver, idx) => <ChatLink />)
// or, alternatively
return receivers.length !== 0 ? (
receivers.map((receiver, idx) => <ChatLink />)
) : (
<p>Loading...</p>
);
}
Alternatively, using for...of, you could do the following:
function App() {
const [conversations, setConversations] = useState<IConversation[]>([]);
const [receivers, setReceivers] = useState<Profile[]>([]);
useEffect(() => {
messagesDatabase.conversations.toArray().then(result => {
setConversations(result);
});
}, []);
useEffect(() => {
if (conversations.length === 0) {
return;
}
async function getReceivers() {
let receivers: Profile[] = [];
const profiles = conversations.map(conversation =>
getProfileById(conversation.conversationWith, token)
);
for (const profile of profiles) {
const receiver = await profile;
receivers.push(receiver);
}
return receivers;
}
getReceivers().then(receivers => {
setReceivers(receivers);
});
}, [conversations]);
return receivers.map((receiver, idx) => <ChatLink />);
}
i think it is happening because for getReceivers() function is asynchronous. it waits for the response, in that meantime your state renders with empty array.
you can display spinner untill the response received.
like
const[isLoading,setLoading]= useState(true)
useEffect(()=>{
getReceivers().then(()=>{setLoading(false)}).catch(..)
} )
return {isLoading ? <spinner/> : <yourdata/>}
Please set receivers initial value as array
const [receivers, setReceivers] = useState<Profile[]>([])
Also foreach will not wait as you expect use for loop instead of foreach
I am not sure it is solution for your question
but it could help you to solve your error
I set up a store containing a list of rides loaded from my API:
const loadRides = () => client.service('rides').find({
query: {
$sort: {
date: -1,
}
}
});
const createRides = () => {
const { subscribe, update } = writable([], async (set) => {
try {
const rides = await loadRides().then((result) => result.data);
set(rides);
} catch (e) {
console.error(e);
}
// Socket update binding?
});
subscribe((rides) => console.debug('rides', rides));
return {
subscribe,
refresh: () => loadRides().then((result) => update(() => result.data)),
};
};
export const rides = createRides();
Then I set a two derived stores for past and future rides:
export const pastRides = derived(
rides,
($rides) => $rides
.filter((ride) => ride.steps.every((step) => step.doneAt))
,
);
export const comingRides = derived(
rides,
($rides) => $rides
.filter((ride) => ride.steps.some((step) => !step.doneAt))
.sort((rideA, rideB) => {
const compare = new Date(rideA.date) - new Date(rideB.date);
console.log(rideA.date, rideB.date, compare);
return compare;
})
,
);
The sort method on the second one does not have any effect.
So I tried to put this method before the filter one. It works, but it also sort $pastRides. In fact, it is sorting the full $rides array and it make sens.
But I does not understand why the sort after filter does not work.
What did I miss?
Array.sort is mutable. Meaning, when you call rides.sort, it will sort and modify rides and return the sorted rides.
When you use derived(rides, ($rides) => ... ), the $rides you received is the original rides array that you call set(rides). So you can imagine that both the pastRides and comingRides received the same $rides array.
you can observe this behavior in this repl
To not having both derived interfere with each other, you can create a new array and sort the new array:
const sorted_1 = derived(array, $a => [...$a].sort());
you can try this out in this repl.
I am using fs.readdir to get a list of directories and then again in the callback to get a list of "subpages" in each of these directories. I would like for the first callback to wait until the second callback is completed but I'm not sure how to do that.
// Array to hold list of pages
const pageList = []
// Get contents of target directory (not recursive)
fs.readdir(targetDir, { withFileTypes: true }, (err, items) => {
// Stop and return if error
if (!!err) return err
// Go through found contents
const theseItems = items.map(item => {
const subpages = []
// Directory name
const dirName = item.name
// Set up published target for this directory
const thisTargetDir = targetDir + '/' + dirName + '/publish'
// Now get pages in the directory's published directory
// (assumes all files within subdirectories are .mdx pages to load)
return (
fs.readdir(thisTargetDir, { withFileTypes: true }, (err, pages) => {
const theseSubpages = pages.map(page => {
const mdxSuffix = /.mdx$/g
const pageName = page.name.replace(mdxSuffix, '')
return subpages.push({ name: pageName })
})
Promise.all(theseSubpages).then(() => {
// Add to page list array
pageList.push({ name: dirName, subpages: subpages })
})
})
)
})
Promise.all(theseItems).then(() => {
console.log('pageList at the end is: ')
console.log(pageList)
})
})
The Promise.all(theseSubpages) works as expected, however the Promise.all(theseItems) resolves before the former has a chance to cycle through. I understand why that's happening and I've tried to do things like return each item as a Promise.resolve(), etc. but these things aren't working.
Wondering if I'm doing something inherently wrong in this approach…
UPDATE
I tried using the fsPromises approach but kept running into the same wrong patterns. Ended up using the node-dir package to go through the directories recursively. Code below, not really the exact answer to what I was trying to do, but this gets the result I was looking for.
const dir = require('node-dir')
const targetDir = __dirname + '/../pages/stuff'
const pageList = []
dir.paths(targetDir, (err, paths) => {
if (err) throw err
const baseMatch = __dirname.replace('/lib', '') + '/pages/stuff'
paths.dirs.map(dir => {
// Only publish paths
if (dir.substr(-7) === 'publish') {
// Get the slug directly before publish path
const contentSlug = dir.split('/').slice(-2)[0]
// Add this to main pageList array as top level objects
pageList.push({ name: contentSlug, subpages: [] })
}
})
paths.files.map(file => {
const filePathArray = file.split('/')
// Only publish paths
if (filePathArray.slice(-2)[0] === 'publish') {
// Get parent content slug for matching purposes
const parentContentSlug = filePathArray.slice(-3)[0]
// Get file name (remove .mdx suffix)
const mdxSuffix = /.mdx$/g
const fileName = filePathArray.slice(-1)[0].replace(mdxSuffix, '')
// Loop through main page list, find match, then add file as subpage
pageList.find((obj, key) => {
if (obj.name === parentContentSlug) {
return pageList[key].subpages.push({ name: fileName })
}
})
}
})
console.log('pageList at end:')
console.log(pageList)
})
Promises work by chaining .then calls (Promise.then(doStuff)). If you start a promise but then don't chain, you can't know when it's done. In order to chain promises from inner functions you have to return promises.
Generally you don't want to mix callbacks and promises.
If I were to do this I would start by using just promises.
const readdir = (target, options) =>
// returns a promise that resolves or rejects when the call finishes
new Promise((resolve, reject) =>
fs.readdir(target, options, (err, result) => {
if (err) reject(err);
resolve(result);
})
);
const collectSubPages = pages =>
// Wait for all the promises in the array to resolve
Promise.all(
// for each page, return a promise that resolves to the page/subpage object
pages.map(({ name }) =>
readdir(targetDir + "/" + name + "/publish", {
withFileTypes: true
})
.then(subpages => subpages.map(({ name }) => ({ name })))
.then(subpages => ({ name, subpages }))
)
);
readdir(targetDir, { withFileTypes: true })
.then(pages => collectSubPages(pages))
.then(console.log);
#David Yeiser, your own "update" code can be written more concisely using Array methods .filter() and .map(), plus various optimisations, as follows :
const dir = require('node-dir');
const targetDir = __dirname + '/../pages/stuff';
dir.paths(targetDir, (err, paths) => {
if (err) {
throw err;
}
const baseMatch = __dirname.replace('/lib', '') + '/pages/stuff';
const mdxSuffix = /.mdx$/g; // define once, use many times
const fileList = paths.files
.map(fileName => fileName.split('/'))
.filter(filePathArray => filePathArray[filePathArray.length - 2] === 'publish'); // Only 'publish' paths
const pageList = paths.dirs
.filter(dir => dir.substr(-7) === 'publish') // Only 'publish' paths
.map(dir => {
const name = dir.split('/').slice(-2)[0];
const subpages = fileList
.filter(filePathArray => filePathArray[filePathArray.length - 3] === name) // select those files whose "parent content slug" matches 'name'
.map(filePathArray => filePathArray[filePathArray.length - 1].replace(mdxSuffix, ''));
return { name, subpages };
});
console.log('pageList at end:');
console.log(pageList);
});
You will see that :
fileList is constructed with a paths.files.map().filter() pattern.
pageList is constructed with a paths.dirs.filter().map() pattern.
for each entry in pageList, subpages is constructed with a fileList.filter().map() pattern.
Barring mistakes on my part, that should give the same result.
untested
I get an Observable<Group[]> from my Firebase collection.
In this Group class is an id which I wanna use to retrieve another dataset array from Firebase, which would be messages for each unique group Observable<Message[]>.(each group has its own chat: Message[])
And it want to return an observable which hold an array of a new Type:
return { ...group, messages: Message[] } as GroupWithMessages
the final goal should be Observable<GroupWithMessages[]>
getGroupWithChat(): Observable<GroupWithMessages[]> {
const groupColl = this.getGroups(); // Observable<Group[]>
const messages = groupColl.pipe(
map(groups => {
return groups.map(meet => {
const messages = this.getMessagesFor(group.uid);
return { messages:messages, ...group} as GroupWithMessages
});
})
);
return messages;
}
}
and here the Message function
getMessagesFor(id: string): Observable<Message[]> {
return this.afs.collection<Message>(`meets/${id} /messages`).valueChanges();
}
sadly that doesnt work because when i create the new Obj I cannot bind messages:messages because messages ist vom typ Observable<Message[]>
I hope that cleares things
UPDATE:
my main problem now comes down to this:
getGroupsWithMessages() {
this.getJoinedGroups()
.pipe(
mergeMap(groups =>
from(groups).pipe(
mergeMap(group => {
return this.getMessagesFor(group.uid).pipe(
map(messages => {
return { ...group, messages } as GroupIdMess;
})
);
}),
tap(x => console.log('reaching here: ', x)),
toArray(),
tap(x => console.log('not reaching here = completed: ', x))
)
),
tap(x => console.log('not reaching here: ', x))
)
.subscribe(x => console.log('not reaching here: ', x));
}
when i call that function my console.log is as follows:
Not sure if I follow what you're doing here but the logic look like you'd want:
getGroupWithChat() {
return this.getGroups.pipe(map(groups=> {
return groups.map(group => this.getMessagesFor(group.uid));
})).subscribe(); // trigger "hot" observable
}
Let me know if I can help further after you clarify.
UPDATE:
So it looks like you need to get the UID of the group before making the call to get the GroupMessages[]?
get Group: Observable
call getMessagesFor(Group.uid)
this example gets groups result$ then
concatMap uses groups result$ to make the messages query
this.getGroups().pipe(
concatMap((group: Group) => this.getMessagesFor(group.uid))
).subscribe((messages: GroupWithMessages[]) => {
console.log(messages);
});
You may still want to map them together but it seems like you know how to do that. concatMap waits for the first to finish, then makes the second call which you need.
Is this closer?
Use forkJoin to wait for messages to be received for all groups. Then map the result of forkJoin to an array of GroupWithMessages like this -
getGroupWithChat(): Observable<GroupWithMessages[]> {
return this.getGroups()
.pipe(
switchMap(groups => {
const messagesForAllGroups$ = groups.map(group => this.getMessagesFor(group.uid));
return forkJoin(messagesForAllGroups$)
.pipe(
map(joined => {
//joined has response like -
//[messagesArrayForGroup0, messagesArrayForGroup1, messagesArrayForGroup2....];
const messagesByGroup = Array<GroupWithMessages>();
groups.forEach((group, index) => {
//assuming that GroupWithMessages has group and messages properties.
const gm = new GroupWithMessages();
gm.group = group;
gm.messages = joined[index];
messagesByGroup.push(gm);
});
return messagesByGroup;
})
)
})
)
}
I usually do that by splitting Observable<any[]> to Observable<any> and then mergeMap the results to inner Observable.
Something like this should work:
getMessagesFor(id: string): Observable<number> {
return of(1);
}
getGroups(): Observable<string[]> {
return of(["1", "2"]);
}
getGroupWithChat() {
this.getGroups().pipe(
mergeMap(groups => from(groups)), // Split the stream into individual group elements instead of an array
mergeMap(group => {
return this.getMessagesFor(group).pipe(
map(messages => {
return Object.assign(group, messages);
})
);
})
);
}
Edit:
Consider BehaviorSubject. It doesn't complete at all:
const behSub: BehaviorSubject<number[]> = new BehaviorSubject([1, 2, 3]);
setTimeout(() => {
behSub.next([4, 5, 6]);
}, 5000);
behSub
.pipe(
mergeMap(arr =>
from(arr).pipe(
tap(), // Do something with individual items, like mergeMap to messages
toArray() // Go back to array
)
)
)
.subscribe(console.log, null, () => {
console.log('Complete');
});
There are few hundred documents in my database. Schema is very simple:
var firmsSchema = mongoose.Schema({
name: String,
sections: [String],
});
I want to query documents and iterate over:
{{#each sections}}
{{sectionName}}
{{#each firms}}
{{firmName}}
{{/each}}
{{/each}}
Simple:
const SECTIONS = ['name_one', 'name_two', 'name_three'];
const UNSORTED_SECTION_NAME = 'unsorted';
router.get('/', function(req, res, next) {
var showFirms = function showSection (i, acc) {
if (i < 0) return;
let query = SECTIONS[i] ? {sections: SECTIONS[i]} : {sections: {$nin: SECTIONS}};
let key = SECTIONS[i] || UNSORTED_SECTION_NAME;
Firms.find(query).
then((result) => {
acc.push({
section: key,
firms: result,
});
if (i === SECTIONS.length) {
acc = acc.sort((a, b) => (a.section > b.section));
res.render('template', {
sections: acc,
});
}
}).
then(showSection (i - 1, acc));
}
showFirms(SECTIONS.length, []);
};
Works fine. Except it returns acc randomly and unpredictably sorted. I mean 'name_two' section can follow 'name_one' or vise versa.
I thought .sort() at the end of promises chain would be a silver bullet here and solve all asynchronous problems, but it didn't.
Of course i can sort acc with handlebars helper after i pass it to my template, but it is so ridiculously strange i can't sort it right after all queries have been done in my showFirms function.
Can you give me some advise please?
Look at this remake of your code. Instead of getting the data one by one, we gotta get them on the same time (asynchronously) and then treat the return.
If you have any questions I am here, this code is untested so give me a feedback. This is an example of how you can change your code.
const showFirms = function showSection() {
return new Promise((resolve, reject) => {
// Get the keys for the queries
const keys = SECTIONS.map(x => x || UNSORTED_SECTION_NAME);
// For each sections we gonna call a find request
const promises = SECTIONS.map((x, xi) => {
const query = x ? {
sections: x,
} : {
sections: {
$nin: SECTIONS,
},
};
const key = keys[xi];
return Firms.find(query);
});
// Resolve all promises
Promise.all(promises)
.then((rets) => {
// Use the finds results to build an acc array
const accs = rets.map((x, xi) => ({
section: keys[xi],
firms: x,
}));
// Change the sort -> ;) #comments
const sortedAccs = accs.sort((a, b) => (a.section > b.section));
resolve(sortedAccs);
})
.catch(reject);
});
};
How to use it
showFirms()
.then(accs => res.render('template', {
sections: accs,
}))
.catch(err => console.log(`I have an error ${err.toString()}`));
Based on Grégory NEUTS excellent solution. I simplified some things and made the "other sections case" work. I even dropped out sort functionality. Eventual result returns in order of sections as they were declared in the initial SECTIONS array, so now i can just reorder it to control output.
const showFirms = function () {
return new Promise ((resolve, reject) => {
const extendedSections = SECTIONS.slice();
extendedSections.push(UNSORTED_SECTION_NAME);
const promises = extendedSections.map((section) => {
const unsortedCase = section === UNSORTED_SECTION_NAME;
const query = unsortedCase ? {sections: {$nin: SECTIONS}} : {sections: section};
return Firms.find(query);
})
Promise.all(promises)
.then((allResponces) => {
const sectionsData = allResponces.map((response, i) => ({
section: extendedSections[i],
firms: response,
}));
resolve(sectionsData);
})
.catch(reject);
});
};