What im trying to do is to get only the source from the elastic search query in order to skip the processing on the javascript, so i could squize some more performance gains.Is it possible to do so?
So here is what i currenly do, i just get the data from the elastic search and iterate every one of those objects in order to construct an array that only contains the what's inside of _source:
const { body } = await this.elsticSearchService.search<any>({
index: this.configService.get('ELASTICSEARCH_INDEX'),
body: {
query: {
match_all: {},
},
size: 10000,
},
});
const hits = body.hits.hits;
return hits.map((item: any) => item._source);
So my question is, is there a way to only get the _source from the elastic search, in order
to skip the processing on the JavaScript?
So the object returned from the elastic search would look like this
[
0:{ "key": "value"}, // key,value from _source object
1:{ "key": "value"}, // key,value from _source object
2:{ "key": "value"}, // key,value from _source object
]
so without all of the other fields like hits, took etc...
It's not possible to change the structure of the response you get from the search call.
However, what you can do is to specify filter_path so you only get the _source content in the response and you wouldn't need to process it since you know you only have _source content
const { body } = await this.elsticSearchService.search<any>({
index: this.configService.get('ELASTICSEARCH_INDEX'),
filter_path: '**._source', <--- add this line
body: {
query: {
match_all: {},
},
size: 10000,
},
});
Related
EDIT: Re-structured question, cleaer, and cleaner:
I have a data object from Sequelize that is sent by node-express:
{
"page": 0,
"limit": 10,
"total": 4,
"data": [
{
"id": 1,
"title": "movies",
"isActive": true,
"createdAt": "2020-05-30T19:26:04.000Z",
"updatedAt": "2020-05-30T19:26:04.000Z",
"questions": [
{
"questionsCount": 4
}
]
}
]
}
The BIG question is, how do I get the value of questionsCount?
The PROBLEM is, I just can't extract it, these two methods give me undefined result:
category.questions[0].questionsCount
category.questions[0]['questionsCount']
I WAS ABLE to get it using toJSON() (From Sequelize lib I think), like so:
category.questions[0].toJSON().questionsCount
But I'd like to know the answer to the question, or at least a clear explanation of why do I have to use toJSON() just to get the questionsCount?
More context:
I have this GET in my controller:
exports.getCategories = (req, res) => {
const page = myUtil.parser.tryParseInt(req.query.page, 0)
const limit = myUtil.parser.tryParseInt(req.query.limit, 10)
db.Category.findAndCountAll({
where: {},
include: [
{
model: db.Question,
as: "questions",
attributes: [[db.Sequelize.fn('COUNT', 'id'), 'questionsCount']]
}
],
offset: limit * page,
limit: limit,
order: [["id", "ASC"]],
})
.then(data => {
data.rows.forEach(function(category) {
console.log("------ May 31 ----> " + JSON.stringify(category.questions[0]) + " -->" + category.questions[0].hasOwnProperty('questionsCount'))
console.log(JSON.stringify(category))
console.log(category.questions[0].toJSON().questionsCount)
})
res.json(myUtil.response.paging(data, page, limit))
})
.catch(err => {
console.log("Error get categories: " + err.message)
res.status(500).send({
message: "An error has occured while retrieving data."
})
})
}
I loop through the data.rows to get each category object.
The console.log outputs are:
------ May 31 ----> {"questionsCount":4} -->false
{"id":1,"title":"movies","isActive":true,"createdAt":"2020-05-30T19:26:04.000Z","updatedAt":"2020-05-30T19:26:04.000Z","questions":[{"questionsCount":4}]}
4
https://github.com/sequelize/sequelize/blob/master/docs/manual/core-concepts/model-querying-finders.md
By default, the results of all finder methods are instances of the model class (as opposed to being just plain JavaScript objects). This means that after the database returns the results, Sequelize automatically wraps everything in proper instance objects. In a few cases, when there are too many results, this wrapping can be inefficient. To disable this wrapping and receive a plain response instead, pass { raw: true } as an option to the finder method.
(emphasis by me)
Or directly in the source code, https://github.com/sequelize/sequelize/blob/59b8a7bfa018b94ccfa6e30e1040de91d1e3d3dd/lib/model.js#L2028
#returns {Promise<{count: number, rows: Model[]}>}
So the thing is that you get an array of Model objects which you could navigate with their get() method. It's an unfortunate coincidence that you expected an array, and got an array so you thought it is "that" array. Try the {raw:true} thing, I guess it looks something like this:
db.Category.findAndCountAll({
where: {},
include: [
{
model: db.Question,
as: "questions",
attributes: [[db.Sequelize.fn('COUNT', 'id'), 'questionsCount']]
}
],
offset: limit * page,
limit: limit,
order: [["id", "ASC"]],
raw: true // <--- hopefully it is this simple
}) [...]
toJSON() is nearby too, https://github.com/sequelize/sequelize/blob/59b8a7bfa018b94ccfa6e30e1040de91d1e3d3dd/lib/model.js#L4341
/**
* Convert the instance to a JSON representation.
* Proxies to calling `get` with no keys.
* This means get all values gotten from the DB, and apply all custom getters.
*
* #see
* {#link Model#get}
*
* #returns {object}
*/
toJSON() {
return _.cloneDeep(
this.get({
plain: true
})
);
}
So it worked exactly because it did what you needed, removed the get() stuff and provided an actual JavaScript object matching your structure (POJSO? - sorry, I could not resist). I rarely use it and thus always forget, but the key background "trick" is that a bit contrary to its name, toJSON() is not expected to create the actual JSON string, but to provide a replacement object which still gets stringified by JSON.stringify(). (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/stringify#toJSON_behavior)
try to do so category.data[0].questions.questionCount
As mentioned by others already, you need category.data[0].questions[0].questionCount.
Let me add to that by showing you why. Look at your object, I annotated it with how each part would be accessed:
category = { // category
"page": 0,
"limit": 10,
"total": 2,
"data": [ // category.data
{ // category.data[0]
"id": 1,
"title": "movies",
"createdAt": "2020-05-30T19:26:04.000Z",
"updatedAt": "2020-05-30T19:26:04.000Z",
"questions": [ // category.data[0].questions
{ // category.data[0].questions[0]
"questionCount": 2 // category.data[0].questions[0].questionCount
}
],
"questionsCount": "newValue here!"
}
]
}
try this
category.data[0].questions[0].questionCount
the reason why you have to use toJSON is because it's sometimes it is used to customise the stringification behavior. like doing some calculation before assinging the value to the object that will be returned , so it is most likley been used here to calculate the "numb of questions and then return an object with the property questionscount and the number calculated
so the object you retreived more or less looks like this
var cathegory = {
data: 'data',
questions:[{
// some calulation here to get the questionsCount
result=4,
toJSON () {
return {"questionsCount":this.result}
}
}
]
};
console.log(cathegory.questions[0].toJSON().questionsCount) //4
console.log(JSON.stringify(cathegory)) // {"data":"data","questions":[{"questionsCount":4}]}
console.log("------ May 31 ----> " + JSON.stringify(cathegory.questions[0]) + " -->" + cathegory.questions[0].hasOwnProperty('questionsCount')) //false
I have an array of Facet Values that I need to gather from an Algolia Indices.
For example, these are: "Beds", "Occupancy". and "Floor".
At the moment, i've got the below code which will go to my Algolia Table, grab me all of the possible values for each of the above but I have to do a query into Algolia for each one. This results in 3 network calls to Algolia.
index.searchForFacetValues(
{
facetName: val,
facetQuery: "",
maxFacetHits: 100,
query: "2019"
},
function(err, content) {
return content
})
Is there a way that I can get all the facet values for "Beds", "Occupancy". and "Floor" in a single query resulting in only one network call?
Also, i'm using https://www.algolia.com/doc/api-client/getting-started/install/javascript/
You can use an empty search and specify which facets you want to receive values for:
client.search({
query: '',
facets: ['Beds', 'Occupancy', 'Floor'],
attributesToRetrieve: [], // a little optimisation for response transfer speed
});
The response will contain something like this:
{
"facets": {
"Beds": {
"2": 1245,
"4": 893,
...
},
"Floor": {
...
},
...
}
}
So the first level of keys in facets are your facet names, and within each nested facet object you have one key/value per facet value/hits count.
If you don't know in advance the list of facet names you want to get, use facets: '*' in your query parameters.
I have parse server which I have this query on
const empList = Parse.Object.extend("EmpList");
const query = new Parse.Query(empList);
query.equalTo("relation", Parse.User.current());
query.find({
success: (results) => {
// results.map((each)=>this.data = each.id)
this.data = results
},
Parse server has 2 classes ( users and EmpList), there is a pointer in EmpList which points correctly to the current user submitted the employee.The returned data also includes the pointer ( which has a name "relation" ) and I can see the username exists as an object in this fashion
Array [
Object {
"createdAt" "2018-07-05T20 17 45.173Z",
"name" "1the latter",
"objectId" "D2kThKcg9z",
"phone" "111",
"relation" Object {
"ACL" Object {
"*" Object {
"read" true,
},
"H2rQJxNyTD" Object {
"read" true,
"write" true,
},
},
"__type" "Object",
"className" "_User",
"createdAt" "2018-07-05T18 43 40.536Z",
"objectId" "H2rQJxNyTD",
"password" undefined,
"sessionToken" "r c45063a034dd81d646bef51ae2055c85",
"updatedAt" "2018-07-05T20 17 35.976Z",
"username" "1",
},
"shift" "tue",
"updatedAt" "2018-07-05T20 22 11.158Z",
},
]
Yet I am unable to extract the username in relation object.
please help me do so by data[0].relation.username or data[0].relation().username
There are two things you can do. The simplest is to eagerly fetch the related object as part of the query using include() ...
const query = new Parse.Query(empList);
query.equalTo("relation", Parse.User.current());
query.include("relation");
(Incidentally, "relation" isn't a great name for that column. A better choice would be something relating to it's meaning, like submittedByUser. Calling it relation is like naming your poodle "Poodle").
The drawback of using include is that it will eagerly fetch all of the related objects on the query, making the query take longer and potentially produce data that you don't need. If you only want the related object on one or a handful of the query results, skip the include() and query the relations individually...
const query = new Parse.Query(empList);
query.equalTo("relation", Parse.User.current());
query.find({
success: results => {
// for one or some of the results...
let submittedByUserRelation = user.relation("relation");
submittedByUserRelation.query().find({
success: user => {
// user.username will be the username
}
});
Check the query() function on the relation that you can use to get all the object in the relation .
What I am trying to do
I am creating a social media app with react native and firebase. I am trying to call a function, and have that function return a list of posts from off of my server.
Problem
Using the return method on a firebase query gives me a hard to use object array:
Array [
Object {
"-L2mDBZ6gqY6ANJD6rg1": Object {
//...
},
},
]
I don't like how there is an object inside of an object, and the whole thing is very hard to work with. I created a list inside my app and named it items, and when pushing all of the values to that, I got a much easier to work with object:
Array [
Object {
//...
"key": "-L2mDBZ6gqY6ANJD6rg1",
},
]
This object is also a lot nicer to use because the key is not the name of the object, but inside of it.
I would just return the array I made, but that returns as undefined.
My question
In a function, how can I return an array I created using a firebase query? (to get the objects of an array)
My Code
runQ(group){
var items = [];
//I am returning the entire firebase query...
return firebase.database().ref('posts/'+group).orderByKey().once ('value', (snap) => {
snap.forEach ( (child) => {
items.push({
//post contents
});
});
console.log(items)
//... but all I want to return is the items array. This returns undefined though.
})
}
Please let me know if I'm getting your question correctly. So, the posts table in database looks like this right now:
And you want to return these posts in this manner:
[
{
"key": "-L1ELDwqJqm17iBI4UZu",
"message": "post 1"
},
{
"key": "-L1ELOuuf9hOdydnI3HU",
"message": "post 2"
},
{
"key": "-L1ELqFi7X9lm6ssOd5d",
"message": "post 3"
},
{
"key": "-L1EMH-Co64-RAQ1-AvU",
"message": "post 4"
}
...
]
Is this correct? If so, here's what you're suppose to do:
var items = [];
firebase.database().ref('posts').orderByKey().once('value', (snapshot) => {
snapshot.forEach((child) => {
// 'key' might not be a part of the post, if you do want to
// include the key as well, then use this code instead
//
// const post = child.val();
// const key = child.key;
// items.push({ ...post, key });
//
// Otherwise, the following line is enough
items.push(child.val());
});
// Then, do something with the 'items' array here
})
.catch(() => { });
Off the topics here: I see that you're using firebase.database().... to fetch posts from the database, are you using cloud functions or you're fetching those posts in your App, using users' devices to do so? If it's the latter, you probably would rather use cloud functions and pagination to fetch posts, mainly because of 2 reasons:
There might be too many posts to fetch at one time
This causes security issues, because you're allowing every device to connect to your database (you'd have to come up with real good security rules to keep your database safe)
Continuing yesterday's saga, now I can retrieve json objects in a response but I can't extract the data from them.
The following node.js snippet is from the file "accounts.js" which is in an ETrade api library that exists in the path /lib. It returns json containing data about the accounts of the authenticated user. The authentication part is working great.
exports.listAccounts = function(successCallback,errorCallback)
{
var actionDescriptor = {
method : "GET",
module : "accounts",
action : "accountlist",
useJSON: true,
};
this._run(actionDescriptor,{},successCallback,errorCallback);
};
The ETrade website says this call will produce the following sample response:
{
"AccountListResponse": {
"Account": [
{
"accountDesc": "MyAccount-1",
"accountId": "83405188",
"marginLevel": "MARGIN",
"netAccountValue": "9999871.82",
"registrationType": "INDIVIDUAL"
},
{
"accountDesc": "MyAccount-3",
"accountId": "83405553",
"marginLevel": "CASH",
"netAccountValue": "100105468.99",
"registrationType": "INDIVIDUAL"
},
{
"accountDesc": "SIMPLE IRA",
"accountId": "83405188",
"marginLevel": "CASH",
"netAccountValue": "99794.13",
"registrationType": "IRA"
}
]
}
}
In my app.js file, I have the following:
var etrade = require('./lib/etrade');
var et = new etrade(configuration);
et.listAccounts(
function(res){
var listAccountsRes = res;
console.log('account list success!');
console.log(listAccountsRes)
},
function(error) {
console.log("Error encountered while attempting " +
"to retrieve account list: " +
error);
});
When I run this code, the console log shows the following message:
{ 'json.accountListResponse':
{ response:
[ [Object],
[ [Object],
[ [Object],
[ [Object],
[ [Object],
[ [Object],
[ [Object],
[ [Object] ] } }
Suppose in app.js I want to put the accounts data in a variable called myAccounts.
One of our members, Jack, solved yesterday's problem and when I commented that I still couldn't access the data in the response, he suggested this: "That property has a dot in it so you'll have to use [ ... ] rather than dot notation to access it. See what's inside the objects with a['json.accountListResponse'].response." So far I have not been able to get that to work, even when I use ['json.accountListResponse'].res like this:
var listAccountsRes = [json.accountListResponse].res;
This returns undefined when printed to the console.
Thanks to Adam for his suggestion which led to this which works:
var listAccountsRes = res['json.accountListResponse'];
var listAccounts = listAccountsRes['response'];
console.log('account list success!');
console.log(listAccounts)
Now the console log reports almost exactly what ETrade says I should get. (They appear to have changed the name "Account" to "response"). I presume my variable listAccounts now contains the json with eight sample accounts in it that I can see in my console log. But I still don't know how to access individual elements. There should be some simple code that will iterate over the json file and produce an array of arrays that I could actually use for something. I tried accessing it like an array: console.log(listAccounts[0]) but that returns undefined. Do I need to stringify it or something?