I need help with prefix autocomplete suggestions from elasticsearch 7.3 documents with a multiple fields search and an associated field type in the output to categorize the suggested term type.
tried with search_as_you_type field type and adding multiple completion analyzers with no luck so far. I wanted a solution that is very easy to implement, eliminates duplicates and reduce my memory footprint. Something like a Redis sorted set.
Here is my el indexed docs and expected auto-complete results.
{
"id": "7cBLdyLTkPNTC8c1Ntq",
"tech": {
"body": "Nikon D600",
"gear": "Nikon AF-S Nikkor 16-35mm f/4 ED VR"
},
"artist": {
"name": " Hilton Paris"
},
"details": {
"place": "Nepal",
"text": "Mount Everest .Natures mirror",
"tags": [
"mountain",
"rock",
"nature",
"wooden trail",
"mirror",
"past and furious"
],
"score": 0.87
},
"createdDate": {
"_seconds": 1566260863,
"_nanoseconds": 12000000
}
}
{
"id": "8cBLdyLTkPNTC8c1Ntq",
"tech": {
"body": "Panasonic Lumix DC-S1R Mirrorless",
"gear": "Panasonic Lumix S PRO 50mm"
},
"artist" : {
"name": "Jhon Gabriel"
},
"details": {
"place": "paris",
"text": "mirror lake reflection",
"tags": [
"lake",
"rock",
"nature",
"water"
],
"score": 0.87
},
"createdDate": {
"_seconds": 1566260863,
"_nanoseconds": 12000000
},
}
search for "pa" should give the following output.
output [ {
"type" :"artist",
"value" : "Paris Peace",
},
{
"type" :"body",
"value" : "Panasonic",
},
{
"type" :"gear",
"value" : " Panasonic Lumix S PRO 50mm"
},
{
"type" :"place",
"value" : "paris"
},
{
"type" : "tag",
"value" : "past and furious"
}
]
What i need is
for a given input prefix e.g 'pa', get all the matching terms from the elsticsearch datastore with the following conditions.
input satisfies a case insensitive prefix to one of the following fields
1.1 "tech.body" ---> token level prefix matching, output only the token value
1.2 "tech.gear" ---> token level prefix matching, output full string value of the filed.
e.g if tehc.gear has a token with 'pa' as the prefix,
output should be the full value of this field ,like "Panasonic Lumix S PRO 50mm"
1.3 "artist.name" --> token level prefix matching
1.4 "details.place" --> token level prefix matching
1.5 "details.tags" --> token level prefix matching, output full string value of the filed, eg. pa--> "past and furious"
1.6 "details.text" --> token level prefix matching
Any help here is highly appreciated, as this is turning out to be a blocker for an upcoming release.
Related
I am using Elastic version 6.8, created one index into whose schema is as follow:
{
"properties": {
"title": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
},
"tags": {
"type": "keyword",
"fields": {
"raw": {
"type": "text"
}
}
}
}}
and I have added following documents into it
[{
"title": "one",
"tags": ["html", "css", "javascript"]
}, {
"title": "two",
"tags": ["java", "jsp", "servlet"]
}, {
"title": "three",
"tags": ["spring", "java"]
}, {
"title": "four",
"tags": ["react", "angular", "javascript"]
}, {
"title": "five",
"tags": ["java"]
}, {
"title": "six",
"tags": []
}]
now I have more than 10 millions document in elastic search. Now I want to search following cases:
List all tags. with unique result (using skip, limit) skip value change but limit is fixed.so here I want result like
html,
css,
javascript,
java,
jsp,
servlet,
spring,
react,
angular
Partil search inside tags, it means if I search using act then it should give result as : react this also using skip limit.
How I can get this using Elastic search query. please help me here?
You can find unique possible value by using term aggregation.
GET yourindex/_search
{
"size": 0,
"aggs": {
"all_tags": {
"terms": {
"field": "tags",
"size": 100
}
}
}
}
"size":100 Get at most 100 unique values. Default is 10. You can increase more but it will include cost. You can check more on doc.
For partial search you can use wildcard query OR you can try N-Gram Tokeninzer. Both will allow to do partial search but wildcard query will be costly. You can evaluate according to your use case.
I needed assistance in order to work out why the aggregate function is not responding the way I'd expect it to respond. This is a RESTful API service I've designed in which I am trying to connect collections with each other. Please note the following:
Collection: Season
{
"_id": {
"$oid": "5c0fc60bfb6fc04dd6ea4e9a"
},
"Season": "1",
"TotalEpisode": "15",
"Name": null,
"Description": "First season with no name for this drama",
"PlayID": "5c0fc4aafb6fc04dd6ea4d81"
}
Collection: Play
{
"_id": {
"$oid": "5c0fc4aafb6fc04dd6ea4d81"
},
"Name": "It was the first time",
"Description": "One of the best action heros in the entertainment industry until this day",
"ReleaseDate": "24/12/2010",
"EndingDate": "12/08/2012",
"Category": "Drama"
}
My implemented code in JavaScript
function getTestLookUp(db, collectionName, response, secondCollectionName){
console.log('First collection name: ' + collectionName + '\n' + 'Second collection name: ' + secondCollectionName);
db.collection(collectionName).aggregate([
{
$lookup:
{
from: secondCollectionName,
localField: 'PlayID',
foreignField: '_id',
as: 'requestedDetails'
}
}
]).toArray((err, res) => {
if(err){
console.log(err);
} else {
console.log(res);
response.status(200).json({
'Items': res
});
}
});
}
The response
{
"Items": [
{
"_id": "5c0fc60bfb6fc04dd6ea4e9a",
"Season": "1",
"TotalEpisode": "15",
"Name": null,
"Description": "First season with no name for this drama",
"PlayID": "5c0fc4aafb6fc04dd6ea4d81",
"requestedDetails": []
}
]
}
The things I've checked so far: the collection names are accurate, the ID is also accurate as I can search it up on the MLabs search feature. I don't understand as to why this is returning a empty 'requestedDetails' as I hoped it would return the item from the Play collection.
In addition to this, I would also appreciate if someone can point out how I can join multiple collections instead of 2.
I welcome any questions regarding this problem.
While still researching for this issue, I accidentally came across a another problem in which someone wrote a comment stating that "you might be comparing a String with ObjectID". This was the cause for this error as I obtain a String variable in return from the database and I am comparing the String variable with the _id which is expecting to see a ObjectID variable to complete the query. Therefore, meaning that my query/lookup is never matching these two variables.
The only way tackle this issue is to do a conversion (string to ObjectID) and then compare the values. However, since I'm using the version of ^3.1.10 of MongoDB, this functionality is not possible. Will need to update the version to 4.0 to be able to implement this functionality.
In order to rectify this issue, I managed to surround the foreign ID within $iod tags.
Before
{
"_id": {
"$oid": "5c0fc60bfb6fc04dd6ea4e9a"
},
"Season": "1",
"TotalEpisode": "15",
"Name": null,
"Description": "First season with no name for this drama",
"PlayID": "5c0fc4aafb6fc04dd6ea4d81"
}
After
{
"_id": {
"$oid": "5c0fc60bfb6fc04dd6ea4e9a"
},
"Season": "1",
"TotalEpisode": "15",
"Name": null,
"Description": "First season with no name for this drama",
"PlayID": {
"$oid": "5c0fc4aafb6fc04dd6ea4d81"
}
}
Response
{
"Items": [
{
"_id": "5c0fc60bfb6fc04dd6ea4e9a",
"Season": "1",
"TotalEpisode": "15",
"Name": null,
"Description": "First season with no name for this drama",
"PlayID": "5c0fc4aafb6fc04dd6ea4d81",
"Details": [
{
"_id": "5c0fc4aafb6fc04dd6ea4d81",
"Name": "It was the first time",
"Description": "One of the best action heros in the entertainment industry until this day",
"ReleaseDate": "24/12/2010",
"EndingDate": "12/08/2012",
"Category": "Drama"
}
]
}
]
}
I am trying to upload my intent everything is working fine, i am writing script in V2 and everything works but for parameter webhookState data type is enum ( [here] ) and we have to enter following of the three values ( [here][1]) now when i enter any one and try to upload it shows.
" Unable to load file: SyntaxError: Unexpected token W in JSON at position 98 "
now when I pass WEBHOOK_STATE_ENABLED(or any one) in quotes code uploads successfully (because as per my knowledge it treats it as string and gets executed) and intent is created for my agent but webhook remains off and also training phrases are not there, i am certain that the parameter webhookState is not being activated when i pass in quotes(because of above mentioned reason) Same is the case with rest, like when i try to put in type of training phrases.
{
"name": "Warehouse_Management",
"displayName": "Warehouse_Management",
"webhookState": "WEBHOOK_STATE_ENABLED_FOR_SLOT_FILLING" ,
"priority": 50000,
"isFallback": false,
"mlDisabled": false,
"trainingPhrases": [
{
"name":"Try1" ,
"type": "EXAMPLE",
"parts": [
{
"text": "for",
"userDefined": true
},
{
"text": "warehouse",
"entityType": "#Properties",
"alias": "Properties",
"userDefined": true
},
{
"text": "management",
"userDefined": true
}
]
},
{
"name":"Try2" ,
"type": "EXAMPLE",
"parts": [
{
"text": "i want app for ",
"userDefined": true
},
{
"text": "warehouse",
"alias": "Properties",
"entityType": "#Properties",
"userDefined": true
}
]
}
],
"outputContexts": [
{
"name": "Yes",
"lifespanCount": 2
},
{
"name": "No",
"lifespanCount": 2
},
{
"name": "Device_Integration",
"lifespanCount": 2
}
],
"resetContexts": false,
"parameters": [
{
"name": "Properties",
"displayName": "Properties",
"value": "$parameter_name",
"entityTypeDisplayName": "#Properties",
"mandatory": false,
"isList": true
}
],
"messages": [
{
"text":"This is sample response"
}
],
"rootFollowupIntentName": "root",
"parentFollowupIntentName": "parent"
}
Please NOTE that in the attached Json i have put it in quotes so it would get successfully executed and will create intent.
In order for it to be valid JSON, the enum value should be wrapped in quotes. It's expecting the enum value as a String.
I am trying to access the card json value, to no avail.
In my scenario, I am asking the bot about "weather in London" and it replies back with "It is currently 9 degrees celcius in London." via the webhook.
Which is correct and dynamic.
However, I am trying to also pass the values to a card too.
In the json reply, I do get the card as so
{
"id": "REMOVED",
"timestamp": "2017-12-05T11:10:52.033Z",
"lang": "en",
"result": {
"source": "agent",
"resolvedQuery": "weather in london",
"action": "sayWeather",
"actionIncomplete": false,
"parameters": {
"geo-city": "London"
},
"contexts": [],
"metadata": {
"intentId": "REMOVED",
"webhookUsed": "true",
"webhookForSlotFillingUsed": "false",
"webhookResponseTime": 626,
"intentName": "Weather"
},
"fulfillment": {
"speech": "It is currently 9 degrees celcius in London.",
"source": "agent",
"displayText": "It is currently 9 degrees celcius in London.",
"messages": [
{
"type": 0,
"speech": "It is currently 9 degrees celcius in London."
}
],
"data": {
"items": [
{
"simpleResponse": {
"textToSpeech": "This is the first simple response for a basic card"
}
},
{
"basicCard": {
"title": "Title: this is a title",
"formattedText": "This is a basic card. Text in a\n basic card can include \"quotes\" and most other unicode characters\n including emoji 📱. Basic cards also support some markdown\n formatting like *emphasis* or _italics_, **strong** or __bold__,\n and ***bold itallic*** or ___strong emphasis___ as well as other things\n like line \nbreaks",
"subtitle": "This is a subtitle",
"image": {
"url": "https://developers.google.com/actions/images/badges/XPM_BADGING_GoogleAssistant_VER.png",
"accessibilityText": "Image alternate text"
},
"buttons": [
{
"title": "This is a button",
"openUrlAction": {
"url": "https://assistant.google.com/"
}
}
]
}
},
{
"simpleResponse": {
"textToSpeech": "This is the 2nd simple response ",
"displayText": "This is the 2nd simple response"
}
}
]
}
},
"score": 1
},
"status": {
"code": 200,
"errorType": "success",
"webhookTimedOut": false
},
"sessionId": "REMOVED"
}
Accessing the value of speech using data.result.fulfillment.speech works fine.
However, when using data.result.fulfillment.data.items.basicCard.image.url it just doesnt work. And if I go up several levels, I do get:
[object Object]
Your help is appreciated.
The items attribute is a list and not an object. As such, you'll have to use a numerical index to retrive the data. In the example you provided the index of the basicCard object is second so your code should look something like this:
data.result.fulfillment.data.items[1].basicCard.image.url
Notice the [1] after items.
Bear in mind that if the order of this list changes you may no longer be retrieving a basicCard object so you may want to add some checking to make sure you're retrieving the data you want.
Does anyone know the format of the arrays or objects that need to be passed to the "to" and "message_tags" fields that are outlined on this page of the documentation https://developers.facebook.com/docs/reference/api/post/
The "to" field should look like this:
"to": {
"data": [
{
"name": "Abc Def",
"id": "##########"
}
]
}
The "message_tags" field looks like the following. Note that what I'm denoting as '#1#' is the offset from the beginning of your message text where the first character of the tag appears and '#2#' is the length of your tag.
"message_tags": {
"#1#": [
{
"id": "##########",
"name": "Abc Def",
"type": "user",
"offset": #1#,
"length": #2#
}
]
}
I used the Facebook Graph Explorer on an existing object to find the syntax.