We Keep Coding

node-redis/search query elements by geospatial position and additional indexes

I want to be able to query elements in a redis cache based on 3 different indexes. Those indexes would be:
A MAC address stored as a String.
A number.
A latitude and longitude(to be able to query spatially).
I have seen that Redis has support for multi indexing using redis search and native geospatial api.
so using nodejs and node-redis I have written the following index:
client.ft.create(
'idx:cits',
{
mid: {
type: SchemaFieldTypes.TEXT
},
timestamp: {
type: SchemaFieldTypes.NUMERIC,
sortable: true
},
position: {
type: SchemaFieldTypes.GEO
}
},
{
ON: 'HASH',
PREFIX: 'CITS'
}
)
Now, i would like to insert records on the database that include those 3 parameters plus an additional String that stores some payload. I have tried using
await client.hSet('CITS:19123123:0:0:00:00:5e:00:53:af', {
timestamp: 19123123,
position: {latitude:0, longitude:0},
mid: '00:00:5e:00:53:af',
message: 'payload'
})
But I get the following error:
throw new TypeError('Invalid argument type');
^
TypeError: Invalid argument type
So, i can't add the latitude and longitude that way, I also tried
using the module ngeohash and computing an 11 character wide geohash like so:
await client.hSet('CITS:19123123:0:0:00:00:5e:00:53:af', {
timestamp: 19123123,
position: geohash.encode(0, 0, 11),
mid: '00:00:5e:00:53:af',
message: 'payload'
})
And it does not give any error but when using redis search querys It does not find points near it.
Is it even possible what I am trying to do? If so, how would you input the data to the redis database?
Here is a minimal reproducible example (Im using "ngeohash": "^0.6.3" and "redis": "^4.5.0"):
const { createClient, SchemaFieldTypes } = require('redis')
const geohash = require('ngeohash')
const client = createClient()
async function start(client) {
await client.connect()
try {
// We only want to sort by these 3 values
await client.ft.create(
'idx:cits',
{
mid: {
type: SchemaFieldTypes.TEXT
},
timestamp: {
type: SchemaFieldTypes.NUMERIC,
sortable: true
},
position: {
type: SchemaFieldTypes.GEO
}
},
{
ON: 'HASH',
PREFIX: 'CITS'
}
)
} catch (e) {
if (e.message === 'Index already exists') {
console.log('Skipping index creation as it already exists.')
} else {
console.error(e)
process.exit(1)
}
}
await client.hSet('CITS:19123123:0:0:00:00:5e:00:53:af', {
timestamp: 19123123,
position: geohash.encode(0, 0, 11),
mid: '00:00:5e:00:53:af',
message: 'payload'
})
await client.hSet('CITS:19123123:0.001:0.001:ff:ff:ff:ff:ff:ff', {
timestamp: 19123123,
position: geohash.encode(0.001, 0.001, 11),
mid: 'ff:ff:ff:ff:ff:ff',
message: 'payload'
})
const results = await client.ft.search(
'idx:cits',
'#position:[0 0 10000 km]'
)
console.log(results)
await client.quit()
}
start(client)
Additionally, I would like to ask if there is maybe another type of database that better suits my needs. I have chosen redis because it offers low latency, and that is the biggest constraint in my environment(I will probably do more writes than reads per second). I only want it to act as a inmediate cache, as persistent data will be stored in another database that does not need to be fast.
Thank you.

You get the Invalid argument type error because Redis does not support nested fields in hashes.
"GEO allows geographic range queries against the value in this attribute. The value of the attribute must be a string containing a longitude (first) and latitude separated by a comma" (https://redis.io/commands/ft.create/)

Retrive some columns of relations in typeorm

I need to retrieve just some columns of relations in typeorm query.
I have an entity Environment that has an relation with Document, I want select environment with just url of document, how to do this in typeorm findOne/findAndCount methods?

To do that you have to use a querybuilder, here's an example:
return this.createQueryBuilder('environment') // use this if the query used inside of your entity's repository or getRepository(Environment)...
.select(["environment.id","environment.xx","environment.xx","document.url"])
.leftJoin("environment.document", "document")
.where("environment.id = :id ", { id: id })
.getOne();

Sorry I can't add comment to post above. If you by not parsed data mean something like "environment.id" instead of "id"
try this:
return this.createQueryBuilder("environment")
.getRepository(Environment)
.select([
"environment.id AS id",
"environment.xx AS xx",
"document.url AS url",
])
.leftJoin("environment.document", "document")
.where("environment.id = :id ", { id: id })
.getRawOne();

Here is the code that works for me, and it doesn't require using the QueryBuilder. I'm using the EntityManager approach, so assuming you have one of those from an existing DataSource, try this:
const environment = await this.entityManager.findOne(Environment, {
select: {
document: {
url: true,
}
},
relations: {
document: true
},
where: {
id: environmentId
},
});
Even though the Environment attributes are not specified in the select clause, my experience is that they are all returned in the results, along with document.url.
In one of the applications that I'm working on, I have the need to bring back attributes from doubled-nested relationships, and I've gotten that to work in a similar way, shown below.
Assuming an object model where an Episode has many CareTeamMembers, and each CareTeamMember has a User, something like the code below will fetch all episodes (all attributes) along with the first and last name of the associated Users:
const episodes = await this.entityManager.find(Episode, {
select: {
careTeamMembers: {
id: true, // Required for this to work
user: {
id: true,
firstName: true,
lastName: true,
},
}
},
relations: {
careTeamMembers: {
user: true,
}
},
where: {
deleted: false,
},
});
For some reason, I have to include at least one attribute from the CareTeamMembers entity itself (I'm using the id) for this approach to work.

Send object as get parameter with Axios

The API expects the following GET request:
/resource?labels[team1]=corona&labels[team2]=virus
My issue is to generate this URL using the Axios params option.
I tried the following params structures and they both generate the wrong url:
{
labels: {
team1: 'corona',
team2: 'virus'
}
}
{
labels: [
team1: 'corona',
team2: 'virus'
]
}
I at least thought it would work with the string indexed array but that generates no get params at all.
So, can anyone tell me how to generate the desired URL?

The solution was to use the paramsSerializer with the same setup as in the axios readme. And I used the first params object from my post above..
paramsSerializer: (params) => {
return qs.stringify(params, { arrayFormat: 'brackets' });
}

Elasticsearch - return only the source data part

I want to get only the json data from elasticsearch through node js.
I used
client.search({
index: 'dashboard',
type: 'test',
filterPath: 'hits.hits._source',
body: {
query: { match_all: {} }
}
})
It will show the result as:
{"_source":{"datatime":"2017-08-21 16:03:00","time_of_day":11},{"_source":{"datatime":"2017-08-21 16:03:00","time_of_day":222}]
May I know how to select only the data part in the source through node js, without "_source"? Only {"datatime":"2017-08-21 16:03:00","time_of_day":11},{"datatime":"2017-08-21 16:03:00","time_of_day":222}

Ok, solved.
can use hits.hits._source.theDataName, such as hits.hits._source.datatime , it will show "datatime":"2017-08-21 16:03:00"

How to create or replace a document in Elasticsearch?

I'm just trying to create or replace a document, but the API docs aren't clear on how to do this.
The upsert example shows a document being created with a counter value of 1:
client.update({
index: 'myindex',
type: 'mytype',
id: '777',
body: {
script: 'ctx._source.counter += 1',
upsert: {
counter: 1
}
}
}, function(error, response) {
// ...
})
I don't need the increment script, so I try to just send an upsert by itself, but I get a 400 Validation Failed: 1: script or doc is missing:
client.update({
index: "testindex",
type: "testType",
id: 1234,
body: {
upsert: {
itworks: true,
ok: "oh yeah"
}
}
})
Well, let's try something silly and include the doc twice; once under doc, which would presumably replace an existing document if there was one, and once under upsert, which would be used to create a new document otherwise:
client.update({
index: "testindex",
type: "testType",
id: 1234,
body: {
upsert: {
itworks: true,
ok: "oh yeah"
},
doc: {
itworks: true,
ok: "oh yeah"
}
}
})
That actually worked, 201 Created. 200 OK when I repeat it to replace the doc with a different one:
client.update({
index: "testindex",
type: "testType",
id: 1234,
body: {
upsert: {
whatever: "man"
},
doc: {
whatever: "man"
}
}
})
But when I check the doc (client.get({index: "testindex", type: "testType", id: 1234})), I see that instead of replacing the existing doc, the new doc was merged with it.
How do you just replace a doc in Elasticsearch, with the standard Node client? The HTTP API makes it look so simple, but I've tried a bunch of permutations of that in the Node client without success. Is there no replace command? Do I have to delete and then create?

In Elasticsearch, to replace a document you simply have to index a document with the same ID and it will be replaced automatically.
If you would like to update a document you can either do a scripted update, a partial update or both.
To do a partial doc update you simply
Partial document update:
client.update({
index: 'myindex',
type: 'mytype',
id: '1',
body: {
// put the partial document under the `doc` key
doc: {
title: 'Updated'
}
}
}, function (error, response) {
// ...
})
Scripted update:
client.update({
index: 'myindex',
type: 'mytype',
id: '1',
body: {
script: 'ctx._source.tags += tag',
params: { tag: 'some new tag' }
}
}, function (error, response) {
// ...
});
More info in Elasticsearch JS documentation

This is one of the reasons I always use index instead of create:
client.index({
"index": 'index-name',
"type": 'mapping type',
"id": 'id you want to create/replace',
'body': JSON.stringify(obj)
}, function (err, resp) {
if (err) {
console.error(err['message'])
}
else {
console.log(resp);
return resp
}
});