We Keep Coding

node-redis/search query elements by geospatial position and additional indexes

I want to be able to query elements in a redis cache based on 3 different indexes. Those indexes would be:
A MAC address stored as a String.
A number.
A latitude and longitude(to be able to query spatially).
I have seen that Redis has support for multi indexing using redis search and native geospatial api.
so using nodejs and node-redis I have written the following index:
client.ft.create(
'idx:cits',
{
mid: {
type: SchemaFieldTypes.TEXT
},
timestamp: {
type: SchemaFieldTypes.NUMERIC,
sortable: true
},
position: {
type: SchemaFieldTypes.GEO
}
},
{
ON: 'HASH',
PREFIX: 'CITS'
}
)
Now, i would like to insert records on the database that include those 3 parameters plus an additional String that stores some payload. I have tried using
await client.hSet('CITS:19123123:0:0:00:00:5e:00:53:af', {
timestamp: 19123123,
position: {latitude:0, longitude:0},
mid: '00:00:5e:00:53:af',
message: 'payload'
})
But I get the following error:
throw new TypeError('Invalid argument type');
^
TypeError: Invalid argument type
So, i can't add the latitude and longitude that way, I also tried
using the module ngeohash and computing an 11 character wide geohash like so:
await client.hSet('CITS:19123123:0:0:00:00:5e:00:53:af', {
timestamp: 19123123,
position: geohash.encode(0, 0, 11),
mid: '00:00:5e:00:53:af',
message: 'payload'
})
And it does not give any error but when using redis search querys It does not find points near it.
Is it even possible what I am trying to do? If so, how would you input the data to the redis database?
Here is a minimal reproducible example (Im using "ngeohash": "^0.6.3" and "redis": "^4.5.0"):
const { createClient, SchemaFieldTypes } = require('redis')
const geohash = require('ngeohash')
const client = createClient()
async function start(client) {
await client.connect()
try {
// We only want to sort by these 3 values
await client.ft.create(
'idx:cits',
{
mid: {
type: SchemaFieldTypes.TEXT
},
timestamp: {
type: SchemaFieldTypes.NUMERIC,
sortable: true
},
position: {
type: SchemaFieldTypes.GEO
}
},
{
ON: 'HASH',
PREFIX: 'CITS'
}
)
} catch (e) {
if (e.message === 'Index already exists') {
console.log('Skipping index creation as it already exists.')
} else {
console.error(e)
process.exit(1)
}
}
await client.hSet('CITS:19123123:0:0:00:00:5e:00:53:af', {
timestamp: 19123123,
position: geohash.encode(0, 0, 11),
mid: '00:00:5e:00:53:af',
message: 'payload'
})
await client.hSet('CITS:19123123:0.001:0.001:ff:ff:ff:ff:ff:ff', {
timestamp: 19123123,
position: geohash.encode(0.001, 0.001, 11),
mid: 'ff:ff:ff:ff:ff:ff',
message: 'payload'
})
const results = await client.ft.search(
'idx:cits',
'#position:[0 0 10000 km]'
)
console.log(results)
await client.quit()
}
start(client)
Additionally, I would like to ask if there is maybe another type of database that better suits my needs. I have chosen redis because it offers low latency, and that is the biggest constraint in my environment(I will probably do more writes than reads per second). I only want it to act as a inmediate cache, as persistent data will be stored in another database that does not need to be fast.
Thank you.

You get the Invalid argument type error because Redis does not support nested fields in hashes.
"GEO allows geographic range queries against the value in this attribute. The value of the attribute must be a string containing a longitude (first) and latitude separated by a comma" (https://redis.io/commands/ft.create/)

Dynamically return config value (node.js)

I'm building a content middleware which gather contents from our external publishers. The publishers will share their contents either in rss or json and the key/value field would be different from each other. To make thing easier, I created a config file where I can pre-defined the key/value and the feed type. The problem is, how can I dynamically return this config value based on publishers name.
Example: To get Publisher #1 feed type, I just can use config.articles.rojak_daily.url_feed
my config file /config/index.js
module.exports = {
batch:100,
mysql: {
database: process.env.database,
host: process.env.host,
username: process.env.username,
password: process.env.password
},
articles:{
rojak_daily:{ // Publisher 1
url: 'xxx',
url_feed: 'rss',
id: null,
Name: 'title',
Description: 'description',
Link: 'link',
DatePublishFrom: 'pubDate',
LandscapeImage: 's3image',
SiteName: 'Rojak Daily',
SiteLogo: null
},
rojak_weekly:{ // publisher 2
url: 'xxx',
url_feed: 'json',
id: null,
Name: 'Name',
Description: 'Desc',
Link: 'link',
DatePublishFrom: 'pubDate',
LandscapeImage: 's3image',
SiteName: 'Rojak Weekly',
SiteLogo: null
}
}
}
my main application script
const config = require('#config'); // export from config file
class Main {
constructor(){
this.publishers = ['rojak_daily','rojak_weekly'];
}
// Main process
async startExport(){
try{
for(let publisher of this.publishers){
const feedType = await this.getFeedType(publisher)
const result = (feedType == 'rss')? await this.rss.start(publisher): await this.json.start(publisher)
return result
}
}catch(err){
console.log("Error occured: ", err)
}
}
// Get feed type from config
async getFeedType(publisher){
return await config.articles.rojak_daily.url_feed;
// this only return publisher 1 url feed.
// my concern is to dynamically passing variable
// into this config file (example: config.articles.<publisher>.url_feed)
}
}
module.exports = Main

async getFeedType(publisher){
return await config.articles[publisher].url_feed;
}
You can access properties of objects by variable

You could either loop over the articles by using Object.entries(articles) or Object.values(articles) in conjunction with Array.prototype.forEach(), or since you already have the name of the publisher you could access that entry with config.articles[publisher].url_feed, like so:
const config = {
articles: {
rojak_daily: { // Publisher 1
url: 'xxx',
url_feed: 'rss',
id: null,
Name: 'title',
Description: 'description',
Link: 'link',
DatePublishFrom: 'pubDate',
LandscapeImage: 's3image',
SiteName: 'Rojak Daily',
SiteLogo: null
},
rojak_weekly: { // publisher 2
url: 'xxx',
url_feed: 'json',
id: null,
Name: 'Name',
Description: 'Desc',
Link: 'link',
DatePublishFrom: 'pubDate',
LandscapeImage: 's3image',
SiteName: 'Rojak Weekly',
SiteLogo: null
}
}
}
const publishers = ['rojak_daily', 'rojak_weekly']
function getFeedType(publisher) {
return config.articles[publisher].url_feed;
}
publishers.forEach(publisher => console.log(getFeedType(publisher)));

JavaScript: TypeError: Converting circular structure to JSON when posting to discord

I have the array sessionCookies which looks like this:
[ { name: 'cookie_name',
value: 'I WANT THIS VALUE ONLY',
domain: 'cookie domain',
path: '/',
expires: -1,
size: 43,
httpOnly: true,
secure: true,
session: true } ]
I am then creating and using the function findCookie to get the value "I WANT THIS VALUE ONLY" seen in the array above using this code:
function findCookie(name) {
for (let i in sessionCookies) {
if (!sessionCookies.hasOwnProperty(i)) {
continue;
}
if (sessionCookies[i].name === name) {
return (sessionCookies[i].value);
}
}
}
const mycookievalue = findCookie('cookie_name');
console.log(mycookievalue); then obviously prints out: I WANT THIS VALUE ONLY
I am then trying to send that variable mycookievalue to discord using the webhook-discord library however it is throwing me the error: TypeError: Converting circular structure to JSON. This is the code I use to send the webhook:
const webhook = require("webhook-discord")
const Hook = new webhook.Webhook("my webhook link")
const msg = new webhook.MessageBuilder()
.setName("Cookie")
.setColor("#2fed5c")
.setText("Got the cookie")
.addField("My Cookie", mycookievalue)
.setTime();
Hook.send(msg);
That sending to discord is throwing me the error as stated above which I do not know how to get rid of since JSON.stringify will not work either. Any help is appreciated!

Error Handling: E11000 duplicate key error collection

I am having a problem with the user model that I'm using with Mongoose and MongoDB to create each profile in my database. It works fine to post one user, but throws the following error if I logout and try again:
{
"name": "MongoError",
"message": "E11000 duplicate key error collection: CourtAPIDev.users index: trackers.case_id_1 dup key: { : null }",
"driver": true,
"index": 0,
"code": 11000,
"errmsg": "E11000 duplicate key error collection: CourtAPIDev.users index: trackers.case_id_1 dup key: { : null }"
}
According to mongoose documentation: If there is more than one document (a second user) without a value for the indexed field or is missing the indexed field, the index build will fail with a duplicate key error. I don't know how to set this _id property for the trackers property –– I thought it generated automatically!
Here's the trackers part of my Schema. And the relevant case_id property, which seems to be throwing the "null" error.
The whole repository can be found on my Github here, but the likely problem spots are the ones I highlighted, I think. Here's the github link: https://github.com/KingOfCramers/node_login_with_trackers
user model:
const UserSchema = new mongoose.Schema({
email: {
type: String,
required: true,
trim: true,
minLength: 1,
unique: true,
validate: {
validator: (value) => {
return validator.isEmail(value);
},
message: '{VALUE} is not a valid email'
}
},
password: {
type: String,
required: true,
minlength: 6
},
tokens: [{
access: {
type: String,
required: true
},
token: {
type: String,
required: true
}
}],
trackers: {
tweets: [TwitterSchema],
legislation: [LegislationSchema],
court_cases: [CourtCaseSchema]
},
frequency: [EmailSchema]
});
Express route:
app.post("/users", (req,res) => {
var body = _.pick(req.body, ['email', 'password']);
body.frequency = {
alert_time: new Date(),
email: req.body.email
}
var user = new User(body);
user.save().then(() => {
return user.generateAuthToken();
}).then((token) => {
res.header("x-auth", token);
res.send(user);
}).catch((e) => {
res.status(400).send(e);
});
});
Test (mocha):
it("Should post a new user", (done) => {
var email = "uniqueemail#example.com"
var password = "9webipasd"
supertest(app)
.post("/users") // Post request to the /todos URL
.send({
email,
password
})
.expect(200)
.expect((res) => {
expect(res.headers).toIncludeKey('x-auth')
expect(res.body._id).toExist();
expect(res.body.email).toBe(email);
})
.end((err) => {
if(err){
return done(err);
}
User.findOne({email}).then((user) => {
expect(user).toExist();
expect(user.password).toNotBe(password);
done();
}).catch((e) => done(e));
});
});

My guess is that there is an index on CourtCaseSchema.case_id which does not allow duplicates.
I think you could check (in a mongo shell) that with CourtAPIDev.court_cases.getIndexes() (I think your db is named CourtAPIDev and the collection is named court_cases but I am not sure about that).
Also if you clean the test db after each run, that would explain why the tests are passing, since there is no more than one user.

Turns out, it was to do with my mongodb database, not any of my code. After searching around online, I found that if I logged into the mongo shell and then dropped all indexes from the users collection, it solved my problem. Could someone explain why this was causing my program to crash? I think it may have to do with an old user model, but I don't really understand. Thanks!

Even if you have all of your keys as unique=False, you may still get E11000 duplicate key error. So in that case, just follow these steps and check if your error is resolved.
Delete all documents from the collection (e.g. db.collection_name.deleteMany({}))
Drop the COLLECTION (NOT THE DATABASE) (e.g db.collection_name.drop())
Cheers !!

How to filter Twit (API) JSON response in node.js?

I'm currently writing a small Twitter app using the Twit API. To do what I need to do, I'd like the data to be able to be filtered by user id, and not get all the other garbage JSON spits out. Here's what the response looks like:
{ created_at: 'Sat Jun 23 03:45:13 +0000 2018',
id: 1010368149466697700,
id_str: '1010368149466697728',
text:
'RT #ClassicIsComing: "Let\'s Talk ETC!" Podcast Series by #chris_seberino of #InputOutputHK \nA deep series of powerful intervie
ws with influ…',
truncated: false,
entities:
{ hashtags: [],
symbols: [],
user_mentions: [ [Object], [Object], [Object] ],
urls: [] },
source:
'TweetDeck',
in_reply_to_status_id: null,
in_reply_to_status_id_str: null,
in_reply_to_user_id: null,
in_reply_to_user_id_str: null,
in_reply_to_screen_name: null,
user:
{ id: 759252279862104000,
id_str: '759252279862104064',
name: 'Ethereum Classic',
screen_name: 'eth_classic',
location: 'Blockchain',
description:
'Latest News and Information from Ethereum Classic (ETC). A crypto-currency with smart contracts which respects immutability a
nd neutrality.',
url: ,
entities: { url: [Object], description: [Object] },
protected: false,
followers_count: 216255,
friends_count: 538,
listed_count: 2147,
etc. The code i'm using to get this is:
T.get('statuses/home_timeline', {count: 1, exclude_replies: true},
function(err, data, response){
if (err){
console.log('Uh oh, we got a problem');
}
else{
console.log('We GUUCie bruh');
}
var tweets = data;
/* for (var i = 0; i < tweets.length; i++) {
console.log(tweets[i]);
} */
console.log(data);
});
the last block of code is commented out because I've attempted to define "tweets" as data.id, data.statuses.id, etc, but everything seems to spit out "undefined." I'm a complete noob to javascript and JSON as I'm only currently learning C++ # school, so any help would be appreciated!
edit
I thought I'd add in the error message to show you what happens when I try to treat the data as an object.
If I try to use JSON.parse(data) as the value for my tweet variable:
T.get('statuses/home_timeline', {count: 1, exclude_replies: true}, callBackFunction)
function callBackFunction(err, data, response){
if (err){
console.log('Uh oh, we got a problem');
}
else{
console.log('We GUUCie bruh');
}
var tweets = JSON.parse(data);
//for (var i = 0; i < tweets.length; i++) {
// console.log(tweets[i].id_str);
// }
console.log(tweets.id_str);
}
I get:
$ node crypt.js
the bot is starting
We GUUCie bruh
undefined:1
[object Object]
^
SyntaxError: Unexpected token o in JSON at position 1
at JSON.parse (<anonymous>)
If I try to treat it as an object right away, with:
function callBackFunction(err, data, response){
if (err){
console.log('Uh oh, we got a problem');
}
else{
console.log('We GUUCie bruh');
}
var tweets = data.id_str;
//for (var i = 0; i < tweets.length; i++) {
// console.log(tweets[i].id_str);
// }
console.log(tweets);
}
I get:
$ node crypt.js
the bot is starting
We GUUCie bruh
undefined

Have you tried JSON.parse?
So your line "var tweets = data;" would be "var tweets = JSON.parse(data);"
From there you should be able to interact with the data as if it were an object and grab specifically the id or whatever you're looking for.
I'm also a noob, so I don't have an in depth explanation as to why this works, but it helped fix an issue I had when pulling data from API.