We Keep Coding

Automatically stringifying object when inserting to a MySQL JSON column with knex

Let's jump straight to an example code:
create table test_json_table
(
data json not null
);
I can insert to the table like this:
const columns = { data: "{ some_json: 123 }" }; // notice that the data column is passed as string
await knex('test_json_table').insert(columns);
And get data from the table like this:
await knex('test_json_table').select();
// returns:
// [
// { data: { some_json: 123 } } // notice that the data is returned as parsed JavaScript object (not a string)
// ]
When inserting a row the JSON column needs to be passed as a serialised string. When retrieving the row, an already parsed object is returned.
This is creating quite a mess in the project. We are using TypeScript and would like to have the same type for inserts as for selects, but this makes it impossible. It'd be fine to either always have string or always object.
I found this topic being discussed at other places, so it looks like I am not alone in this (link, link). It seems like there is no way to convert the object to string automatically. Or I am missing something?
It'd be nice if knex provided a hook where we could manually serialise the object into string when inserting.
What would be the easiest way to achieve that? Is there any lightweight ORM with support for that? Or any other option?

You could try objection.js that allows you to declare certain columns to be marked as json attributes and those should be stringified automatically when inserting / updating their values https://vincit.github.io/objection.js/api/model/static-properties.html#static-jsonattributes
I haven't tried if it works with mysql though. I don't see any reason why it wouldn't.

I think the easiest way using jsonb data type. mysql json type
We prefer postgresql for this kind of problem at office, easier and solid database for your problem.

Well you could call your own function before inserting that converts all objects to string and call it every time before you insert.
You can probably wrap knex to do it automatically as well.

JSON Array from String

I have an array of JSON plots which I store in MySQL. When I retrieve this information from MySQL it is given as one long string. How can I restore this back into an array of JSON objects using Javascript? I'm running this using NodeJS and MySQL package.
My data is returned like the following:
'[{"x":0,"y":0},{"x":1,y:1},{"x":2,"y":2}]'
What I would like to be able to do is use the data like:
var data = [{"x":0,"y":0},{"x":1,"y":1},{"x":2,"y":2}];
console.log(data[0].x);
I've had a try using JSON.parse and originally stored the data using JSON.stringify on the array, but it is not behaving as I would expect.
Are there any methods or packages available to handle this?
Edit: I realize now that this is not JSON but rather objects. Apologies for the wrong terminology here, but my problem still remains.

var data = new Function ('return ' + dataString)();

How can I remove dynamic data in string before a value in that string? / using Lodash's _.trimStart with dynamic data?

I'm working urls returned from a server that I have no control over where and sometimes the urls return with extra data at the front.
For instance
sometimes it returns this
https://example.com/image/5119b3905.jpg
and this I can use, but sometimes it will return something like this
https://d1yww.cloudfront.net/9MA=/670x670/example.com/image/5119b3905.jpg
where I'd like to use remove everything before the example.com and to do that I could use something like lodash's _.trimStart method something like
_.trimStart('https://d1yww.cloudfront.net/9MA=/670x670/example.com/image/5119b3905.jpg',
'd1yww.cloudfront.net/9MA=/670x670');
but the d1yww.cloudfront.net/9MA=/670x670' is never static for me to do this and I don't know how to grab the dynamic data to use _.trimStart and I don't see any other useful lodash's methods and I don't know of any vanilla javascript ones.
TLDR: How can I remove dynamic data in string before a value in that string (in this example everything before the example.com)

You don't need lodash to do that
var str = 'https://d1yww.cloudfront.net/9MA=/670x670/example.com/image/5119b3905.jpg'
str.substr(str.indexOf('example.com'))

You could search for a Regular Expression
For Example:
/\/([-a-zA-Z0-9#:%._\+~#=]{2,256}\.[a-z]{2,6}\b)\//g
and look for the second match

converting Mongoose returned array to regular JSON

In a Node.js App, i want to fetch some data from database via Mongoose. i do it like this:
Users.find({_id:userID}).exec(function(err, users){
});
because users is an Array like Mongoose object, i cant do this:
users.toJSON();
this works if i use findOne but right now doesn't work. also users.toObject() doesn't work and i get this error:
object has no method 'toObject'
if i use lean() in query like this:
Users.find({_id:userID}).lean().exec(function(err, users){
});
this works but it has other problems. for example, if an array value has no value, instead of showing it like this:
myval:[]
doesn't show that key/value pair at all!!
What i want is that i have to edit that users result and apparently, its Mongoose object and i cant. because of that i have to convert it to a regular JSON but how?

Just use findOne instead of find and you'll be able to edit your instance.
If you absolutely have to use find then know that users parameter is the array of user instances, so just loop through them and edit them one by one.
Users.find({_id:userID}).exec(function(err, users){
users.forEach(function(user){
// edit my user here
});
});

Expressjs - parsing form array/bracket fields to real arrays

I'd like to either find some existing middleware or some handy code to transform form fields with square brackets (i.e. 'contact[21][name]')into real arrays for ExpressJS 3.
Something along the lines of:
for(key in req.body){
if(key.isArrayField()) {
//add new array field to req.body and delete old string version from req.body
...
I'd like to to parse infinite depth of brackets. I'm still learning JS/Node so would love some direction.
Thanks

Can you provide some clarification about what you're trying to achieve with this?
Depending on how you're getting the input from the web form in the first place you can easily parse JSON syntax into real objects (including arrays) using JSON.parse() (docs here) which is built into node.js
Using that method you can have an infinite level of nested values (hashes / objects, and arrays) but you'll need some client-side script to put the values there in the first place (jQuery for example).
If you give a fuller example, including the context of the form being submitted, I can probably give better direction...

In my experience you can simply do something like this...
Jade
input(type="text" name="foo[]" value="foo1")
input(type="text" name="foo[]" value="foo2")
JS
app.post('/test', function(req,res){
console.log(req.body)// { foo: [ 'foo1', 'foo2' ] }
console.log(req.body.foo)//["foo1","foo2"]
console.log(req.body.foo[0])//foo1
console.log(req.body.foo[1])//foo2
});
I don't know if this makes sense but this is how you treat them... at least on express 3.0+

I've sort of solved this by using https://github.com/marioizquierdo/jquery.serializeJSON - the issue is/was that if I just submit a POST form to express it doesn't handle brackets as an array so they have to be regexed manually. By parsing the form into JSON using the the plugin I can then submit it through ajax.