I'm trying to create a mat from an 2d array using the cv.matFromArray(rows, cols, type, array); method.
It works with really small arrays like this:
let mat = cv.matFromArray(2, 2, cv.CV_8UC1, [255, 255, 128, 128]);
return mat; // works
But when i basically do the same thing with my image array (1024x1024px, values range from 20 to 230) it just fills every Mat value to 0
const mat = cv.matFromArray(img_array.length, img_array[0].length, cv.CV_8UC1, img_array);
return mat; // every value is 0
Why is that?
Okay, I converted the 2d array to 1d like this:
[].concat(...img_array);
and its working now:
img_array; // 1024x1024px 2d array
const mat = cv.matFromArray(img_array.length, img_array[0].length, cv.CV_8UC1, [].concat(...img_array));
return mat;
Related
I am new to TensorflowJS and I try to code something but I am stuck...
I have two input layers like that:
const input1 = tf.input({ shape: [64, 64, 3] });
const input2 = tf.input({ shape: [1536] });
The first one is for an image of 64 by 64 and the 3 is for RGB.
The second one is for an array that contains 1536 numbers (floats).
I tried to concatenate them with .concatenate().apply(input1, input2) but got the following error:
ValueError: A `Concatenate` layer requires inputs with matching shapes except for the concat axis. Got input shapes: [[null,64,64,3],[null,1536]]
I also tried to add { axis: -1 } or { axis: 1 } (found that on stack overflow but that doesnt work too).
I also tried that (answer by chat gpt) :
const flatten1 = tf.layers.flatten().apply(input1);
const flatten2 = tf.layers.flatten().apply(input2);
const concat = tf.layers.concatenate({ axis: -1 }).apply([flatten1, flatten2]);
but same error...
Can someone help me? I just want to add this to my tf.sequential() as an input...
PS: This is the module I use:
const tf = require('#tensorflow/tfjs-node');
This just shows you what concatenation is, and how can be done for those specific inputs.
This is a textual description of the code at the bottom of the post:
create an image-like shape (width, height, colorChannels)
create a random one dimensional array (aka vector) of 1536 values
1536 elements can be reshaped into a little "image" of 8 x 64 and 3 channels
Result is 72 x 64 x 8 so this should hint you what the code did (extend the axis with different number of values, that is it.
const original = tf.ones([64, 64, 3]);
const random = tf.randomNormal([1536]);
const reshaped = tf.reshape(random, [8, 64, 3]);
const axis = 0;
const concat = tf.concat([original, reshaped], axis);
console.log(concat);
<script src="https://cdn.jsdelivr.net/npm/#tensorflow/tfjs#2.0.0/dist/tf.min.js"></script>
Another possibility is flat, extend and then back re-shape but I find this simple enough.
Does anybody know how to convert an opencv.js mat into a tensor so I can feed it into my tensorflow.js classifier?
The following code shows what I did to read in and preprocess the image I want to classify:
img_array = cv.imread(document.getElementById('picture1'), cv.IMREAD_GRAYSCALE);
cv.cvtColor(img_array, img_array, cv.COLOR_RGBA2GRAY);
let dsize = new cv.Size(100, 100);
cv.resize(img_array, img_array, dsize);
My classifier needs a tensor of shape (1, 100, 100, 1) as an input and I do not know how to convert the cv mat into a tensorflow.js tensor.
An image object has a properties data that can be used to get all pixels values in a flattened array. To construct a tensor, the following can be used
const src = cv.imread(imageSource)
const tensor = tf.tensor(src.data, [src.rows, src.cols, -1])
Using Opencv.js I have acquired the contours in an image.
using some selection function I have collected a subset of these contours.
Say they are in the list of contours:
var suitableContours = [];
I need to know draw these contours using opencv.
If this were python I could do
cv.drawContours(img, [suitableContours[cnt]], 0, [255, 0, 0, 255], 2);
Assume by the point I want to draw the contours I don't have access to the original image nor the mask nor the original contour list, only the selected contours.
How can I draw these contours?
The above code results in: BindingError: Cannot pass "[object Object],[object Object]" as a MatVector
From
https://docs.opencv.org/3.4/d5/daa/tutorial_js_contours_begin.html
Try something like this:
let color = new cv.Scalar(255,0,0,255);
for (let i = 0; i < suitableContours.size(); ++i) {
cv.drawContours(src, suitableContours, i, color, 1, cv.LINE_8, hierarchy, 100);
}
cv.imshow('canvasOutput', src);
Im trying to create a convex hull with opencv.js based on an array with points, does anyone know a way to do this correctly and efficient? An array would look like this:
[
[5,5],
[10,10],
[15,15]
...
]
-> where the first value would be the x and the second the y value, but it wouldn't be a problem to change this format to something more suitable.
Thnx for the help :)
As far I could experiment OpenCV stores contour/hull data in Mat format with type CV_32SC2: essentially a flat list of 32bit short integers in [x1,y1,x2,y2,x3,y3,...] order.
Note the two channels/planes part of 32SC2: one channel for all the x values and another for all the y values
You can manually create such a Mat, access it's data32S property and fill in each value:
let testHull = cv.Mat.ones(4, 1, cv.CV_32SC2);
testHull.data32S[0] = 100;
testHull.data32S[1] = 100;
testHull.data32S[2] = 200;
testHull.data32S[3] = 100;
testHull.data32S[4] = 200;
testHull.data32S[5] = 200;
testHull.data32S[6] = 100;
testHull.data32S[7] = 200;
However OpenCV.js comes with a handy method to convert a flat array of values to such a Mat:
let testHull = cv.matFromArray(4, 1, cv.CV_32SC2, [100,100,200,100,200,200,100,200])
If your array is nested, you can simply use JS Array's flat() method to flatten it from a 2D array([[x1,y1]...]) to a 1D array ([x1,y1,...]).
So you don't have to worry about the Mat type and all that you can wrap it all into a nice function, for example:
function nestedPointsArrayToMat(points){
return cv.matFromArray(points.length, 1, cv.CV_32SC2, points.flat());
}
Here's a quick demo:
function onOpenCvReady(){
cv.then(test);
}
function nestedPointsArrayToMat(points){
return cv.matFromArray(points.length, 1, cv.CV_32SC2, points.flat());
}
function test(cv){
console.log("cv loaded");
// make a Mat to draw into
let mainMat = cv.Mat.zeros(30, 30, cv.CV_8UC3);
// make a fake hull
let points = [
[ 5, 5],
[25, 5],
[25,25],
[ 5,25]
]
let hull = nestedPointsArrayToMat(points);
console.log("hull data", hull.data32S);
// make a fake hulls vector
let hulls = new cv.MatVector();
// add the recently created hull
hulls.push_back(hull);
// test drawing it
cv.drawContours(mainMat, hulls, 0, [192,64,0,0], -1, 8);
// output to canvas
cv.imshow('canvasOutput', mainMat);
}
<script async src="https://docs.opencv.org/4.4.0/opencv.js" onload="onOpenCvReady();" type="text/javascript"></script>
<canvas id="canvasOutput" width="30" height="30"></canvas>
Note that the above is a rough example, there's no data validation or any other fancier checks, but hopefully it illustrates the idea so it can be extended robustly as required.
Lets say that your points represent a contour:
var contours = new cv.MatVector();
for (var i = 0; i < points.size(); ++i) {
contours.push_back(new cv.Mat(points[i][0], points[i][1])
}
Now following this tutorial from opencv website:
// approximates each contour to convex hull
for (var i = 0; i < contours.size(); ++i) {
var tmp = new cv.Mat();
var cnt = contours.get(i);
// You can try more different parameters
cv.convexHull(cnt, tmp, false, true);
hull.push_back(tmp);
cnt.delete(); tmp.delete();
}
I have a question regarding the use of segmentation LUTs in AMI JS (not XTK but there is no ami js tag yet!). Particularly what I want to do is to load a segmentation / labelmap layer and display it with the right colors, one for each label.
My labelmap layer consists of N integer labels that define different structures (e.g from 0 to 14000), which are also the voxel values of the labelmap. Each one of the labels has a different color associated (they are generated by Freesurfer and can be seen on: https://surfer.nmr.mgh.harvard.edu/fswiki/FsTutorial/AnatomicalROI/FreeSurferColorLUT ).
What I would like is a LUT that, for each different label, paints it with the correspondant color. I have had trouble finding the right way to do it and have not had success so far. What I have done is to get all the colors and store them into an array (colors normalized between 0 and 1 and the first component being the position inside the texture from 0 to 1, with a step of 1/total labels, which results in a really small step as there are 1200 labels!). From what I've seen then, the HelpersLUT class takes all the colors and maps them discretely into the texture, but the colors appear messed up and I can't seem to get the opacities right either...
I have seen also that the StackModels also have some functionalities such as prepareSegmentation() and such but do not know how to specify the LUT in there and cannot get it to work either (it is not used on the example).
Which is the best way to create a discrete LUT with a different color for each integer label and the 0 value being transparent and the other labels opaque?
The procedure used to generate the LUTs is: First I read a JSON with the information of the Freesurfer and store it into a variable, the first component of each one is the index of the label between 0 and 1, and the other ones are the associated color to the label between 0 and 1 as well. I have also generated a LUT of opacities.
let customLUT = {
"fsLUT": [],
"default": [[0, 0, 0, 0], [0.25, 0.3, 0.4, 0.5], [0.5, 0.2, 0.5, 0.4],
[0.75, 0.1, 0.2, 0.3], [1, 0.5, 0.5, 0.8]],
"fsLUT0": [[0, 0], [0.01, 1], [0.6, 1], [1, 1]]
};
$.getJSON("https://cdn.rawgit.com/YorkeUtopy/ami-viewerData/e773d737/FreesurferInfo.json", function (data) {
FsInfo = data;
FsInfo.forEach(function (value, i) {
customLUT.fsLUT.push([i / FsInfo.length, (value.color[0] / 255), (value.color[1] / 255.000), (value.color[2] / 255.000)]);
});
});
Then I create a helpers LUT with the LUT0 defined and the LUT with the colors and apply it to the texture. Everythink else is just as the labelmap example createing the layer mix, etc...
lutLayerLblmap = new HelpersLut(
"my-lut-canvases-l1",
"default",
"linear", [[0, 0, 0, 0], [1, 1, 1, 1]],
customLUT.fsLUT0,
false
);
lutLayerLblmap.luts = customLUT;
lutLayerLblmap.lut = "fsLUT";
refObj.uniformsLayerLblmap.uLut.value = 1;
refObj.uniformsLayerLblmap.uTextureLUT.value = lutLayerLblmap.texture;
With that some colors appear but there are not correct and the opacities are messed up (I know the LUT0 is not correct and that it is not discrete!). However, when I make the helpersLUT discrete and put a LUT0 like [0,0],[1,1], the colors are messed up and the opacities do not apply correctly... maybe it is that the voxel values are not between 0 and 1 but have values such as 1100,1200... ? or that I am not correctly generating the LUTs (step size too small?).... Here are some examples of the LUT.
[0]: 0,0,0,0
[1]:0.0008319467554076539,0.27450980392156865,0.5098039215686274,0.7058823529411765
[2]:0.0016638935108153079,0.9607843137254902,0.9607843137254902,0.9607843137254902
[3]:0.0024958402662229617,0.803921568627451,0.24313725490196078,0.3058823529411765
[last -2]:0.997504159733777,0.08235294117647059,0.7058823529411765,0.7058823529411765
[last-1]:0.9983361064891847,0.8745098039215686,0.8627450980392157,0.23529411764705882
[last]:0.9991680532445923,0.8666666666666667,0.23529411764705882,0.23529411764705882
this is the sample data I use:
T1 Volume + Labelmap + Freesurfer JSON
You seem to be making everything fine.
It is a current limitation in AMI side.
It currently only supports 256 colors and on top of that, it requires values to be normalized.
In AMI, we need to support a new type of LUT (Segmentation LUT seems a good name).
Live fiddle based on you approach.
const fsLUT = [];
fetch("https://cdn.rawgit.com/YorkeUtopy/ami-viewerData/e773d737/FreesurferInfo.json")
.then(response => response.json())
.then(jsonLUT => {
jsonLUT.forEach(function (value, i) {
fsLUT.push([
i / json.length,
(value.color[0] / 255),
(value.color[1] / 255.000),
(value.color[2] / 255.000)]);
});
return fsLUT;
})
http://jsfiddle.net/agoyre4e/20/