Does anybody know how to convert an opencv.js mat into a tensor so I can feed it into my tensorflow.js classifier?
The following code shows what I did to read in and preprocess the image I want to classify:
img_array = cv.imread(document.getElementById('picture1'), cv.IMREAD_GRAYSCALE);
cv.cvtColor(img_array, img_array, cv.COLOR_RGBA2GRAY);
let dsize = new cv.Size(100, 100);
cv.resize(img_array, img_array, dsize);
My classifier needs a tensor of shape (1, 100, 100, 1) as an input and I do not know how to convert the cv mat into a tensorflow.js tensor.
An image object has a properties data that can be used to get all pixels values in a flattened array. To construct a tensor, the following can be used
const src = cv.imread(imageSource)
const tensor = tf.tensor(src.data, [src.rows, src.cols, -1])
Related
I get the following error:
Error when checking : expected dense_Dense_1_input to have 2 dimensions, but got array with shape [224, 224, 3]
When I run the following code:
const modelJson = require("../offline_model/pose-model.json");
const modelWeights = require("../offline_model/weights.bin");
let imageTensor = images.next().value as tf.Tensor3D;
tf.loadLayersModel(bundleResourceIO(modelJson, modelWeights)).then(model => {
try {
imageTensor = tf.image.resizeBilinear(
imageTensor,
[224, 224],
false
);
const normalized = imageTensor.cast('float32').div(127.5).sub(1);
model.predict(normalized)
Maybe this code is not the most correct but I already tried every other code from stackoverflow and youtube videos and they all give some error. I think the problem is that in my model.json file (I trained the model with teachable machine) I find this value:
"name":"dense_Dense1", "batch_input_shape":[null,14739]
This is why it expects 2 dimensions and I have 3. I guess I am not sure how to change the 3 dimension tensor I have to fit the dimensions of the batch input shape.
I am new to TensorflowJS and I try to code something but I am stuck...
I have two input layers like that:
const input1 = tf.input({ shape: [64, 64, 3] });
const input2 = tf.input({ shape: [1536] });
The first one is for an image of 64 by 64 and the 3 is for RGB.
The second one is for an array that contains 1536 numbers (floats).
I tried to concatenate them with .concatenate().apply(input1, input2) but got the following error:
ValueError: A `Concatenate` layer requires inputs with matching shapes except for the concat axis. Got input shapes: [[null,64,64,3],[null,1536]]
I also tried to add { axis: -1 } or { axis: 1 } (found that on stack overflow but that doesnt work too).
I also tried that (answer by chat gpt) :
const flatten1 = tf.layers.flatten().apply(input1);
const flatten2 = tf.layers.flatten().apply(input2);
const concat = tf.layers.concatenate({ axis: -1 }).apply([flatten1, flatten2]);
but same error...
Can someone help me? I just want to add this to my tf.sequential() as an input...
PS: This is the module I use:
const tf = require('#tensorflow/tfjs-node');
This just shows you what concatenation is, and how can be done for those specific inputs.
This is a textual description of the code at the bottom of the post:
create an image-like shape (width, height, colorChannels)
create a random one dimensional array (aka vector) of 1536 values
1536 elements can be reshaped into a little "image" of 8 x 64 and 3 channels
Result is 72 x 64 x 8 so this should hint you what the code did (extend the axis with different number of values, that is it.
const original = tf.ones([64, 64, 3]);
const random = tf.randomNormal([1536]);
const reshaped = tf.reshape(random, [8, 64, 3]);
const axis = 0;
const concat = tf.concat([original, reshaped], axis);
console.log(concat);
<script src="https://cdn.jsdelivr.net/npm/#tensorflow/tfjs#2.0.0/dist/tf.min.js"></script>
Another possibility is flat, extend and then back re-shape but I find this simple enough.
Trying to predict with TensorflowJS. But, in input image above error is showing? My image is Uint8Array type. How can I pass an Uint8Array type for making a tensor?
async predict(imageData: any) {
let img = tf.browser.fromPixels(imageData, 3).resizeBilinear([256, 256]) # problem showing here
img = imageData.reshape([256, 256, 3])
img = tf.cast(img, 'float32')
const segmentation = this.model.predict(img) as any
console.log('success')
}
loadImage(file: FileList) {
this.fileToUpload = file.item(0);
let reader = new FileReader();
reader.readAsDataURL(this.fileToUpload);
reader.onload = (event: any) => {
this.imageUrl = reader.result
this.predict(this.convertDataURIToBinary(this.imageUrl)); # passing Unit8Array image from here
}
}
Any idea how can I overcome it? Thanks for your suggestion.
Update
Solved this issue using '#ViewChild('ImageRef') ImageRef: ElementRef;' . Lastly I converted Unit8Array into imageData then drew into canvas with putImageData.
But facing problem in another part. As I am doing image segmentation, the result is just the inverse of what I made in python. Any idea?
In python I did - >
img_face = cv2.resize(frame,(256,256))
img_face = cv2.cvtColor(img_face, cv2.COLOR_BGR2RGB)
img_face = img_face / 255.0
img_face = img_face.astype(np.float32)
mask = model.predict(np.expand_dims(img_face , axis=0))[0]
And my current js part is already mentioned above as predict() func.
If imageData is a UInt8Array, it can be easily converted to a tensor using tf.tensor
tf.tensor(imageData)
Additionnally the image width and height can be specified while creating the tensor
tf.tensor(imageData, [height, width, channels])
Now regarding the image processing, in python you are doing
img_face / 255.0
You are not doing the same thing in js. you need to divide the js tensor by 255
I'm trying to create a mat from an 2d array using the cv.matFromArray(rows, cols, type, array); method.
It works with really small arrays like this:
let mat = cv.matFromArray(2, 2, cv.CV_8UC1, [255, 255, 128, 128]);
return mat; // works
But when i basically do the same thing with my image array (1024x1024px, values range from 20 to 230) it just fills every Mat value to 0
const mat = cv.matFromArray(img_array.length, img_array[0].length, cv.CV_8UC1, img_array);
return mat; // every value is 0
Why is that?
Okay, I converted the 2d array to 1d like this:
[].concat(...img_array);
and its working now:
img_array; // 1024x1024px 2d array
const mat = cv.matFromArray(img_array.length, img_array[0].length, cv.CV_8UC1, [].concat(...img_array));
return mat;
I'm new to tensorflowjs (and js in general), however I need to run a trained model on it.
Currently I converted the model to json format, but struggle to feed data to it:
const tf = require('#tensorflow/tfjs')
const tfn = require('#tensorflow/tfjs-node-gpu')
async function start() {
const handler = tfn.io.fileSystem("./model/model.json");
const model = await tf.loadGraphModel(handler);
let latents = tf.randomNormal([1,512], 'float32');
let labels = tf.zeros([1, 0]);
model.predict([latents, labels]);
}
start();
But I receive an error saying The Conv2D op currently supports NHWC tensor format on the CPU. The op was given the format: NCHW
So as I understood, it is a tfjs issue, so I tried to create a float32 array and pass it to model like this:
var f32array = new Float32Array(512);
model.predict([f32array, labels]);
But then I see an error saying the dtype of dict['Gs/latents_in'] provided in model.execute(dict) must be float32, but was undefined
With python, I'm running inference by using this code:
graph = load_graph("dash/frozen_model.pb")
x = graph.get_tensor_by_name('prefix/Gs/latents_in:0')
x2 = graph.get_tensor_by_name('prefix/Gs/labels_in:0')
y = graph.get_tensor_by_name('prefix/Gs/images_out:0')
with tf.Session(graph=graph, config = config) as sess:
while True:
start_time = time.time()
latents = np.random.randn(1, 512).astype(np.float32)
labels = np.zeros([latents.shape[0], 0], np.float32)
y_out = sess.run(y, feed_dict = { x: latents, x2: labels})
Would appreciate any help
Passing the data as Float32Array will not work since model.predict expects either a tensor or an array of tensors.
As indicated by the error:
The Conv2D op currently supports NHWC tensor format on the CPU. The op was given the format: NCHW
the conv2D as of the version 1.6 in js only supports the format NHWC. The only thing you can do is to change the model in python in order to use only the NHWC format.