So I'm using tensorflow JS and python for training models. Now I'm working on the website so that abstract doctors could upload an MRI image and get the prediction. Here's my code:
<script>
async function LoadModels(){
model = undefined;
model = await tf.loadLayersModel("http://127.0.0.1:5500/modelsBrain/modelBrain.json");
const image = document.getElementById("image");
const image1 = tf.browser.fromPixels(image);
const image2 = tf.reshape(image1, [1,200,200,3]);
const prediction = model.predict(image2);
const softmaxPred = prediction.softmax().dataSync();
alert(softmaxPred);
let top5 = Array.from(softmaxPred)
.map(function (p, i) {
return {
probability: p,
className: TARGET_CLASSES_BRAIN[i]
};
}).sort(function (a, b) {
return b.probability - a.probability;
}).slice(0, 4);
const pred = [[]];
top5.forEach(function (p) {
pred.push(p.className, p.probability);
alert(p.className + ' ' + p.probability);
});
}
const fileInput = document.getElementById("file-input");
const image = document.getElementById("image");
function getImage() {
if(!fileInput.files[0])
throw new Error("Image not found");
const file = fileInput.files[0];
const reader = new FileReader();
reader.onload = function (event) {
const dataUrl = event.target.result;
const imageElement = new Image();
imageElement.src = dataUrl;
imageElement.onload = async function () {
image.setAttribute("src", this.src);
image.setAttribute("height", this.height);
image.setAttribute("width", this.width);
await LoadModels();
};
};
reader.readAsDataURL(file);
}
fileInput.addEventListener("change", getImage);
</script>
This error occurrs not every (!) Live Server open. I am confused, what seems to be the problem?
Error CONTEXT_LOST_WEBGL is 99% due to low GPU memory - what kind of HW do you have available? Alternatively, you can try WASM backend which runs computation on CPU and doesn't require GPU resources.
Btw, you're not deallocating your tensors anywhere so if you're running this in a loop for multiple inputs, you do have a massive memory leak. But if error occurs on the first input already, your GPU simply is not good enough for this model.
Related
I'm trying to validate images before being sent to the DALL E API. The API rejects certain image properties that I need to validate:
file type
image dimensions
file size
whether the image has an alpha channel
What is the best way to do this?
This is a solution that works for me, although I would guess there are more efficient ways to accomplish it:
Validate items 1-3:
// Takes an imageBase64 url (we create locally, within javascript, from a file) and checks the dimensions once rendered.
function checkImageDims(imageBase64: string) {
const img = new Image();
img.src = imageBase64;
const promise = new Promise<string | null>((resolve, reject) => {
img.onload = () => {
const { width, height } = img;
if (height !== width) {
resolve(
`Image needs to be square. Current dimensions are ${width}x${height}`
);
} else {
resolve(null);
}
img.onerror = reject;
};
});
return promise;
}
// I am using AWS Amplify with S3. This gets the URL to the image:
const getS3Url = await Storage.get(`myFolder/${s3ObjectName}`);
// Once we have the URL
const fetched = await fetch(getS3Url);
const blobbed = await fetched.blob();
const imageBase64 = URL.createObjectURL(blobbed);
const dimensionsError = await checkImageDims(imageBase64);
if (dimensionsError) return dimensionsError;
console.log({
size: blobbed.size,
type: blobbed.type,
});
Validate item 4 (alpha)
// Checks image for alpha channel (transparency) https://stackoverflow.com/a/41302302/11664580
function checkForAlphaChannel(buffer: ArrayBuffer) {
const view = new DataView(buffer);
// is a PNG?
if (view.getUint32(0) === 0x89504e47 && view.getUint32(4) === 0x0d0a1a0a) {
// We know format field exists in the IHDR chunk. The chunk exists at
// offset 8 +8 bytes (size, name) +8 (depth) & +9 (type)
const type = view.getUint8(8 + 8 + 9);
return type === 4 || type === 6; // grayscale + alpha or RGB + alpha
}
return false;
}
const arrayBuffer = await blobbed.arrayBuffer();
const checkAlpha = checkForAlphaChannel(arrayBuffer);
console.log({checkAlpha})
Credit https://stackoverflow.com/a/41302302/11664580 for the Alpha validation.
For SEO optimizaion I'm attemting to low off the size of the files that the user attempts to send (I know I could have some size limitation or something not doing so because of the UX). and I'm doing it in the front-end cause I want to use pre-signed URL method (AWS S3)
process(event: any, imageInputElement: any, maxWidth: number): any {
return new Promise<any>((resolve, reject) => {
try {
const file = event.target.files[0]
console.log('🚀 ~ file: index.vue ~ line 143 ~ process ~ file', file)
const fileSize = file.size
if (fileSize < 100000) return
if (!file) return
const reader = new FileReader()
reader.readAsDataURL(file)
reader.onload = function (event: any) {
const src = event.target.result
const canvas = document.createElement('canvas') as any
const imgElement = document.createElement('img') as any
imgElement.src = src
imageInputElement.src = event.target?.result
console.log(maxWidth)
imageInputElement.onload = function (e: any) {
const scaleSize = maxWidth / e.target.width
canvas.width = maxWidth
canvas.height = e.target.height * scaleSize
const ctx = canvas.getContext('2d')
ctx.drawImage(e.target, 0, 0, canvas.width, canvas.height)
const compressPer = (data: number) => {
const result = 10000000 / data
if (Math.trunc(result) >= 100) {
return 100
} else if (Math.trunc(result) < 1) {
return 1
} else {
return Math.trunc(result)
}
}
const srcEncoded = ctx.canvas.toDataURL(
e.target,
'image/jpeg',
compressPer(fileSize)
)
const result = new File([srcEncoded], `${file.name}`, {
type: 'image/jpeg',
})
console.log(
'🚀 ~ file: index.vue ~ line 186 ~ process ~ result',
result
)
resolve(result)
}
}
} catch (error: any) {
reject(error)
}
})
},
This function gets called every time the user changes a file input.
event: is the default change event that includes the file itself.
imageInputElement: is the element that I want to render the new file in it. and maxWidth is the width that I pass to the function to specify the max width
The actual problem: the file will become visible in the browser and gets uploaded to the s3 bucket but the file is crashed when I want to download it again.
instead of
const file = event.target.files[0]
I should have used
var blobBin = atob(srcEncoded.split(',')[1]);
var array = [];
for(let i = 0; i < blobBin.length; i++) {
array.push(blobBin.charCodeAt(i));
}
const result:any = new Blob([new Uint8Array(array)], {type: 'image/png'});
got my answer from here
I'm trying to write some text on a simple jpg or png with nodejs and JIMP but i'm having issues on making it work.
The picture is gotten from a telegram bot, it gets merged with another picture with canvas and then i must write some simple text on it.
Here is my code:
const Jimp = require("jimp");
var imageCaption = 'WRITE THIS ON PICTURE';
var loadedImage;
const image = await Jimp.read(finalCanvas)
.then(function (image) {
loadedImage = image;
return Jimp.loadFont(Jimp.FONT_SANS_16_BLACK);
})
.then(function (font) {
loadedImage.print(font, 10, 10, imageCaption)
.write(finalCanvas);
})
.catch(function (err) {
console.error(err);
});
I keep getting an error about a matching contstructor overloading not found.
Also had troubles getting JIMP read my local files.
Complete error i get:
Error: No matching constructor overloading was found. Please see the docs for how to call the Jimp constructor.
at Jimp.throwError (/home/smp0/ifsbadge/node_modules/#jimp/utils/dist/index.js:33:13)
at new Jimp (/home/smp0/ifsbadge/node_modules/#jimp/core/dist/index.js:412:85)
at _construct (/home/smp0/ifsbadge/node_modules/#babel/runtime/helpers/construct.js:19:21)
at /home/smp0/ifsbadge/node_modules/#jimp/core/dist/index.js:926:32
at new Promise (<anonymous>)
at Function.Jimp.read (/home/smp0/ifsbadge/node_modules/#jimp/core/dist/index.js:925:10)
at TelegramBot.<anonymous> (/home/smp0/ifsbadge/index.js:51:32)
at processTicksAndRejections (internal/process/task_queues.js:93:5) {
methodName: 'constructor'
}
Complete context:
var needle = require('needle');
const Telegram = require('node-telegram-bot-api')
const { createCanvas, loadImage, ImageData } = require('canvas')
var Jimp = require("jimp");
var fs = require('fs');
const factions = {}
token="1234:BLABLA"
const bot = new Telegram(token, { polling: true })
bot.on('message', async (msg) => {
if (msg.photo) {
if (factions[msg.chat.id]) {
console.log(`Generating badge for ${msg.from.first_name} (${msg.from.username})...`)
bot.sendChatAction(msg.chat.id, 'upload_photo').catch(console.error)
const pictureCanvas = createCanvas(559, 772)
const pictureCtx = pictureCanvas.getContext('2d')
const { file_path } = await bot.getFile(msg.photo[msg.photo.length - 1].file_id)
const picture = await loadImage(`https://api.telegram.org/file/bot${token}/${file_path}`)
// PICTURE CALCULATIONS
pheight = picture.height
pwidth = picture.width
aspectratiow = (pwidth/pheight)
aspectratioh = (pheight/pwidth)
oheight = pheight*aspectratioh
owidth = (pwidth) / (pwidth/559)
newheight = 559*pheight/pwidth
var scale = Math.min(559/pwidth, 772/pheight);
var posx = (559 / 2) - (559 / 2) * scale;
var posy = (772 / 2) - (pheight / 2) * scale;
// END OF CALCULATIONS
// MERGING TWO PICTURES
pictureCtx.drawImage(picture, 10 , posy, 559, newheight)
const finalCanvas = createCanvas(559, 772)
const finalCtx = finalCanvas.getContext('2d')
const frame = await loadImage(`./frames/${factions[msg.chat.id]}.png`)
finalCtx.drawImage(pictureCanvas, 0, 0, 559, 772)
finalCtx.drawImage(frame, 0, 0, 559, 772)
factions[msg.chat.id] = null
// END OF MERGING PICTURES
//APPLYING TEXT ON PICTURE
const Jimp = require("jimp");
var imageCaption = 'WRITE THIS ON PICTURE';
var loadedImage;
const image = await Jimp.read(finalCanvas)
.then(function (image) {
loadedImage = image;
return Jimp.loadFont(Jimp.FONT_SANS_16_BLACK);
})
.then(function (font) {
loadedImage.print(font, 10, 10, imageCaption)
.write(finalCanvas);
})
.catch(function (err) {
console.error(err);
});
//END OF APPLYING TEXT ON PICTURE
bot.sendPhoto(msg.chat.id, finalCanvas.toBuffer('image/jpeg', { quality: 1 }))
} else {
bot.sendMessage(msg.chat.id, 'Write /enl1 /enl2 /enl3 o /res1 /res2 /res3 o /xf1 /xf2 !').catch(console.log)
}
}
})
bot.onText(/\/start/, async (msg) => {
bot.sendMessage(msg.chat.id, "Welcome! Select your badge Write /enl1 /enl2 /enl3 o /res1 /res2 /res3 o /xf1 /xf2 !").catch(console.log)
})
bot.onText(/\/(enl1|enl2|enl3|res1|res2|res3|xf1|xf2)/, async (msg, match) => {
factions[msg.chat.id] = match[1]
bot.sendMessage(msg.chat.id, 'Good! Now send me your picture').catch(console.log)
})
srcs on github: https://github.com/pedrofracassi/badgemaker and https://github.com/FerdinandoLM/IngressFSBadgeMaker
From what I can see, finalCanvas is an instance of Canvas. I don't think Jimp would take an instance of Canvas, but Canvas#toBuffer() might be the thing you want as Jimp would take a buffer: Jimp.read(finalCanvas.toBuffer())
Both http.get and response.pipe(writeStream) are asynchronous. At the time you try Jimp.read, the file isn't in the file system yet or it is not completely written onto disk.
To do it after file is written, listen to finish event.
file.on('finish', async () => {
await Jump.read(...);
});
I am using tensorflowjs to do some front-end image classification. I am trying to use tf.browser.fromPixels to convert an img element to a tensor. However, I am getting all zeros of shape [160, 160, 3]. I am using the FileReader api to read an image from the file system via the <input type="file"> element. Here's some of the code:
function getFiles(event) {
const files = event.target.files;
let tempStore = [];
for (let i = 0; i < files.length; ++i) {
tempStore.push(files[i]);
}
return tempStore;
}
const imageElement = document.getElementById("upload");
imageElement.addEventListener("change", event => {
const files = getFiles(event);
Promise.all(files.map(loadImg)).then(d => {
console.log("All done !!!", d);
});
});
const loadImg = imgFile => {
return new Promise((resolve, reject) => {
let reader = new FileReader();
let imgEl = document.createElement("img");
reader.onload = async e => {
imgEl.src = e.target.result;
imgEl.setAttribute("width", 160);
imgEl.setAttribute("height", 160);
document.body.append(imgEl);
const fromPixels = tf.browser.fromPixels(imgEl);
resolve(fromPixels);
};
reader.onerror = reject;
reader.readAsDataURL(imgFile);
});
};
The image gets appended to document body properly.
The imageElement is of the form:
<img src="data:image/jpeg;base64,....." width=160 height=160>
You are creating the tensor from the image when the img tag has not yet been loaded. Here is the way to go
imgEl.src = e.target.result;
imgEl.setAttribute("width", 160);
imgEl.setAttribute("height", 160);
document.body.append(imgEl);
im.onload = () => {
// create the tensor after the image has loaded
const fromPixels = tf.browser.fromPixels(imgEl);
resolve(fromPixels);
}
var images;
function preloadTrial(actor, event) {
return new Promise(function(res) {
var i = 0;
images = [];
var handler = function(resolve, reject) {
var img = new Image;
var source = '/static/videos/' + actor + '/' + event + '/' + i + '.png';
img.onload = function() {
i++;
resolve(img);
}
img.onerror = function() {
reject()
}
img.src = source;
}
var _catch = function() { res(images) }
var operate = function(value) {
if (value) images.push(value);
new Promise(handler).then(operate).catch(_catch);
}
operate();
})
}
function playSequence(time){
var delta = (time - currentTime) / 1000;
currentFrame += (delta * FPS);
var frameNum = Math.floor(currentFrame);
if (frameNum >= numFramesPlay) {
currentFrame = frameNum = 0;
return;
}else{
requestAnimationFrame(playSequence);
currentImage.src = images[frameNum];
currentTime = time;
console.log("display"+currentImage.src);
}
};
function rightNow() {
if (window['performance'] && window['performance']['now']) {
return window['performance']['now']();
} else {
return +(new Date());
}
};
currentImage = document.getElementById("instructionImage");
// Then use like this
preloadTrial('examples', 'ex1').then(function(value) {
playSequence(currentTime=rightNow());
});
I wrote a Javascript function that is suppose to load a directory full of numbered .png files. However, I do not know the number of items inside the directory beforehand. So I made a function that continues to store images until the source gives me an error. But when I run the code the program does not even enter the .onload and .onerror functions, resulting in an infinite loop.
Edit: This is my current code. It appears that images are correctly assigned and pushed into the array images. But when I attempt to load it onto a img tag (currentImage.src) and run playSequence, it does not display.
You could use promises to handle the pre-loading of the images.
Chain the resolves on the onload event and reject onerror to end the cycle.
function preloadImages(baseurl, extension, starter) {
return new Promise(function(res) {
var i = starter;
var images = [];
// Inner promise handler
var handler = function(resolve, reject) {
var img = new Image;
var source = baseurl + i + '.' + extension;
img.onload = function() {
i++;
resolve(img);
}
img.onerror = function() {
reject('Rejected after '+ i + 'frames.');
}
img.src = source;
}
// Once you catch the inner promise you resolve the outer one.
var _catch = function() { res(images) }
var operate = function(value) {
if (value) images.push(value);
// Inner recursive promises chain.
// Stop with the catch resolving the outer promise.
new Promise(handler).then(operate).catch(_catch);
}
operate();
})
}
To simulate a video player, you can draw on a HTML5 canvas.
function play(canvas, imagelist, refreshRate, frameWidth, frameHeight) {
// Since we're using promises, let's promisify the animation too.
return new Promise(function(resolve) {
// May need to adjust the framerate
// requestAnimationFrame is about 60/120 fps depending on the browser
// and the refresh rate of the display devices.
var ctx = canvas.getContext('2d');
var ts, i = 0, delay = 1000 / refreshRate;
var roll = function(timestamp) {
if (!ts || timestamp - ts >= delay) {
// Since the image was prefetched you need to specify the rect.
ctx.drawImage(imagelist[i], 0, 0, frameWidth, frameHeight);
i++;
ts = timestamp;
}
if (i < imagelist.length)
requestAnimationFrame(roll);
else
resolve(i);
}
roll();
})
}
To test I used ffmpeg to cut a video with the following command:
ffmpeg -i input.mp4 -ss 00:00:14.435 -vframes 100 %d.png
And I used devd.io to quickly create a static folder containing the script and images and a basic index.html.
imageroller.js - with the above code.
var preload = preloadImages('/static/videos/examples/testvid/', 'png', 1);
preload.then(function(value) {
console.log('starting play');
var canvas = document.getElementById("canvas");
play(canvas, value, 24, 720, 400) // ~480p 24fps
.then(function(frame){
console.log('roll finished after ' + frame + ' frames.')
})
});
While the preloading of the images was pretty slow, if you keep the number of frames to an acceptable level you can make some nice loops.
I haven't tested the snippet below (and there are probably cleaner solutions) but the idea should be correct. Basically we have a recursive function loadImages(), and we pass in the images array and a callback function. We wait for our current image to load; if it loads, we push it into images and call loadImages() again. If it throws an error, we know we are finished loading, so we return our callback function. Let me know if you have any questions.
function preloadTrial(actor, event) {
let images = [];
loadImages(images, actor, event, function () {
// code to run when done loading
});
};
function loadImages (images, actor, event, callback) {
let img = new Image();
let i = images.length;
let source ='/static/videos/'+actor+'/'+event+'/'+i+'.png';
img.onload = function() {
images.push(img);
return loadImages(images, actor, event, callback);
}
img.onerror = function() {
return callback(images);
}
img.src = source;
}
The optimal solution would be to provide a server-side API that tells you beforehand, how many Images there are in the directories.
If that is not possible, you should load the images one after the other to prevent excess requests to the server. In this case i would put the image loading code in a separate function and call it if the previous image was loaded successfully, like so:
function loadImage(actor, event, i, loadCallback, errorCallback) {
var image = new Image();
var source ='/static/videos/'+actor+'/'+event+'/'+i+'.png';
image.onload = loadCallback;
image.onerror = errorCallback;
image.src = source;
return image;
}
and then call this function in your while loop and in the loadCallback.