I am trying to draw a rectangle with a label slightly above it on a canvas. The x, y, width, and height were generated to be around an object that was detected using the coco-ssd model in tensorflow. The problem is that the coordinates generated by the coco-ssd model in tensorflow is relative to a different origin from the canvas itself. More specifically, the origin for the coco-ssd model is at the top right corner, and the origin for the canvas is at the top left corner.
I am able to move the origin of the canvas, but not the model's origin (That I know of). To move the canvas' origin, I translated the canvas to the right 410px, 4px smaller than the width of the canvas, and then reflected it horizontally. This draws the rectangle at the correct position. If I were to also create the text at this point it would be inverted and unreadable (but at the proper position). If it were possible to get the x and y position of the rectangle after translating the canvas back left 410px and reflecting it horizontally once more I could easily use those coordinates to fill in the text at the proper position. From what I have learned about canvas, this is not possible. (Please correct me if im wrong)
Another solution I considered would be to use the x position generated and to apply this formula, -x+xLim, where xLim is the largest possible value of x. The problem here is that obtaining xLim is not possible either, it is not static, and it will change depending on the distance away from the detected object. I know this from trying to obtain what xLim could be by simply positioning the object on the leftmost side of the screen. (The largest value of x that is currently viewable with respect to the coco-ssd model's origin) Keep in mind, that if I create distance from the object, the value of x on the leftmost side of the screen will increase. If I were able to somehow grab the largest x value that is actively viewable on the canvas then this would be another viable solution.
Here is the function in-charge of drawing to the canvas.
export const drawRect = (x, y, width, height, text, ctx, canvas) => {
ctx.clearRect(0, 0, canvas.current.width, canvas.current.height);
ctx.transform(-1, 0, 0, 1, 410, 0);
//draw rectangle
ctx.beginPath();
const r = ctx.rect(x,y,width,height)
ctx.stroke()
//draw text
ctx.save();
ctx.scale(-1,1);
ctx.translate(-410, 0)
//update x and y to point to where the rectangle is currently
ctx.fillText(text,x,y-5)
ctx.stroke()
ctx.restore()
})
I feel heavily limited by the API available to react native and I hope that there is something I've simply overlooked. I've spent lots of time trying to resolve this issue and have found many relatable questions on stack overflow, but none of them gave insight as to how to solve this problem with so many unknown variables.
Here are some images to provide a visual of the issue at hand.
Leftmost
Rightmost
Without Restoring canvas to its original origin
SUMMARY:
The origin for the coco-ssd model is at the top right corner, and the origin for the canvas is at the top left corner.
I need to
A.)Somehow grab the largest x value that is actively viewable on the canvas
OR
B.) get the x and y position of the rectangle after translating the canvas back left 410px and reflecting it horizontally once more
This is in a react native expo environment
Public repo:
https://github.com/dpaceoffice/MobileAppTeamProject
The transforms that you are using seem unnecessary. Here is a simple proof of concept using the embed code for coco-ssd and an example on manipulating videos:
https://www.npmjs.com/package/#tensorflow-models/coco-ssd
https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API/Manipulating_video_using_canvas
The following code does not use any transforms and everything works as expected. The video, canvas and model all use the same coordinate system:
(0, 0) is top left corner of the frame
(width, 0) is top right
(0, height) is bottom left
(width, height) is bottom right
And the rectangle (x, y, width, height) also use the familiar coordinates everywhere:
(x, y) is top left
(x + width, y) is top right
(x, y + height) is bottom left
(x + width, y + height) is bottom right
The only thing that is a bit different, is the context.fillText(text, x, y) method:
(x, y) is bottom left corner of the text. However, this is actually nice, because we just can draw the rectangle and the text with the same coordinates and the text will sit directly above the rectangle.
(x, y - fontSize) is usually the top left corner
(x, y + fontSize) is the coordinate for the next row.
If you want to place the text at a different position, the context.measureText(text) might be of interest. It returns a TextMetrics object. Usually, the .width property of this object is of most interest:
(x + context.measureText(text).width, y) is the bottom right corner of the text.
const processor = {}
processor.doLoad = function (model)
{
const video = document.getElementById("video")
this.video = video
this.canvas = document.getElementById("canvas")
this.context = this.canvas.getContext("2d")
this.model = model
video.addEventListener("play", () => {
this.canvas.width = video.videoWidth
this.canvas.height = video.videoHeight
this.timerCallback()
}, false)
}
processor.timerCallback = async function ()
{
if (this.video.paused || this.video.ended)
return
await this.computeFrame()
setTimeout(() => this.timerCallback(), 0)
};
processor.computeFrame = async function ()
{
// detect objects in the image.
const predictions = await this.model.detect(this.video)
const context = this.context
// draws the frame from the video at position (0, 0)
context.drawImage(this.video, 0, 0)
context.strokeStyle = "red"
context.fillStyle = "red"
context.font = "16px sans-serif"
for (const { bbox: [x, y, width, height], class: _class, score } of predictions)
{
// draws a rect with top-left corner of (x, y)
context.strokeRect(x, y, width, height)
// writes the class directly above (x, y), outside the rectangle
context.fillText(_class, x, y)
// writes the class directly below (x, y), inside the rectangle
context.fillText(score.toFixed(2), x, y + 16)
}
}
// Load the model.
const model = cocoSsd.load()
model.then(model => processor.doLoad(model))
<!-- Load TensorFlow.js. This is required to use coco-ssd model. -->
<script src="https://cdn.jsdelivr.net/npm/#tensorflow/tfjs"> </script>
<!-- Load the coco-ssd model. -->
<script src="https://cdn.jsdelivr.net/npm/#tensorflow-models/coco-ssd"> </script>
<div style="display: flex; flex-flow: column; gap: 1em; width: 200px;">
<!-- Replace this with your image. Make sure CORS settings allow reading the image! -->
<video id="video" src="https://mdn.github.io/dom-examples/canvas/chroma-keying/media/video.mp4" controls crossorigin="anonymous"></video>
<canvas id="canvas" style="border: 1px solid black;"></canvas>
</div>
As it seems, that you want to draw the image into some part of the canvas, here is some code that showcases how to correctly use the transforms. There are two options here:
Use the transforms (1x translate + 1x scale), so that you can use the coordinate system from the image. Note, that it does not require negative scales. The drawing is handled by the browser. You have to correct the font and line widths for the scaling.
Also use transforms (1x translate + 1x scale). Then restore the canvas, and transform the points manually. This involves some further math to transform the points. However, the upside is, that you don't have to correct the font and line widths.
const image = document.createElement("canvas")
image.width = 1200
image.height = 600
const image_context = image.getContext("2d")
image_context.fillStyle = "#4af"
image_context.fillRect(0, 0, 1200, 600)
const circle = (x, y, r) =>
{
image_context.beginPath()
image_context.arc(x, y, r, 0, Math.PI*2)
image_context.fill()
}
image_context.fillStyle = "#800"
image_context.fillRect(500-40/2, 400, 40, 180)
image_context.fillStyle = "#080"
circle(500, 400, 100)
circle(500, 300, 70)
circle(500, 220, 50)
const prediction = { bbox: [500-100, 220-50, 100*2, (400+180)-(220-50)], class: "tree", score: 0.42 } // somehow get a prediction
const canvas = document.getElementById("canvas")
const context = canvas.getContext("2d")
// we want to draw the big image into a smaller area (x, y, width, height)
const x = 50
const y = 80
const width = 220
const height = width * (image.height / image.width)
// debug: mark the area that we want to draw the image into
{
context.save() // save context, so that stroke properties can be restored
context.lineWidth = 5
context.strokeStyle = "red"
context.setLineDash([10, 10])
context.strokeRect(x, y, width, height)
context.restore()
}
{
// save context, before doing any transforms (even before setTransform)
context.save()
// Move top left corner to (x, y)
context.translate(x, y)
// This is the scale factor, it should be less than one, because the image is bigger that the target area. The idea is to increase the scale by the target area width, and then decrease the scale by the image width.
const f = width / image.width
context.scale(f, f)
// Draws the image, note that the coordinates are just (0, 0) without scaling.
context.drawImage(image, 0, 0)
// option 1: draw the prediction using the native transforms
if (true)
{
context.strokeStyle = "red"
context.fillStyle = "red"
context.lineWidth = 1 / f // linewidth and font-size has to be adjusted by scaling
const fontSize = 16 / f
context.font = fontSize.toFixed(0) + "px sans-serif"
const [p_x, p_y, p_width, p_height] = prediction.bbox
context.strokeRect(p_x, p_y, p_width, p_height) // draw the prediction
context.fillText(prediction.class, p_x, p_y) // draw the text
context.fillText(prediction.score.toFixed(2), p_x, p_y + fontSize) // draw the text
}
const matrix = context.getTransform() // save transform for option 2, needs to be done before restore()
context.restore()
// option 2: draw the prediction by manually transforming the corners
if (false)
{
context.save()
context.strokeStyle = "red"
context.fillStyle = "red"
context.lineWidth = 1
const fontSize = 16
context.font = fontSize + "px sans-serif"
let [p_x, p_y, p_width, p_height] = prediction.bbox
// manually transform corners
const topleft = matrix.transformPoint(new DOMPoint(p_x, p_y))
const bottomright = matrix.transformPoint(new DOMPoint(p_x + p_width, p_y + p_height))
p_x = topleft.x
p_y = topleft.y
p_width = bottomright.x - topleft.x
p_height = bottomright.y - topleft.y
context.strokeRect(p_x, p_y, p_width, p_height) // draw the prediction
context.fillText(prediction.class, p_x, p_y) // draw the text
context.fillText(prediction.score.toFixed(2), p_x, p_y + fontSize) // draw the text
context.restore()
}
}
<canvas id="canvas" width=400 height=300 style="border: 1px solid black;"></canvas>
Now, if you need to mirror the image using the transforms, you just have to make sure to mirror it back before writing the text.
const image = document.createElement("canvas")
image.width = 1200
image.height = 600
const image_context = image.getContext("2d")
image_context.fillStyle = "#4af"
image_context.fillRect(0, 0, 1200, 600)
const circle = (x, y, r) =>
{
image_context.beginPath()
image_context.arc(x, y, r, 0, Math.PI*2)
image_context.fill()
}
image_context.fillStyle = "#800"
image_context.fillRect(500-40/2, 400, 40, 180)
image_context.fillStyle = "#080"
circle(500, 400, 100)
circle(500, 300, 70)
circle(500, 220, 50)
const prediction = { bbox: [500-100, 220-50, 100*2, (400+180)-(220-50)], class: "tree", score: 0.42 } // somehow get a prediction
const canvas = document.getElementById("canvas")
const context = canvas.getContext("2d")
// we want to draw the big image into a smaller area (x, y, width, height)
const x = 50
const y = 80
const width = 220
const height = width * (image.height / image.width)
// debug: mark the area that we want to draw the image into
{
context.save() // save context, so that stroke properties can be restored
context.lineWidth = 5
context.strokeStyle = "red"
context.setLineDash([10, 10])
context.strokeRect(x, y, width, height)
context.restore()
}
{
// save context, before doing any transforms (even before setTransform)
context.save()
// Move top left corner to (x, y)
context.translate(x, y)
// This is the scale factor, it should be less than one, because the image is bigger that the target area. The idea is to increase the scale by the target area width, and then decrease the scale by the image width.
const f = width / image.width
context.scale(f, f)
// mirror the image before drawing it
context.scale(-1, 1)
context.translate(-image.width, 0)
// Draws the image, note that the coordinates are just (0, 0) without scaling.
context.drawImage(image, 0, 0)
// option 1: draw the prediction using the native transforms
if (true)
{
const [p_x, p_y, p_width, p_height] = prediction.bbox
// move to correct position and only then undo the mirroring
context.save()
context.translate(p_x + p_width, p_y) // move to top "right" (that is now actually at the left, due to mirroring)
context.scale(-1, 1)
context.strokeStyle = "red"
context.fillStyle = "red"
context.lineWidth = 1 / f // linewidth and font-size has to be adjusted by scaling
const fontSize = 16 / f
context.font = fontSize.toFixed(0) + "px sans-serif"
context.strokeRect(0, 0, p_width, p_height) // draw the prediction
context.fillText(prediction.class, 0, 0) // draw the text
context.fillText(prediction.score.toFixed(2), 0, 0 + fontSize) // draw the text
context.restore()
}
const matrix = context.getTransform() // save transform for option 2, needs to be done before restore()
context.restore()
// option 2: draw the prediction by manually transforming the corners
if (false)
{
context.save()
context.strokeStyle = "red"
context.fillStyle = "red"
context.lineWidth = 1
const fontSize = 16
context.font = fontSize + "px sans-serif"
let [p_x, p_y, p_width, p_height] = prediction.bbox
// manually transform corners, note that compared to previous snippet, topleft now uses the top right corner of the rectangle
const topleft = matrix.transformPoint(new DOMPoint(p_x + p_width, p_y))
const bottomright = matrix.transformPoint(new DOMPoint(p_x, p_y + p_height))
p_x = topleft.x
p_y = topleft.y
p_width = bottomright.x - topleft.x
p_height = bottomright.y - topleft.y
context.strokeRect(p_x, p_y, p_width, p_height) // draw the prediction
context.fillText(prediction.class, p_x, p_y) // draw the text
context.fillText(prediction.score.toFixed(2), p_x, p_y + fontSize) // draw the text
context.restore()
}
}
<canvas id="canvas" width=400 height=300 style="border: 1px solid black;"></canvas>
I've been experimenting with the <canvas> recently, and I noticed a strange behaviour when stroking rectangles near the origin (0, 0) of the canvas.
// canvas context
var ctx = document.querySelector('#screen').getContext('2d');
// draw a rectangle
ctx.fillStyle = 'orange';
ctx.fillRect(0, 0, 100, 100);
// stroke a border for the rectangle
ctx.lineWidth = 20;
ctx.strokeRect(0, 0, 100, 100);
<canvas id="screen"></canvas>
What went wrong?
In the example above, the rectangle itself was drawn at (0, 0) as intended, but its border (the stroked rectangle) seems to be drawn at an offset.
Generally, when stroking a rectangle at a position away from the origin, this effect is omitted —
Meaning that the stroked rectangles aren't being drawn starting at the position specified, but at an offset, I suppose.
Why is that?
The stroke is centered around the coordinates that your primitve is defined at. In the case of your rectangle with stroke width of 20, drawing this at the top left of the canvas will cause half of the strokes width to be drawn outside of the canvas boundary.
Adjusting the coordinates of strokeRect() to 10,10,.. causes the rectangle to be offset from the canvas origin, meaning that the full stroke of 20 pixels will be visible from the top-left of the canvas:
ctx.lineWidth = 20;
ctx.strokeRect(10, 10, 100, 100);
Consider the following adjustments, made to ensure the stroke is fully visible around the drawn rectangle:
var canvas = document.querySelector('#screen');
// Set the width and height to specify dimensions of canvas (in pixels)
// Choosing a 100x100 square matches the strokeRect() drawn below and
// therefore achieves the appearance of a symmetric stroke
canvas.width = 100;
canvas.height = 100;
// canvas context
var ctx = canvas.getContext('2d');
// draw a rectangle
ctx.fillStyle = 'orange';
ctx.fillRect(10, 10, 90, 90);
// stroke a border for the rectangle
ctx.lineWidth = 20;
var halfStroke = ctx.lineWidth * 0.5;
ctx.strokeRect(halfStroke, halfStroke, 100 - (halfStroke * 2), 100 - (halfStroke * 2));
<canvas id="screen"></canvas>
Update
Here is a visualisation of the stroke in relation to the line/rectangle edge provided by Ibrahim Mahrir:
Here is an example!
I am trying to reset the green arc inside drawValueArc() so that each time you click the change button, the green arc is removed and redrawn. How can I remove it without removing the entire canvas? Also, as an aside, I have noticed that Math.random() * 405 * Math.PI / 180 doesn't actually always result in an arc that fits inside the gray arc, sometimes it is larger than the gray arc, why is this?
var canvas = document.getElementById("canvas");
var ctx = canvas.getContext("2d");
var cx = 150;
var cy = 150;
var startRadians = 135 * Math.PI / 180;
var endRadians = 405 * Math.PI / 180;
//main arc
ctx.beginPath();
ctx.arc(cx, cy, 58, startRadians, endRadians, false);
ctx.strokeStyle="rgb(220,220,220)";
ctx.lineWidth = 38;
ctx.stroke();
$('#setRandomValue').click(function(){
drawValueArc(Math.random() * 405 * Math.PI / 180);
});
function drawValueArc(val){
//ctx.clearRect(0, 0, W, H);
ctx.beginPath();
ctx.arc(cx, cy, 58, startRadians, val, false);
ctx.strokeStyle = "green";
ctx.lineWidth = 38;
ctx.stroke();
}
Drawing past boundary
The problem you are facing is in first instance the fact you are drawing before and after a 0-degree on the circle. This can be complicated to handle as you need to split in two draws: one for the part up to 0 (360) and one 0 to the remaining part.
There is a simple trick you can use to make this easier to deal with and that is to deal with all angles from 0 and use an offset when you draw.
Demo using redraw base (I moved it to jsfiddle as jsbin did not work for me):
http://jsfiddle.net/3dGLR/
Demo using off-screen canvas
http://jsfiddle.net/AbdiasSoftware/Dg9Jj/
First, some optimizations and settings for the offset:
var startRadians = 0; //just deal with angles
var endRadians = 300;
var deg2rad = Math.PI / 180; //pre-calculate this to save some cpu cycles
var offset = 122; //adjust this to modify rotation
We will now let the main function, drawArc() do all calculations for us so we can focus on the numbers - here we also offset the values:
function drawArc(color, start, end) {
ctx.beginPath();
ctx.arc(cx, cy, 58,
(startRadians + offset) * deg2rad,
(end + offset) * deg2rad, false);
ctx.strokeStyle = color;
ctx.lineWidth = 38;
ctx.stroke();
}
Clearing the previous arc
There are several techniques to clear the previous drawn arc:
You can draw the base arc to an off-screen canvas and use drawImage() to erase the old.
You can do as in the following example, just re-draw it with the base color
As with 2. but subtracting the green arc and draw the base color from the end of the green arc to the end of the base arc.
clearing the whole canvas with fillRect or clearRect.
1 and 3 are the fastest, while 4 is the slowest.
With out re-factored function (drawArc) it's as easy as this:
function drawValueArc(val) {
drawArc("rgb(220,220,220)", startRadians, endRadians);
drawArc("green", startRadians, val);
}
As everything now is 0-based concerning start we really don't need to give any other argument than 0 to the drawArc instead of startRadians. Use the new offset to offset the start position and adjust the endRadians to where you want it to stop.
As you can see in the demo, using this technique keeps everything in check without the need to draw in split.
Tip: if you notice green artifacts on the edges: this is due to anti-alias. Simply reduce the line width for the green color by 2 pixels (see demo 2, off-screen canvas).
I'm attempting to make a program that takes the information gathered from some calculations and plots it on a canvas graph. I need to scale the graph, however, so that it can accommodate larger numbers. But every time I put ctx.scale(); the whole canvas blanks out! I thought I could stop this by scaling the canvas first, but nothing is drawn on the canvas after I scale it.
Here's the coding for my canvas:
var c=document.getElementById("graph_");
var ctx=c.getContext("2d");
graph_.style.backgroundColor="white";
var z0=Math.max(Math.abs(a),Math.abs(b));
var z=Math.round(z0);
var z1=Math.round(z);
var z2=z*2
// alert(z1);
// alert(z2);
ctx.scale(3200/z,3200/z)
var xnew=360/2+360/2*a
var ynew=360/2-360/2*b
alert(xnew);
alert(ynew);
ctx.font = '10px Calibri';
ctx.fillText("( 0 , 0 )", 125, 85);
ctx.fillText(z1, 210, 87);
ctx.fillText(z2, 270, 87);
ctx.fillText(z1*-1, 75, 87);
ctx.fillText(z2*-1, 0, 87);
ctx.fillText(z1, 120, 43.5);
ctx.fillText(z2, 120, 10);
ctx.fillText(z1*-1, 120, 120);
ctx.fillText(z2*-1, 120, 145);
ctx.lineWidth = 1;
ctx.beginPath()
ctx.moveTo(150, 0);
ctx.lineTo(150, 400);
ctx.closePath();
ctx.lineWidth = .2;
ctx.moveTo(0, 75);
ctx.lineTo(400, 75);
ctx.strokeStyle = "#8B8682";
ctx.stroke();
ctx.closePath();
ctx.beginPath();
ctx.lineWidth = 2;
ctx.moveTo(xnew, 180);
ctx.lineTo(180, ynew);
ctx.strokeStyle = "red";
ctx.stroke();
Actually, the stuff is being drawn to the canvas, you just can't see it because you're both too far zoomed in and still in the upper left corner of the graph since the default origin points for drawing are in the top left as 0,0.
So if you want to zoom in that far (even though you probably want to zoom out to show bigger numbers, i.e. larger drawings on the graph) you need to translate the canvas origin point to your new origin point (the top left of what you want to see) before you scale the context.
You can use the translate method like
ctx.translate(newX,newY);
But before you do you're going to what to save the context's state so you can revert back to it.
Say you wanted to zoom in on the center of the graph you would translate to the point that is:
ctx.translate((-c.width /2 * scale) + offsetX,(-c.height / 2 * scale) + offsetY);
where the offsetX is the canvas width / 2 and offsetY is the canvas height / 2 and the scale is by the amount that you're scaling in you ctx.scale call.
What is the value of 3200/z, exactly?
I'm guessing that you are scaling your canvas by an enormous amount, so much so that the only thing visible on your screen would be the first few pixels of the canvas. Since you don't draw anything in the top-left 5 pixels of the screen, you don't see anything.
I'm trying to flip/mirror an image as I paint it on an HTML canvas; I found a game tutorial showing a sprite sheet per direction a character has to face, but this doesn't seem quite right to me. Especially since each frame has a different size.
What would be the best technique to reach this goal?
I tried to call the setScale(-1, 1); on my canvas with no success. Maybe that isn't meant for this.
Thanks
You can do this by transforming the context with myContext.scale(-1,1) before drawing your image, however
This is going to slow down your game. It's a better idea to have a separate, reversed sprite.
You need to set the scale of the canvas as well as inverting the width.
drawToCanvas : function(v, context, width, height){
context.save();
context.scale(-1, 1);
context.drawImage(v, 0, 0, width*-1, height);
context.restore();
}
There are probably some performance issues with this but for me that was not an issue.
If you just flip it horizontally it will get off of bounds... so use translate to adjust its position:
ctx.translate(canvas.width, 0);
ctx.scale(-1, 1);
ctx.drawImage(img, 0, 0);
For a shorter code you can remove the translate and use the image size as negative offset in the second parameter of the drawImage (x coordinate) instead:
ctx.scale(-1, 1);
ctx.drawImage(img, canvas.width * -1, 0);
If you want to restore the context later, add save/restore before and after it all:
ctx.save();
ctx.scale(-1, 1);
ctx.drawImage(img, canvas.width * -1, 0);
ctx.restore();
You don't need to redraw the entire image when creating a reflection. An original reflection simply shows the bottom part of the image. This way you are redrawing a smaller part of the image which provides better performance and also you don't need to create linear gradient to hide the lower part of the image (since you never draw it).
var img = new Image();
img.src = "//vignette2.wikia.nocookie.net/tomandjerryfan/images/9/99/Jerry_Mouse.jpg/revision/latest?cb=20110522075610";
img.onload = function() {
var thumbWidth = 250;
var REFLECTION_HEIGHT = 50;
var c = document.getElementById("output");
var ctx = c.getContext("2d");
var x = 1;
var y = 1;
//draw the original image
ctx.drawImage(img, x, y, thumbWidth, thumbWidth);
ctx.save();
//translate to a point from where we want to redraw the new image
ctx.translate(0, y + thumbWidth + REFLECTION_HEIGHT + 10);
ctx.scale(1, -1);
ctx.globalAlpha = 0.25;
//redraw only bottom part of the image
//g.drawImage(image, sx, sy, sWidth, sHeight, dx, dy, dWidth, dHeight);
ctx.drawImage(img, 0, img.height - REFLECTION_HEIGHT, img.width, REFLECTION_HEIGHT, x, y, thumbWidth, REFLECTION_HEIGHT);
// Revert transform and scale
ctx.restore();
};
body {
background-color: #FFF;
text-align: center;
padding-top: 10px;
}
<canvas id="output" width="500" height="500"></canvas>