We Keep Coding

HTML Canvas coordinate systems and rendering process

I'm playing with drawing on html canvas and I'm little confused of how different coordinate systems actually works. What I have learned so far is that there are more coordinate systems:
canvas coordinate system
css coordinate system
physical (display) coordinate system
So when I draw a line using CanvasRenderingContext2D
ctx.lineWidth = 1;
ctx.beginPath();
ctx.moveTo(3, 1);
ctx.lineTo(3, 5);
ctx.stroke();
before drawing pixels to the display, the path needs to be
scaled according to the ctx transformation matrix (if any)
scaled according to the ratio between css canvas element dimensions (canvas.style.width and canvas.style.height) and canvas drawing dimensions (canvas.width and canvas.height)
scaled according to the window.devicePixelRatio (hi-res displays)
Now when I want to draw a crisp line, I found that there are two things to fight with. The first one is that canvas uses antialiasing. So when I draw a line of thikness 1 at integer coordinates, it will be blurred.
To fix this, it needs to be shifted by 0.5 pixels
ctx.moveTo(3.5, 1);
ctx.lineTo(3.5, 5);
The second thing to consider is window.devicePixelRatio. It is used to map logical css pixels to physical pixels. The snadard way how to adapt canvas to hi-res devices is to scale to the ratio
const ratio = window.devicePixelRatio || 1;
const clientBoundingRectangle = canvas.getBoundingClientRect();
canvas.width = clientBoundingRectangle.width * ratio;
canvas.height = clientBoundingRectangle.height * ratio;
const ctx = canvas.getContext('2d');
ctx.scale(ratio, ratio);
My question is, how is the solution of the antialiasing problem related to the scaling for the hi-res displays?
Let's say my display is hi-res and window.devicePixelRatio is 2.0. When I apply context scaling to adapt canvas to the hi-res display and want to draw the line with thickness of 1, can I just ignore the context scale and draw
ctx.moveTo(3.5, 1);
ctx.lineTo(3.5, 5);
which is in this case effectively
ctx.moveTo(7, 2);
ctx.lineTo(7, 10);
or do I have to consider the scaling ratio and use something like
ctx.moveTo(3.75, 1);
ctx.lineTo(3.75, 5);
to get the crisp line?

Antialiasing can occur both in the rendering on the canvas bitmap buffer, at the time you draw to it, and at the time it's displayed on the monitor, by CSS.
The 0.5px offset for straight lines works only for line widths that are odd integers. As you hinted to, it's so that the stroke, that can only be aligned to the center of the path, and thus will spread inside and outside of the actual path by half the line width, falls on full pixel coordinates. For a comprehensive explanation, see this previous answer of mine.
Scaling the canvas buffer to the monitor's pixel ratio works because on high-res devices, multiple physical dots will be used to cover a single px area. This allows to have more details e.g in texts, or other vector graphics. However, for bitmaps this means the browser has to "pretend" it was bigger in the first place. For instance a 100x100 image, rendered on a 2x monitor will have to be rendered as if it was a 200x200 image to have the same size as on a 1x monitor. During that scaling, the browser may yet again use antialiasing, or another scaling algorithm to "create" the missing pixels.
By directly scaling up the canvas by the pixel ratio, and scaling it down through CSS, we end up with an original bitmap that's the size it will be rendered, and there is no need for CSS to scale anything anymore.
But now, your canvas context is scaled by this pixel ratio too, and if we go back to our straight lines, still assuming a 2x monitor, the 0.5px offset now actually becomes a 1px offset, which is useless. A lineWidth of 1 will actually generate a 2px stroke, which doesn't need any offset.
So no, don't ignore the scaling when offsetting your context for straight lines.
But the best is probably to not use that offset trick at all, and instead use rect() calls and fill() if you want your lines to fit perfectly on pixels.
const canvas = document.querySelector("canvas");
// devicePixelRatio may not be accurate, see below
setCanvasSize(canvas);
function draw() {
const dPR = devicePixelRatio;
const ctx = canvas.getContext("2d");
// scale() with weird zoom levels may produce antialiasing
// So one might prefer to do the scaling of all coords manually:
const lineWidth = Math.round(1 * dPR);
const cellSize = Math.round(10 * dPR);
for (let x = cellSize; x < canvas.width; x += cellSize) {
ctx.rect(x, 0, lineWidth, canvas.height);
}
for (let y = cellSize; y < canvas.height; y += cellSize) {
ctx.rect(0, y, canvas.width, lineWidth);
}
ctx.fill();
}
function setCanvasSize(canvas) {
// We resize the canvas bitmap based on the size of the viewport
// while respecting the actual dPR
// Thanks to gman for the reminder of how to suppport all early impl.
// https://stackoverflow.com/a/65435847/3702797
const observer = new ResizeObserver(([entry]) => {
let width;
let height;
const dPR = devicePixelRatio;
if (entry.devicePixelContentBoxSize) {
width = entry.devicePixelContentBoxSize[0].inlineSize;
height = entry.devicePixelContentBoxSize[0].blockSize;
} else if (entry.contentBoxSize) {
if ( entry.contentBoxSize[0]) {
width = entry.contentBoxSize[0].inlineSize * dPR;
height = entry.contentBoxSize[0].blockSize * dPR;
} else {
width = entry.contentBoxSize.inlineSize * dPR;
height = entry.contentBoxSize.blockSize * dPR;
}
} else {
width = entry.contentRect.width * dPR;
height = entry.contentRect.height * dPR;
}
canvas.width = width;
canvas.height = height;
canvas.style.width = (width / dPR) + 'px';
canvas.style.height = (height / dPR) + 'px';
// we need to redraw
draw();
});
// observe the scrollbox size changes
try {
observer.observe(canvas, { box: 'device-pixel-content-box' });
}
catch(err) {
observer.observe(canvas, { box: 'content-box' });
}
}
canvas { width: 300px; height: 150px; }
<canvas></canvas>

Preventing anti-aliasing requires that the pixels of the canvas, which is a raster image, are aligned with the pixels of the screen, which can be done by multiplying the canvas size by the devicePixelRatio, while using the CSS size to hold the canvas to its original size:
canvas.width = pixelSize * window.devicePixelRatio;
canvas.height = pixelSize * window.devicePixelRatio;
canvas.style.width = pixelSize + 'px';
canvas.style.height = pixelSize + 'px';
You can then use scale on the context, so that the drawn images won't be shrunk by higher devicePixelRatios. Here I am rounding so that lines can be crisp on ratios that are not whole numbers:
let roundedScale = Math.round(window.devicePixelRatio);
context.scale(roundedScale, roundedScale);
The example then draws a vertical line from the center top of one pixel to the center top of another:
context.moveTo(100.5, 10);
context.lineTo(100.5, 190);
One thing to keep in mind is zooming. If you zoom in on the example, it will become anti-aliased as the browser scales up the raster image. If you then click run on the example again, it will become crisp again (on most browsers). This is because most browsers update the devicePixelRatio to include any zooming. If you are rendering in an animation loop while they are zooming, the rounding could cause some flickering.

Use drawImage to scale a canvas above

I'm trying to make a pixel editor with 2 canvas. The first canvas displays a second canvas which contains the pixels. The first canvas uses drawImage to position and scale the second canvas.
When the second canvas is scaled smaller than it's original size, it starts to glitch.
Here is the canvas displayed at it's original size. When I zoom in, the second canvas get bigger and everything works perfectly.
However when I zoom out, the grid and the background (transparency) act very strangely.
To draw the second canvas on the first canvas, I use the function
ctx.drawImage(drawCanvas, offset.x, offset.y, width * pixelSize, height * pixelSize);
I have read that scaling in multiple iterations might give a better quality with images but I am not sure about a canvas.
I could fully redraw the second canvas in a lower resolution when the user zooms out, but it is a bit heavy on the cpu.
Is there any better solution that I don't know of?

Your problem comes from anti-aliasing.
Pixels aren't sub-divisible, and when you ask the computer to draw something outside of the pixel boundaries, it will try its best to render something that usually looks good to eyes, by mixing the colors so that what should have been a black 0.1 pixel line will become a light-gray pixel for instance.
This generally works good, particularly with pictures of the real word, or complex shapes like circles. However with grids... That's not so great as you experienced it.
Your case is dealing with two different cases, and you will have to deal with hem separately.
In the canvas 2D API (and a lot of 2D APIs) stroke do bleed from both sides of the coordinates you did set it. So when drawing lines of 1px wide, you need to account for a 0.5px offset to be sure it won't get rendered as two gray pixels. For more info about this, see this answer. You are probably using such a stroke for the grid.
fill on the other hand only covers the inside of the shape, so if you fill a rectangle, you need to not offset its coords from the px boundaries. This is required for the checkerboard.
Now, for boh these drawings, the best is probably to use patterns. You only need to draw a small version of it, and then the pattern will repeat it automatically, saving a lot of computation.
Scaling of a pattern can be done by calling the transform methods of the 2D context. We can even take advantage of the closest-neighbor algorithm to avoid antialising when drawing this pattern by setting the imageSmoothingEnabled property to false.
However for our grid, we may want to keep the lineWidth constant. For this we will need to generate a new pattern at every draw call.
// An helper function to create CanvasPatterns
// returns a 2DContext on which a simple `finalize` method is attached
// method which does return a CanvasPattern from the underlying canvas
function patternMaker(width, height) {
const canvas = document.createElement('canvas');
canvas.width = width;
canvas.height = height;
const ctx = canvas.getContext('2d');
ctx.finalize = (repetition = "repeat") => ctx.createPattern(canvas, repetition);
return ctx;
}
// The checkerboard can be generated only once
const checkerboard_patt_maker = patternMaker(2, 2);
checkerboard_patt_maker.fillStyle = "#CCC";
checkerboard_patt_maker.fillRect(0,0,1,1);
checkerboard_patt_maker.fillRect(1,1,1,1);
const checkerboard_patt = checkerboard_patt_maker.finalize();
// An helper function to create grid patterns
// Since we want a constant lineWidth, no matter the zoom level
function makeGridPattern(width, height) {
width = Math.round(width);
height = Math.round(height);
const grid_patt_maker = patternMaker(width, height);
grid_patt_maker.lineWidth = 1;
// apply the 0.5 offset only if we are on integer coords
// for instance a <3,3> pattern wouldn't need any offset, 1.5 is already perfect
const x = width/2 % 1 ? width/2 : width/2 + 0.5;
const y = height/2 % 1 ? height/2 : height/2 + 0.5;
grid_patt_maker.moveTo(x, 0);
grid_patt_maker.lineTo(x, height);
grid_patt_maker.moveTo(0, y);
grid_patt_maker.lineTo(width, y);
grid_patt_maker.stroke();
return grid_patt_maker.finalize();
}
const canvas = document.getElementById('canvas');
const ctx = canvas.getContext('2d');
const checkerboard_input = document.getElementById('checkerboard_input');
const grid_input = document.getElementById('grid_input');
const connector = document.getElementById('connector');
function draw() {
ctx.clearRect(0, 0, canvas.width, canvas.height);
const checkerboard_zoom = checkerboard_input.value;
const grid_zoom = grid_input.value;
// we generate a new pattern for the grid, so the lineWidth is always 1
const grid_patt = makeGridPattern(grid_zoom, grid_zoom);
// draw once the rectangle covering the whole canvas
// with normal transforms
ctx.beginPath();
ctx.rect(0, 0, canvas.width, canvas.height);
// the checkerboard
ctx.fillStyle = checkerboard_patt;
// our path is already drawn, we can control only the fill
ctx.scale(checkerboard_zoom, checkerboard_zoom);
// avoid antialiasing when painting our pattern (similar to rounding the zoom level)
ctx.imageSmoothingEnabled = false;
ctx.fill();
// done, reset to normal
ctx.imageSmoothingEnabled = true;
ctx.setTransform(1, 0, 0, 1, 0, 0);
// paint the grid
ctx.fillStyle = grid_patt;
// because our grid is drawn in the middle of the pattern
ctx.translate(Math.round(grid_zoom/2), Math.round(grid_zoom/2));
ctx.fill();
// reset
ctx.setTransform(1, 0, 0, 1, 0, 0);
}
draw();
checkerboard_input.oninput = grid_input.oninput = function(e) {
if(connector.checked) {
checkerboard_input.value = grid_input.value = this.value;
}
draw();
};
connector.oninput = e => checkerboard_input.oninput();
<label>checkerboard-layer zoom<input id="checkerboard_input" type="range" min="2" max="50" step="0.1"></label><br>
<label>grid-layer zoom<input id="grid_input" type="range" min="2" max="50" step="1"></label><br>
<label>connect both zooms<input id="connector" type="checkbox"></label>
<canvas id="canvas"></canvas>

JS canvas white lines when scaling

Using JavaScript I am displaying an array on an html 5 canvas. The program uses c.fillRect() for each value in the array. Everything looks normal until I scale it using c.scale(). After being scaled white lines are visible between the squares. I do know their white because that is the color of the background (When the background changes their color changes too).
Since the squares are 5 units apart I tried setting their width to 5.5 instead of 5; this only remove the white lines when zoom in far enough, but when zooming out the white lines were still there.
This is my code (unnecessary parts removed):
function loop()
{
c.resetTransform();
c.fillStyle = "white";
c.fillRect(0, 0, c.canvas.width, c.canvas.height);
c.scale(scale, scale);
c.translate(xViewportOffset, yViewportOffset);
...
for(var x = 0; x < array.length; x++)
{
for(var y = 0; y < array[x].length; y++)
{
...
c.fillStyle = 'rgb(' + r + ',' + g + ',' + b + ')';
c.fillRect(0 + x * 5, 200 + y * 5, 5, 5);
}
}
...
}
No scaling:
Zoomed in:
Zoomed out:
(the pattern changes depending on the amount of zoom)
Thanks for any help and if any other information is needed please let me know.
Update:
I am using Google Chrome
Version 71.0.3578.98 (Official Build) (64-bit)

This is probably because you are using non-integer values to set the context's scale and/or translate.
Doing so, your rects are not on pixel boundaries anymore but on floating values.
Let's make a simple example:
Two pixels, one at coords (x,y) (11,10) the other at coords (12,10).
At default scale, both pixels should be neighbors.
Now, if we apply a scale of 1.3, the real pixel-coords of the first square will be at (14.3,13) and the ones of the second one at (15.6,13).
None of these coords can hold a single pixel, so browsers will apply antialiasing, which consist in smoothing your color with the background color to give the impression of smaller pixels. This is what makes your grids.
const ctx = small.getContext('2d');
ctx.scale(1.3, 1.3);
ctx.fillRect(2,10,10,10);
ctx.fillRect(12,10,10,10);
const mag = magnifier.getContext('2d');
mag.scale(10,10);
mag.imageSmoothingEnabled = false;
mag.drawImage(small, 0,-10);
/* it is actually transparent, not just more white */
body:hover{background:yellow}
<canvas id="small" width="50" height="50"></canvas><br>
<canvas id="magnifier" width="300" height="300"></canvas>
To avoid this, several solutions, all dependent on what you are doing exactly.
In your case, it seems you'd win a lot by working on an ImageData which would allow you to replace all these fillRect calls to simpler and faster pixel manipulation.
By using a small ImageData, the size of your matrix, you can replace each rect to a single pixel. Then you just need to put this matrix on your canvas and redraw the canvas over itself at the correct scale after disabling the imageSmootingEnabled flag, which allows us to disable antialiasing for drawImage and CanvasPatterns only.
// the original matrix will be 20x20 squares
const width = 20;
const height = 20;
const ctx = canvas.getContext('2d');
// create an ImageData the size of our matrix
const img = ctx.createImageData(width, height);
// wrap it inside an Uint32Array so that we can work on it faster
const pixels = new Uint32Array(img.data.buffer);
// we could have worked directly with the Uint8 version
// but our loop would have needed to iterate 4 pixels every time
// just to draw a radial-gradient
const rad = width / 2;
// iterate over every pixels
for(let x=0; x<width; x++) {
for(let y=0; y<height; y++) {
// make a radial-gradient
const dist = Math.min(Math.hypot(rad - x, rad - y), rad);
const color = 0xFF * ((rad - dist) / rad) + 0xFF000000;
pixels[(y * width) + x] = color;
}
}
// here we are still at 50x50 pixels
ctx.putImageData(img, 0, 0);
// in case we had transparency, this composite mode will ensure
// that only what we draw after is kept on the canvas
ctx.globalCompositeOperation = "copy";
// remove anti-aliasing for drawImage
ctx.imageSmoothingEnabled = false;
// make it bigger
ctx.scale(30,30);
// draw the canvas over itself
ctx.drawImage(canvas, 0,0);
// In case we draw again, reset all to defaults
ctx.setTransform(1,0,0,1,0,0);
ctx.globalCompositeOperation = "source-over";
body:hover{background:yellow}
<canvas id="canvas" width="600" height="600"></canvas>

Canvas grid gets blurry at different zoom levels

I am trying to create a simple canvas grid which will fit itself to the player's current zoom level, but also to a certain canvas height/width proportional screen limit. Here is what I got so far:
JS:
var bw = window.innerWidth / 2; //canvas size before padding
var bh = window.innerHeight / 1.3; //canvas size before padding
//padding around grid, h and w
var pW = 30;
var pH = 2;
var lLimit = 0; //9 line limit for both height and width to create 8x8
//size of canvas - it will consist the padding around the grid from all sides + the grid itself. it's a total sum
var cw = bw + pW;
var ch = bh + pH;
var canvas = $('<canvas/>').attr({width: cw, height: ch}).appendTo('body');
var context = canvas.get(0).getContext("2d");
function drawBoard(){
for (var x = 0; lLimit <= 8; x += bw / 8) { //handling the height grid
context.moveTo(x, 0);
context.lineTo(x, bh);
lLimit++;
}
for (var x = 0; lLimit <= 17; x += bh / 8) { //handling the width grid
context.moveTo(0, x); //begin the line at this cord
context.lineTo(bw, x); //end the line at this cord
lLimit++;
}
//context.lineWidth = 0.5; what should I put here?
context.strokeStyle = "black";
context.stroke();
}
drawBoard();
Now, I succeeded at making the canvas to be at the same proportional level for each screen resolution zoom level. this is part of what I am trying to achieve. I also try to achieve thin lines, which will look the same at all different zooming levels, and of course to remove the blurriness. right now the thickness
of the lines change according to the zooming levels and are sometimes blurry.
Here is jsFiddle (although the jsFiddle window itself is small so you will barely notice the difference):
https://jsfiddle.net/wL60jo5n/
Help will be greatly appreciated.

To prevent blur, you should account for window.devicePixelRatio when setting dimensions of your canvas element (and account for that dimensions during subsequent drawing, of course).
width and height properties of your canvas element should contain values that are proportionally higher than values in CSS properties of the same names. This can be expressed e.g. as the following function:
function setCanvasSize(canvas, width, height) {
var ratio = window.devicePixelRatio,
style = canvas.style;
style.width = '' + (width / ratio) + 'px';
style.height = '' + (height / ratio) + 'px';
canvas.width = width;
canvas.height = height;
}

To remove blurry effect on canvas zoom/scale i used image-rendering: pixelated in css

The problem is that you are using decimal values to draw. Both the canvas width and the position increments in your drawBoard() loop use fractions. The canvas is a bitmap surface, not a vectorial drawing. When you set the width and height of the canvas, you set the actual number of pixels stored in memory. That value cannot be decimal (browsers will probably just trim the decimal part). When you try to draw at decimal positions, the canvas will use pixel interpolation to avoid aliasing, hence the occasional blur.
See a version where I round x before drawing:
https://jsfiddle.net/hts7yybm/
Try rounding the values just before you draw them, but not in your actual logic. That way, the imprecision won't stack as the algorithm keeps adding to the value.
function drawBoard(){
for (var x = 0; lLimit <= 8; x += bw / 8) {
var roundedX = Math.round(x);
context.moveTo(roundedX, 0);
context.lineTo(roundedX, bh);
lLimit++;
}
for (var x = 0; lLimit <= 17; x += bh / 8) {
var roundedX = Math.round(x);
context.moveTo(0, roundedX);
context.lineTo(bw, roundedX);
lLimit++;
}
context.lineWidth = 1; // never use decimals
context.strokeStyle = "black";
context.stroke();
}
EDIT: I'm pretty sure all browsers behave as if the canvas was an img element, so there's no way to prevent aliasing when the user zooms with their browser's zoom function, other than with prefixed css. And even then, I'm not sure the browsers's zoom feature takes that into account.
canvas {
image-rendering: -moz-crisp-edges;
image-rendering: -o-crisp-edges;
image-rendering: -webkit-optimize-contrast;
image-rendering: crisp-edges;
-ms-interpolation-mode: nearest-neighbor;
}
Also, make sure the canvas doesn't have any CSS-set dimensions. That only stretches the image after it's been drawn instead of increasing the drawing surface. If you want to fill a block with the canvas by giving it 100% width and height, then you need some JS to compute the CSS-given height and width and set the value of the canvas's width and height property based on that. Then you can make your own implementation of a zoom function within your canvas drawing code, but depending on what you're doing it might be overkill.

Image jumping at the beginning of scrolling

I wrote some code to zoom in my image, but when I scroll at the very beginning this picture jumps a little. How to fix the problem?
Full page view.
Editor view.
HTML
<canvas id="canvas"></canvas>
JS
function draw(scroll) {
scroll = (window.scrollY || window.pageYOffset) / (document.body.clientHeight - window.innerHeight) * 3000;
canvas.setAttribute('width', window.innerWidth);
canvas.setAttribute('height', window.innerHeight);
//The main formula that draws and zooms the picture
drawImageProp(ctx, forest, 0, (-scroll * 3.9) / 4, canvas.width, canvas.height + (scroll * 3.9) / 2);
}

Not a bug fix
I had a look at the Codepen example and it does jump at the top (sometimes). I have a fix for you but I did not have the time to locate the source of your code problem. I did notice that the jump involved a aspect change so it must be in the scaling that your error is. (look out for negatives)
GPU is a better clipper
Also your code is actually doing unnecessary work, because you are calculating the image clipping region. Canvas context does the clipping for you and is especially good at clipping images. Even though you provide the clip area the image will still go through clip as that is part of the render pipeline. The only time you should be concerned about the clipped display of an image is whether or not any part of the image is visible so that you don't send a draw call, and it only really matters if you are pushing the image render count (ie game sprite counts 500+)
Code example
Anyway I digress. Below is my code. You can add the checks and balances. (argument vetting, scaling max min, etc).
Calling function.
// get a normalised scale 0-1 from the scroll postion
var scale = (window.scrollY || window.pageYOffset) / (document.body.clientHeight - window.innerHeight);
// call the draw function
// scale 0-1 where 0 is min scale and 1 is max scale (min max determined in function
// X and y offset are clamped but are ranged
// 0 - image.width and 0 - image.height
// where 0,0 shows top left and width,height show bottom right
drawImage(ctx, forest, scale, xOffset, yOffset);
The function.
The comments should cover what you need to know. You will notice that all I am concerned with is how big the image should be and where the top left corner will be. The GPU will do the clipping for you, and will not cost you processing time (even for unaccelerated displays). I personally like to work with normalised values 0-1, it is a little extra work but my brain likes the simplicity, it also reduces the need for magic numbers (magics number are a sign that code is not adaptable) . Function will work for any size display and any size image. Oh and I like divide rather than multiply, (a bad coding habit that comes from a good math habit) replacing the / 2 and needed brackets with * 0.5 will make it more readable.
function drawImage(ctx, img, scale, x, y){
const MAX_SCALE = 4;
const MIN_SCALE = 1;
var w = canvas.width; // set vars just for source clarity
var h = canvas.height;
var iw = img.width;
var ih = img.height;
var fit = Math.max(w / iw, h / ih); // get the scale to fill the avalible display area
// Scale is a normalised value from 0-1 as input arg Convert to range
scale = (MAX_SCALE - MIN_SCALE) * scale + MIN_SCALE;
var idw = iw * fit * scale; // get image total display size;
var idh = ih * fit * scale;
x /= iw; // normalise offsets
y /= ih; //
x = - (idw - w) * x; // transform offsets to display coords
y = - (idh - h) * y;
x = Math.min( 0, Math.max( - (idw - w), x) ); // clamp image to display area
y = Math.min( 0, Math.max( - (idh - h), y) );
// use set transform to scale and translate
ctx.setTransform(scale, 0, 0, scale, idw / 2 + x, idh / 2 + y);
// display the image to fit;
ctx.drawImage(img, ( - iw / 2 ) * fit, (- ih / 2 ) * fit);
// restore transform.
ctx.setTransform(1, 0, 0, 1, 0, 0)
}
Sorry I did not solve the problem directly, but hopefully this will help you redesign your approch.
I recently added a similar answer involving zooming and panning (and rotation) with the mouse which you may be interested in How to pan the canvas? Its a bit messy still "note to self (my clean it up)" and has no bounds clamping. But shows how to set a zoom origin, and convert from screen space to world space. (find where a screen pixel is on a pan/scale/rotated display).
Good luck with your project.