I have an array of RGB value integers/whole numbers, ranging from 0 to 255. Eight Different lists.
For example one list has 8,349,042 values in it. OR 2,783,014 RGB sets of three.
The core objective is the user selects a pixel in an image.
That pixel's (R,G,B) value is grabbed and searched for within these lists. It exists in one of these lists, as all the lists together contain all the possible RGB combinations (16,777,216)
I'm trying to figure out what's the best way to store and search through these values.
again: these values don't change, they are hard coded lists(see Bullet Point below), with a known range.
The search query would be at minimum 3 times every event which would be every 10-30 seconds or so if the user was spamming.
OR Best case scenario, if the storage and search technique is fast enough I would like to: run the search on every pixel in an image (of maybe 800 x 600 or smaller resolutions) to have more data to play with. If I run into any memory limitations, I plan to work with it and use it as restrictions for my game design.
I used Javascript to generate these lists, going through and assigning each value based on how close it was to a base color.
[maybe unimportant how I made these lists]
I first assigned black and white RGB lists based on hard numeric limitations, then the rest of the RGB values were looped through and assigned to their closest base colors, red, yellow, cyan, green, blue and magenta. If there was a tie in distance I gave it to currently shortest list to try to keep it somewhat balanced. I may try to optimize this later and generate a new list, but not during runtime, just raw data.
I saved the results in hard text, and they are currently stored as a text that I can dump into large array.
At first I was trying to store the data as a JSON file along with my scripts. But I struggled to read the data and save it to an array. I ran into issues with using fetch and async and not being able to have the array where I needed it. Testing with console.log(arr) and getting undefined. I'm guessing because it wasn't loaded yet.
I can just paste it hard coded into the array but it's ginormous and I know there has to be a better way.
Also, hearing about differences in arrays vs sets vs objects
and different searching techniques within them.
Most of them seemed to be more tactful for multi variable arrays like name and age and location databases.
Since my data is all numeric I was thinking it could be a more bit/byte based approach?
I was reading some things on trees and hashs, bit crunching encryption?
Trees seemed nice for quicker searching as I could try to assign each branch of the tree to each R, G and B of the value, but I would also need to figure out how to convert my Giant single list of numbers into that, it also could be just the search type and that may depend on how I store the data.
I also struggled to understand the difference between front end and back end. I believe everything I've tried would be considered front end as I'm only testing my code in a browser.
I was pointed towards Node for backend but got lost in trying to get the console to run things.
I'm willing to give any of these things a try but I don't want to go down a path and find out it can't do what I want, or not optimally enough, like a server burden, or user burden with waiting too long or unable to do things because of user data security, requiring the user to do something more than just load the game, permissives wise.
SO I'm hoping someone can give me suggestions on what I should pursue so I can knuckle down and have a better focus on what I need to learn to be well versed and best tackle problem.
EDIT: Simplified question: In Javascript, I have an array of 2 million (x,y,z) numbers. What's the best way to search that array for a specific (x,y,z) value?
Would it be better to store the data in a different format than an array for constant searches?
I'm not certain I've understood the overall goal here but have a suggestion to consider if you are trying to assign a predetermined value to some (almost random as it is picked as a pixel colour from an image) rgb number set.
I assume the list/dictionary you have made allocates some value to each rgb number set and that it can be regenerated or reformatted if needed.
There are a maximum of 16,777,216 rgba values (256^3). Javascript arrays can have up to 2^32-2 (almost 4.3 billion) elements. Therefore the suggestion is to reformat your dictionary to be a 3-dimensional array where each dimension is indexed 0-255. The array can be declared and assigned as an array literal in a regular js script (text file) like colourDictionary= [[[val0]..[val255]]..[[]..[]],...[[val0]..[val255]]..[[]..[]]]; and each value accessed arithmetically in constant time using the pixel values as colourDictionary[r][g][b]
To be useful without writing lines to cater for missing values, your gaps (you mention a list of 8,349,042, around half the available number combinations) could be filled with the values of nearest neighbours.
Apologies if I've missed the point.
Related
Edit: Since I'm just planning my approach to the project, I have no code to show for it. My question is more about how i could approach the problem (arrays, objects or if there are smarter ways to do it) than about the actual code. Since I'm pretty new Im looking for a "way in" how I could tackle the project and look at tutorials.
(Since I'm pretty new to coding I'm trying to solidify the basics of html/css/js by building my own projects. With this project my problem is finding my way in, since I struggle with finding the correct mechanics, so I cant find any tutorials because i dont know where to start)
The project
I'm trying to build a project which pairs 3 images ("background", "base" and "addition"), starting from one which is selected.
For example: I'm selecting in image with the "background" woods and it will pair it with a small river as "base" and foxes as "addition".
If the user doesn't like the pairing, he can click a btn and it will pair the same selected image (in this case the wood) with other two compatible images (for example: a mossy stone and squirrels).
However: it wouldn pair my selected image, which in this case is a wood with a desert floor and dolphins.
The user also could start by selecting a "addition", for example a crab, which will be paired with a beach and smaller pebbles.
The problem
I'm pretty sure that the main focus with this project should be on arrays.
I thought i could do 3 arrays (one for the background, one for the base and one for the addition) and from there just pair random values from the different arrays. However, should i make 3 arrays for every probability or is there a way I could filter the pairs I want and the pairs i dont want?
An other approach i thought of was to let every image in the array be an object, so that i can assign attributes (for example: climate, color, sea level..) to the objects and filter them based on that.
In this case however I'm struggling with the code to first filter the correct pairings and then select a random pairing from those which were selected.
Every bit of help is appreciated,
thank you!
I need to generate several objects on page load with unique names that includes a method that generates the following code in my html:
<a-entity gltf-model="#filename" modify-materials position="0 0 !insert location here!"></a-entity>
The !insert location here! is a z-axis value, which can be positive or negative. As the user moves across the z-axis, I would like it to endlessly generate new objects with location values at intervals of 12 units that are within a range, and unload the objects as they get out of range.
I need to be able to add child objects to each placed model object, and I will need to go back and add (and store) properties to specific instances as well, so including a couple of those in the code to generate at the same time would be nice too.
It needs to be efficient, with as few server requests and user-position checks as possible. Possibly lazy-loading. I am having a breakdown with this seemingly herculean task, I'm sure I'm missing something that makes it easy.
I am willing to use libraries and the like, but prefer the lightest weight option available, even if it means writing more code by hand. (just need to figure out how) PLEASE, YOUR GUIDANCE IS APPRECIATED.
Every time I try and code this up, it gets suuuper bloated, and at some point the code just breaks down and I get lost.
I've been performing some research, in order to find the best approach to identify break points (trend direction change) in a dataset (with pairs of x/y coordinates), that allow me to identify the trend lines behind my data collections.
However I had no luck, finding anything that brings me some light.
The yellow dots in the following image, represent the breakpoints I need to detect.
Any suggestion about an article, algorithm, or implementation example (typescript prefered) would be very helpful and appreciated.
Usually, people tend to filter the data by looking only maximums (support) or only minimums (resistance). A trend line could be the average of those. The breakpoints are when the data crosses the trend, but this gives a lot of false breakpoints. Because images are better than words, you can look at page 2 of http://www.meacse.org/ijcar/archives/128.pdf.
There are a lot of scripts available look for "ZigZag" in
https://www.tradingview.com/
e.g. https://www.tradingview.com/script/lj8djt1n-ZigZag/ https://www.tradingview.com/script/prH14cfo-Trend-Direction-Helper-ZigZag-and-S-R-and-HH-LL-labels/
Also you can find an interesting blog post here (but code in in python):
https://towardsdatascience.com/programmatic-identification-of-support-resistance-trend-lines-with-python-d797a4a90530
with code available: https://pypi.org/project/trendln/
If you can identify trend lines then can't you just identify a breakpoint as when the slope changes? If you can't identify trend lines, then can you for example, take a 5-day moving average and see when that changes slope?
This might sound strange, or even controversial, but -- there are no "breakpoints". Even looking at your image, the fourth breakpoint might as well be on the local maximum immediately before its current position. So, different people might call "breakpoints" different points on the same graph (and, indeed, they do).
What you have in the numbers are several possible moving averages (calculated on a varying interval, so you might consider MA5 for five-day average, or MA7 for a week average) and their first and maybe second derivatives (if you feel fancy you can experiment with third derivatives). If you plot all these lines, suitably smoothed, over your data, you will notice that the salient points of some of them will roughly intersect near the "breakpoints". Those are the parameters that your brain considers when you "see" the breakpoints; it is why you see there the breakpoints, and not somewhere else.
Another method that the human vision employs to recognize features is trimming outliers: you discard in the above calculations either any value outside a given tolerance, or a fixed percentage of all values starting from those farther from the average. You can also experiment not trimming those values that are outliers for longer-period moving averages but are not for shorter periods (this gives better responsivity but will find more "breakpoints"). Then you run the same calculations on the remaining data.
Finally you can attribute a "breakpoint score" based on weighing the distance from nearby salient points. Then, choose a desired breakpoint distance and call "breakpoint" the highest scoring point in that interval; repeat for all subsequent breakpoints. Again, you may want to experiment with different intervals. This allows for a conveniently paced breakpoint set.
And, finally, you will probably notice that different kinds of signal sources have different "best" breakpoint parameters, so there is no one "One-Size-Fits-All" parameter set.
If you're building an interface to display data, leaving the control of these parameters to the user might be a good idea.
can someone please tell me the benefits of using UintArrays and also why 8,16, and 32 bit arrays are used?
As if we just want to store 1's and 0's then we can just use the 8 bit one whats the use of others.
Also can you provide a short description of the flow of GROUP functionality in crossfilter.
Also is there any source where i can easily understand the entire flow of crossfilter's internal code.
Many thanks.
IIRC, the Uint arrays are just used for filters. The width of the array is determined by the number of dimensions you have and each bit of each row of the array is flipped to 1 if the record at that position in the array is excluded by the filter on the dimension corresponding to the bit position. The fact that the width of the array needs to be equal to or greater than the number of dimensions is the source of Crossfilter's 32-dimension cap. There are a few advantages, again, if I remember correctly:
The array takes up a lot less space in memory
Applying a new filter becomes a matter of bit operations and doesn't require updating the full integer
Checking if a given record should be displayed under the current filters is just a matter of comparing the filter array to 0 at the index corresponding to the record
My in-browser LED Display object has LED objects stored in it's map property. Each LED has x and y properties so as to know where it is on canvas, while map is a 2D array of the LEDs (which can be of different sizes) that maps them on the Display matrix. The display object also has properties that define the LEDs' area and the space between them.
The display should be able to show characters, such as numbers and letters, from a pre-made font.
What I can't get my head around is: what is the best way to display characters? For example, here is an example of how the letter 'S' might look like (. is a LED display turned off, O is a LED display turned on:
..OO..
.OOOO.
OO..OO
OO....
.OO...
..OO..
...OO.
OO.OO.
.OOO..
..OO..
How should I map which LEDs are to be switched on?
My current solution is: create a Character object that has the properties x and y (which determine it's place on the display matrix) and width and height (which determine it's area in LEDs). The LEDs that are supposed to be on are then mapped in a 2D array. When we need to show a character, we get the Character's x and y, find the 'source' LED and then switch the LEDs on as mapped.
Is there a better way? Using paths, perhaps?
I'm not entirely sure from your description how you're doing it, but here's how I'd do it based on what I think you're after.
First step, get all your characters worked out in arrays defining which points need leds on or off. If this is going to be a big set you could try some techniques to compress the data.
Depending on the size of your LEDs on the canvas I'd either create a function that calculates which leds need changing and only wipe the display where there are LEDs that need changing. Pre-render the LEDs (on and off state) once, like a display list in openGL (see link below).
Otherwise if the characters themselves are quite small on the screen and there are lots of them I would pre-render each entire character and use like a font. This is would require much more clearing of the screen, but if there are hundreds of LEDs the overhead of calculation and display clearing might be high.
This page has some good information about efficient canvas rendering:
http://www.html5rocks.com/en/tutorials/canvas/performance/
I've assumed the LED's are rendered in the program and are not just images? If they're just images then you have effectively pre-rendered the LED already.
From what I understood, you are trying to have a "pixelized" led display.
So basically, storing your characters as array of pixels seems the best way to define them... I don't really see what is your problem.
If your problem is about getting the pixel map, one thing you could do is get a pixel like font, draw a character on an offscreen canvas, get the pixel datas and then use them to create a pixel map of your character. Then you can do a loop to map all the characters without having to creating every map by yourself...