Bellman's equation's loss in TFjs - javascript

I am trying to implement a simple Pong Q-Network in JS using p5 and tfjs.
To train the network, I first created a custom loss function where I passed a label tensor only containing the labels for the actions a_t
TFjs didn't really seemed to like the fact that the shape was different from the predictions from the model, so I crafted another custom loss function where the label input is a tensor of shape [batchSize, 3] (3 actions up, down and none) where each element is in the form [0,0,y_j] or [0,y_j,0] or [y_j,0,0] (y_j at the place I am supposed to compare to the predictions tensor and 0 elsewhere). Here it is:
function bellmanLoss(predictions, labels)
{
let predictions_buffer = predictions.buffer();
let labels_buffer = labels.buffer();
let length = labels.shape[0];
predictions.dispose();
labels.dispose();
let loss = 0;
for(let i = 0; i < length; i++)
{
for(let j = 0; j < 3; j++)
{
if(labels_buffer.get(i,j) != 0)
{
loss += Math.pow(labels_buffer.get(i,j) - predictions_buffer.get(i,j), 2);
break;
}
}
}
return tf.tensor(loss);
}
But here I am crafting a tensor from scratch and so I get this error using model.fit: "Error: Cannot find a connection between any variable and the result of the loss function y=f(x). Please make sure the operations that use variables are inside the function f passed to minimize()".
Is there a way to have a loss compatible with model.fit or do I have to tune the model weights manually (which would be a pain)?
Update : I've made some changes to be more "tensory" and it seem to go in the right direction:
function bellmanLoss(preds, labels)
{
let mask = tf.cast(labels, 'bool');
let zeros = tf.zerosLike(preds);
let clean_preds = preds.where(mask, zeros);
return tf.squaredDifference(clean_preds, labels).mean();
}
However, I still need to find another alternative because
"Error: Cannot compute gradient: gradient function not found for where"
Final update: I've found a way without using tf.where and it works
function bellmanLoss(preds, labels)
{
let mask_b = tf.cast(labels, 'bool');
let mask = tf.cast(mask_b, 'float32');
let clean_preds = preds.mul(mask);
return tf.squaredDifference(clean_preds, labels).mean();
}

Related

How can I get the particular point in a THREE.Points object to change his position?

I work on a personal project to try out equations to try to simulate the behavior of a galaxy. I have so far managed to place the Points as I wanted, but now I want to take each point individually to change its position.
The goal for now is just to successfully try to apply a Random Vector to each of the points.
I tried:
var direction = new THREE.Vector3(0.00003, 0.000005, 0);
points.position.add(direction);
but this applies to all Points.
Then I tried something like that:
for (let i = 0; i < points.geometry.attributes.position.count; i++) {
points.geometry.attributes.position[i] = Math.random() * 500
}
points.geometry.attributes.position.needsUpdate = true;
But nothing append :( I thing I missed something but I dind't know what
Here the full code on codepen:
Codepen
When you access:
points.geometry.attributes.position[i]
you're not getting the array of the vertex positions. You're getting the BufferAttribute. What you probably want is the array inside the BufferAttribute:
points.geometry.attributes.position.array[i]
However, this is still not the recommended approach. Three.js recommends you use the .getAttribute() method:
// Get the attribute
const posAttribute = points.geometry.getAttribute("position");
// Get the array inside the attribute
const posArray = posAttribute.array;
// Increment by 3 at a time to access XYZ separately
for(let i3 = 0; i3 < posArray.length; i3 += 3) {
posArray[i3 + 0] = xPosition;
posArray[i3 + 1] = yPosition;
posArray[i3 + 2] = zPosition;
}
// Tell the attribute it needs updatin
posAttribute.needsUpdate = true;

Iterate over a Range fast in Excelscript for web

I want to check that a range of cell are empty or has any values in them, I use this for loop :
for (let i = 0; i <= namesRange.getCellCount(); i++) {
if (namesRange.getCell(i,0).getText() == "")
{
break;
}
bookedCount += 1;
}
However this iteration is extremely slow (as is the use of Range.getValue, but the console warns you that iterating with .getValue is slow, does not warn you with getText) It takes several seconds to iterate over a very short list of 10 elements.
Is there any way to check for the values of a cell in a speedy manner using ExcelScripts?
Does this mean that, even if I develop a UDF or a ribbon Add-In with office.js and Node.js it will also be this extremely slow for iterating over cells?
Is there any way to make this faster?
The reason your code is likely performing slowly is that the calls to getCell() and getText() are expensive. Instead of performing these calls every time in the loop you can try a different approach. One approach is to get an array of the cell values and iterate over that. You can use your namesRange variable to get the array of values. And you can also use it to get the row count and the column count for the range. Using this information, you should be able to write nested for loops to iterate over the array. Here's an example of how you might do that:
function main(workbook: ExcelScript.Workbook) {
let namesRange: ExcelScript.Range = workbook.getActiveWorksheet().getRange("A1");
let rowCount: number = namesRange.getRowCount();
let colCount: number = namesRange.getColumnCount();
let vals: string[][] = namesRange.getValues() as string[][];
for (let i = 0; i < rowCount; i++) {
for (let j = 0; j < colCount; j++) {
if (vals[i][j] == "") {
//additional code here
}
}
}
}
Another alternative to the first answer is to use the forEach approach for every cell in the range of values.
It can cut down the amount of variables you need to achieve the desired result.
function main(workbook: ExcelScript.Workbook)
{
let worksheet = workbook.getActiveWorksheet();
let usedRange = worksheet.getUsedRange().getValues();
usedRange.forEach(row => {
row.forEach(cellValue => {
console.log(cellValue);
});
});
}

Dynamic variable declaration. Is this even the right method?

A little new to JS so be gentle :)
I'm trying to create a program that holds 5000+ boolean values that dynamically change based on other vars.
const chars = "abcdefghijklmnopqrstuvwxyz0";
const charsC = "ABCDEFGHIJKLMNOPQRSTUVWXYZ0"
const maxNum = 48;
const maxTile = 6;
var tile1, tile2, tile3, tile4, tile5, tile6
// test vars
var tile4 = "A27"
var t4a27 = false
// this snippet will be in an interval loop
for (let i = 1; i <= maxTile; ++i) {
for (let n = 0; n < chars.length; ++n) {
for (let j = 1; j <= maxNum; ++j) {
// this obviously doesnt work
var t[i][`${chars[n]}`][j];
// ^ ^ ^
if (tile[i] == `${charsC[n]}${j}`) {
t[i][h][j] = true;
console.log(t4a27)
} else {
t[i][h][j] = false;
}
}
}
}
For clarification a better word than "tile" for the vars could be "sprite" rather because its a point on the sprite.
The basic concept is the tile vars are designed to output their current position as a string value e.g. "A27". Then this loop will take that information and scan each tile subset to be true/false. So if the sprite lower right quadrant is inside "A27" the output would be t4a27 = true
In practice I can do this with just a lot of code (over 20,000 lines) but I figured their has to be an easier way that requires far less code.
This is probably not the right approach for your problem.
If you really need to store this amount of variables, it is probably best to put them in an object like so:
var tiles = {}
var tileName = 'abc'
// Dynamic setting:
tile[tileName] = true
// Dynamic reading:
console.log(tile[tileName])
I am wondering if you really want to store 5000 variables or if there is another way to calculate them at the time you need time, but that requires a bit more knowledge of the problem.
Javascript doesn't have this kind of ability to reflect local variables.
What you can do is attach all those variables to a global object, and proceed with: Object.keys(your_object) and your_object[key_name_here] = ...
I think you should use a 2-dim array for this. Or use a regular array of booleans with the appropriate size and do the index-magic yourself.
As you said, you are running on coordinates. A-27 is the same as row(1)+field(27) -- considering A is 1
If your field is 500x100, you create an Array as such: let gamefield = Array(500*100);
Warning: I have not tested this for syntax errors, but you should get the idea.
let gamefield = Array(500*100);
// optional gamefield.fill(true);
let row = idx => idx * 500;
let posIdx = (r, c) => row(r) + c;
// there is a sprite with a tiles property that returns
// 4 index positions for the sprite's quadrants as [r,c]
let quadrants = sprite.tiles.reportPositions()
// filter the quadrants where the gamefield at r,c is true
// this might also be a good case for some() instead of filter()
let collisions = quadrants.filter(pos => return gamefield[posIdx(...pos)]);
// if there is any of these, you can kill the sprite.
if(collisions.length > 0) sprite.kill();

Tensorflow.js Please make sure the operations that use variables are inside the function f passed to minimize()

I have started to use javascript and TensorFlow.js to work on some machine learning projects, I am working on creating a linear regression model, however, I cannot figure out what is causing this error
Can not find a connection between any variable and the result of the loss function y=f(x). Please make sure the operations that use variables are inside the function f passed to minimize()."
I have created two Tensors
globalTensorXs = tf.tensor2d(globaArrayXs); //input data
globalTensorYs = tf.tensor1d(globaArrayYs); //y output
I have created the coefficients/weights as below as an array of tf scalars.
function createWeights(_numWeights)
{
for ( var x = 0; x < _numWeights; x++)
{
globalWeightsTensorArr.push(tf.variable(tf.scalar(Math.random())));
}
}
There is A training function which I pass the x and y Tensors into, it is the call to the optimise.minimize that causes the issue. it does not detect the variable for training, which are stored in globalWeightsTensorArr
async function train(xsTensor, ysTensor, numIterations)
{
/*//////OPTIMISER.MINIMISE/////////////
Minimize takes a function that does two things:
It predicts y values for all the x values using the predict
model function.
It returns the mean squared error loss for those predictions
using the loss function. Minimize then automatically adjusts any
Variables used by thi predict/loss function in order to minimize
the return value (our loss), in this case the variables are in
"globalWeightsTensorArr" which contains the coefficient values
to be altered by the modeld during "numIterations" iterations of
SGD.
*/
for (let iter = 0; iter < numIterations; iter++)
{
optimiser.minimize(function ()
{
return loss(predict(xsTensor), ysTensor);
}, globalWeightsTensorArr);
}
}
// the predict and loss function are here...
//The following code constructs a predict function that takes inputs(X's) //and returns prediction Y: it represents our 'model'. Given an input //'xs' it will try and * predict the appropriate output 'y'.
function predict(_Xs)
{
return tf.tidy(() => {
for ( var x = 0; x < 8; x++)
globalWeightsArr[x] = globalWeightsTensorArr[x].dataSync();
const weightTensor = tf.tensor1d(globalWeightsArr);
const prediction = tf.dot(_Xs, weightTensor);
return prediction;
});
}
//The loss function takes the predictions from the predict function
//and the actual lables and adjusts the weights
//the weights are considered to be any tensor variable that impact the //function We can define a MSE loss function in TensorFlow.js as follows:
function loss(_predictedTensor, _labels)
{
const meanSquareError =_predictedTensor.sub(_labels).square().mean();
return meanSquareError ;
}
can anyone please help explain the problem?
Regards
Aideen
We resolved the issue by changing the way the weights/coefficients were created. Now minimize can detect the variables used by predict, and it adjusts them accordingly. later I will post the entire solution to codepen. still learning!
function createWeights(_numWeights) {
const randomTensor = tf.randomUniform([_numWeights, 1]);
globalWeightsTensorVar = tf.variable(randomTensor);
}
here is the predict function used b
function predictLogical(_Xs) {
return tf.dot(_Xs, globalWeightsTensorVar);
}
The issue is related to tf.variable. One needs to use tf.variable to create the weights that will be updated by the function created by optimiser.minimize().
A variable created by tf.variable is mutable contrary to tf.tensor that is immutable. As a result if one uses tf.tensor to create the weights they could not be updated during the training

Error in tensorflow.js multiple regression sgd minimize

this is my first post. I am facing an error with fitting a curve in tensorflow.js which I can't seem to fix. I have spent two days on it so far. Since tensorflow.js is pretty new, there's not a whole lot of answers to this sort of question out there, so I'm sure many people are interested in this. I have tried to replicate the example from the tensorflow.js project's website:
https://js.tensorflow.org/tutorials/fit-curve.html .
The difference is that I am using multiple predictors to predict an outcome variable. I have 20 prices, and I am using the previous 4 prices to predict the fifth one. So I start out with price number 5 and go up to price number 20 where price 5 is predicted by price 1 to 4 and so forth in a weighted time-series prediction model. I am using a multiple linear regression framework where I set up 4 random parameters (one weight for each of the four previous prices). My goal is to train the variable to minimize my loss function (using minimum least square criterion). I have tried following the example from the link as closely as possible. Whenever I run my code I get:
Error: The f passed in variableGrads(f) must be a function
which is generated by the call of .minimize in line 59 (right before return in the train function at the end). Basicially what I'm doing is fit a linear regression which could be more easily done in R but we aim at very large data sets and more complex machine learning procedures. I'm sure this is interesting to a lot of other people who are getting started woth tensorflow.js.
here's my code with some comments:
const tf = require('#tensorflow/tfjs');
require('#tensorflow/tfjs-node');
module.exports = function tensorFlow() {
//the trainable variable with initial random numbers
let lag = tf.variable(tf.tensor([Math.random(), Math.random(), Math.random(), Math.random()], [4]));
//20 observed prices
let priceData = [21.00397, 21.29068, 22.80492, 23.40646, 24.06598, 23.89722, 25.40211, 24.63436, 25.83449, 26.44832, 26.25194, 27.34009, 27.90455, 27.14175, 28.12549, 29.99411, 30.43631, 30.39753, 30.16104, 31.14931];
//the prices from price 5 on that are to be predicted
let toBePredictedList = [24.06598, 23.89722, 25.40211, 24.63436, 25.83449, 26.44832, 26.25194, 27.34009, 27.90455, 27.14175, 28.12549, 29.99411, 30.43631, 30.39753, 30.16104, 31.14931];
//set up tensor of labels to compare predictions with
let toBePredicted = tf.tensor(toBePredictedList, [16]);
//a list of predictors with 16 rows and four columns for 16 predictions to be made using 4 previous prices each
let predictorsList = [];
for (let predictorIndex = 0; predictorIndex < 16; predictorIndex++) {
for (let predictionsIndex = 0; predictionsIndex < 4; predictionsIndex++) {
predictorsList.push(priceData[predictorIndex + predictionsIndex]);
}
}
//make it a tensor
let predictors = tf.tensor(predictorsList, [16, 4]);
//predict multiplies all predictors in all lines with the parameters from lag to be trained and adds up the four elements to generate an estimate of the fifth price
function predict(predictors) {
function modelMaker() {
let modelList = [];
for (let rowIndex = 0; rowIndex < 16; rowIndex++) {
let prediction = 0;
for (let colIndex = 0; colIndex < 4; colIndex++) {
prediction += lag.get(colIndex) * predictors.get(rowIndex, colIndex);
console.log({prediction});
}
modelList.push(prediction);
}
return tf.tensor(modelList, [16]);
}
return tf.tidy(modelMaker);
}
//means square error of my prediction when compared to actual outcome price
function loss(predictions, toBePredicted) {
return tf.losses.meanSquaredError(toBePredicted, predictions);
}
function train(predictors, toBePredicted, numIterations) {
function computeLoss (predictors, toBePredicted) {
let predictions = predict(predictors);
return loss(predictions, toBePredicted);
}
let learningRate = 0.5; //suggested by Google Developers
const OPTIMIZER = tf.train.sgd(learningRate); //suggested by Google Developers
for (let iter = 0; iter < numIterations; iter++) {
OPTIMIZER.minimize(computeLoss(predictors, toBePredicted));
}
return {
a: lag.get(0),
b: lag.get(1),
c: lag.get(2),
d: lag.get(3)
};
};
//75 suggested by google developers
return train(predictors, toBePredicted, 75);
};
The problem is with minimze in the end as I said. The above code works fine and computes everything it is supposed to.
Thanks for any suggestions!
Chris
optimizer.minimize() updates the weights during each cycle of training. For the weights to be updated, they need to be created using tf.variable. A variable created using tf.variable is mutable whereas tf.tensor creates immutable variable.
It is also noteworthy to point out that the predict() should return a function whose coefficient are created using tf.variable that will be updated to minimize the loss function.

Categories