I was wondering if it were possible to implement some kind of crude JavaScript anti-flood protection.
My code receives events from a server through AJAX, but sometimes these events can be quite frequent (they're not governed by me).
I have attempted to come up with a method of combating this, and I've written a small script: http://jsfiddle.net/Ry5k9/
var puts = {};
function receiverFunction(id, text) {
if ( !puts[id] ) {
puts = {};
puts[id] = {};
}
puts[id].start = puts[id].start || new Date();
var count = puts[id].count = puts[id].count + 1 || 0;
var time = (new Date() - puts[id].start) * 0.001;
$("text").set("text", (count / time.toFixed()).toString() + " lines/second");
doSomethingWithTextIfNotSpam(text);
}
};
which I think could prove effective against these kinds of attacks, but I'm wondering if it can be improved or perhaps rewritten?
So far, I think everything more than 3 or 2.5 lines per second seems like spam, but as time progresses forward (because start mark was set... well... at the start), an offender could simply idle for a while and then commence the flood, effectively never passing 1 line per minute.
Also, I would like to add that I use Mootools and Lo-Dash libraries (maybe they provide some interesting methods), but it would be preferable if this can be done using native JS.
Any insight is greatly appreciated!
If you are concerned about the frequency a particular javascript function fires, you could debounce the function.
In your example, I guess it would be something like:
onSuccess: function(){ _.debounce(someOtherFunction, timeOut)};
where timeout is the maximum frequency you want someOtherFunction to be called.
I know you asked about native JavaScript, but maybe take a look at RxJS.
RxJS or Reactive Extensions for JavaScript is a library for
transforming, composing, and querying streams of data. We mean all
kinds of data too, from simple arrays of values, to series of events
(unfortunate or otherwise), to complex flows of data.
There is an example on that page which uses the throttle method to "Ignores values from an observable sequence which are followed by another value before dueTime" (see source).
keyup = Rx.Observable.fromEvent(input, 'keyup').select(function(ev) {
return ev.target.value;
}).where(function(text) {
return text.length > 2;
}).throttle(500)
.distinctUntilChanged()
There might be a similar way to get your 2.5-3 per second and ignore the rest of the events until the next second.
I've spent many days pondering on effective measures to forbid message-flooding, until I came across the solution implemented somewhere else.
First, we need three things, penalty and score variables, and a point in time where last action occured:
var score = 0;
var penalty = 200; // Penalty can be fine-tuned.
var lastact = new Date();
Next, we decrease score by the distance between the previous message and current in time.
/* The smaller the distance, more time has to pass in order
* to negate the score penalty cause{d,s}.
*/
score -= (new Date() - lastact) * 0.05;
// Score shouldn't be less than zero.
score = (score < 0) ? 0 : score;
Then we add the message penalty and check if it crosses the threshold:
if ( (score += penalty) > 1000 ) {
// Do things.
}
Shouldn't forget to update last action afterwards:
lastact = new Date();
Related
I have about 50k lines of data that need to be queried over an API service. The service limites it's API calls as follows:
per second: max 50
per minute: max 2.500
per hour: max 25.000
per day: max 50.000
per month: max 100.000
I am focusing on the limits per second and minute for now, looping through all rows and stopping if the rate limit went over:
var app = SpreadsheetApp;
var sheet = app.getActiveSpreadsheet().getSheetByName("data");
var maxRows = sheet.getLastRow();
var maxColumns = sheet.getLastColumn();
var data = sheet.getrange(2,1,maxRows,maxColumns).getValues();
var rateLimitSec = 50;
var rateLimitMin = 2500;
var rateLimitSecUsed = 0;
var rateLimitMinUsed = 0;
function example () {
for (
i=0;
(i<=maxRows) || (rateLimitSecUsed<rateLimitSec) || (rateLimitMinUsed<=RateLimitMin);
i++
)
{ // start loop code
// do the API request
rateLimitSecUsed = rateLimitSecUsed+1 // and increase Rate limit used marker
rateLimitMinUsed = rateLimitMinUsed+1
if ( !(rateLimitMinUsed < RateLimitMin) ) {
// stop for now, or add better solution later. show some warning;
} else if ( !( rateLimitSecUsed !< rateLimitSec) ) {
-->> Utilities.sleep(1000); // HERE I WANT A DELAY FOR 1S, <<---
rateLimitSec = 0; // reset limit after delay.
example(); // call function again after delay
}
} // end loop code
}; // end function example
This is not working as intended though. I read somewhere that this would not work because of asynchronicity, and that I would need to use the class lock. However, I can't figure out how the lock class works. Can anyone help guide me in the right direction on this? Is the lock class the right solution and, if yes, how does it work?
Also, is my approach for keeping the rate limits in check good or are there better ways of doing this?
This should be a pretty common issue, but I can't find any decent examples on this.
EDIT: found some typo's in the example code, fixes what i found. Don't worry too much about all that, the focus of the question is "how do i add a delay between one loop and the next, to make sure I don't go over the API-limit?"
I would approach this logic in a different way - one which is much simpler, and which avoids that recursive call you currently are using (calling example() from within example()).
Let's assume you have between 25,000 and 50,000 rows of data to process. That means you need to stay within the "50,000 per day" limit - but you don't need to spread these call out across the entire 24 hour period. You can spread them across 2 hours, to avoid the "25,000 per hour" limit.
That is: up to 25,000 API calls within 7,200 seconds, or no more than 1 call every 0.144 seconds. Let's call that our "throttle value". We can even increase that number slightly to ensure we are safely within the allowed limit: 1 call every 0.2 seconds, which is 18,000 calls per hour (or whatever safety margin works for you).
In that case, we can use code like this:
var maxRows = 48987; // calculated from the spreadsheet
var throttle = 200; // millis
function example() {
for (i = 1; i <= maxRows; i++) {
// do your API call here
Utilities.sleep(throttle);
}
}
Now, if you have between 50,000 and 100,000 lines of data to handle, that requires a different "throttle value" calculation.
50,000 calls per day = 50,000 calls per 86,400 seconds = 1 call every 1,728 milliseconds (again, you can adjust for safety - let's say 1,800 millis.
You can modify the above code to account for this:
var maxRows = ...; // calculated from the spreadsheet
var throttle;
if (maxRows <= 100000 && maxRows > 50000) {
throttle = 1800;
} else if (maxRows <= 50000 && maxRows > 25000) {
throttle = 200;
} // and more else/if statements as needed for different row counts...
function example() {
for (i = 1; i <= maxRows; i++) {
// do your API call here
Utilities.sleep(throttle);
}
}
You can certainly improve on this by using a switch statement for more throttle values, and by using variables instead of those hard-coded numbers - but this gives you the idea.
This is deliberately not a throughput-optimized solution. It trades optimization for simplicity.
You could get better throughput by re-evaluating the "left-over" row count after processing an initial chunk of data. For example, if you have 50,123 rows of data then you could use one throttle value for the first 50,000 rows, and then a much better throttle value for the remaining 123 rows.
It also does not account for the time of day at which your process starts (I don't know what constitutes a "day" here - probably midnight UTC to midnight UTC.) But there is the potential for further optimizations based on the starting time, also.
I would keep it simple.
In examples I see two different ways how to handle animations using requestAnimationFrame():
The first one using setTimeout()
const e = document.getElementById('e');
let count = 0;
function move(timestamp) {
e.style.left = ++count + 'px';
setTimeout(f=>{
requestAnimationFrame(move);
}, 200);
};
requestAnimationFrame(move);
Try it in this jsfiddle.
The second one by calulating the ellapsed time yourself
const e = document.getElementById('e');
let count = 0;
let past = null;
function move(timestamp) {
if (past !== null && timestamp - past < 200) {
requestAnimationFrame(move);
return;
}
past = timestamp;
e.style.left = ++count + 'px';
requestAnimationFrame(move);
};
requestAnimationFrame(move);
Try it in this jsfiddle.
Now my question is: Which one performs better?
My guess is, that if an interval is applicable, it will perform better. There aren't so many logical expressions to evaluate and so many calculations to do. But that's just a guess. So, are their any benefits of one way over the other?
You should do the first one. However you shouldn't assume it will be exactly 200ms. If you need to know exactly how many milliseconds it's been, you must calculate it every time like in your second example, but keep using the structure of your first example.
The problem with the second example is that it calculates the new frames just as often as the first example, but it runs the function just to check that it doesn't need to run the function every other time it runs. Lots of wasted computation there.
You need to calculate the time elapsed (if you need accuracy) because the setTimeout combined with the requestAnimationFrame will not be guaranteed to run in 200ms. The requestAnimationFrame in particular will add time onto that if it feels it needs to before it's ready to give you permission to animate (which is basically what requestAnimationFrame is -- it's giving you permission to animate, saying 'I'm ready now').
I am working on a very time-sensitive application that uses key presses for user input. As I am talking milliseconds here, I went ahead and tried a version like this:
function start() {
//stim.style.display = "block";
rt_start = new Date().getTime();
response_allowed = 1;
}
function end() {
var t = rt_end - rt_start;
//stim.style.display = "none";
log.innerHTML = t;
i++;
if (i < iterations) {
setTimeout('start();', 1000);
}
}
var rt_start;
var rt_end;
var iterations = 100;
var i = 0;
var response_allowed = 0;
var stim;
var log;
$(document).ready(function() {
document.onkeydown = function(e) {
if (response_allowed == 1) {
rt_end = new Date().getTime();
response_allowed = 0;
end();
}
};
stim = document.getElementById('stim');
log = document.getElementById('log');
start();
});
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
<div id="log"></div>
<img src="https://www.gravatar.com/avatar/cfefd93404e6b0eb3cde02b4b6df4e2b?s=128&d=identicon&r=PG&f=1" id="stim" />
And it works fine, usually sub-5ms timers (just holding down a key). But as soon as I modify the code to display the image (uncommenting the two lines), this slows down a lot to about 30ms.
Can someone point me into the direction why exactly this is the case and how to possibly avoid this additional delay?
Thanks
I would recommend using a DOMHighResTimeStamp where available (with a polyfill for browsers that don't provide it).
It's a high-resolution timestamp (designed with accurate measurement in mind) to be used (e.g.) with the Navigation Timing and Web Performance APIs (search for this in the Mozilla Developer Network, as I can't share more than two links within a single post).
The quick way to get a DOMHighResTimeStamp - much like you do with var ts = new Date().getTime(); to get a regular millisecond timestamp - is:
var ts = performance.now();
As I said above, take a look at the Web Performance API at MDN. It will be very helpful if your application is really time-sensitive.
EDIT:
About your snippet, it seems to me that if you hold a key down, you will be always limited to the resolution of the keydown event (which fires continuously, but not every milissecond). You can easily see this behavior if you press a character key down (continuously) with a text editor and check for how many times per second the character is written. This, I guess, is controlled via an OS setting.
You are also limited to the "drift" associated with setTimeout/setInterval. You see, setTimeout queues something for execution after a given delay, but it does not guarantee timely execution. It's a "best effort" scenario and, if the browser is busy doing something, it will drift significantly. Meaning: if you use a setTimeout to re-enable a response_allowed variable after 1 second, you can expect it to re-enable it after "about" (but not exactly) 1 second.
First of all I want to mention two things,
One: My code isn't perfect (esspechially the eval parts) - but I wanted to try something for my self, and see if I could duplicate the jQuery Animation function, so please forgive my "bad" practices, and please don't suggest that I'll use jQuery, I wanted to experiment.
Two: This code isn't done yet, and I just wanted to figure out what makes it work badly.
So the animation runs for about 12 seconds while the duration parameter I entered was 15 seconds, What am I doing wrong?
function animate(elem, attr, duration){
if(attr.constructor === Object){//check for object literal
var i = 0;
var cssProp = [];
var cssValue = [];
for(key in attr) {
cssProp[i] = key;
cssValue[i] = attr[key];
}
var fps = (1000 / 60);
var t = setInterval(function(){
for(var j=0;j<cssProp.length;j++){
if(document.getElementById(elem).style[cssProp[j]].length == 0){
//asign basic value in css if the object dosn't have one.
document.getElementById(elem).style[cssProp[j]]= 0;
}
var c = document.getElementById(elem).style[cssProp[j]];
//console.log(str +" | "+c+"|"+cssValue[j]);
if(c > cssValue[j]){
document.getElementById(elem).style[cssProp[j]] -= 1/((duration/fps)*(c-cssValue[j]));
}else if(c < cssValue[j]){
document.getElementById(elem).style[cssProp[j]] += 1/((duration/fps)*(c-cssValue[j]));
}else if(c == cssValue[j]){
window.clearInterval(t);
}
}
},fps);
}
}
animate('hello',{opacity:0},15000);
html:
<p id="hello" style="opacity:1;">Hello World</p>
Note: I guess there is a problem with the
(duration/fps)*(c-cssValue[j])
Part or/and the interval of the setInterval (fps variable).
Thanks in advance.
I'm not gonna try and refactor that and figure it out, cause it's pretty wonky. That said... a few things.
Don't rely on the value you are animating to let you know animation progress
In general your approach is unsound. You are better off keeping track of progress yourself. Also, as a result of your approach your math seems like it's trying too hard, and should be much simpler.
Think of it like this: your animation is complete when the time has elapsed, not when the animated value seems to indicate that it's at the final position.
Don't increment, set
Floating point math is inexact, and repeated addition cumulation like this is going accumulate floating point errors as well. And it's far more readable to make some variables to keep track of progress for you, which you can use in calculations.
animatedValue += changeOnThisFrame // BAD!
animatedValue = valueOnThisFrame // GOOD!
Don't do the positive/negative conditional dance
It turns out that 10 + 10 and 10 - (-10) is really the same thing. Which means you can always add the values, but the rate of change can be negative or positive, and the value will animate in the appropriate direction.
timeouts and intervals aren't exact
Turns out setTimeout(fn, 50) actually means to schedule the fn to be call at least 50ms later. The next JS run loop to execute after those 50ms will run the function, so you can't rely on it to be perfectly accurate.
That said it's usually within a few milliseconds. But 60fps is about 16ms for frame, and that timer may actually fire in a variable amount of time from 16-22ms. So when you do calculations based on frame rate, it's not matching the actual time elapsed closely at all.
Refactor complex math
Deconstructing this line here is gonna be hard.
document.getElementById(elem).style[cssProp[j]] -= 1/((duration/fps)*(c-cssValue[j]));
Why for more complex break it up so you can easily understand what's going on here. refactoring this line alone, I might do this:
var style = document.getElementById(elem).style;
var changeThisFrame = duration/fps;
var someOddCalculatedValue = c-cssValue[j];
style[cssProp[j]] -= 1 / (changeThisFrame * someOddCalculatedValue);
Doing this makes it clearer what each expression in your math means and what it's for. And because you didn't do it here, I had a very hard time wondering why c-cssValue[j] was in there and what it represents.
Simple Example
This is less capable than what you have, but it shows the approach you should be taking. It uses the animation start time to create the perfect value, depending on how complete the animation should be, where it started, and where it's going. It doesn't use the current animated value to determine anything, and is guaranteed to run the full length of the animation.
var anim = function(elem, duration) {
// save when we started for calculating progress
var startedAt = Date.now();
// set animation bounds
var startValue = 10;
var endValue = 200;
// figure out how much change we have over the whole animation
var delta = endValue - startValue;
// Animation function, to run at 60 fps.
var t = setInterval(function(){
// How far are we into the animation, on a scale of 0 to 1.
var progress = (Date.now() - startedAt) / duration;
// If we passed 1, the animation is over so clean up.
if (progress > 1) {
alert('DONE! Elapsed: ' + (Date.now() - startedAt) + 'ms');
clearInterval(t);
}
// Set the real value.
elem.style.top = startValue + (progress * delta) + "px";
}, 1000 / 60);
};
anim(document.getElementById('foo'), 5000);
JSFiddle: http://jsfiddle.net/DSRst/
You cannot use setInterval for accurate total timing. Because JS is single threaded and multiple things compete for cycles on the one thread, there is no guarantee that the next interval call will be exactly on time or that N intervals will consume the exact duration of time.
Instead, pretty much all animation routines get the current time and use the system clock to measure time for the total duration. The general algorithm is to get the start time, calculate a desired finish time (starttime + duration). Then, as you've done, calculate the expected step value and number of iterations. Then, upon each step, you recalculate the remaining time left and the remaining step value. In this way, you ensure that the animation always finishes exactly (or nearly exactly) on time and that you always get exactly to the final position. If the animation gets behind the ideal trajectory, then it will self correct and move slightly more for the remaining steps. If it gets ahead for any reason (rounding errors, etc...), it will dial back the step size and likewise arrive at the final position on time.
You may also need to know that browsers don't always support very small timing amounts. Each browser has some sort of minimum time that they will allow for a timer operation. Here's an article on minimum timer levels.
Here's an article on tweening (the process of continually recalculating the step to fit the duration exactly).
I'd also suggest that you look at the code for doing animation in some libraries (jQuery, YUI or any other one you find) as they can all show you how this is done in a very general purpose way, including tweening, easing functions, etc...
What’s the best way to get monotonically increasing time in JavaScript? I’m hoping for something like Java’s System.nanoTime().
Date() obviously won’t work, as it’s affected by system time changes.
In other words, what I would like is for a <= b, always:
a = myIncreasingTime.getMilliseconds();
...
// some time later, maybe seconds, maybe days
b = myIncreasingTime.getMilliseconds();
At best, even when using the UTC functions in Date(), it will return what it believes is the correct time, but if someone sets the time backward, the next call to Date() can return a lesser value. System.nanoTime() does not suffer from this limitation (at least not until the system is rebooted).
Modification: [2012-02-26: not intended to affect the original question, which has a bounty]
I am not interested knowing the “wall time”, I’m interested in knowing elapsed time with some accuracy, which Date() cannot possibly provide.
You could use window.performance.now() - since Firefox 15, and window.performance.webkitNow() - Chrome 20]
var a = window.performance.now();
//...
var delay = window.performance.now() - a;
You could wrap Date() or Date.now() so as to force it to be monotonic (but inaccurate). Sketch, untested:
var offset = 0;
var seen = 0;
function time() {
var t = Date.now();
if (t < seen) {
offset += (seen - t);
}
seen = t;
return t + offset;
}
If the system clock is set back at a given moment, then it will appear that no time has passed (and an elapsed time containing that interval will be incorrect), but you will at least not have negative deltas. If there are no set-backs then this returns the same value as Date.now().
This might be a suitable solution if you're writing a game simulation loop, for example, where time() is called extremely frequently — the maximum error is the number of set-backs times the interval between calls. If your application doesn't naturally do that, you could explicitly call it on a setInterval, say (assuming that isn't hosed by the system clock), to keep your accuracy at the cost of some CPU time.
It is also possible that the clock will be set forward, which does not prevent monotonicity but might have equally undesirable effects (e.g. a game spending too long trying to catch up its simulation at once). However, this is not especially distinguishable from the machine having been asleep for some time. If such a protection is desired, it just means changing the condition next to the existing one, with a constant threshold for acceptable progress:
if (t > seen + leapForwardMaximum) {
offset += (seen - t) + leapForwardMaximum;
}
I would suggest that leapForwardMaximum should be set to more than 1000 ms because, for example, Chrome (if I recall correctly) throttles timers in background tabs to fire not more than once per second.
Javascript itself does not have any functionality to access the nanoTime. You might load a java-applet to aqcuire that information, like benchmark.js has done. Maybe #mathias can shed some light on what they did there…
Firefox provides "delay" argument for setTimeout...
this is the one of ways to implement monotonically increased time counter.
var time = 0;
setTimeout(function x(actualLateness) {
setTimeout(x, 0);
time += actualLateness;
}, 0);