We Keep Coding

Prevent malicious users from abusing and spamming unauthenticated open APIs

Here's a security problem I've encountered a couple of times when building small web-based projects interacting with a REST API service. For example, let's say you're building a casual JavaScript-based game where you want a leaderboard of highscores, so you need to post the scores of users to a database.
The easiest solution would be to build a simple web service, e.g. using PHP, Node.js or Python, that accepts GET request and saves the results to a database. Let's imagine the API looks something like this:
GET https://www.example.com/api/highscore?name=SuperGoat31&score=500
Creating such an API for posting highscores has some obvious drawbacks. A malicious user could write a three-line piece of PHP code to spam the database full of false results, for example:
for ($i = 0; $i < 100; i++) {
file_get_contents("https://www.example.com/api/highscore?name=SuperGoat31&score=5000000");
}
So, I'm looking for a way to prevent that. This mostly relates to small hobby or hackathon projects that just need some kind of protection that will prevent the most obvious of attacks, not large enterprise applications that need strict security. A couple of things I could think of:
1. Some form of authentication
An obvious way to solve this would be to have user accounts and only allow requests from logged-in users. This unfortunately has the drawback of putting up a large barrier for users, who need to get an account first. It would also require building a whole authentication workflow with password recovery and properly encrypting passwords and the like.
2. One-time token based protection
Generate a token on the server side and serve that to the user on first load, then only allow requests that serve that specific token. Simple enough, but also very easy to circumvent by finding the requests in a browser web inspector and using that for the three-line PHP script.
3. Log IP address's and ban when malicious use happens
This could work, but I feel it's not very privacy friendly. Also, logging IP addresses would require GDPR consent from users in Europe. Also doesn't prevent the actual spamming itself so you might to first clean up the mess before you start banning IP addresses.
4. Use an external service
There are services that provide solutions to this problem. For example, in the past I've used Google's reCAPTCHA to prevent malicious use. But that also means integrating an external service, making sure you keep it up to date, concerns about the privacy aspects (esp. regarding a service like reCAPTCHA), etc. It feels a bit much for a weekend project.
5. Throttle requests
I feel this is probably the easiest solution that actually works for a bit. This does require some form of IP address logging (which might give the problems stated in 3), but at least you can delete those IP addresses pretty quickly afterwards.
But I'm sure there are other methods I've missed, so I would be curious to see other ways of tackling this problem.

Taking into account all mentioned limitations, I would recommend using a combination of methods:
Simple session authentication based on one-time token
Script obfuscation
Request encryption with integrity control
Example:
let req_obj = {
user: 'SuperGoat31',
score: 123456,
sessionId: '4d2NhIgMWDuzarfAY0qT3g8U2ax4HCo7',
};
req_obj.hash = someCustomHashFunc(JSON.stringify(req_obj));
// now, req_obj.hash = "y0UXBY0rYkxMrJJPdoSgypd"
let req_string = "https://www.example.com/api/cmd?name=" +
req_obj.user +
"&data=" +
Buffer.from(JSON.stringify(req_obj)).toString('base64');
// now, your requests will look like that:
"https://www.example.com/api/cmd?name=SuperGoat31&data=eyJ1c2VyIjoiU3VwZXJHb2F0MzEiLCJzY29yZSI6MTIzNDU2LCJzZXNzaW9uSWQiOiI0ZDJOaElnTVdEdXphcmZBWTBxVDNnOFUyYXg0SENvNyIsImhhc2giOiJ5MFVYQlkwcllreE1ySkpQZG9TZ3lwZCJ9"
For casual players, this allows start playing very quickly, as no explicit registration is required. Upon generation, token might be saved as cookie for repetitive use, but this is not necessary, single-time use would also suffice. No personal info gathered.
However, if short-term storage of some client information is an option, the token might be not just some random bytes, but an encrypted string, containing some parameters, such as random salt + IP address + nickname + agent id + etc. In this case you may start silently ignore certain requests from fraudulent clients upon detection.
Obviously, this would be very easy to crack for a professional, but this is not our goal. When such simple methods are mixed with several kilobytes of logic of the game and obfuscated, figuring out how to deal with it would require significant amount of knowledge and time, which might serve as a sufficient barrier.
As it is all about balance between convenience and protection, you may implement some additional scoring logic to detect cheating attempts, like final score cannot end with '0', or cannot be even, etc. This would allow you to count cheating attempts (in addition to counting forged requests) and then estimate efficiency of implemented combination of methods.

Your list of solutions are mostly mitigations, and they are good ideas if they are your only tools. The list seems pretty exhaustive.
2 major ways to actually solve this problem are:
Remove the incentive of cheating. There's no point submitting a fake score if you are the only person who can see the score. Think about the purpose of why you even want a global high-score list. Maybe there's another way you can reach your objective that makes it uninteresting (or undesirable) to cheat.
Have the server completely manage (or duplicate) the game state. You can't cheat if the server calculates the score. For example, if you're modelling a chess game the server can compute every valid move, preventing clients from submitting moves that wouldn't be possible.
It's possible that for your specific case neither are possible, but if you can't adopt either of these strategies you are stuck to imperfect detection mechanisms.

I suspect that a perfect solution will be elusive because two of
your wishes are, perhaps, contradictory:
"You need to post the scores of users to a database" but... "prevent
the most obvious of attacks" without "Some form of authentication."
The most obvious of attacks are those from users without some form
of authentication.
You wish this system to work without placing an undue burden on
your users. You wish to avoid the usual login and password
authentication which can be cumbersome for users.
I think there is a way to accomplish what you want by creating a
very simple form of authentication by the use of a one-time token
based protection. And I would also incorporate IP tracking against
abuse. In other words, let's combine your options 1 and 2 and 3 in
the following way.
You already have implied that you will maintain a database, and that
within the database, user names will be unique (otherwise you couldn't
record unique high scores). Let people sign up freely by submitting
their requested user name, which you'll accept if not already used
by someone prior. Track the sign-up requests by IP address to detect
and prevent abuse: too many sign-ups from one IP address within a given timeframe. So far, the burden is all at the server end, not on the user.
When you process a valid sign-up (i.e. new user name) into the
database, you will also generate, record into the database, and return to the user a shared secret (a token) that will be used by the
Time-based One-time Password (TOTP) algorithm.
Don't reinvent this.
See:
Time-based One-Time Password
FreeOTP
OneTimePass
When you return a token to the user, it will be in the form of a "QR Code"
QR code
which the user will scan and store with his "Google Authenticator" or
equivalent TOTP application.
When the user returns to your web site to update his high score, he
will authenticate himself using his Google Authenticator" or
equivalent TOTP application. These are often used for "second factor"
authentication, 2FA (Multi-factor authentication), but because
of your need for less strict security, you'll be using the TOTP
authentication as the primary and only form of authentication.
So we have combined a form of authentication which doesn't place a
very high burden on the user (apps already widely available and in
use), with one-time token based protection (provided by the TOTP
app) and a little bit of IP address-based abuse protection for the initial sign-ups.
On of the weaknesses of my proposal is that a user may share his
TOTP token with another person, who may then impersonate him. But this
is no different from the risk of password sharing. And there will
be no "recover my lost password" option.

I would tackle this in a slightly different way: usernames/gamertags. Depending on how frequently you find gamertags and usernames sharing the same IP. So if you only accept a maximum of, say, 5 gamertags per IP, and you also throttle the frequency of updates per gamertag, you have a fairly spam-resistant system.

I would recommend a mix of code obfuscation and using web sockets to request the score, rather than post the score. Something like socket.io (https://socket.io/) where the server sends a request with a code in it and your game responds with the score and that code changed in some way.
Obviously a hacker could look through your code for how your game responds to requests and rewrite it, which is where the obfuscation is important, but it does at least hide the obvious network traffic and prevents them posting scores whenever they feel like it.

I would suggest using reCAPTCHA V2.
Admittedly, v3 provides better protection, but it is hard to implement, so go with v2.
Come on, it is just a few lines of code.
How it should work (according to me):
You are at the main page willing to play the game
You solve the reCAPTCHA
Then the app sends a one-time token with a script tag which establishes a websocket request with your server (using socket.io) with the one-time token and then it is destroyed immediately (from the server as well as the client) after establishment of a connection
Your server validates the token and accepts the request of websocket and then it will send the HTML content
Just create a div and set the value using obj.innerHTML
You can use styles in body (I guess)
And the most important point is obfuscating your code.
Security
Websockets are harder to reverse engineer in a test environment
Even if they create a web socket, it won't respond, because they don't know the one-time token
It prevents script blocking (as the script loads everything on the page)
It provides real-time communication
The only way out is to somehow get your hands on Google's reCAPTCHA token which is impossible, because it means going against Google
You can’t reuse any token (however immediate it be), because it was destroyed from both the sides
One more last tip: set a timeout for the one-time token to about 15 seconds
How will it help? It will prevent someone (extremely malicious) from pausing the Chrome debugger and get the token and put it in their stuff as 15 seconds is ok for slow networks also, but not a human

Google Oauth - Where to sign in users, backend/frontend

What is the difference between using Google OAuth and signing in users at the frontend of the application, and signing in users at the backend side of the application?
Example: Signing in users and getting the ID and a auth object in React or letting Node/Express handle everything, the prosess, redirecting and store it in a database.
Is there any advantages for one or the other methods, which are most common and are there any "rules"?

What is the difference between using Google OAuth and signing in users
at the frontend of the application, and signing in users at the
backend side of the application?
So to clarify to everyone reading this, signing in users at the backend side of the application is another way of saying OAuth for servers and signing in users at the frontend of the application is OAuth for JavaScript browser applications such as React.
Signing in users via OAuth for servers and browsers results in a 'token' that a server/browser can use to make requests on behalf of the user. We are asking the user to provide us access to their information via an outside service provider such as an email address, identification of some sort and it will be inside that token as well as some possible permissions that allow us to take actions on behalf of that user.
Is there any advantages for one or the other methods, which are most
common and are there any "rules"?
Server-side OAuth is usually used when we have an app that needs to access user data when they are not logged in. So if you are attempting to build an application that needs to access the users email every ten minutes and possibly delete emails on the users' behalf, you would want to be able to access their email account every 10 minutes regardless of whether they are logged into our application. So for this use case we want to make use of Oauth for server.
Otherwise, if we are not trying to do anything on behalf of the user except sign them into our app, then we can just go with OAuth for browser.

Those two things are very different. Without knowing the specifics of what you're trying to accomplish, you should know as a general rule that front-end-only authentication and authorization leaves you extremely vulnerable.
If someone knows JavaScript well, or is using editing plugins, or any of a million different things, they can bypass much front-end authorization. Or they could just watch to see what calls your app makes to the back end, then simulate those calls from something like Postman, bypassing your web front end entirely.
If you're not securing your backend, you're not secure. Typically systems do both.

It is just a different ways of registering users to our website.
Signing in users at the frontend using OAuth can be very handy as it enables our users to not go through our boring forms in our website. Using OAuth is just one click away from registering users in our website. Beneficial to both the clients and developer.We are provided with the general information about clients by the provider(google,facebook) just clicking on one button.
Signing in users at the backend side is the traditional way of registering our clients. Here we force our client to fill the forms in our website(which may be painfull if it is a long form) and all the filled data is stored in our database.
So they are both different ways to register clients to our websites. Both are used very often. It depends on usecase and needs. Oauth can be used if you want to attract more clients by registering them just by one click.

Firebase Authentication via "Magic-Link"

I want to send notification mails to users of my community platform coded with react and firebase. I was wondering is there is a way to authenticate the user via a magic link which is contained in his or her personal notification mails.
E.g. You've got 5 new comments on your post click HERE to read them
Clicking "here" should redirect the user to the page and automatically authenticate him without having to enter their individual passwords or usernames.

Due to the cancelation of the privacy shield framework and Schrems II, you might no longer be able to store customer details in the GCP.
A solution for it can be pseudonymization.
I am running an open-source project that can help you with that.
Check out the following article for more information:
https://databunker.org/use-case/schrems-ii-compliance/

I think your question can be divided into 2 different questions.
1. How do I send notification mails to my community platform?
I have never used notification mails in Firebase, I usually use google's SMTP server directly and send a request to the server to send an email on behalf of me. So, in your case, I would search up the library on your tech stack (for me it was Go's gomail), then you can use Google's SMTP server (host:smtp.gmail.com, port:587).
2. How do I make a link that automatically authenticate your users whenever they click the link?
I feel like this is the harder question. I think you need to consider a few things:
Your links need to be short lived. Why? to prevent unauthorized brute force login attempt (this also depends on the structure of your link). However, best practices suggests that links should be short lived (less than 24 hours).
Given that links are short lived, this depends on whether the user checked their email frequently for notification from you! There's a pretty good chance that they will miss that email in the 24 hour window.
I'm pretty much against sending time sensitive notification like that through emails. But if it is something that you still want to do, it is pretty easy to create the link, the simplest way I did it was:
Generate a random uuid for a link.
Associate it to the user who will login with the link. You should save this association in a persistent data storage.
Create an endpoint for magic link, for example /magic/:link_id that takes link_id then checks what user should be logged in.
Voila! You got yourself a magic link!
Finally, you can send the link through email like the first part of the problem!

Office365 IMAP Access: User is Authenticated but not Connected

I'm building my own IMAP client using node-imap, and I'm testing it against accounts on office365.
Since it's under development, I tend to hammer on the IMAP server more that I would in production, but not necessarily what I would consider to be too much.
Occasionally, I start getting the IMAP response "BAD User is authenticated but not connected". This comes after the first command after a successful login.
This is not related to password changes as other similar articles indicate, as my passwords are not changing during this time.
When this happens, the account seems to get locked up for a period of time. I seem to be able to jiggle it free by disabling then re-enabling IMAP on the account via the office365 admin panel.
So it looks like there's a circuit breaker of some sort in the IMAP interface on office365. Microsoft documents a limit of 20 concurrent sessions. In my case I am only opening one session at a time, but I am then closing it and re-opening it again soon after (debugging), so if I am running afoul of a limitation, it's not documented anywhere i have read.
Microsoft support has been helpful but unable to help me root cause this.
In order to take this app to production, I need to know what causes this to happen.
Has anyone else had a similar experience with office365?

You may want to take a look at the Outlook REST APIs rather than IMAP. The REST APIs are the recommended development API for Office 365 and are much better suited for app development with features like webhook based notifications, and are constantly evolving with the new features and functionality in the service. They also don't have the session oriented nature that IMAP has that results in the kinds of problems you are hitting.
There is a NodeJS quickstart guide to O365 here: https://dev.outlook.com/RestGettingStarted/Tutorial/node

you need to perform 2 operations on exchange online using powershell
You need to create the service principal
New-ServicePrincipal -AppId {clientid} -ServiceId {enterprise application objectid} -Organization {tenantid}
you need to grant your service principal access to the mailbox,
If the principal does not have the access, you get "user not connected"
Add-MailboxPermission -Identity {email} -User -ServiceId {enterprise application objectid} -AccessRights FullAccess
See "Register service principals in Exchange" at
https://learn.microsoft.com/en-us/exchange/client-developer/legacy-protocols/how-to-authenticate-an-imap-pop-smtp-application-by-using-oauth

If you have already verified that your password is correct (by logging into your mailbox from the web) then the issue might be Exchange server keeping a list of all your active login sessions.
Try signing out from all other locations by clicking "sign out everywhere" on the account security page https://mysignins.microsoft.com/security-info and then retry signing in again.

Office365 will accept connections and successfully authenticate even when the user doesn't have permissions to access the server by IMAP, and this is the exact error it gives in that situation.
To resolve it, go into the Office365 panel, open the user, and ensure that the IMAP permission is checked.

how to make sure only my own website (clientside code) can talk to Firebase backend?

I've read about Firebase and it looks awesome for what I want to do.
I've read about authentication and how based on rules certain logged-in users are authorized to do different stuff. Al good.
However, I'm unsure about another type of security: how do I make sure that only my own site (using client-side javascript) can talk to my firebase-backend? I'm asking because afaik there's no way to prevent anyone from looking up my firebase endpoint from the client-side code (url pointing to my specific firebase backend) and start using that for god knows what.
This is especially worrisome in situations in which I want to open up writes to the anonymous user role. (e.g: some analytics perhaps)
Any help in clearing my mind on this much appreciated.

Update (May 2021): Thanks to the new feature called Firebase App Check, it is now actually possible to limit calls to your backend service to only those requests coming from iOS, Android and Web apps that are registered in your Firebase project.
You'll typically want to combine this with the user authentication based security that Kato describes below, so that you have another shield against abusive users that do use your app.
In my opinion, this isn't so much a question about Firebase's security as a general discussion of the internet architecture as it stands today. Since the web is an open platform, you can't prevent anyone from visiting a URL (including to your Firebase) any more than you can prevent someone from driving past your house in the real world. If you could, a visitor could still lie about the site of origin and there is no way to stop this either.
Secure your data with authentication. Use the Authorized Domains in Forge to prevent CSRF. Put security rules in place to prevent users from doing things they should not. Most data writes you would use a server to prevent can be accomplished with security rules alone.
This is actually one of the finer qualities of Firebase and API services in general. The client is completely isolated and thus easily replaced or extended. As long as you can prove you're allowed in, and follow the rules, where you call in from is unimportant.
Regarding anonymous access, if you could make them visit only from your site, that still won't stop malicious writes (I can open my JavaScript debugger and write as many times as I want while sitting on your site). Instead, place tight security rules on the format, content, and length of data writable by anonymous users, or save yourself some time and find an existing service to handle your analytics for you, like the ubiquitous Google Analytics.
You can, of course, use a server as an intermediary as you would with any data store. This is useful for some advanced kinds of logic that can't be enforced by security rules or trusted to an authenticated user (like advanced game mechanics). But even if you hide Firebase (or any database or service) behind a server to prevent access, the server will still have an API and still face all the same challenges of identifying clients' origins, as long as it's on the web.
Another alternative to anonymous access is to use custom login, which would allow a server to create its own Firebase access tokens (a user wouldn't necessarily have to authenticate for this; the signing of the tokens is completely up to you). This is advantageous because, if the anonymous user misbehaves, the access token can then be revoked (by storing a value in Firebase which is used by the security rules to enforce access).
UPDATE
Firebase now has anonymous authentication built into simple login, no need to use custom login for common use cases here.