We Keep Coding

Pseudo real-time data stream javascript and php

Update: I am getting the impression that this is not even the right website to post this. If someone can point me in the right direction, I'd be appreciative...
I have an existing PHP+MySQL application that wasn't built to render "real-time" or similarly live-style data. But now I need to build in a way to pull nearly real-time data into the application and keep the data on the page fresh. This live data is only for 1 page in the application.
Looked at things like socket.io and PHP-based websockets libraries, but it seemed like overkill because the data is basically coming from 1 source and being delivered to 1 person (the client). Multiple other users could have this process running, but each one would bring their own data endpoint. That's... like a year down the road. But good to think about. Would ideally have hundreds, or thousands of users on the system, pulling their live-ish data. So I want this to be as streamlined and low-impact as possible.
Users must be authenticated and authorized to consume the data. This is already baked into the current system.
The API to get the data (which has already been built by another vendor) is also NOT streaming. It's set on a 20-second cron, so the new data is available every 20 seconds, which satisfies the client's needs.
My current plan is to do something like this...
Data is pulled on a cron every 20 seconds, organized, and stored into the database (complete)
Adjust #1 so it also does any additional proprietary calculations on data AND compiles + writes a JSON file on the server (unique to the user) which is the exact data needed for the front end (DB data is needed for other pages)
Create small PHP-based service which validates a client-provided JWT and reads the JSON file out
Write AJAX front end to poll endpoint from #3 every X seconds using a JWT for authorization
This all seems sort of like I might be reinventing the wheel, or missing something. The fact that this is an existing PHP based application (LAMP) does have some limiting factors, but I feel like there's got to be a more efficient way to handle this... It's pretty new to me. Also, I'm open to other technologies that'll run on the LAMP stack, if it'll make things better.

I would say go for the API solution in the beginning :) Since it fits the architecture more and is for sure the least amount of work. Also if there will be problem with the "live" feeling of the data you can fix it by polling more often or introducing long polling, assuming you change the cron job time.
I mean in the end it is all about impact for the time spent, don't start implement features that customers don't care about :)
The biggest problem to solve is to implement it in a way that fits your requirements and is somewhat future extendable. You still have to deal with issues like resolution, time outs, reducing server processing when requesting data and so on!

For me, if you need to maintain a global service state because a single client(s) request could affect all other connected client request(s) then most all server-side scripting languages are not the best choice! Also to further add, if you plan on implementing something like this with PHP, you will be setting your self up for a living nightmare! Why, because simply put, PHP(s) socket(s) implementation is that bad!

Adding server-side libraries for javascript

I am currently coding my offline game into an online game by use of node.js and socket.io
In my game, I use vectors from a library called p5.js to store position of player, collision related movement, etc.
However, the server side (a txt file called "server.js") does not have p5.js like the client, so I can't send information about the player's with vectors.
Here is my question: How could I make the server.js file have access to my p5.js library?
Note: Simply sending x and y values, and then using them to make a vector would be a difficult solution, as I would no longer be able to send a single array holding all the information of all players. Also, enemies, food, trail positions, and much more also depend on vectors. Coverting all of these would be difficult to code.

What you want to do is simple, but it won't feel simple until you fully understand all the parts involved, and for that I'm afraid it's going to take you at least a few months given your current level.
How could I make the server.js file have access to my p5.js library?
Is your p5.js a browser-only library, or you can import it as a module in your server? If it's the second option, all you have to do for your server to access it is:
const p5 = require('p5.js');
Keep in mind that:
Server should handle the position, movement, and actions of players so they can't cheat.
The client should be just a visual display of the information coming from the server, plus sending the player key inputs to the server.
So unless you want to make client side prediction and entity interpolation, which I doubt because you are starting out, keep it simple. No need to share libraries between client and server yet.

Back-end solution for pure-Javascript site with link sharing

I'm developer-beginner and I would like to do simple card-sorting site.
hundreds of cards with plain text
no log-ins, it must be as simple for user as possible
few dozens users per day
final state of sorting should be saved under unique link, like site.com/q56we1cfuz4 or site.com/link.php?unique=q56we1cfuz4
User sorts cards as he/she wishes and the result is saved into browser sessions storage (few kb of data in JSON or so). So far, so good.
Question:
As I know only Javascript:
If I need to share the state of the page (data in session storage) with some unique link, is something like Firebase.com good solution for such kind of back-end? Or some simple DB with help od Node.js?
In order to "catch" the link when someone click at unique URL (site.com/link.php?unique=q56we1cfuz4), I still need some server-side script which will ask the DB, like PHP header redirect, right?

Your questions are a little fuzzy, no problem tho. You are just getting into web dev, so there's a lot to wrap your head around and all of the options can be pretty confusing. Some people will complain about opinionated answers, and I'm not going to claim to be objective here, but here are a few tips that I think will get you pointed in a better direction...
Firstly, yes - firebase is a good solution for you to try working with. Aside from the fact that it will give you the db/storage features you need, it's a realtime db, which will give you a lot more than just storage in the long run, and will better equip you for the future web development trends. The firebase API is (mostly) designed to work asynchronously, so from a javascript perspective, it falls right in line with the kind of code you'll end up learning to write.
Re: the other aspect of your question - server-side - check out nodeJS. It's basically a server-side javascript platform that will allow you to use the same skills you're learning to write client-side code for the server. Also check out expressJS, a nodeJS package that provides you the http-server, and allows you to handle dynamic urls, etc. - the bits you were thinking about when you made a reference to PHP.
Hopefully this will help you focus on a few specific tools to familiarize yourself with as you learn web development, rather than having to struggle with everything from new languages, platforms, and way too many libraries, frameworks and techniques to wrap your head around. Good luck!

What is the best way I can scale my nodejs app?

The basics
Right now a few of my friends and I are trying to develope a browser game made in nodejs. It's a multiplayer top-down shooter, and most of both the client-side and server-side code is in javascript. We have a good general direction that we'd like to go in, and we're having a lot of fun developing the game. One of our goals when making this game was to make it as hard as possible to cheat. Do do that, we have all of the game logic handled server-side. The client only sends their input the the server via web socket, and the server updates the client (also web socket) with what is happening in the game. Here's the start of our problem.
All of the server side math is getting pretty hefty, and we're finding that we need to scale in some way to handle anything more than 10 players (we want to be able to host many more). At first we had figured that we could just scale vertically as we needed to, but since nodejs is single threaded, is can only take advantage of one core. This means that getting a beefier server won't help that problem. Our only solution is to scale horizontally.
Why we're asking here
We haven't been able to find any good examples of how to scale out a nodejs game. Our use case is pretty particular, and while we've done our best to do this by ourselves, we could really benefit from outside opinions and advice
Details
We've already put a LOT of thought into how to solve this problem. We've been working on it for over a week. Here's what we have put together so far:
Four types of servers
We're splitting tasks into 4 different 'types' of servers. Each one will have a specific task it completes.
The proxy server
The proxy server would sit at the front of the entire stack, and be the only server directly accessible from the internet (there could potentially be more of these). It would have haproxy on it, and it would route all connections to the web servers. We chose haproxy because of its rich feature set, reliability, and nearly unbeatable speed.
The web server
The web server would receive the web-requests, and serve all web-pages. They would also handle lobby creation/management and game creation/management. To do this, they would tell the game servers what lobbies it has, what users are in that lobby, and info about the game they're going to play. The web servers would then update the game servers about user input, and the game server would update the web servers (who would then update the clients) of what's happening in the game. The web servers would use TCP sockets to communicate with the game servers about any type of management, and they would use UDP sockets when communicating about game updates. This would all be done with nodejs.
The game server
The game server would handle all the game math and variable updates about the game. The game servers also communicate with the db servers to record cool stats about players in game. This would be done with nodejs.
The db server
The db server would host the database. This part actually turned out to be the easiest since we found rethinkdb, the coolest db ever. This scales easily, and oddly enough, turned out to be the easiest part of scaling our application.
Some other details
If you're having trouble getting your head around our whole getup, look at this, it's a semi-accurate chart of how we think we'll scale.
If you're just curious, or think it might be helpful to look at our game, it's currently hosted in it's un-scaled state here.
Some things we don't want
We don't want to use the cluster module of nodejs. It isn't stable (said here), and it doesn't scale to other servers, only other processors. We'd like to just take the leap to horizontal scaling.
Our question, summed up
We hope we're going in the right direction, and we've done our homework, but we're not certain. We could certainly take a few tips on how to do this the right way.
Thanks
I realize that this is a pretty long question, and making a well thought out answer will not be easy, but I would really appreciate it.
Thanks!!

Following my spontaneous thoughts on your case:
Multicore usage
node.js can scale with multiple cores as well. How, you can read for example here (or just think about it: You have one thread/process running on one core, what do you need to use multiple cores? Multiple threads or multiple processes. Push work from main thread to other threads or processes and you are done).
I personally would say it is childish to develop an application, which does not make use of multiple cores. If you make use of some background processes, ok, but if you until now only do work in the node.js main event loop, you should definitely invest some time to make the app scalable over cores.
Implementing something like IPC is not that easy by the way. You can do, but if your case is complicated maybe you are good to go with the cluster module. This is obviously not your favorite, but just because something is called "experimental" it does not mean it's trashy. Just give it a try, maybe you can even fix some bugs of the module on the way. It's most likely better to use some broadly used software for complex problems, than invent a new wheel.
You should also (if you do not already) think about (wise) usage of nextTick functionality. This allows the main event loop to pause some cpu intensive task and perform other work in the meanwhile. You can read about it for example here.
General thoughts on computations
You should definitely take a very close look at your algorithms of the game engine. You already noticed that this is your bottleneck right now and actually computations are the most critical part of mostly every game. Scaling does solve this problem in one way, but scaling introduces other problems. Also you cannot throw "scaling" as problem solver on everything and expect every problem to disappear.
Your best bet is to make your game code elegant and fast. Think about how to solve problems efficiently. If you cannot solve something in Javascript efficiently, but the problem can easily be extracted, why not write a little C component instead? This counts as a separate process as well, which reduces load on your main node.js event loop.
Proxy?
Personally I do not see the advantage of the proxy level right now. You do not seem to expect large amount of users, you therefore won't need to solve problems like CDN solves or whatever... it's okay to think about it, but I would not invest much time there right now.
Technically there is a high chance your webserver software provides proxy functionality anyway. So it is ok to have it on the paper, but I would not plan with dedicated hardware right now.
Epilogue
The rest seems more or less fine to me.

Little late to the game, but take a look here: http://goldfirestudios.com/blog/136/Horizontally-Scaling-Node.js-and-WebSockets-with-Redis
You did not mention anything to do with memory management. As you know, nodejs doesn't share its memory with other processes, so an in-memory database is a must if you want to scale. (Redis, Memcache, etc). You need to setup a publisher & subscriber event on each node to accept incoming requests from redis. This way, you can scale up x nilo amount of servers (infront of your HAProxy) and utilize the data piped from redis.
There is also this node addon: http://blog.varunajayasiri.com/shared-memory-with-nodejs That lets you share memory between processes, but only works under Linux. This will help if you don't want to send data across local processes all the time or have to deal with nodes ipc api.
You can also fork child processes within node for a new v8 isolate to help with expensive cpu bound tasks. For example, players can kill monsters and obtain quite a bit of loot within my action rpg game. I have a child process called LootGenerater, and basically whenever a player kills a monster it sends the game id, mob_id, and user_id to the process via the default IPC api .send. Once the child process receives it, it iterates over the large loot table and manages the items (stores to redis, or whatever) and pipes it back.
This helps free up the event loop greatly, and just one idea I can think of to help you scale. But most importantly you will want to use an in-memory database system and make sure your game code architecture is designed around whatever database system you use. Don't make the mistake I did by now having to re-write everything :)
Hope this helps!
Note: If you do decide to go with Memcache, you will need to utilize another pub/sub system.

What good ways are there to prevent cheating in JavaScript multiplayer games?

Imagine a space shooter with a scrolling level. What methods are there for preventing a malicious player from modifying the game to their benefit? Things he could do that are hard to limit server-side is auto-aiming, peeking outside the visible area, speed hacking and other things.
What ways are there of preventing this? Assume that the server is any language and that the clients are connected via WebSocket.
Always assume that the code is 100% hackable. Think of ways to prevent a client completely rewritten (for the purposes of cheating) from cheating. These can be things such as methods for writing a secure game protocol, server-side detection, etc.

The server is king. Clients are hackable.
What you want to do is two things with your websocket.
Send game actions to the server and receive game state from the server.
You render the game state. and you send input to the server.
auto aiming - this one is hard to solve. You have to go for realism. If a user hits 10 headshots in 10ms then you kick him. Write a clever cheat detection algorithm.
peeking outside the visibile area - solved by only sending the visible area to each client
speeding hacking - solved by handling input correctly. You receive an event that user a moved forward and you control how fast he goes.
You can NOT solve these problems by minifying code. Code on the client is ONLY there to handle input and display output. ALL logic has to be done on the server.
You simply need to write server side validation . The only thing is that a game input is significantly harder to validate then form input due to complexity. It's the exact same thing you would do to make forms secure.
You need to be really careful with your "input is valid" detection though. You do not want to kick/ban highly skilled players from your game. It's very hard to hit the balance of too lax on bot detection and too strict on bot detection. The whole realm of bot detection is very hard overall. For example Quake had an auto aim detection that kicked legitedly skilled players back in the day.
As for stopping a bots from connecting to your websocket directly set up a seperate HTTP or HTTPS verification channel on your multiplayer game for added security. Use multiple Http/https/ws channels to validate a client as being "official", acting as some form of handshake. This will make connecting to the ws directly harder.
Example:
Think of a simple multiplayer game. A 2D room based racing game. Upto n users go on a flat 2D platformer map and race to get from A to B.
Let's say for arguments sake that you have a foolsafe system where there's a complex authetication going over a HTTPS channel so that users can not access your websocket channel directly and are forced to go through the browser. You might have a chrome extension that deals with the authentication and you force users to use that. This reduces the problem domain.
Your server is going to send all the visual data that the client needs to render the screen. You can not obscure this data away. No matter what you try a silled hacker can take your code and slow it down in the debugger editing it as he goes along until all he's left with is a primitive wrapper around your websocket. He let's you run the entire authentication but there is nothing you can do to stop him from stripping out any JavaScript you write from stopping him doing that. All you can achieve with that is limit the amount of hackers skilled enough of accessing your websocket.
So the hacker now has your websocket in a chrome sandbox. He sees the input. Of course your race course is dynamically and uniquely generated. If you had a set amount of them then the hacker could pre engineer the optimum race route. The data you send to visualise this map can be rendered faster then human interaction with your game and the optimum moves to win your racing game can be calculated and send to your server.
If you were to try and ban players who reacted too fast to your map data and call them bots then the hacker adjusts this and adds a delay. If you try and ban players who play too perfectly then the hacker adjusts this and plays less then perfect using random numbers. If you place traps in your map that only algorithmic bots fall into then they can be avoided by learning about them, through trial and error or a machine learning algorithm. There is nothing you can do to be absolutely secure.
You have only ONE option to absolutely avoid hackers. That is to build your own browser which cannot be hacked. Build the security mechanisms into the browser. Do not allow users to edit javascript at runtime in realtime.

At the server-side, there are 2 options:
1) Full server-side game
Each client sends their "actions" to the server. The server executes them and sends relevant data back. e.g. a ship wants to move north, the server calculates its new position and sends it back. The server also sends a list of visible ships (solving maphacks), etcetera.
2) Full client-side game
Each client still sends their actions to the server. But to reduce workload on the server, the server doesn't execute the actions but forwards them to all other clients. The clients then resolve all actions simultaneously. As a result, each client should end up with an identical game. Periodically, each client sends their absolute data (ship positions, etc.) to the server and the server checks if all client data is identical. Otherwise, the games are out of sync and someone must be hacking.
Disadvantage of the second method is that some hacks remain undetected: A maphack for example. A cheater could inject code so he sees everything, but still only sends the data he should normally be able to see to the server.
--
At the client-side, there is 1 option:
A javascript component that scans the game code to see if anything has been modified (e.g. code modified to render objects that aren't visible but send different validation data to the server).
Obviously, a hacker could easily disable this component. To fix that, you could force the client to periodically reload the component from the server (The server can check if the script file was requested by the user periodically). This introduces a new problem: the hacker simply periodically requests the component via AJAX but prevents it from running. To avoid that: have the component redownload itself, but a slightly modified version of itself.
For example: have the component be located at yoursite/cheatdetect.js?control=5.
The server will generate a slightly modified cheatdetect.js so that in the next iteration, cheatdetect.js?control=22 (for example) must be downloaded. If the control mechanism is sufficiently complicated, the hacker won't be able to predict which control number to request next, and cheatdetect.js must be executed in order to continue the game.

There's nothing you can really do to prevent anyone from modifying your JS or writing a GreaseMonkey script. However you can make it hard for them by minifying your script as well as making your code as cryptic as possible. Maybe even throwing in some fake methods or variables that do nothing but are used to throw an attacker off. But given enough time, none of these methods are completely foolproof, as once your code goes to the client, it is no longer yours.

The only way I can even think of implementing this is by modifying your Javascript to function as a client and then designing a central server mechanism to validate data sent from that client. This is probably a big change to implement and will most likely make your project more complex. However, as was said earlier, if the application runs entirely on the client, the client can pretty much do whatever they want with your script. The only way to secure it to use a trusted machine to handle validation.

They don't have to touch your client-side code -- they could just sniff and implement your Websocket protocol and write a tiny agent that pretends to be a human player.
Update: The problem has a few parts, and I don't have answers off the top of my head, but the various options could be evaluated with these questions in mind:
How far are you willing to go to prevent cheating? If you only care about casual cheating, how many barriers are enough to discourage the casual cheater? The intermediate Javascript programmer? A serious expert? Weighing this against the benefits of cheating, is there anything of real value at stake, like cash and prizes, or just reputation?
How do you get a high confidence that a human is providing inputs to your game? For example, with a good enough computer vision library I could model your game on a separate machine feed inputs to the computer pretending to be the mouse, but this has a high relative cost (not worth my time).
How can you create a chain of trust in your protocol such that knowledge of (2) can be passed to the server, and that your server is relatively confident your client code is sending the messages?
Sure many of the roadblocks you throw up can be side-stepped, but what is the cost to the player and you? See "Attrition warfare".

Some other methods that can be implemented:
Make the target elements difficult for a script to distinguish from other elements. Avoid divs with predictable class and id names if possible. Inject styling using JavaScript instead of using classes. Think like a hacker and make it hard on yourself.
Use decoys that a script will fire on. For instance, if the threat vector is a screen scraping algorithm using pixel colors, throw some common pixel colors in non-target elements. Hits on these non-targets could seem inconsequential to the cheater, but would be detectable. You don't want the cheater to know why you know.
Limit the minimum time between actions to slightly below the best human levels. The best players will hit that plateau, and it won't matter as much who's cheating, and immediately be able to detect anyone scripting faster than that by side-calling method calls.
Random number generators are typically uniform. Human nature is not. Likely a random number generator will have values within a set limit and even distribution. Natural distribution is a Gaussian curve. If you sampled the distribution and it looks like a square wave in the x and y axis, 100% it's a cheater. This will be fairly difficult for the cheater to detect the threshold for the algorithm because it's a derivative of the random, and not the random distribution itself. You're also using aggregate data and not individual plays to detect it, so reverse engineering the algorithm would be extremely difficult without knowing your detection algorithm.
Utilize entropy whenever possible. Avoid predictable game plays. Imagine a racing game on a set collection of race tracks. Each game play could have slightly differing levels of traction, horsepower, and momentum. The script would have to be extremely good to beat it. In a scrolling game, you can alter factors that are instinctual to humans, but difficult for computers, such as wind force, changes in gravity, etc. It would also make it more fun as a side benefit.
Server generated tokens can be used to validate UI elements were used and not calls to the code itself. Validation can be handled in one call at the end of the game comparing events to hashed codes of UI elements. The token should be a hash with a server private key and some value of the UI element.
Decoy the cheater with data they think you're using to detect cheats. Such as calls to a DetectCheat method with dummy calls to a fake backend. It's the old magician's trick. Wave your hand over here, while you slip a card into the deck with the other hand. Let them waste days on end in a maze that has no exit, with lot's of hair pulling.

I'd use a combination of minification and AJAX. If all of the functions and data aren't loaded into the page, it'd be more difficult to cheat.
On the other hand, modding turned out to be a very profitable tool for companies like Id Software. Perhaps allowing the system to be modded might make the game that much more enjoyable to the community at large.

Obfuscate your client exposed code as much as possible. Additionally, use some magic.

You can edit the javascript on the browser and make it work.
Some people suggest that make a call to check with the server. So after making a call to the server, it will be validated in the server. Once validated, it will come to client side and do actions. But I think even this is not foolproof.
For eg.,. for a Basic login action : in angular while making a call to server, the backend validates username & pwd and if validated, it will come back to the client and let the user login using angular.
When I say login using angular, it is going to store things in cookies, like user objects and other things. But still the user can remove the JS code which is making the call to backend, and return TRUE(wherever needed) and insert user object(dummy) to cookies and other objects(whatever needed) and login. It is a very difficult thing to do, but it is doable. In many scenarios, this is not desirable even if it takes hours to edit/hack the code.
This is possible in single page applications, where JS files dont get reloaded for each page. To mitigate the possibility of getting hacked we can use minified codes. And I guess if actions like this is done in backend(like login in Django) it is much safer.
Please correct me if I am wrong.