The Express node.js server application I'm currently developing will sometimes go off into the void and stops returning requests. I very much suspect somewhere in it I am doing something that is ultimately skipping a res.end that I need to be doing.
Does anyone have any tips or tricks on how to best go about looking for where I'm missing the res.end (assuming that's the problem)? Is there anyway to view the currently open requests in real time in the node event loop?
When things hang up, everything is still running in that Express will still accept incoming requests. And, these requests are logged to the console via express.logger just fine. But, the request is never returned to the client and no error is being thrown.
Thank you in advance for your thoughts.
Follow-up:
There doesn't seem to be much documentation on node's debug mode beyond the raw API docs. I did find this article which seems to provide a good overview of several options. But, there doesn't seem to be the kind of thing I was thinking of for watching the event loop, something a la vtop's process list.
More Follow-up:
Yesterday, Daniel Hood published Debugging in Node.js stating that, "The state of debugging node.js apps has always been a bit unfortunate..." So, perhaps my question is not so silly as to merit down votes as it might seem. Anyway, in his next article he promises to cover some tools for debugging asynchronous operations.
I posted the last article you mentioned. If you need to debug async stuff, I'll be going over things like long stack traces (modules such longjohn), the try-catch module, and things like zones/domains. Hopefully those links will be of some help (might be a week before I write part 2).
However, for your particular issue, have you tried setting a breakpoint at some known location and stepping through the process? This way you can just watch how the app is executed, and maybe find where some function is returning nothing when it should be returning the req/res object.
If it isn't very reproducible, maybe some simple logging would be helpful.
Related
We have a NodeJS application deployed with forever. We don't know why, but this solution is generating too many defunct processes. In some cases, we need to restart manually the application because this situation collapses our server. We'are logging now the PID and timestamp to investigate the reasons, but I'm not confident that we're going to get something with this.
Has anyone this problem? What type of things do we need to check in our NodeJS application to remove this issue? I'm completely stuck and I don't know which type of things can generate this defunct processes. Our code is really simple, with JS promises, WS request, React, ...
If anyone needs some code (like package.json or whatever file) or more info, please just tell me.
Thanks!
Suppose you're new to a project that uses a lot of AJAX requests and other asynchronous methods (e.g. Observable subscriptions). If you're seeing an intermittent issue, and you suspect the problem is due to the timing of AJAX responses and resulting observable updates, how would you even begin debugging this?
Here's what I've done. First, I drew sequence diagrams for the relevant code. Then I tried to imagine scenarios that would cause problems, like out-of-order AJAX responses. This takes a lot of time, so instead I often try introducing delays into various REST endpoints in an attempt to expose the problem consistently. Another shotgun-debugging technique I've tried is logging all events and observable updates, but this is way too verbose to be practical.
Is there a better way? It's hard to visualize all of the complexity, and coming up with potential causes just seems like a shot in the dark.
Simply put, you HAVE to document all your async dependencies and then you have to code to make absolutely sure that regardless of timing, all your dependencies are fulfilled. This is the place to start. If this information isn't 100% known, then you can't even begin to ensure the design is correct.
So, this means figuring out exactly which operations are independent and can run on their own and complete whenever and which operations have dependent async operations that must complete first. For the operations which have dependencies, you MUST make sure the code enforces that dependency. This isn't the step where you imagine out of sequence operations or weird timing. This is the step where you fully understand and document for every async operation what has to complete before this operation can either be started or before the result can be processed.
Promises are often a nice architecture for both stating dependencies in the code and then enforcing that dependency.
So, if a and b must complete before c runs, then you have to code such that your code enforces that dependency.
I would NOT suggest that you just try to test the heck out of it to "find" all the problems. Not only is that hard to do, but it isn't really the structured way to make sure you understand the design requirements and that you have code designed to implement those design requirements.
Once you think you fully understand what is required and think you have implemented it, then and only then is the time to start figuring out if you can test what you've built to see if it reacts the way you expect and also as a check to see if you've missed any dependencies. This is the step where you start figuring out how to create out of sequence results in order to test your dependencies. But, you're only doing this to test if the dependencies you've already identified and coded for are working and implemented properly, not to find new things to design for.
If you have a project with a lot of async operations, I would strongly suggest that you use some sort of library to help you manage those async operations and dependencies between them. Depending upon what else you're using in your project and whether it's client or server, you have many different choices. The whole promise architecture looks like it's one of the main design directions this type of thing is headed in the future - ES6 will have promises built-in, many libraries like jQuery have some sort of promises implementation built in, etc... Promises allow you to either sequence async operations or declare that multiple async operations must complete before starting the next one and so on.
One more thing. Once you've identified a problem and need help debugging it, you often cannot use breakpoints in the code because they tend to just mess up the timing of everything that comes afterwards which totally changes the very situation you're trying to look at (kind of like a Heisenberg uncertainty principle for async debugging). As such, I tend to resort to detailed logging so once you see a problem, you can go back and study the logs and see what actually happened. With a more detailed understanding of what might have happened, you can then often either add more targeted logging or perhaps then set a breakpoint to examine the issue or maybe by then, you'll see what dependency in the code you don't have working right or missed that you had to code for.
"Welcome friend, to the asynchronous world of AJAX", that's my opening statement to you...
In short, you need a debugger and a truck load of breakpoints. Get firebug for Firefox my friend or try the google chrome developer console (hit F12 when using Chrome!). Both of these debugger are great for examining responses from the server, the timing and breaking during code execution. They also allow you to examine the object structure if you're working with JSON which saves hours of mentally picturing problems, literally.
My honest advice is:
Get Firebug or Use Chrome's Developer console
Set breakpoints in the script and watch the logical flow and paths of execution of your code.
Fix/Adjust/Re-order code blocks, then repeat step 1 and 2.
This may take time, cause you to restructure your scripts a lot, but it is definitely better than setTimeouts, setIntervals or other shotgun techniques, which will break down if the server responses take longer than usual.
Recently, I have a chance to work on a project contains some large javascript files. I would say 4000-5000 lines per file. For example, there are 3 big files(custom plugin) build on top of each other. I got a debug task that needs to resolve(logically and it is not a js error). when I tried to debug and understand the logic under chrome dev tool such as step through or tracing where the variable came from and goes, I always get lost at some point because the files are so big. I thought maybe I need to sit down for 1 or 2 day to read through all the files and draw the logic on the paper, I guess that may be not a good solution. I wonder if there is any technique that I missed for debugging and tracing the variables or the logic. please share your experience with me. thanks
Sometimes when I look at something like that, I start by creating a test. Try to test only the defect. Take a working copy and try to reduce it down until you have isolated the problem.
Good luck!
Specifically, for advanced step-debugging, there is:
using the call stack to inspect the local variables from the calling scope without having to step out of the function.
using conditional breakpoints.
https://developers.google.com/chrome-developer-tools/docs/javascript-debugging
If your JavaScript is making a lot of HTTP requests, it might also be useful to use the network tab to check the requests and responses are as expected.
Here is the situation: A complex web app is not working, and it is possible to produce undesired behavior consistently. The cause of the problem is not known.
Proposal: Trace the execution paths of all javascript code. Essentially, produce two monstrous logs which can then be fed into a diff algorithm to determine where the behavior related to the bug begins to diverge (as the cause is not apparent from application behavior, and both comprehending and obtaining a copy of the actual JS code being run is difficult, due to the many pages that must be switched to and copied out from the web inspector. Making it difficult is the fact that all pages are dynamically spliced together with Perl code, where significant portions of JS code exist only as (dynamic...) Perl strings).
The Web Inspector in Chrome does not have an option that I know about for logging an execution trace. Basically what I would like is a log of every line of JS that is executed, in the order that they are executed. I don't see this as being a difficult thing to obtain given that the JS VM is single-threaded. The problem is simply that the existing user-facing tools are not designed for quite this much hardcore debugging. If we look at the Profiler in the Dev Tools, it's clearly capable of the kind of instrumentation that I need, but it is fundamentally designed to do profiling instead of tracing.
How can I get started with this? Is there some way I can build Chrome from source, where I can
switch off JIT in V8?
log every single javascript expression evaluated by V8 to a file
I have zero experience with the development side of Chrome. So e.g. links to dev-builds/branches/versions/distros of Chrome/Chromium/Canary (what's the difference?) are welcome.
At this point it appears that instrumenting the browser with powerful js tracing is still likely to be easier than redesigning the buggy app. The architecture of the page is a disaster, but the functionality is complex, and it almost fully works. I just have to find the one missing piece.
Alternatively, if tools of this sort already exist, what are some other keywords I can search for them with? "Code Tracing" is pretty much the only thing I can come up with.
I tested dynaTrace, which was a happy coincidence as our app supports IE (indeed Chrome support just came out of beta), but this does not produce a text dump, it basically produces a massive Win32 UI expando-tree, which is impossible to diff. This makes me really sad because I know how much more difficult it was to make the representation of the trace show up that way, and yet it turns out being almost utterly useless. Who's going to scroll up and down that tree view and see anything really useful in it, in anything other than a toy example of a web app?
If you are developing a big web app, it is always good to follow a test driven strategy for the coding part of it. Using just a few tips allows you to make a simple unit testing script (using QUnit) to test pretty much all aspects of your app. Here are some potential errors and some ways of solving them.
Make yourself handlers to register long living Objects and a handler to close them the safe way. If the safe way does not succeed then it is the management of the Object itself failing. One example would be Backbone zombie views. Either the view has bad code in the close section, the parent close is not hooked or an infinite loop happened. Testing all view events is also good, although tedious.
By putting all the code for data fetching inside a certain module (I often use a bunch of Backbone.Model objects for each table/document in my DB) and handlers for each using a reqres pattern, you can test them 1 by 1 to see if they all fetch and save correctly.
If complex computation is needed, abstract it in a function or module so that it can be easily tested with known data.
If your app uses data binding, a good policy is to have a JSON schema for all data to be tested against your views containing your bindings. Check against the schema all the data required. This is applied to your Backbone.Model also.
Using a good IDE also helps. PyCharm (if you use Python for backend) or WebStorm are extremely good for testing and developing JavaScript/CoffeeScript. You can breakpoint and study your code at specific locations, inside your browser! Also it runs your code for auto-completion and you can see some of the errors that way.
I just cannot encourage enough the use of modules in your code. Although there is no JavaScript official way of doing it (next ECMAScript draft has it), you can still use good libraries for it. Good ones are: RequireJS, CommonJS or Marionette.Module (if you use Marionette as your framework). I think Ember/AngularJS also offers this kind of functionality but I did not work with them personally so I am not sure.
This might not give you an immediate solution to your problem and I don't think (IMO) there is an easy one either. My focus was to show you ways to develop so that errors can be easily spotted and countered, and all of it (depending on your Unit Testing) during development phase. Errors will always happen, as much as our programmer ego wants us to believe the contrary. Hope I helped :)
I would suggest a divide and conquer strategy, first via logging, and second via code. Wrap suspect methods of the code with console logging in and out events and when the bug occurs hopefully it is occurring between or after some event. If event logging will not shed light, bring parts of the code into a new page/app. Divide and conquer will find when the error starts occurring.
So it seems you're in the realm of weird already, so I'm thinking of a weird solution. I know nothing about the backend of chrome myself so we're in the same boat, but if you're feeling bold here's an idea. Maybe you could find/replace every newline in your javascript files with a piece of code that logs either to a global string or to the console a) what file you're in, b) the contents of "this" or something useful to you, and maybe even c) the time. This would at least get you started. Make sure it's wrapped in something distinct so you can just as easily remove it.
I am new to Node.js and am currently questioning its reliability.
Based on what I've seen so far, there seems to be a major flaw: any uncaught error/exceptions crashes the server. Sure, you can try to bullet-proof your code or put try/catch in key areas, but there will almost always be bugs that slip through the crack. And it seems dangerous if one problematic request could affect all other requests. There are 2 workarounds that I found:
Use daemon or module like forever to automatically restart the server when it crashes. The thing I don't like about this is that the server is still down for a second or two (for a large site, that could be hundreds (of thousands?) of request).
Catch uncaught exceptions using process.on('uncaughtException'). The problem with this approach (as far as I know) is that there is no way to get a reference to the request that causes the exception. So that particular request is left hanging (user sees loading indicator until timeout). But at least in this case, other non-problematic requests can still be handled.
Can any Node.js veteran pitch in?
For automatic restarts and load-balancing, I'd suggest you check out Learnboost's up balancer.
It allows you to reload a worker behind the load-balancer without dropping any requests. It stops directing new requests towards the worker, but for existing requests that are already being served, it provides workerTimeout grace period to wait for requests to finish before truly shutting down the process.
You might adapt this strategy to be also triggered by the uncaughtException event.
You have got full control of the base process, and that is a feature.
If you compare Node to an Apache/PHP setup the latter is really just equivalent to having a simple Node server that sends each incoming request to it's own process which is terminated after the request has been handled.
You can make that setup in Node if you wish, and in many cases something like that is probably a good idea. The great thing about Node is that you can break this pattern, you could for instance have the main process or another permanent process do session handling before a request is passed to it's handler.
Node is a very flexible tool, that is good if you need this flexibility, but it takes some skill to handle.
Exceptions don't crash the server, they raise exceptions.
Errors in node.js that bring down the entire process are a different story.
Your best bet (which you should do with any technology), is just test it out with your application as soon as possible to see if it fits.
An uncaught exception will, if not caught, crash the server. Something like calling a misspelled function. I use process.on('uncaughtException') to capture such exceptions. If you use this then, yes, the error sent to process.on('uncaughtException') is less informative.
I usually include a module like nomnom to allow for command-line flags. I include one called --exceptions which, when set, bypasses process.on('uncaughtException'). Basically, if I see that uncaught exceptions are happening then I, in development, start up the app with --exceptions so that when that error is raised it will not be captured, which causes Node to spit out the stack trace and then die. This tells you what line it happened on, and in what file.
Capturing the exceptions is one way to deal with it. But, like you said, that means that if an error happens it may result in users not receiving responses, et cetera. I would actually recommend letting the error crash the server. (I use process.on('uncaughtException') in apps, not webservers). And using forever. The fact is that it is likely better for the webserver to crash and then expose what you need to fix.
Let's say you used PHP instead of Node. PHP does not abruptly crash the server (since it doesn't really serve). It spits out really ugly errors. Sure, it doesn't result in a whole server going down and then having to come back up. Nobody wants their clients to have any downtime. But it also means that a problem will persist and will be less noticeable. We've all seen sites that have said errors, and they don't get patched very fast. If such a bug were to take everything down for one small blip (which honestly its not all that bad in the larger picture) then it would surely call attention to itself. You would see it happen and would track that bug down.
The fact is that bugs will exist in any system, independent of language or platform. And it is arguably better for them to be fatal in order for you to know they happened. And over time it causes you to become more aware of how these error occur. I don't know about you, but I know a lot of PHP devs who make the same common mistakes time after time.