We Keep Coding

Extending JavaScript syntax in Monaco Editor with full integration

I'm trying to extend Monaco Editor to allow users to write in a hybrid of JavaScript and another language, using delimiters to separate them within the same file, similarly to how Markdown allows writing multiple languages using fenced code blocks.
The difference is that I want to keep all the other IDE features that Monaco has built in for JavaScript, such as linting (done via diagnostics), smart auto-completion, jump-to-definition, auto-formatting helpers, and every other IDE feature that comes with Monaco's built-in JavaScript mode. I'd like these features to still work within the JavaScript portion of the code that Monaco is editing, and be disabled for the sub-language portion.
My first attempt was to call setMonarchTokensProvider, passing in a modified version of TypeScript's tokenizer rules. Specifically, I was able to add the beginning-fence delimiter to the root rule and create a new rule for the sub-language in the same way the documentation for Monarch (Monaco's syntax highligher) describes, using #nextEmbedded. (For testing purposes, I've been hard-coding CSS as the embedded language.)
When I call setMonarchTokensProvider like this for the language "javascript", it completely ignores this syntax highlighting tokenizer, and colors the code-fences of CSS as invalid JavaScript, indicating that you cannot override the built-in JavaScript mode this way.
When I call setMonarchTokensProvider with a new language (e.g. "mylang") and set the editor to use that language, it provides correct syntax highlighting (!) for this CSS-in-JS hybrid language. But all other advanced features that were found in the JavaScript mode are no longer present. The editor didn't have any smart auto-completion for methods defined on classes in the same file, or any in-editor error-reporting for invalid syntax, or any of its trademark JavaScript IDE features.
So my next attempt was to modify the pre-bundled Monaco code's TypeScript definition to include my custom syntax highlighting rules. This correctly highlighted my CSS-in-JS code completely (!), when setting the language to "typescript", and left all the other features intact (!) including diagnostics reporting (live-validation and underlining of errors), auto-completion, all of it! (I didn't try it with "javascript" but it's safe to assume it probably works or is trivial to get it working, since JavaScript is actually implemented as a variant configuration of the TypeScript mode in Monaco.)
Unfortunately, it also considered the entire CSS portion of it, including the fence around it, to be invalid JavaScript code.
I know that this is theoretically doable, because within HTML mode, you can embed CSS or JS with full support for proper validation and auto-completion and every other IDE feature; basically, every sub-language in an HTML file works like it's in its own file: HTML features in the root of the file, CSS features within style tags, JS features within script tags.
But digging into the TypeScript plugin's implementation inside Monaco, it's not clear where to begin editing this, either as a user of Monaco the library, or by forking it and patching it up where necessary. I started at trying to modify the DiagnostcsAdapter [sic] and tracing where it's actually implemented, but I got stuck two function-calls deep, where it seems to push a promise of syntax validation that returns a value that's used later, but the implementation of getSyntacticDiagnostics just shells the work out to some other implementation that I can't find anywhere else in the repo, nor in the monaco-languages repo or the vscode repo.

I make the similar thing. My solution is to place non-JS code inside a block comment:
regularJsCode()
/*
[your-syntax-identifying-start-token]
place any syntax you want here
[your-syntax-identifying-end-token]
*/
regularJsCode()
Then you can process it with your tools, parsers, IDE extension etc. And the coolest part you can make VSCode to syntax-highlight it like you want so it won't seem like some hack.
This approach is preferrable because your JS file still remains a 100% valid JS file.
If you still don't want to put your syntax into comments, then you should create your own file extension like .jsx/.tsx. In practical, VSCode terms this means you need to create VSCode extension with language server and stuff. This is not so easy, but the documentation is good. You could assemble your own JS highlighting code inside your VSCode extension using language server: https://github.com/sourcegraph/javascript-typescript-langserver

According to the creator of Monaco:
Architecturally, you can do the following:
use monaco-editor-core directly
define a new language to the editor
fork monaco-typescript and change it to work with your newly defined langauge id. Then, modify the TS language host code to not pass the original models to TypeScript, but first run a preprocess which strips your custom language out of the text, and then only passes on valid TypeScript to the TS compiler. One idea is to replace each character that you remove with a space. This will leave all the position/offset computation work without any effort on your side.
Good luck!

How to parse a Javascript file and get all the used variables?

I am gathering a lot of Javascript code from untrusted people and have to integrate it in my project. As is is untrusted, I would like to check if it doesn't do something nasty.
My main concern is the variables the code uses.
To check it is OK, I would like to parse all the code and verify the name of the variables. For instance, that all the variables are included in window.sandboxedVariables.
Is it possible to parse a Javascript code (in any language but preferably Javascript or bash) and get the list of all the variables ? Is it possible to do the same with the imported libraries ?
Is it possible to do with Uglify ? I read a bit the API documentation and found nothing specific.
Thank you very much !

Assuming you're talking about global variables, you can do the following:
clone the window object
load/run the untrusted script
compare the window object to the cloned one
move all newfound items into window.sandboxedVariables
However, this won't work if the untrusted script overrides one of the existing properties (variables) of window.

eslint is a JavaScript source code linting tool that lets you write custom plugins. You should be able to write a plugin that meets your needs. Plus, the plugins can be written in JavaScript.
http://eslint.org/docs/developer-guide

It is impossible to write an algorithm that verifies untrusted JavaScript code. You can parse it, you can run it in a sandbox and analyze its actions. But you can never be sure you've identified everything it might do or every variable it might use once you run it in your real environment.
If you don't trust it then either only run it in a secure sandbox or don't use it.

You could use Mozilla Rhino. It is a JavaScript engine written in Java.
Here you can find an example similar to what you are trying to do:
http://ramkulkarni.com/blog/parsing-javascript-code-using-mozilla-rhino/

ctags, jsctags/doctorjs, Tagbar step by step

I need help on setting up ctags, jsctags, and tagbar so I can have a workable Javascript editing envrionment. I got everything installed but couldn't get an idea how ctags and jsctags work together so I don't know how to configure properly. I have done quite a bit Google around but the information is pretty broken and lacks consistency. I got an error similar to this post, ctags and tagbar configuration are out of sync. I'm on OS X mountain lion and iterm2.
Any help would be greatly appreciated. A step by step instructions would be excellent.
Thx.

First, you have to understand that jsctags and ctags will probably never be in sync. Even if you have only a simple function, both programs may output slightly different informations. Thus it is advised to use one or the other but not both.
Second, AFAIK, TagBar doesn't need to be configured to use jsctags instead of ctags. If you are doing JavaScript it will simply default to jsctags if it's available. TagBar shouldn't complain if you have a working jsctags.
Third, for its own tag-related functionalities (:tag, :tselect, <C-]>, etc.), Vim only relies on the presence of one or more physical tags files. Whether those files are generated by ctags, jsctags or whatevertags is not a concern as long as they are generated correctly.
And now we arrive at the fourth point which is where the pain really is: neither Tagbar nor its older cousin TagList actually use a physical tags file. They tap ctags or jsctags directly without even trying to use or update any existing tags file. Because of that, and the fact that TagBar only deals with the current buffer, your own physical tags file (and thus Vim's tag-related functionalities) and TagBar are almost guaranteed to be out of sync. Even if you use the same indexer.
So… I'd advise you to use either ctags or jsctags and forget anything about syncing TagBar's and Vim's tags-related stuff as both things are completely separate:
Use TagBar to understand/navigate in your current buffer.
Use Vim's tag-related functionalities to move around your project.
Random thoughts…
For Vim to locate your tags file(s) easily, you should put this line in your ~/.vimrc:
set tags=./tags,tags;/
./tags means "look for a tags file in the directory of the current file", tags means "look for a tags file in the working directory", ;/ means "keep looking up and up until you reach /".
The TagBar wiki talks about a bug in jsctags, make sure that you are not concerned by it.
jsctags is better than ctags when you write crazy "modern" JavaScript with lots of callbacks and self-executing functions. If your JavaScript is more traditional, ctags may be enough.

What are the advatanges of less?

http://lesscss.org/
Yeah the language syntax looks cool, but are there any significant advantages?
Because from what I see there you actually write more CSS, and since the less stylesheet gets compiled by javascript you get a un-styled site for a few seconds...

That is if you use lesscss for client side sheet generation.
You can also use lesscss on the server side or even compile your sheets before hand.
For example, if you are using node.js you can generate your sheets on the server.
Additionally, if you want, you can use it as a way to write more manageable templates and then "compile" your sheets to generate a formed style sheet for when you are reading for production.
I think it makes perferct sense, because it means that while you are in development, you can change a collection of colors in a single hex color change. When you are ready to deploy, you can compile before hand and then distribute that style sheet with your project.
Think about manageability and using the command line compiler.
Command Line
http://lesscss.org/
Less comes with a binary, which lets you invoke the compiler from the
command-line, as such:
$ lessc styles.less
This will output the compiled CSS to stdout, you may then redirect it to a file of your choice:
$ lessc styles.less > styles.css
To output minified CSS, simply pass the -x option.

Why would you write more CSS using LESS? In the contrary, you write less. And the code that you write is way more verbose than normal css. You can create groups better, use inheritance... You just have a way better overview over your styling than you do with CSS.
My CSS-file for a project currently has over 800 lines... in LESS, it's just around 150.
And to the javascript-compiling: I generally use an offline compiler and upload just the compiled CSS, so I don't have to fuss with the JS client.

You can use a precompiler (ruby's got a cool one, got it running under win).
Try writing a site in less and you'll never want to get back to normal css again :) my own experience...

There is actually a LESS PHP compiler, which I use all the time. I think the best way to work out why it's better than CSS is to use it yourself, but I'll summarise some things:
If you have theme colours used all over the place, for example, and want to change them, it's very easy with LESS's variable system.
You actually write less markup than vanilla CSS, due to not having to retype long selector strings.
If you want to, say, use cross-browser rounded corners, the mixins reduce repetition to almost nothing; all you do is create a class with style definitions in it, and then simply "mix it in". Essentially it's a re-usable function. If you need to change something specific to all rounded corners elements, you just change the single function.
Yeah the language syntax looks cool Personally I think this is rather dismissive and ignorant; if you have a look around the LESS examples, you will see it is miles ahead of normal CSS in terms of usability and speed of development.
There are more reasons, but these should be enough to get you started.

Using text/html javascript templates in production code

Has anyone implemented John Resig's micro-templating in production code? In particular I'd like to know if embedding the templates with <script type="text/html"> has caused any problems.
A bit more information: here's the sort of thing I'm worried about:
My users are mostly in corporate environments - could an unusual proxy server mangle or strip out the templates
The embedded templates seem to fail W3C validation. What if, for instance, the next version of IE decides it doesn't like them?

I abandoned inlining templates through scripttags as my view layer would create redundant duplicate script templates. Instead I placed the templates back into the JavaScript as string:
var tpl = ''.concat(
'<div class="person">',
'<span class="name">${name}</span>',
'<span class="lastname">${lastName}</span>',
'</div>'
);
I used the string concat trick so I could make the template string readable. There are variations to the trick like an array join or simply additive concatenation. In any case, inline script templates works and works well, but in a php/jsp/asp view layer, chances are you will create redundant duplicate script templates unless you do even more work to avoid it.
Furthermore, these templates become rather complex the more logic you have to add to them, so I looked further and found mustache.js which imo. is far superior and keeps the logic (conditions and dynamic variable definitions) in the JavaScript scope.
Another option would be to retreive template strings through ajax, in which case you can put each template inside it's own file and simply grant it a .tpl extention. The only thing you have to worry about is the http request roundtrip, which should not take too long for small .tpl files and is imo. insignificant enough.

While I haven't used #jeresig's micro-templating, I did roll my own which uses <script type="text/html">. I've used it in a couple of (albeit basic) production sites without problems. The HTML5 spec itself refers to something similar (near the end of the section I've linked to). AFAIK the block only executes when the MIME type is a recognized script type, otherwise the block is just part of the document.

I have actually, and it works great. Though only for webkit powered browsers so can't vouch for others. But what problems are you expecting? The approach is simple and I can't think of how it might break.

If you're using jQuery I can recommend jQuery templates instead:
http://github.com/nje/jquery-tmpl
I think it's supposed to be introduced officially in jQuery 1.5
BTW. I think both Resigs script and jQuery templates relies on using innerHTML, so as long as that works in your browser you should be OK.