For context I am trying to get a webserver up and running with low specs (think Raspberry Pi level) and want it to serve text content to as many users as possible.
I have tried looking up general tips, and this was helpful...but I can't believe that it's completely pointless to use Javascript to de-encode a website that's been compressed. If I took a simple cipher to replace each instance of '<div' with a special character I know I would never serve (some of these maybe), wouldn't I already be saving some bandwidth? A JS search and replace on the client side can't take more than a fraction of a second.
So my question is are any of these novel methods for speeding up simple websites worth it in the end? Are there other methods I am not considering?
The http-protocoll supports gzip compression natively. There is no further benefit in replacing
"<div " with "x". In order to improve performance you can tell your web-server to serve files that are already gzip-compressed. I did that once and it cut down page loading time about 10 milliseconds on a shared hoster using apache.
Related
Application:
I wish to publish a web-application that takes input strings, searches for the string in about 5,000 plain-text files and returns the name of the files with matches. Each text file is about 4MB (uncompressed).
Problem:
In PHP, I could use exec(grep -l pattern dir/* ) and get the job done. However, for cost reasons, I would go for a shared web-hosting plan which normally do not allow for executing programs.
Could you please suggest any alternative to grep for web environment?
I have understood following so far:
A binary program file for any grep-alternative (e.g sift) could work. However, the problem of executing on a shared server would remain.
PHP function preg_match is inappropriate considering a large number of files and their size.
I am open to implementations of grep-like function in other languages (e.g perl or javascript). However, I am not sure if the performance would be comparable to grep and whether the problem of execution would still remain.
I have tried looking for different web-hosting providers and understood that a virtual-private server (VPS) might be the solution. However, the price for a VPS plan by all hosting providers I have come across is unaffordable.
Any solutions or guidance for this problem?
Possible solutions depend on what your hosting provider offers and your budget.
Will you have a RDBMS available? You could then use full texts search which many offers. If not you could use SQLite, which has full texts search support.
If you have to stick to low tech solutions, than the PHP solution linked on the right might work for you.
Perl has a File::Find module, which you could use.
Background:
I am building a node.js-based Web app that needs to make use of various fonts. But it only needs to do so in the backend since the results will be delivered as an image. Consequently, the client/browser does not need access to the fonts at all in my case.
Question:
I will try to formulate the question as little subjective as possible:
What are the typical options to provide a node.js backend with a large collection of fonts?
The options I came up with so far are:
Does one install these hundreds or thousands of fonts in the operating system of the (in my case: Ubuntu) server?
Does one somehow serve the fonts from a cloud storage such as S3 or (online) database such as a Mongo DB server?
Does one use a local file system to store the fonts and retrieve them?
...other options
I am currently leaning towards Option 1 because this is the way a layman like me does it on a local machine.
Without starting a discussion here, where could I find resources discussing the (dis-)advantages of the different options?
EDIT:
Thank you for all the responses.
Thanks to these, I noticed that I need to clarify something. I need the fonts to be used in SVG processing libraries such as p5.js, paper.js, raphael.js. So I need to make the fonts available to these libraries that are run on node.js.
The key to your question is
hundreds or thousands of fonts
Until I took that in there is no real difference between your methods. But if that number is correct (kind of mind-boggling though) I would:
not install them in the OS. What happens if you move servers without an image? Or move OS?
Local File system would be a sane way of doing it, though you would need to keep track manually of all the file names and paths for your code.
MongoDB - store file names+paths in the collection..and store the actual fonts in your system.
In advent of moving servers you would have to pick up the directory where all the actual files are stored and the DB where you hold the file name+paths.
If you want you can place it all in a MongoDB but then that file would also be huge, I assume - that is up to you.
Choice #3 is probably what I would do in such a case.
If you have a decent enough server setup (e.g. a VPS or some other VM solution where you control what's installed) then another option you might want to consider is to do this job "out of node". For instance, in one of my projects where I need to build 175+ as-perfect-as-can-be maths statements, I offload that work to XeLaTeX instead:
I run a node script that takes the input text and builds a small but complete .tex file
I then tell node to call "xelatex theFileIJustMade.tex" which yields a pdf
I then tell node to call "pdfcrop" on that pdf, to remove the margins
I then tell node to call "pdf2svg", which is a free and amazingly effective utility
Then as a final step mostly to conserve space and bandwidth, I use "svgo" which is a nodejs based svg optimizer that can run either as normal script code, or as CLI utility.
(some more details on that here, with concrete code here)
Of course, depending on how responsive a system you need, you can do entirely without steps 3 and 5. There is a limit to how fast we can run, but as a server-side task there should never be the expectation of real-time responsiveness.
This is a good example of remembering that your server runs inside a larger operating system that might also offer tools that can do the job. While you're using Node, and the obvious choice is a Node solution, Node is also a general purpose programming language and can call anything else through spawn and exec, much like python, php, java, C#, etc. As such, it's sometimes worth thinking about whether there might be another tool that is even better suited for your needs, especially when you're thinking about doing a highly specialized job like typesetting a string to SVG.
In this case, LaTeX was specifically created to typeset text from the command line, and XeLaTeX was created to do that with full Unicode awareness and clean, easy access to fonts both from file and from the system, with full OpenType feature control, so would certainly qualify as just as worthwhile a candidate as any node-specific solution might be.
As for the tools used: XeLaTeX and pdfcrop come with TeX Live (installed using whatever package manager your OS uses, or through MiKTeX on Windows, but I suspect your server doesn't run on windows) pdf2svg is freely available on github, and svgo is available from npm)
I'm looking for a robust way of creating a zip archive on the fly from information on a given page and making this available for download. Client-side zipping is a must since my script runs from a bookmarklet.
My first approach while I was more concerned with writing the rest of the script was just to post the information to a few lines of PHP running on my local server which zipped it and sent it back. This is obviously not suitable for a bookmarklet worth sharing.
I found JSZip earlier today, and I thought that'd be the end of it. This library works great when it works; unfortunately, the archives I'm creating frequently exceed a couple of MBs, and this breaks JSZip. (Note: I've only tested this on Chrome.)
Pure JS downloads also have the limitation of funky names due the data URI, which I intended to solve using JSZip's recommended method, using Downloadify, which uses Flash. This made me wonder whether the size limitations on JS zip generating could be / have been overcome by using a similar interplay of Flash & JS.
I Googled this, but having no experience with Actionscript I couldn't figure out quickly whether what I'm asking is possible. Is it possible to use a Flash object from JS to create relatively large (into the 10s of MBs) zip file on the client-side?
Thanks!
First of all some numbers:
Flash promises that uploads will work if the file is smaller than 100 Mb (I don't know whether it means base 10 or base 16).
There are two popular libraries in Flash for creating ZIP archives, but read on first.
ZIP archiver is a program that both compresses and archives the data, and it does it in exactly this order. I.e. it compresses each file separately and then appends it to the entire archive. This yields worse compression rate but allows for iterative creation of the archive. With the benefit being that you can even start sending the archive before it is entirely compressed.
An alternative to ZIP is first to use a dedicated archiver and then to compress the entire archive at once. This, some times can achieve few times better compression, but the cost is that you have to process the data at once.
But Flash ByteArray.compress() method offers you native implementation of deflate algorithm, which is mostly the same thing you would use in ZIP archiver. So, if you had implemented something like tar, you could significantly reduce the size of the files being sent.
But Flash is a single-thread environment, so, you would have to be careful about the size of the data you compress, and, probably, will have to find it out empirically. Or just use ZIP - more redundancy, but easier to implement.
I've used this library before: nochump. Didn't have any problems. Although, it is somewhat old, and it might make sense to try to port it to use Alchemy opcodes (which are used for fast memory access significantly reducing the cost of low-level binary arithmentic operations such as binary or, binary and etc.) This library implements CRC32 algorithm, which is an essential part of ZIP archive and it uses Alchemy - so it should be considerably faster, but you would have to implement the rest on your own.
Yet another option you might consider is Goole's NaCl - there you would be able to choose from archiver and compression implementations because it essentially runs the native code, so you could even use bz2 and other modern stuff - unfortunately, only in Chrome (and users must enable it) or Firefox (need plugin).
PageSpeed and Yslow suggest that to combine javascripts file to reduce HTTPRequest. But this is becuase (I think) pre ie8 browser has no more than 2 serverhost connection.
But nowaday, browser has 6 serverhost connections, which means it has download javascripts in parrallel. So let say we have 1MB of javascript, should we break it down into 6 different files in similar size to obtain max download speed? Please let me know.
Micahel.S
No, because each HTTP request involves overhead (less if pipelining is used)
The answer to your question is no. However, assuming you are able to serve your content in a completely isolated environment where only IE8 is used (like company intranet), then the answer to your question becomes: no.
Since you aren't designing for IE6-7 I assume you are in an isolated environment (otherwise you are making a poor designing decision). In this environment, yes you might see small benefits from breaking down JavaScript files, but I recommend against it.
Why? Since you are optimizing for speed, I assume you are putting JavaScript at the bottom of the body tag in your HTML document, in order to prevent JS from blocking download of DOM. This is a fundamental practice to make the page appear to be loading faster. However, by placing the content in the bottom of the body, your question becomes moot. Since the DOM is no longer being blocked by the script tags, whatever speed benefits you could achieve by using parallel downloading would be lost on the user because they see the page load before the browser even requests the JavaScript files.
tl;dr: There is no practical speed advantage to break JS into multiple files for parallel downloading.
Splitting the files won't make too much of a difference really. If you want to see performance gains in terms of download times for your Production environment what I always do is use YUI Compressor (http://developer.yahoo.com/yui/compressor/) to get my JS file size down as small as possible then serve a gzipped version of the js to browsers that support it.
In a development environment you shouldn't be too worried about it though. Just split them logically based on their purpose so they're easier to work on then bring them all together into one file once you're ready and optimize it for production.
Most browsers will cache your JavaScript files anyway, so after the first page load, it won't matter.
But, you should split your JavaScript logically and in a way that would most help you in development, since browsers vary in the number of simultaneous connections they allow.
For speed, you can obfuscate your code (via a minimization routine) and serve it in a way no human would have the patience to read.
Is it really possible to achieve grade "A" in yslow for all things for a dynamic and CMS(PHP/Asp.net) based websites? and using same server.
(source: haacked.com)
http://developer.yahoo.com/yslow/help/index.html#performance_view
Yes, I guess it is possible to achieve this on one server, except of course for the CDN part which relies on an external service. You'll probably need full control over your server to configure things like ETags and such.
I think it's rarely worth the effort to fulfill all this literally down to the last percent except if you're a huge site like Google or Yahoo themselves, where every saved byte can mean tens or hundreds of thousands in savings. Just get a proper grade so things work fast and reliably - much like in school :)
Yes. First of all, try making any and all of your JS external, and load it on demand, only preloading the components that you really need. Then monitor when each javascript file is loaded, in order. Run that through JSBuilder (JavaScript packaging and compressor tool).
Turn on GZIP on your server. Gzip compression was able to reduce my static filesizes (css, js, etc.) by 73.43%.
Cache, cache, cache. Anything that doesn't change between application deployments needs to have a far-future expires header.
If you can afford it, serve your files from a cdn. They are distributed networks that make delivering content easier.
Get rid of your cookies, or combine them by encoding their values in JSON or using a caching service server-side to cache the values, and only store the cache key in the cookie. That way you only have one cookie instead of hundreds.
Put your css at the top, and optimize it by stripping out any unused selectors and properties.
Oh, and consider switching to a thin client...reloading the web page is so 1999. Using a thin client allows you to try different page download optimization techniques, and decouples your view (the web client) from your server and middleware api, allowing you to develop a front end in just about any RIA environment of your choice. You can go extremely lightweight with JQuery, or go with the more robust pre-built UI's of Ext or Dojo.
Reduce the amount of unused HTML. Tables are evil unless absolutey necessary or inserted into the dom after page load.
I'm sure that some of this will require some major re-work, which your application architecture and developer skill set may not be geared for at this time. The good news is that you can improve your user-experience just by cacheing cookies server-side (like I mentioned above), gzipping your static components, and combining and minifying any and all JS, and optimizing your CSS and layout, without getting into re-structuring your web application.
Sure, why not?
Each item follows through to a link with more details on how to acheive a higher grade
Yes, it is possible.
Here is a guided success example of an optimisation of a Typo3 installation. Take a look at this Yahoo! page for optimisation goodies.
Is it worth trying to optimise for bandwidth reduction and server responsiveness? Sure!
A lot of people connect through hand-held devices with expensive mobile phone plans. The school system where I live even have limited and expensive broadband plans. Waiting for a website to load is a waste of time.