Almost 6 months ago, I asked a question on stackoverflow "Software to help in log analysis?"
Please look at that question before reading ahead.
It turned out that there is no good software available currently that can intermix log files based on timestamps and present them in a good UI.
I wanted to take initiative and develop something and open source it once its completed.
Earlier, I worked around by writing a quick and dirty piece of code in c++, that would generate a tab separated file (like csv but tab separated) which I would later opened in Excel.
I am not satisfied with my c++ code for the following reasons:
1. It totally depends on Excel to view the output file later.
2. Since there is no UI involved, its not easy to write its commandline everytime.
3. Because of the learning curve of the commandline, its not so much sharable with other team members (and the world).
For the above reasons (and a few more), I was thinking to develop that as a web solution. That way I can share the working instance with everyone.
What I have in mind is a web based solution something like this:
The user will be able to give the input log files using HTML5's File API.
And then user would probably tell the format of the timestamp associated with each log file.
Thereafter, the javascript would process those log files into intermixed HTML output in a table.
I am just a beginner in web based technologies. So I need your help in determining if this would be the best way to go about it?
I want a web solution, but that doesn't mean I want user to upload his log files for backend processing. I want a web-based client only solution.
Thanks for your inputs.
EDIT: Based on comment below by Raynos
#bits You do realise that browsers
were not meant to handle large pieces
of data. There was
stackoverflow.com/questions/4833480/…
which shows that this can cause
problems.
I feel that doing this in browsers isn't the best deal. Probably, I should explore backend based solutions.
Any ideas or suggestions?
Your looking for an online diff tool which takes n files containing a list of timestamps in some order including a extra data to be displayed in place but not parsed in the diffing.
The file upload would involve
<input id="upload" type="file">
Along with snippets of javascript
$("#upload").change(function(files) {
var files = this.files;
for (var i = 0; i < files.length; i++) {
(function() {
var file = files[i];
var reader = new FileReader;
reader.onload = function(e) {
var text = reader.result;
console.log(text);
};
reader.readAsText(file);
}());
}
});
See live example.
So you have all the text you just need to work on a parser. I hope that helps a bit.
As for the markup of the diff I would suggest something like:
<table>
<!-- one tr per unique timestamp -->
<tr>
<!-- one td/textarea per file -->
<td> <textarea /> </td>
<td> <textarea /> </td>
</tr>
...
</table>
I would recommend making this a template and using a template engine to do some of the heavy lifting.
Let's say we want to use jquery-tmpl.
Here's an example to get you started. (I spend zero time on making it look good. That's your job).
All that's left is generating JSON data to insert into the template.
So given your file input you should have an array of fileTexts somewhere.
We want to have some kind of deliminator to split it up into individual time stamp records. For simplicities sake let's say that the new line character would work.
var fileTexts = [file];
var regex = new RegExp("(timestampformat)(.*)");
for (var i = 0; i < fileTexts.length; i++) {
var text = fileTexts[i];
var records = text.split("\n");
for (var j = 0; j < records.length; j++) {
var match = regex.exec(records[j]);
addToTimestamps(match[1], match[2], i);
}
}
function addToTimestamps(timestamp, text, currFileCount) {
console.log(arguments);
// implement it.
}
As per example.
These are the basic building blocks. Get the data from the File API. Manipulate the data into a normalised data format then use some kind of rendering tool on the data format.
This would be fairly easy to do using javascript. You mentioned above using the html5 file api, and that is a great place to start (html file api), you can use unobtrusive javascript to have a callback fire when the a file is uploaded. Inside the callback you could use any of the great javascript templating libraries to construct a table of elements from the uploaded file. Then on subsequent file uploads, you could dynamically interleave them into the table using their timestamp. Detecting the timestamp using js regular expressions would be fairly straightforward, and reasonably efficient if you used a compiled form.
This is a fairly high level answer to the question, and if you have any questions about particular details, I'd be happy to answer those as well.
Hope this helps
Related
Intro
Hi, I was looking for answer in the whole Internet (in some way I kind of feel that I know every question in Stack Overflow), but the answers were never appropriate to what I'm looking for. I was trying to avoid posting question here, but situation forced me to do this.
Sorry if the answer is simpler than I think.
I'm in the middle of building my first app in Electron using JavaScript. I think that I should describe it in few words, so flam:ngo™ (which is projects name) should work like this:
User will upload two files:
file with tables (like XLSX or DOC)
file with data and blank spaces (which will be used as a template)
App will import from tables.
Now app should let user choose which rows he's interested in and where in uploaded file he wants them to be placed.
flam:ngo save document in PDF (or DOC).
Clue
Right now I need solutions just for myself and in little simpler form. For now I need flam:ngo just to work with one specify XLSX and with one DOC template, but I stuck. I know which rows in document I will always need, but I don't know what should I write to specify in JS's code that I need exactly this ones (like hey, app, pick only this one, this one and maybe this one) while JS is reading file and I don't know how to create new DOC (or PDF) file, how to write data in specified blank spaces and then at the end: how to save it in direction which I should choose for every time I'm using an app - everything in one, maybe two, processes.
Ending
Could you, please, help me: for now I have implemented file uploader which is importing file in XLSX and which is saving it as CSV, XML oraz HTML. What should I do to keep moving forward?
I really appreciate your help!
PS. For better explanation:
For now this should look like this:
1. "template.docx" is uploaded in the core
2. user uploads .xlsx
3. app reads tables and select rows chosen in code
4. app puts data from chosen rows in specify places
5. app saves file > new life of the app :)
Ok, so it's been a while and app has been already written by me. Maybe this will help someone in the future:
Solution
Packages that helps me with this issue was:
js-xlsx which converts my file to simple HTML file, where cells from my XLSX template file have always the same ID (which was important for me to work with document templates).
officegen which helps me write brand new document based on earlier prepared template.
Rest of it was just Vanilla JS.
I have been given a PDF file with a form. The form is not formatted as a table. My requirement is to extract the form field values, and write them to a CSV file which can be imported into Excel. I have tried using the automated "Merge data files to Spreadsheet" menu item in Acrobat Pro, but the output includes both the labels and form field values. I am interested in mostly just the form field values.
I would like to use JavaScript to extract the form data, and instruct JavaScript how to write the CSV (since I know what the end spreadsheet should look like). I got as far as extracting the form fields:
this.getField("Today_s_Date").value;
And following this post: How to write a text file in Acrobat Javascript , I tried to write to CSV using:
var cMyC = "abc";
var doc = this.createDataObject({cName: "test.txt", cValue: cMyC});
but I get the following error:
"SyntaxError: syntax error
1:Console:Exec"
Ideally, I do not want to use an online third party tool to do this, because the data is sensitive. But please let me know if you have suggestions. The ideal output will be a CSV file that an end business user can open in Excel to see the spreadsheet format of her choice.
Has anyone done this before? Open to hearing any alternative solutions as well. Thanks in advance!
Your code should work, make sure you are selecting the entire code when running it in the console.
For security reasons you are limited in what you can output from Acrobat without user interaction. There is a good discussion of what can be output from PDF's here, and if you haven't already, be sure to check out what's possible with exportDataObject() in the reference.
An example to get you going -- you could place a button on the form that would iterate through each of the fields in the form, adding them to an array that could then be output as a csv.
Something like:
var fieldValues = [];
for (var i = 0; i < this.numFields; i++)
fieldValues.push(this.getField(this.getNthFieldName(i)).value);
this.createDataObject('output.csv', fieldValues.join());
this.exportDataObject({ cName:'output.csv', nLaunch:'2'});
In this example the .csv would be opened as a temporary file by the default csv program on the machine. Alternatively you could omit nLaunch, and give the user a file save dialog.
I have a Google Spreadsheet and would like to encrypt the content of a few cells (I do not care which encryption method is being used as long as there is an equivalent decryption method for iOS).
Unfortunately there are no built-in encryption functions in Google Apps Script.
For this reason I would like to use a open source Javascript library like Crypto-JS and sjcl.
How can I use one of these libraries with Google Apps Script?
In the Google Apps Script documentation, I have not found any clue on how to use external JavaScript libraries with my Google Apps Script.
Well I'll say this, because this is the method that I used with Date JS. You can do the following:
Download the source .js file(s).
Open the .js file(s) in a text editor
Copy/paste all code into a new Script Project
here you can "recreate" the original .js files (copy/paste source individually) with the same names
Include the project key of that Script Project as a library of the project in which you want to use those functions.
Even if the projects are open-source you will want to make sure you comply with the licenses of those projects if you are going to use them.
This is basically a small "hack" around not being able to upload .js files into GAS Projects. Assuming that the JS is standard, this method will work with Google's system.
The other option is to simple find a light-weight one- or two-function crypto package, or a single crypto algorithm like AES-128 (taht you are given permission to use, of course). It really depends on how much encryption you want, if you need to reverse the cipher text to get the plain values, etc.
If this is a for some kind of password system, I would recommend using a simple hash. For example:
function stringHash (someString) {
var hash = 0;
if (this.length == 0) return hash;
for (i = 0; i < this.length; i++) {
char = this.charCodeAt(i);
hash = ((hash << 5) - hash) + char;
hash = hash & hash;
}
return Math.abs(hash); // Personally I don't like negative values, so I abs'd it
}
in which you would ask for a user's password, and if the password hash matched the hash stored in the spreadsheet or wherever, then you would validate. You can use this to simulate logging into a UiApp GUI, for example: store usernames/password hashes in a database and validate a user before loading the "real" app.
However, as Serge mentioned, Spreadsheets will contain revision history of the original value before it was hashed, as well as the value after it was hashed. If you want to avoid this, use ScriptDB.
PS - in addition to this work-around, I'll say that it's not currently possible to "import" a non-GAS code library into your Script Project, unless you manually copy the source file-by-file into your Script Project. There may be a feature request on the Issue Tracker already, if not you can create one and I'll star it.
EDIT: As per request, I've included an open source AES encryption "package" (contains base64 as well, which is nice) in the answer, to act as a reference for others who want to encrypt in GAS. Make sure you follow the author's request, which is to retain his original copyright and link back to the source.
Other than the AES I linked and the simple hash (equivalent to Java's String.hashCode()), whose resource can be found here, there is Crypto-JS as you mentioned in your question and, if you took the time to fully copy/paste all the code (assuming that agrees with the terms of the license - I haven't read it), you could use that by the steps I described in the top half of my answer.
MD5 in Javascript is also an algorithm that you could use. If you use the code in md5.js which is located at the top of the page, you'll have what you need. Again, make sure you're following licensing rules if you use it.
Personally I would probably just use the hash and the base-64 patterns, as most of what you would use this encryption for is probably not incredibly important. AES might take a bit longer to compute - you can probably benchmark it yourself to see if it will cause major problems with triggers running for an extended period of time, but I doubt it would be a problem anyway.
Note: base-64 is 2-way, so is AES. MD5 is a type of hash, and the simple hash function I provided is also (of course) a hash. Hash functions are one-way. So if you need two-way functionality (encrypt/decrypt), then use base-64 or AES. Base-64 is essentially the kid version of AES. And the simple hash function is the kid version of MD5. Keep this in mind :)
Edit again: I'm not familiar with iOS development or its internals, but it seems to me that iOS can at least do some cryptographic operations. You may want to read more into those methods though, because I'm not really sure how you're putting GAS and iOS together; I can't give you any more help in that area unfortunately.
The functions above don't work for me. Here is something what you can copy and paste into google sheets (spreadsheet) script editor
function enc(str) {
var encoded = "";
for (i=0; i<str.length;i++) {
var a = str.charCodeAt(i);
var b = a ^ 123; // bitwise XOR with any number, e.g. 123
encoded = encoded+String.fromCharCode(b);
}
return encoded;
}
This is what you get when you use it =ENC in your spreadsheet
Based on this post here
External libraries can be used by using JavaScript's built in eval() function
(ex. eval(UrlFetchApp.fetch('path/to/library'))).
Of course, the library must have no dependencies for this to work.
I'm trying to use localization in my project but I can't find a way to access my resx files from javascript. I have been looking around a bit and I don't think the 'AJAX call' method would be ideal for my project since I have quiet a lot of string that need to be fetched and it would just have to spam the server hard!
if I just put it in my HTML then it works with this code:
#using Resources
<p>#Html.Raw(ISt_Localization.January)</p>
I guess one of the things I could do is put all the strings in a hidden div and then get the content from the divs in my javascript but this wouldn't be very effective..
I had a similar situation and in my case, I created a separate partial view which only contained a javascript block where I put all the resource strings required for use in client side logic. Every resource string was defined as a javascript variable. You could also create an associative array.
In your partial view:
var Resources = {
January : "#Html.Raw(ISt_Localization.January)",
February : "#Html.Raw(ISt_Localization.February)",
...
};
You can also try the below thing directly
#using Resources
<script>
var value = '#Resource.January';
/* work with value
.......
.....
*/
</script>
I took a totally different approach.
I want to have the resource strings required by my Javascript files be part of my resx files.
Every key in my resource file which starts with js has to become available in Javascript.
In global.asax, in Application_OnStart I build Javascript files for all supported languages on the fly.
Because this only happens at the start of the application, it does not matter if it takes a few seconds.
Advantages:
All translations in one place (in the rex files which you use for your .NET application (C# or VB), but also for your Javascript code
Always up to date
Very fast, because we are going to use variables to get the translations
Building the Javascript file is easy.
Iterate through all key value pairs in all resx-files. Just pick out the keys starting with _js_.
Save the key value pair in the Javascript file.
So if the key value pair in the resx-file (languageSupport_es.resx) is '_js_Hello', 'Hola',
I write in my Javascript file (languageSupport_es.js) var Hello = 'Hola';
So alert(Hello) will give you 'Hola' if the current language is Spanish.
The only thing I now have to take care of, is using the right 'language Javascript file' is loaded before my other Javascript files.
So if the language is Spanish, I ONLY include my 'Spanish language Javascript file' (languageSupport_es.js) first.
Easy no? If somebody is interested, I can show some example code...
Is there any way to read a file line by line in javascript, specifically this file which is a dictionary. I was trying to build a replica of a java anagram solver I made a few months ago, but hit this problem of not being able to read a file line by line. I could download the file and store it locally if that would make any difference to being able to read it.
Use YQL:
http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20html%20where%20url%3D%22http%3A%2F%2Fdocs.oracle.com%2Fjavase%2Ftutorial%2Fcollections%2Finterfaces%2Fexamples%2Fdictionary.txt%22&format=json&diagnostics=true&callback=cbfunc
Here's what the fiddle looks like:
window.callback = function(a) { window.file = a.query.results.body.p; go(); };
$.getScript('http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20html%20where%20url%3D%22http%3A%2F%2Fdocs.oracle.com%2Fjavase%2Ftutorial%2Fcollections%2Finterfaces%2Fexamples%2Fdictionary.txt%22&format=json&diagnostics=true&callback=callback');
window.go = function() {
var terms = file.split(' ');
for (var i = 0; i < 100; i++)
console.log(terms[i])
};
The fiddle only does the first 100 but you get the idea (I hope).
In most circumstances, you could just read the file into memory and then parse it into lines. If you read the whole thing into memory with an ajax call, you could just use data.split("\n") to convert it to an array of lines.
You should initiate Ajax Request for such Operation. Reading a file line by line although is not recommended via Ajax, as you will end up create loads of server requests in the process;as JavaScript is all about Client side and limited access also. Ajax request to server is recent years add-on to it.
Definitely you are searching some information using some keywords; so it would be wise if you add the search logic on server functions; call the specific function via Ajax and return back result-set to the browser. That function could be a file that generates result or, a web service. You choose your flavor.
Alternate option would be to re-code the file information to JSON (about JSON) at start and let it roll to client js script. I would not recommend XML for this as its gonna eatup loads of browser memory in the processing terms. Been there! :(. Since JavaScript has native support to JSON, it will go smooth. Warning this exposes data to local.