We Keep Coding

Render an image byte stream on angular client side app

I have a NodeJS / Express RESTful API that proxies requests from an Active Directory LDAP Server. I do this because LDAP queries tend to be slow. I use the RESTful API to cache and refresh data intermittently. I recently attempted to add the thumbnail photo. In research it appears the library that I am using ldapjs is converting the native ldap byte array to a string.
Example of what this looks like:
\ufffd\ufffd\ufffd\ufffd\u0000\u0010JFIF\u0000\u0001\u0000\u0001\u0000x\u0000x\u0000\u0000\ufffd\ufffd\u0000\u001fLEAD
Technologies Inc.
V1.01\u0000\ufffd\ufffd\u0000\ufffd\u0000\u0005\u0005\u0005\b\
Due to this fact the image does not render correctly on the angular client app. So based on my research, here are some attempts in correcting this problem:
Convert the string to a byte array using different methods (See code examples)
Modify the ldapjs library to render the data as a byte array in the RESTFUL as in the following, then bind the byte stream to the angular page:
https://github.com/joyent/node-ldapjs/issues/137
https://csjdpw.atlassian.net/wiki/spaces/~Genhan.Chen/pages/235044890/Display+LDAP+thumbnail+photos
html binding:
<div>
<img *ngIf="userImage" [src]="userImage" alt="{{dataSource.sAMAccountName}}">
</div>
controller:
public get userImage() {
let value = null;
if(this.dataSource.thumbnailPhoto) {
const byteArray = this.string2Bin(this.dataSource.thumbnailPhoto);
const image = `data:image/jpeg;base64,${Buffer.from(byteArray).toString('base64')}`;
value = this.domSanitizer.bypassSecurityTrustUrl(image);
}
return value;
}
private string2Bin(str) {
var result = [];
for (var i = 0; i < str.length; i++) {
result.push(str.charCodeAt(i));
}
return result;
}
and
alternate version of controller:
public get userImage() {
let value = null;
if(this.dataSource.thumbnailPhoto) {
const byteArray = new TextEncoder().encode(this.dataSource.thumbnailPhoto);
const image = `data:image/jpeg;base64,${Buffer.from(byteArray).toString('base64')}`;
value = this.domSanitizer.bypassSecurityTrustUrl(image);
}
return value;
}
another alternate version of controller:
public get userImage() {
let value = null;
if(this.dataSource.thumbnailPhoto) {
const blob = new Blob( [Buffer.from(this.dataSource.thumbnailPhoto).toString('base64')], { type: 'image/jpeg' } );
const value = window.URL.createObjectURL(blob);
return value;
}
I expected a rendered image on the angular page but all I get is the non-rendered placeholder.
Here are the versions of the libraries I am using
Angular - 8.0.3
NodeJS - 10.15.0
ldapjs - 1.0.2
I am sure I am missing something, I am just not sure what it is. Any assistance would be appreciated.

So after some guidance provided by #Aritra Chakraborty, I checked the RESTful api source code. It appears to be a problem with a ldapjs library. When using the entry object conversion, it is doing something strange with the data to which it is not usable. I then realized, I had access to the entry raw format which is the byte array . Instead of trying to convert to base64 on the client, I moved this to the API. Then just mapped it back on the client binding and bang it worked.
Here is some example code:
RESTFul api
_client.search(this._search_dn, opts, (error, res) => {
res.on("searchEntry", (entry) => {
let result = {};
result.id = string_service.formatGUID(JSON.parse(JSON.stringify(entry.raw)).objectGUID);
result = Object.assign({}, result, entry.object);
if(entry.raw.thumbnailPhoto) {
result.thumbnailPhoto = entry.raw.thumbnailPhoto.toString('base64');
}
// The previous 3 lines did not exist previously
On the Angular 8 client I simplified the binding:
public get userImage() {
let value = null;
if(this.dataSource.thumbnailPhoto) {
const image = `data:image/jpeg;base64,${this.dataSource.thumbnailPhoto}`;
value = this.domSanitizer.bypassSecurityTrustUrl(image);
}
return value;
}
I hope someone finds some value out of this.

How do I write a LZ compressed string to text file using JXA?

I am trying to write a JXA script in Apple Script Editor, that compresses a string using the LZ algorithm and writes it to a text (JSON) file:
var story = "Once upon a time in Silicon Valley..."
var storyC = LZString.compress(story)
var data_to_write = "{\x22test\x22\x20:\x20\x22"+storyC+"\x22}"
app.displayAlert(data_to_write)
var desktopString = app.pathTo("desktop").toString()
var file = `${desktopString}/test.json`
writeTextToFile(data_to_write, file, true)
Everything works, except that the LZ compressed string is just transformed to a set of "?" by the time it reaches the output file, test.json.
It should look like:
{"test" : "㲃냆੠Њޱᐈ攀렒삶퓲ٔ쀛䳂䨀푖㢈Ӱນꀀ"}
Instead it looks like:
{"test" : "????????????????????"}
I have a feeling the conversion is happening in the app.write command used by the writeTextToFile() function (which I pulled from an example in Apple's Mac Automation Scripting Guide):
var app = Application.currentApplication()
app.includeStandardAdditions = true
function writeTextToFile(text, file, overwriteExistingContent) {
try {
// Convert the file to a string
var fileString = file.toString()
// Open the file for writing
var openedFile = app.openForAccess(Path(fileString), { writePermission: true })
// Clear the file if content should be overwritten
if (overwriteExistingContent) {
app.setEof(openedFile, { to: 0 })
}
// Write the new content to the file
app.write(text, { to: openedFile, startingAt: app.getEof(openedFile) })
// Close the file
app.closeAccess(openedFile)
// Return a boolean indicating that writing was successful
return true
}
catch(error) {
try {
// Close the file
app.closeAccess(file)
}
catch(error) {
// Report the error is closing failed
console.log(`Couldn't close file: ${error}`)
}
// Return a boolean indicating that writing was successful
return false
}
}
Is there a substitute command for app.write that maintains the LZ compressed string / a better way to accomplish what I am trying to do?
In addition, I am using the readFile() function (also from the Scripting Guide) to load the LZ string back into the script:
function readFile(file) {
// Convert the file to a string
var fileString = file.toString()
// Read the file and return its contents
return app.read(Path(fileString))
}
But rather than returning:
{"test" : "㲃냆੠Њޱᐈ攀렒삶퓲ٔ쀛䳂䨀푖㢈Ӱນꀀ"}
It is returning:
"{\"test\" : \"㲃냆੠Њޱᐈ攀렒삶퓲ٔ쀛䳂䨀푖㢈Ӱນꀀ\"}"
Does anybody know a fix for this too?
I know that it is possible to use Cocoa in JXA scripts, so maybe the solution lies therein?
I am just getting to grips with JavaScript so I'll admit trying to grasp Objective-C or Swift is way beyond me right now.
I look forward to any solutions and/or pointers that you might be able to provide me. Thanks in advance!

After some further Googl'ing, I came across these two posts:
How can I write UTF-8 files using JavaScript for Mac Automation?
read file as class utf8
I have thus altered my script accordingly.
writeTextToFile() now looks like:
function writeTextToFile(text, file) {
// source: https://stackoverflow.com/a/44293869/11616368
var nsStr = $.NSString.alloc.initWithUTF8String(text)
var nsPath = $(file).stringByStandardizingPath
var successBool = nsStr.writeToFileAtomicallyEncodingError(nsPath, false, $.NSUTF8StringEncoding, null)
if (!successBool) {
throw new Error("function writeFile ERROR:\nWrite to File FAILED for:\n" + file)
}
return successBool
};
While readFile() looks like:
ObjC.import('Foundation')
const readFile = function (path, encoding) {
// source: https://github.com/JXA-Cookbook/JXA-Cookbook/issues/25#issuecomment-271204038
pathString = path.toString()
!encoding && (encoding = $.NSUTF8StringEncoding)
const fm = $.NSFileManager.defaultManager
const data = fm.contentsAtPath(pathString)
const str = $.NSString.alloc.initWithDataEncoding(data, encoding)
return ObjC.unwrap(str)
};
Both use Objective-C to overcome app.write and app.read's inability to handle UTF-8.

Can't save/create files using Store.js

So I wanted to save a file on the client storage using Store.js.
I can change the date using store.set and i can log it to console to see the change, but then it's supposed to be saved in app data where it's not created.
I tried to get the Path where it's being saved and it's :
C:\Users\USER\AppData\Roaming\stoma2/Categories.json
I noticed that there is a "/" so I tried :
C:\Users\USER\AppData\Roaming\stoma2\Categories.json
and :
C:/Users/USER/AppData/Roaming/stoma2/Categories.json
But all 3 of them didn't work.
This is my Store.js :
const fs = require('browserify-fs');
var fs2 = require('filereader'),Fs2 = new fs2();
const electron = window.require('electron');
const path = require('path');
class Store {
constructor(opts) {
// Renderer process has to get `app` module via `remote`, whereas the main process can get it directly
// app.getPath('userData') will return a string of the user's app data directory path.
//const userDataPath = (electron.app || electron.remote.app).getPath('userData');
var userDataPath = (electron.app || electron.remote.app).getPath('userData');
for(var i=0;i<userDataPath.length;i++){
if(userDataPath.charAt(i)=="\\"){
userDataPath = userDataPath.replace("\\","/");
}
}
// We'll use the `configName` property to set the file name and path.join to bring it all together as a string
this.path = path.join(userDataPath, opts.configName + '.json');
this.data = parseDataFile(this.path, opts.defaults);
console.log(this.path);
}
// This will just return the property on the `data` object
get(key) {
return this.data[key];
}
// ...and this will set it
set(key, val) {
this.data[key] = val;
// Wait, I thought using the node.js' synchronous APIs was bad form?
// We're not writing a server so there's not nearly the same IO demand on the process
// Also if we used an async API and our app was quit before the asynchronous write had a chance to complete,
// we might lose that data. Note that in a real app, we would try/catch this.
fs.writeFile(this.path, JSON.stringify(this.data));
}
}
function parseDataFile(filePath, data) {
// We'll try/catch it in case the file doesn't exist yet, which will be the case on the first application run.
// `fs.readFileSync` will return a JSON string which we then parse into a Javascript object
try {
return JSON.parse(Fs2.readAsDataURL(new File(filePath)));
} catch(error) {
// if there was some kind of error, return the passed in defaults instead.
return data;
}
}
// expose the class
export default Store;
There might be a probleme fith js.writeFile() (well that's the source of probleme).
and this is my call :
//creation
const storeDefCat = new Store({
configName: "Categories",
defaults: require("../data/DefaultCategorie.json")
})
//call for the save
storeDefCat.set('Pizza',{id:0,path:storeDefCat.get('Pizza').path});
For now if possible,I might need to find another way to save the file.
And i tried : fs : It doesn't work for me for some reason (I get strange errors that they don't want to be fixed..) .
If anyone has an Idea then please I would be grateful.

So I managed to fix the probleme, Why fs was sending me errors about undefined functions?Why file wasn't getting created ? It has NOTHING to do with the code it self, but the imports...
To clearify, I was using :
const fs = require('fs');
And the solution is to make it like :
const fs = window.require('fs');
Just adding window. fixed all the problems .Since it's my first time using electron I wasn't used to import from the window but it seems it's necessary.And more over...There was no posts saying this is the fix.

Executing mongodb scripts via mongoid Rails

I have a mongo db script in a js file:
query.js
//conn = new Mongo();
//db = conn.getDB("dbName");
functionFoo = function (arg){
//----process arg
}
also I have an array of args known as args_array, (that I fetch from database using mongoid) for which I want to do something like this:
args_array.each do |arg|
//some how call functionFoo(arg) from the query.js file
end
is this possible in rails?
I am able to execute the file from terminal but I want to wrap it in my application so that I can use it from rails console.

I know this old question but in case you still need answer or any one else. This answer works with gem mongo ~> 2.3.
The key to answer you do not need mongoid in this case - in my case I use it for rails model, so I use mongoid (5.1.0) only to get DB connection db = Mongoid.default_client.database - or you can get/create database using mongo gem.
To execute javascript on database you need to call command method db.command({ eval: 'js' }) or db.command({ eval: 'function(n){return db.projects.find({name: n}).toArray();}', args: ['beskhai'], nolock: true })
To get the result you can call .documents db.command(...).documents, The return is a hash {retval: it will be return of you script, ok: is 1 if success} the return object of command call is [Mongo::Operation::Result] https://github.com/mongodb/mongo-ruby-driver/blob/master/lib/mongo/operation/result.rb.

I'm using MongoID 6.0.1, and it easy to query everything you want like that:
db ||= Mongoid.default_client.database
f = """
functionFoo = function (arg){
//----process arg
}
"""
result = db.command({:$eval => f, args: [arg1, arg2, ...arg_n], nolock: true})
#result_data = result.first['retval']
It not only a function, just every thing you want to do with command.
My example is:
db ||= Mongoid.default_client.database
f = """
var collectionNames = db.getCollectionNames(), stats = [];
collectionNames.forEach(function (n) { stats.push(db[n].stats()); });
stats = stats.sort(function(a, b) { return b['size'] - a['size']; });
return stats;
"""
result = db.command({:$eval => f, args: [], nolock: true})
#result_data = result.first['retval']

How to create streams from string in Node.Js?

I am using a library, ya-csv, that expects either a file or a stream as input, but I have a string.
How do I convert that string into a stream in Node?

As #substack corrected me in #node, the new streams API in Node v10 makes this easier:
const Readable = require('stream').Readable;
const s = new Readable();
s._read = () => {}; // redundant? see update below
s.push('your text here');
s.push(null);
… after which you can freely pipe it or otherwise pass it to your intended consumer.
It's not as clean as the resumer one-liner, but it does avoid the extra dependency.
(Update: in v0.10.26 through v9.2.1 so far, a call to push directly from the REPL prompt will crash with a not implemented exception if you didn't set _read. It won't crash inside a function or a script. If inconsistency makes you nervous, include the noop.)

Do not use Jo Liss's resumer answer. It will work in most cases, but in my case it lost me a good 4 or 5 hours bug finding. There is no need for third party modules to do this.
NEW ANSWER:
var Readable = require('stream').Readable
var s = new Readable()
s.push('beep') // the string you want
s.push(null) // indicates end-of-file basically - the end of the stream
This should be a fully compliant Readable stream. See here for more info on how to use streams properly.
OLD ANSWER:
Just use the native PassThrough stream:
var stream = require("stream")
var a = new stream.PassThrough()
a.write("your string")
a.end()
a.pipe(process.stdout) // piping will work as normal
/*stream.on('data', function(x) {
// using the 'data' event works too
console.log('data '+x)
})*/
/*setTimeout(function() {
// you can even pipe after the scheduler has had time to do other things
a.pipe(process.stdout)
},100)*/
a.on('end', function() {
console.log('ended') // the end event will be called properly
})
Note that the 'close' event is not emitted (which is not required by the stream interfaces).

From node 10.17, stream.Readable have a from method to easily create streams from any iterable (which includes array literals):
const { Readable } = require("stream")
const readable = Readable.from(["input string"])
readable.on("data", (chunk) => {
console.log(chunk) // will be called once with `"input string"`
})
Note that at least between 10.17 and 12.3, a string is itself a iterable, so Readable.from("input string") will work, but emit one event per character. Readable.from(["input string"]) will emit one event per item in the array (in this case, one item).
Also note that in later nodes (probably 12.3, since the documentation says the function was changed then), it is no longer necessary to wrap the string in an array.
https://nodejs.org/api/stream.html#stream_stream_readable_from_iterable_options

Just create a new instance of the stream module and customize it according to your needs:
var Stream = require('stream');
var stream = new Stream();
stream.pipe = function(dest) {
dest.write('your string');
return dest;
};
stream.pipe(process.stdout); // in this case the terminal, change to ya-csv
or
var Stream = require('stream');
var stream = new Stream();
stream.on('data', function(data) {
process.stdout.write(data); // change process.stdout to ya-csv
});
stream.emit('data', 'this is my string');

Edit: Garth's answer is probably better.
My old answer text is preserved below.
To convert a string to a stream, you can use a paused through stream:
through().pause().queue('your string').end()
Example:
var through = require('through')
// Create a paused stream and buffer some data into it:
var stream = through().pause().queue('your string').end()
// Pass stream around:
callback(null, stream)
// Now that a consumer has attached, remember to resume the stream:
stream.resume()

There's a module for that: https://www.npmjs.com/package/string-to-stream
var str = require('string-to-stream')
str('hi there').pipe(process.stdout) // => 'hi there'

Another solution is passing the read function to the constructor of Readable (cf doc stream readeable options)
var s = new Readable({read(size) {
this.push("your string here")
this.push(null)
}});
you can after use s.pipe for exemple

in coffee-script:
class StringStream extends Readable
constructor: (#str) ->
super()
_read: (size) ->
#push #str
#push null
use it:
new StringStream('text here').pipe(stream1).pipe(stream2)

I got tired of having to re-learn this every six months, so I just published an npm module to abstract away the implementation details:
https://www.npmjs.com/package/streamify-string
This is the core of the module:
const Readable = require('stream').Readable;
const util = require('util');
function Streamify(str, options) {
if (! (this instanceof Streamify)) {
return new Streamify(str, options);
}
Readable.call(this, options);
this.str = str;
}
util.inherits(Streamify, Readable);
Streamify.prototype._read = function (size) {
var chunk = this.str.slice(0, size);
if (chunk) {
this.str = this.str.slice(size);
this.push(chunk);
}
else {
this.push(null);
}
};
module.exports = Streamify;
str is the string that must be passed to the constructor upon invokation, and will be outputted by the stream as data. options are the typical options that may be passed to a stream, per the documentation.
According to Travis CI, it should be compatible with most versions of node.

Heres a tidy solution in TypeScript:
import { Readable } from 'stream'
class ReadableString extends Readable {
private sent = false
constructor(
private str: string
) {
super();
}
_read() {
if (!this.sent) {
this.push(Buffer.from(this.str));
this.sent = true
}
else {
this.push(null)
}
}
}
const stringStream = new ReadableString('string to be streamed...')

In a NodeJS, you can create a readable stream in a few ways:
SOLUTION 1
You can do it with fs module. The function fs.createReadStream() allows you to open up a readable stream and all you have to do is pass the path of the file to start streaming in.
const fs = require('fs');
const readable_stream = fs.createReadStream('file_path');
SOLUTION 2
If you don't want to create file, you can create an in-memory stream and do something with it (for example, upload it somewhere). ​You can do this with stream module. You can import Readable from stream module and you can create a readable stream. When creating an object, you can also implement read() method which is used to read the data out of the internal buffer. If no data available to be read, null is returned. The optional size argument specifies a specific number of bytes to read. If the size argument is not specified, all of the data contained in the internal buffer will be returned.
const Readable = require('stream').Readable;
const readable_stream = new Readable({
​read(size) {
​// ...
​ }
});
SOLUTION 3
When you are fetching something over the network, that can be fetched like stream (for example you are fetching a PDF document from some API).
const axios = require('axios');
const readable_stream = await axios({
method: 'get',
url: "pdf_resource_url",
responseType: 'stream'
}).data;
SOLUTION 4
Third party packages can support creating of streams as a feature. That is a way with aws-sdk package that is usually used for uploading files to S3.
const file = await s3.getObject(params).createReadStream();

JavaScript is duck-typed, so if you just copy a readable stream's API, it'll work just fine. In fact, you can probably not implement most of those methods or just leave them as stubs; all you'll need to implement is what the library uses. You can use Node's pre-built EventEmitter class to deal with events, too, so you don't have to implement addListener and such yourself.
Here's how you might implement it in CoffeeScript:
class StringStream extends require('events').EventEmitter
constructor: (#string) -> super()
readable: true
writable: false
setEncoding: -> throw 'not implemented'
pause: -> # nothing to do
resume: -> # nothing to do
destroy: -> # nothing to do
pipe: -> throw 'not implemented'
send: ->
#emit 'data', #string
#emit 'end'
Then you could use it like so:
stream = new StringStream someString
doSomethingWith stream
stream.send()