I am just starting to work with clojurescript on the node-js side. I'm using it for building command line tools that will run on node.
Now that I have my proof of concept already setup and doing more or less what I want is time to organize the code a bit better.
On JS when I need something similar to a configured http client I usually export a single function that accepts the basic parameters and returns an object with methods bound to that parameters (normally using the revealing module pattern). Something similar to creating a new instance on OOP.
Here is a small example of how I would do this on JS:
const request = require('request')
module.exports = (user, pass, baseUrl) => {
const client = request.defaults({baseUrl, auth: {user, pass}})
const getSomething = (name) => client.get('resources/' + name)
const createSomething = (name, options) => client.post('resources', {...})
return { getSomething, createSomething }
}
However on clojurescript I can't find a proper way of doing this. All defines are top level declarations computed at compile time, and making an structure such the one above would require to declare all my functions with a client parameter, then partially apply them and them use them on the body of the logic. This can be something like this:
(ns some-client [:require ["request" :as request]])
(defn get-something [client, name]
(.get client (str "resources/" name)))
(defn create-something [client, name, options]
(.post client (str "resources") {:name name :data: options}))
(defn make-client [usr, pass, baseUrl]
(let [client (.defaults request {:auth {:user usr :pass pass} :baseUrl baseUrl})]
{:get-something (partial get-something client)
:create-something (partial create-something client)}))
This may not look so bad, but as soon as you need to use it on another place where all the functions would require such client things start to get messy. You will need to accept the client on all the functions and if that other namespace is just a collection of functions that you will need to use on another place, you will be forced to follow the same schema of return a client creator, accept the client you depend on and make sure you pass it to every function that could need it. I can become as horrible as this:
(ns other-helper)
(defn trivial-stuff [client name bla]
(let [get-something (client :get-something)]
(get-something name))) ; make things like filtering and that
(defn more-trivial-stuff [client name bla]
(let [get-something (client :get-something)])
(get-something name)) ; make things like filtering and that
(defn non-trivial-stuff [client name bla]
(->>
(trivial-stuff client name bla)
(more-trivial-stuff client name)))
(defn more-non-trivial-stuff [client name bla]
(->>
(trivial-stuff client name bla)
(more-trivial-stuff client name)))
(defn compile-utils [client]
{:more-non-trivial (partial more-non-trivial-stuff client)
:non-trivial (partial non-trivial-stuff client)})
I can't make any def for the clients because I will need the credentials at runtime, so I have to accept all that stuff as parameters and bind the results, To me that looks like a lot of boilerplate and repetitive code that is not maintainable at all.
Does clojurians have a better approach ? Is any style guide on this regard ?
This is the second time I approach clojurescript and it looks very appealing at first, but as soon as you start building non trivial stuff it starts to become messy.
NOTE: for the shake of simplicity I didn't managed any js interop or used channels for async handling. I just declared js objects as normal cljs maps and took everything as it were synchronous, but including js interop and all that stuff will make things even worse.
EDIT (for clarification):
My question is not about if this is possible on clojure, I know it is possible since CLJS and JS share the required set of functionalities to make it possible. However, using the same patter on a totally different language not only feels wrong, but it also looks ugly due to the lisp syntax. The other alternatives I can think about also looks ugly, and involves a lot of repetition because it requires to get the client use it and pass it around on every single function, which leads to very repetitive and distracting code.
To make clear how would I use this on js it will be like this
const makeClient = require('./my-http')
const client = makeClient('my-user','my-pass','http://google.es')
client.getSomething('dude')
As you can see, I can create as many clients with different settings as I may need, I can even make some destructuring and pick only the methods I need since they do not depend on their bindings at all.
Note: I haven’t used Clojure/Script “in anger” yet, so this is a learning experience for me as well :) I haven't validated the code in a REPL unfortunately.
If I understand correctly, the JS module pattern is a function that returns a dictionary of two functions. At some point in your code you “create” this module, perhaps giving it a name, and you pass it around in your code like so:
let client = require("mymodule")("user", "pass", "http://example.com");
client.getSomething("foo")
You could do the same thing with ClojureScript:
(defn create-client [user pass base]
(let [get-something (fn [name] ...)
create-something (fn [name other] ...)]
{:get get-something :create create-something}))
(let [client (create-client "user" "pass" "http://example.com")]
((client :get) "foo"))
Now arguably this might look a bit more clunky but it is exactly the same code: close over a few variables, stick two functions in a map, get a function out of a map and call it.
The second part of your question looks like it's about global state – you have to carry around the client object everywhere, and it feels clunky. I don't think though that it looks any better in Javascript?
let client = require("mymodule")("user", "pass", "http://example.com");
let trivialStuff = (client, name, blah) => { client.getSomething(name); ... };
let moreTrivialStuff = (client, name, blah) => { client.getSomething(name); ... };
let nonTrivialStuff = (client, name, blah) => {
let result = trivialStuff(client, name, blah)
return moreTrivialStuff(client, name, result)
}
i.e. you are still passing client around. You could make it a module-level variable after initialisation, but then you lose the ability to make two different clients at runtime.
Are you saying that with the revealing module pattern you would also expose nonTrivialStuff, therefore you could do client.nonTrivialStuff()?
How about creating a namespace with all the functions that expect a client (which could be just a plain map containing the JS requests client), and just using them directly?
e.g.
(ns some-client [:require ["request" :as request]])
(defn make-client [usr pass base-url]
{:client (.defaults request {:auth {:user usr :pass pass} :baseUrl baseUrl})}) ;; you might want to use #js here, since you usually cannot pass CLJS maps to JS directly
(defn get-something [client name]
(.get (client :client) (str "resources/" name)))
(defn create-something [client name options]
(.post (client :client) (str "resources") {:name name :data options}))
Then in other namespaces:
(ns other-code [:require [some-client :as client]])
(def c (atom nil))
;; get user and pass from runtime, then call
(reset! c (client/make-client user pass base-url))
;; use like so
(client/get-something #c "name")
I opted into putting the JS client object into a CLJS map to allow for flexibility – in the future you may want to add more data into this map. Of course, the client code would not change since it it should treat it as an opaque value.
Related
I'm interested in using Fauna from browser from pure JS in read-only mode.
Either official documentation lacks understandable information for this case or I can't figure it out.
Please help me to understand.
I'm trying to run such query:
Map(Paginate(Documents(Collection('spells'))), Lambda('x', Get(Var('x'))));
Running this query from the shell gives me the result.
I want to push this query to JavaScript variable.
But I don't understand what to do.
Here is my code:
var client = new faunadb.Client({
secret: '320438826900127936',
keepAlive: false,
domain: 'db.eu.fauna.com',
// NOTE: Use the correct domain for your database's Region Group.
port: 443,
scheme: 'https',
});
var helper = client.paginate(
q.Match(
q.Index('spells'),
'101'
)
)
paginate.then(function(response) {
console.log(response.ref) // Would log the ref to console.
})
Please help me to get output from DB using pure JavaScript.
Using the JavaScript driver, all of the FQL functions are in faunadb.query namespace. Typically, developers would do this at the top of their scripts:
const faunadb = require('faunadb')
const q = faunadb.query
After that, the FQL functions can be access using the q object. For example: q.Add(1, 2).
Otherwise, you can use destructuring to import FQL functions into the script's namespace, like so:
const faunadb = require('faunadb')
const {
Documents,
Collection,
Get,
Index,
Lambda,
Match,
Paginate
} = faunadb.query
and then you can call the FQL functions "directly", e.g. Add(1, 2).
If you use destructuring, be sure to import all of the FQL functions that you intend to use in your queries. Not all FQL functions can be imported "as-is", since some of their names would collide with built-in JavaScript names, such as Function. For those, you have to assign them directly:
const FaunaFunction = faunadb.query.Function
And then use the local name that you specified (FaunaFunction in this case) when composing queries.
Either way, the query itself isn't sent to Fauna until the client.query(), or in your case, client.paginate(), function is called.
That means that you can assign the FQL function calls to JavaScript variables. With the desired query, that could look like this:
const spellsQuery = q.Map(
q.Paginate(
q.Documents(q.Collection('spells'))
),
q.Lambda('x', q.Get(q.Var('x')))
)
Then you can do this:
client.query(spellsQuery)
.then(res => console.log(res))
You can use the client.paginate if you are only performing pagination. Your desired query calls Map, so the client.pagination helper gets in your way. My example uses client.query.
Note that when you configure your client connection object, you don't need to specify the port and scheme options; you're using the default values already. And you likely shouldn't change the keepAlive option unless your workflow specifically requires it.
Plus, I see that you didn't include an actual secret in your example, which is a good thing: if you did include an actual secret, anyone with that secret could access your database just like you can.
If you want to see a complete (but simple) Fauna-based application that runs in a web page, see: https://github.com/fauna-labs/todo-vanillajs
Suppose I have a bunch of same-origin windows or tabs A, B, C, D, and E, that don't hold references to each other. (e.g. a user opened them independently). Suppose A sends a BroadcastChannel message to the others, and as a result, D needs to send some data back to A, ideally without involving B, C, or E.
Is this possible, using any of the message-passing APIs?
There's an event.source property on the broadcast message event, which looked as if it should maybe contain a WindowProxy or MessagePort object in this context, but (in my tests with Firefox 78 at least) it was simply null. There's also a ports array, but that was empty.
...I'm aware that you could start up a SharedWorker to assign each window a unique ID and act as a waystation for passing messages between them, but (a) that seems very complicated for the functionality desired, and (b) every message sent that way is going to need 2 hops, from window to sharedWorker and back to a window, crossing thread boundaries both times, and (usually) getting serialized & unserialized both times as well - even when the two windows share the same javascript thread! So it's not very efficient.
This seems like such an obvious thing to want to do, I'm finding it hard to believe there isn't something obvious I'm missing... but I don't see it, if so!
Looks like the standards require source to be null for a BroadcastChannel. But it shares the MessageEvent interface with several other APIs that do use source, hence why it exists, but is null.
The postMessage(message) method steps are:
...
5. Remove source from destinations.
Looks like they intentionally kept BroadcastChannel very lightweight. Just a guess, but the functionality you're looking for might have required additional resources that they didn't want to allocate. This guess is based on a general note they have in the spec:
For elaborate cases, e.g. to manage locking of shared state, to manage synchronization of resources between a server and multiple local clients, to share a WebSocket connection with a remote host, and so forth, shared workers are the most appropriate solution.
For simple cases, though, where a shared worker would be an unreasonable overhead, authors can use the simple channel-based broadcast mechanism described in this section.
SharedWorkers are definitely more appropriate for complicated cases, think of the BroadcastChannel really just as a one-to-many simple notification sender.
It isn't able to transfer data — Which of the receivers should become the owner then? — so except in the case of Blobs (which are just small wrappers with no data of their own), passing data through a BroadcastChannel means it has to be fully deserialized by all receivers, not the most performant way of doing.
So I'm not sure what kind of data you need to send, but if it's big data that should normally be transferable, then probably prefer a SharedWorker.
One workaround though if your data is not to be transfered, is to create a new BroadcastChannel that only your two contexts will listen at.
Live demo
In page A:
const common_channel = new BroadcastChannel( "main" );
const uuid = "private-" + Math.random();
common_channel.postMessage( {
type: "gimme the data",
from: "pageB",
respondAt: uuid
} );
const private_channel = new BroadcastChannel( uuid );
private_channel.onmessage = ({data}) => {
handleDataFromPageB(data);
private_channel.close();
};
In page B:
const common_channel = new BroadcastChannel( "main" );
common_channel.onmessage = ({ data }) => {
if( data.from === "pageB" && data.type === "gimme the data" ) {
const private_channel = new BroadcastChannel( data.respondAt );
private_channel.postMessage( the_data );
private_channel.close();
}
};
Regarding why you can't have a ports value on MessageEvent firing on BroadcastChannels it's because MessagePorts must be transfered, but as we already said, BroadcastChannels can't do transfers.
For why there is no source, it's probably because as you expected that should have been a WindowProxy object, but WorkerContexts can also post messages to BroadcastChannels, and they don't implement that interface (e.g their postMessage method wouldn't do the same thing at all than for a WindowContext).
Note that this is not a duplicate of a similar question for go, since this uses grpc-node. For some reason, there seems to be differences in the API
I do the standard procedure of creating my APIPackageDefinitions and APIPackagePbjects, and create two separate clients from each one, individually.
let grpc = require('grpc')
let protoLoader = require('#grpc/proto-loader')
async function createGrcpConnection() {
const HOST = 'localhost'
const PORT = '50053'
const PORT2 = '50054'
let physicalProjectAPIPackageDefinition = await protoLoader.load(
'./physical_project_api.proto',protoLoaderOptions
)
let configAPIPackageDefinition = await protoLoader.load(
'./config_api.proto', protoLoaderOptions
)
let physicalProjectAPIPackageObject = grpc.loadPackageDefinition(
physicalProjectAPIPackageDefinition
).package.v1
let configAPIPackageObject = grpc.loadPackageDefinition(
configAPIPackageDefinition
).package.v1
let grpcClient1 = physicalProjectAPIPackageObject.PhysicalProjectAPI(
`${HOST}:${PORT}`,
grpc.credentials.createInsecure()
)
let grpcClient2 = configAPIPackageObject.ConfigAPI(
`${HOST}:${PORT2}`,
grpc.credentials.createInsecure()
)
return { grpcClient1, grpcClient2 }
}
I am looking for a way to create two clients that share the same connection. I think I am close to the solution by creating a new Channel and replacing the last two let statements with
let cc = new grpc.Channel(
`${HOST}:${PORT}`,
grpc.credentials.createInsecure()
)
let grpcClient1 = physicalProjectAPIPackageObject.PhysicalProjectAPI(cc)
let grpcClient2 = configAPIPackageObject.ConfigAPI(cc)
However, I received a TypeError: Channel's first argument (address) must be a string. I'm not sure how to incorporate the newly instantiated Channel to create new clients for each service. I couldn't find any useful methods on the docs. Any help would be appreciated.
P.S. At the moment I am trying to use two services, and create a client for each service, and have those two clients share a connection on the same channel. Is it possible to use two service, and create a single client for both services? Maybe I can use .proto package namespaces to my advantage here? My internet search fu failed me on this question.
There is an API to do this, but it is a bit more awkward than what you were trying. And you don't actually need to use it to get what you want. The grpc library internally pools connections to the same server, as long as those connections were created with identical parameters. So, the Client objects created in your first code block will actually use the same TCP connection.
However, as mentioned, there is a way to do this explicitly. The third argument to the Client constructor is an optional object with various additional options, including channelOverride. That accepts a Channel object like the one you constructed at the beginning of your second code block. You still have to pass valid values for the first two arguments, but they will actually be ignored and the third argument will be used instead. You can see more information about that constructor's arguments in the API documentation.
I am writing a standalone web scraper in Node, run from command line, which looks for specific data on a set of pages, fetches page views data from Google Analytics and saves it all in an MySQL database. Almost all is ready, but today I found a problem with the way I write data in the db.
To make thing easier let's assume I have an index.js file and two controllers - db and web. Db reads/writes data to db, web scraps the pages using configurable amount of PhantomJs instances.
Web exposes one function checkTargetUrls(urls, writer)
where urls is an array with urls to be checked and writer is an optional parameter, called only if it is a function and there is data to be written.
Now the way I pass the writer is obviously wrong, but looks as follows (in index.js):
some code here
....
let pageId = 0;
... some promises code,
which checks validy of urls,
creates new execution in the database, etc.
...
.then(ulrs => {
return web.checkTargetUrls(urls,
function(singleUrl, pageData) {
...
a chain of promisable functions from db controller,
which first lookup page id in the db, then its
puts in the pageId variable and continues with write to db
...
}).then(() => {
logger.info('All done captain!');
}).catch(err => {logger.error(err})
In the effect randomly pageId gets overwritten by id of preceeding/succeeding page and invalid data is saved. Inside web there are up to 10 concurrent instances of PhantomJs running, which call writer function after they analyzed a page. Excuse me my language, but for me an analogy for that situation would be if I had, say, 10 instances of some object, which then rely for writing on a singleton, which causes the pageId overwriting problem (don't know how to properly express in JS/Node.js terms).
So far I have found one fix to the problem, but it is ugly as it introduces tight coupling. If I put the writer code in a separate module and then load it directly from inside the web controller all works great. But for me it is a bad design pattern and would rather do it otherwise.
var writer = require('./writer');
function checkTargetUrls(urls, executionId) {
return new Promise(
function(resolve, reject) {
let poolSize = config.phantomJs.concurrentInstances;
let running = 0;
....
a bit of code goes here
....
if (slots != undefined && slots != null && slots.data.length > 0) {
return writer.write(executionId, singleUrl, slots);
}
...
more code follows
})
}
I have a hard time findng a nicer solution, where I could still pass writer as an argument for checkTargetUrls(urls, writer) function. Can anyone point me in the right direction or suggest where to look for the answer?
The exact problem around your global pageId is not entirely clear to me but you could reduce coupling by exposing a setWriter function from your 'web' controller.
var writer;
module.exports.setWriter = function(_writer) { writer = _writer };
Then near the top of your index.js, something like:
var web = require('./web');
web.setWriter(require('./writer'));
I am doing some development on the firefox both with javascript and C++ for some XPCOM components.
I am trying to monitor the http activity with nsIHttpActivityDistributor.
The problem now is , is there any flag or id that belong to nsIHttpChannel that I can use to identify a unique nsHttpChannel object?
I want to save some nsIHttpChannel referred objects in C++ and then process later in Javascript or C++. The thing is that currently I cannot find a elegent way to identify a channel object that can used both in js and C++, which is used to log it clearly into a log file.
Any idea?
You can easily add your own data to HTTP channels, they always implement nsIPropertyBag2 and nsIWritablePropertyBag2 interfaces. Something along these lines (untested code, merely to illustrate the principle):
static PRInt64 maxChannelID = -1;
...
nsCOMPtr<nsIWritablePropertyBag2> bag = do_QueryInterface(channel);
if (!bag)
...
nsAutoString prop(NS_LITERAL_STRING("myChannelID"));
PRInt64 channelID;
rv = bag->GetPropertyAsInt64(prop, &channelID);
if (NS_FAILED(rv))
{
// First time that we see that channel, assign it an ID
channelID = ++maxChannelID;
rv = bag->SetPropertyAsInt64(prop, channelID)
if (NS_FAILED(rv))
...
}
printf("Channel ID: %i\n", channelID);
You might want to check what happens on HTTP redirect however. I think that channel properties are copied over to the new channel in that case, not sure whether this is desirable for you.