I have an Express application that spawns a Python process to execute a Python script. When I do a firebase serve, I can see that my endpoint is being hit, which then runs the Python process. However, the process doesn't seem to be executing.
const runPythonScript = () => {
return new Promise((resolve, reject) => {
let value;
const spawn = require('child_process').spawn;
const pythonProcess = spawn('python', ['./myScript.py']);
pythonProcess.stdout.on('data', (data: string) => {
console.log('Am I being hit?') // This line is not being hit
value = JSON.parse(data);
});
pythonProcess.on('exit', (code: number) => {
if (code === 0) {
resolve(value);
}
else {
reject(value);
}
});
});
}
From the comment in the code above, the listener for stdout 'data' is not being hit. I'm not too familiar with Firebase, but my idea is to use Firebase Hosting for my frontend and then Firebase Cloud Functions to run my Express server. Is there anything that I need to do in order for my application to run the Python script?
From what I've gathered from other StackOverflow posts (here), I can't run a Python process, perhaps because Firebase Cloud Functions does not have Python installed. So instead, I need to package my Python script into an executable (as described here), so that Firebase Cloud Functions can just run the executable. Is this correct? If so, I would prefer not to have to package all of my Python scripts. Is there a better approach to handling this? Is it free?
From what I've gathered from other StackOverflow posts, I can't run a Python process, perhaps because Firebase Cloud Functions does not have Python installed.
This is true.
So instead, I need to package my Python script into an executable (as described here), so that Firebase Cloud Functions can just run the executable. Is this correct?
You can certainly try this, but I don't recommend it. It sounds like a lot of work for little benefit, especially when you have other options.
Is there a better approach to handling this?
You can write Cloud Functions natively in python. You just won't be able to use Firebase tools to test and deploy them. Google Cloud has everything you need to get started.
Related
I would like to insert data into a SQlite3 db by pressing a button from an HTML file.
In order to achieve this, I wrote a JS script.
Unfortunately, when I press the button, I get this error message in my console.log:
script.js:5 Uncaught ReferenceError: require is not defined
Then, I tried to convert my JS file with browserify but then I got this error:
Cannot read property '_handle' of undefined
Here my HTML and JS codes to reproduce the error:
HTML:
<head>
</head>
<body>
<div>
<label>SQLite3</label>
<button type="button">Connection</button>
</div>
<script src="script.js"></script>
</body>
JS:
function addData() {
const sqlite3 = require('sqlite3').verbose();
let db = new sqlite3.Database('./data.db');
db.run('INSERT INTO info (result) VALUES (10)', function(err, row) {
if(err) {
console.log(err.message);
}
console.log("entry added to table");
});
}
addData();
Browserify works around the problem that browsers don't have native support for the Node.js module system.
Browserify does not work around most other APIs that are provided but Node.js but not browsers. For example, Node.js has the fs module which allows you to access the file system of the computer the code is running on. Browsers, on the other hand, don't allow access except in very particular and limited ways.
Since new sqlite3.Database('./data.db') tries to read data from a file, it will not work in a browser.
Use a different API that is supported by browsers instead (such as IndexedDB or Web Storage.
Alternatively, if you want to share data between browsers, use Node.js to run a web server (e.g. with Express.js) and write a web service through which the database can be accessed.
You can't connect to sqlite databawe from client side you have to connect from server side language. Perhaps you can use NodeJS as server side programming language.
follow those steps then it will work
STEP1
Download and install node js from https://nodejs.org/en/download/
STEP2
Install npm by folloy this instruction
https://www.npmjs.com/get-npm
STEP3
Create Node js project by following this guides http://cassandrawilcox.me/setting-up-a-node-js-project-with-npm/
STEP4
Now you can install sqlite3 via npm.
After that you have to run server. If you are beginner then follow this
https://www.w3schools.com/nodejs/
I want to test locally my firebase functions.
These functions make firestore queries.
So i start the emulator firebase emulators:start and in my client i use firebase.functions().useFunctionsEmulator('http://localhost:5001').
My functions work well when i call them in my client. I can read/write data inside the firestore emulator.
The problem :
I want to read the firestore emulator data directly inside my client, like :
firebase.firestore().collection('tests').get().then(tests => {
console.log( tests.docs.map(test=>test.map) )
})
but i can't find how to set the firestore emulator inside my client.
here what i tried:
1) Firestore setting
firebase.firestore().settings({
host:'http://localhost:8080',
ssl:false
})
result :
i get #firebase/firestore: Firestore (6.3.5): Could not reach Cloud Firestore backend. Backend didn't respond within 10 seconds. inside my client console.
The http request returns 'Not found'
2) Set the emulator url inside my firebaseConfig
var firebaseConfig = {
// ...
databaseURL: "http://localhost:8080",
// ...
}
firebase.initializeApp(firebaseConfig)
in this case, the remote server (https://firestore.googleapis.com..) is requested.
So i want to setup one of these two cases :
1) Using the remote firestore inside my functions emulators
OR
2) Using the local firestore emulator inside my client code.
Anyone has already done this ?
Install the testing lib
npm i -D #firebase/testing
Setup and start the emulator in another terminal:
firebase setup:emulators:firestore
firebase serve --only firestore
Setup the tests
const firebase = require("#firebase/testing");
// Helper function to setup test db
function authedApp(auth) {
return firebase
.initializeTestApp({ projectId: FIRESTORE_PROJECT_ID, auth })
.firestore();
}
// Setup methods
beforeEach(async () => {
// Clear the database between tests
await firebase.clearFirestoreData({ projectId: FIRESTORE_PROJECT_ID });
});
// Clean up apps between tests.
afterEach(async () => {
await Promise.all(firebase.apps().map(app => app.delete()));
});
Run the tests
it("should retrieve correct item", async () => {
// Init test db
const db = authedApp(null);
// Manually add item to collection
const ref = await db.collection(COLLECTION_NAME).add({name: 'test item'});
// Fetch item by id
const resp = await db.collection(COLLECTION_NAME).doc(ref.id).get();
// test the output
expect(resp).toBeDefined();
expect(resp).toEqual(expect.objectContaining({name: 'test item'}));
});
Of course your particular setup and circumstances will differ, but this at least should give you a general idea. More info: https://firebase.google.com/docs/rules/unit-tests
Note from 'Test your Cloud Firestore Security Rules'
Data written to the Cloud Firestore emulator is held in memory until
the emulator is stopped. If the emulator is run continuously, this may
have an impact on test isolation. To ensure that data written in one
test is not read in another, either explicitly clear your data with
clearFirestoreData, or assign a different project ID for each
independent test: when you call firebase.initializeAdminApp or
firebase.initializeTestApp, append a user ID, timestamp, or random
integer to the projectID.
Edit: I wrote a blog post a while back, which goes into more detail about the subject.
I had the same problem too. I found the following example, which looks like its still work in progress:
https://github.com/firebase/quickstart-nodejs/tree/master/firestore-emulator/browser-quickstart
They didn't use #firebase/testing directly in their example. When I did try to embed #firebase/testing in my webpack code, it tries to connect via grpc, which attempts to do fs.existsSync, which doesn't exist in webpack browser context. They prefer to enable the functionality via WebChannel instead.
Some caveats as of Nov 2019:
You might see errors in the console connecting to localhost:8080 with the X-Goog-API. I am not sure what is that for.
You might get the following error: #firebase/firestore: Firestore (6.3.5): Could not reach Cloud Firestore backend. Backend didn't respond within 10 seconds.
Despite having those errors, my functions were still able to connect to the local firestore.
Versions I had at time I was testing:
firebase-js-sdk 6.3.1
firebase-tools 7.3.1
Update 11/18:
I raised an issue within quickstart-nodejs github and it seems I just needed to use the latest versions of everything.
Versions:
firebase-js-sdk v7.4.0
firebase-tools v7.8.0
Ok i found how to do it :
1) launch the functions emulator locally:
set GOOGLE_APPLICATION_CREDENTIALS=./privatekey.json && firebase serve --only functions
2) then client side:
if (process.env.NODE_ENV === 'development') {
firebase.functions().useFunctionsEmulator('http://localhost:5001')
}
Okay, this is trivial... In your firestore cient config you should have provided the host, not the origin for firestore (the protocol is set using the ssl parameter):
firebase.firestore().settings({
host: 'localhost:8080',
ssl: false
})
At least this solved it for me when I had the exact same error.
Just FYI to anyone who's reading this - if you run into problems with the firestore client, you can use debug level logging, just set firebase.firestore.setLogLevel('debug'). Had the OP done that, he might have noticed that firebase is accessing firestore at http://http://localhost:8080/...
Define the FIRESTORE_EMULATOR_HOST environment variable
If you are using a library that supports the FIRESTORE_EMULATOR_HOST environment variable, run:
export FIRESTORE_EMULATOR_HOST=localhost:8080
or just add FIRESTORE_EMULATOR_HOST=localhost:8080 to your .env file
What would be the best way to execute command line tools from nodejs to python? I'm working on a app that can generate blockchain certificates.
https://github.com/blockchain-certificates/cert-tools
This is a python based command line tool, where you can generate blockchain certificates. I have followed the steps and everything is working fine in the virtual env. But now I want to know how to implement this in to a app, where I can run the command line tools external from NodeJS. Is this possible? I have found some libraries where you can run python script from nodejs. Example I have used python shell. Whenever I run the setup script, I get missing file errors.
Can someone guide me what the best way will be to solve this problem. Thanks in advance.
You can use python shell in node, you would go about using it somewhat like the following:
var PythonShell = require('python-shell');
PythonShell.run('my_script.py', function (err) {
if (err) throw err;
console.log('finished');
});
for more documentation visit their github repository, hope i could help :)
You should just be able to execute the commands using the child_process.exec() function.
Make sure to only use the file once the python script has finished.
child_process.exec('py path/to/script.py arg1 arg2', (err, stdout, stderr) => {
if (err) {
console.error(err);
return;
}
console.log(stdout);
// check if generated file exists
let fileExists = require("fs").existsSync("/path/to/generated/file");
if (fileExists) {
// do something with the file
}
});
I am very much enjoying AWS Lambda functions, and I'm wondering if what I want to do here is possible. On my local machine, I have a Protractor config file :
// conf.js
exports.config = {
framework: 'jasmine',
seleniumAddress: 'http://127.0.0.1:4444/wd/hub',
specs: ['automation-script.js'],
capabilities: {
browserName: 'chrome'
}
}
and a script that loads up a browser window with a certain url:
describe('Protractor Demo App', function() {
it('should have a title', function() {
browser.driver.get('https://github.com/');
// Click around and do things here.
});
});
The purpose my scripts right now are not to black-box test an application that I'm developing, but instead to automate common browser tasks that I don't feel like doing.
Currently, I'm running the protractor script through my local command shell like this:
protractor protractor.conf.js
I'm wondering if it is possibly to run protractor from within another node.js script. My thinking is that I could have the Lambda function kick off a protractor job, possibly by using the browsers available from Browserstack or Sauce Labs, but I can't figure out how to run protractor from a Node.js script.
This is a really interesting question. Our organization has been probing how much of our CI/CD pipeline can be done in a serverless fashion. This is right up that alley.
Unfortunately, I don't think there is an elegant way to run protractor from another Node script. That is, protractor doesn't seem to expose an API that makes it easy to consume in such a manner.
It's been asked for, but (as a relative newcomer to protractor) the comment right before the issue was closed doesn't contain enough detail for me to know how to take that approach. So, the not-so-elegant approach:
Child Process
Prior comments notwithstanding, you can indeed run protractor from within another Node script, including a Node script executing in AWS' Lambda environment. There may be prettier/better ways to do this, but I took this answer and based the following Lambda function on it:
'use strict';
module.exports.runtest = (event, context, callback) => {
var npm = require('npm');
var path = require('path');
var childProcess = require('child_process');
var args = ['conf.js'];
npm.load({}, function() {
var child = childProcess
.fork(path.join(npm.root, 'protractor/bin/protractor'), args)
.on('close', function(errorCode) {
const response = {
statusCode: 200,
body: JSON.stringify({
message: `Selenium Test executed on BrowserStack! Child process Error Code: ${errorCode}`,
}),
};
callback(null, response);
});
process.on('SIGINT', child.kill);
});
};
var args = ['conf.js']; points to the protractor config file, which in turn points to the test (index.js in this case):
exports.config = {
'specs': ['./index.js'],
'seleniumAddress': 'http://hub-cloud.browserstack.com/wd/hub',
'capabilities': {
'browserstack.user': '<BROWSERSTACK_USER>',
'browserstack.key': '<BROWSERSTACK_KEY>',
'browserName': 'chrome'
}
};
Repository here.
Notes
npm is a runtime dependency using this approach, meaning it has to be packaged into your deployable. This makes for a relatively large lambda function. At ~20mb, it's big enough that you don't get to edit code inline in the AWS console anymore. An approach that didn't package npm as a runtime dependency would be much nicer.
Don't forget Lambda has a hard 5 minute time limit. Your tests will need to complete in less time than that.
Watch the clock. In many instances, my toy example only uses a browser for a couple of seconds, but the overhead (of connecting to BrowserStack, mostly, I presume) makes the Lambda take 12-30 seconds altogether. Paying for 30 seconds of compute to use a browser for 2.5 seconds doesn't sound like a win. Larger batches of tests might be less wasteful.
You do get CloudWatch logging of the child process without doing any extra plumbing yourself, which is nice.
Disclaimer: My example has only been happy-path tested, and I'm no expert on child processes in Node.
I have a node.js application that receives a file, via a web request and then will apply a conversion process to this file. Since the task is long running this needs to run separate to the main thread.
At the moment I have just called the necessary code via a setTimeout() call. To isolate the main application from the conversion process I would like to move it out into a child process, since it is long running and I would like to isolate the main code from the work being done (am I worrying too much?). At the moment I am calling:
const execFile = require('child_process').execFile;
const child = execFile('node', './myModule.js', (error, stdout, stderr) => {
if (error) {
throw error;
}
console.log(stdout);
});
Is this the right approach in node.js, or is there of simply starting a child process with the module and params specified, but not have to specify 'node' as the executable?
Just seen that node.js provides the 'fork' function, for executing modules, though they will need to be written as if they were expecting command line arguments, processing the process.argv array.
The command call being:
child_process.fork(modulePath[, args][, options])
More details here.
In my specific case forking probably doesn't make sense, since there is already a fork being made by the node.js library I am using.