We Keep Coding

Electron nedb-promises storage file gets replaced on every app start

I'm now trying for a few hours to understand why this happens, in my electron app i would like to use the nedb-promises package("nedb-promises": "^6.2.1",). Installation and configuration works so far but on every app start (dev & prod) the db file got replaced by a new / empty one. Should'nt the package not handle that?
I've took the code from this example:
https://shivekkhurana.medium.com/persist-data-in-electron-apps-using-nedb-5fa35500149a
// db.js
const {app} = require('electron');
const Datastore = require('nedb-promises');
const dbFactory = (fileName) => Datastore.create({
filename: `${process.env.NODE_ENV === 'development' ? '.' : app.getPath('userData')}/data/${fileName}`,
timestampData: true,
autoload: true
});
const db = {
customers: dbFactory('customers.db'),
tasks: dbFactory('tasks.db')
};
module.exports = db;
import db from './db'
....
// load task should not be important because file is already replaced when arriving here
ipcMain.handle('Elements:Get', async (event, args) => {
// 'Select * from Customers
let data = await db.customers.find({});
console.log(data);
return data;
})
...
// Set an item
ipcMain.handle('Element:Save', async (event, data) => {
console.log(data)
const result = db.customers.insertOne(data.item).then((newDoc) => {
console.log(newDoc)
return newDoc
}).catch((err) => {
console.log("Error while Adding")
console.log(err)
});
console.log(result);
return result;
})
Note: After "adding" newDoc contains the new element and when checking the file manually in the filesystem it is added. When i now close the app and open again the file got replaced.
I've checked the docs up and down - i have no clue what i'm doing wrong - thanks for your help.

Sveltekit & Fleek (IPFS) import syntax problem?

I have managed to use fleek to update IPFS via straight javascript. I am now trying to add this functionality to a clean install of a svelteKit app. I think I am having trouble with the syntax around imports, but am not sure what I am doing wrong. When I click the button on the index.svelte I get the following error
Uncaught ReferenceError: require is not defined
uploadIPFS upload.js:3
listen index.mjs:412..........(I truncated the error here)
A few thoughts
I am wondering if it could be working in javascript because it is being called in node (running on the server) but running on the client in svelte?
More Details
The index.svelte file looks like this
<script>
import {uploadIPFS} from '../IPFS/upload'
</script>
<button on:click={uploadIPFS}>
upload to ipfs
</button>
the upload.js file looks like this
export const uploadIPFS = () => {
const fleek = require('#fleekhq/fleek-storage-js');
const apiKey = 'cZsQh9XV5+6Nd1+Bou4OuA==';
const apiSecret = '';
const data = 'pauls test load';
const testFunctionUpload = async (data) => {
const date = new Date();
const timestamp = date.getTime();
const input = {
apiKey,
apiSecret,
key: `file-${timestamp}`,
data
};
try {
const result = await fleek.upload(input);
console.log(result);
} catch (e) {
console.log('error', e);
}
};
testFunctionUpload(data);
};
I have also tried using the other import syntax and when I do I get the following error
500
global is not defined....
import with the other syntax is
import fleekStorage from '#fleekhq/fleek-storage-js';
function uploadIPFS() {
console.log('fleekStorage',fleekStorage)
};
export default uploadIPFS;
*I erased the api secret in the code above. In future I will store these in a .env file.
Even more details (if you need them)
The file below will update IPFS and runs via the command
npm run upload
That file is below. For my version that I used in svelte I simplified the file by removing all the file management and just loading a variable instead of a file (as in the example below)
const fs = require('fs');
const path = require('path');
const fleek = require('#fleekhq/fleek-storage-js');
require('dotenv').config()
const apiKey = process.env.FLEEK_API_KEY;
const apiSecret = process.env.FLEEK_API_SECRET;
const testFunctionUpload = async (data) => {
const date = new Date();
const timestamp = date.getTime();
const input = {
apiKey,
apiSecret,
key: `file-${timestamp}`,
data,
};
try {
const result = await fleek.upload(input);
console.log(result);
} catch(e) {
console.log('error', e);
}
}
// File management not used a my svelte version to keep it simple
const filePath = path.join(__dirname, 'README.md');
fs.readFile(filePath, (err, data) => {
if(!err) {
testFunctionUpload(data);
}
})

Module not found: Can't resolve 'child_process' - google-spreadsheet

I am trying to save form data to a spreadsheet in Next.js but I keep getting this error which appears as soon as I import google-spreadsheet
Error
./node_modules/google-spreadsheet/node_modules/google-auth-library/build/src/auth/googleauth.js:17:0
Module not found: Can't resolve 'child_process'
Bellow is what I have that is causing the error.
// The error appears when I do this import
import { GoogleSpreadsheet } from "google-spreadsheet";
const SPREADSHEET_ID = process.env.NEXT_PUBLIC_SPREADSHEET_ID;
const SHEET_ID = process.env.NEXT_PUBLIC_SHEET_ID;
const CLIENT_EMAIL = process.env.NEXT_PUBLIC_GOOGLE_CLIENT_EMAIL;
const PRIVATE_KEY = process.env.NEXT_PUBLIC_GOOGLE_SERVICE_PRIVATE_KEY;
const doc = new GoogleSpreadsheet(SPREADSHEET_ID);
const appendSpreadsheet = async (row) => {
try {
await doc.useServiceAccountAuth({
client_email: CLIENT_EMAIL,
private_key: PRIVATE_KEY,
});
// loads document properties and worksheets
await doc.loadInfo();
const sheet = doc.sheetsById[SHEET_ID];
const result = await sheet.addRow(row);
return result;
} catch (e) {
console.error("Error: ", e);
}
};

I just solve it.
Please create next.config.js file in your root.
And fill it below.
module.exports = {
webpack: config => {
config.node = {
fs: 'empty',
child_process: 'empty',
net: 'empty',
dns: 'empty',
tls: 'empty',
};
return config;
},
};
Hoorai!

I was having this problem with nextjs 12. Here's what fixed it for me:
My code:
const doc = new GoogleSpreadsheet(SPREADSHEET_ID);
await doc.useServiceAccountAuth({
client_email: process.env.GOOGLE_SERVICE_ACCOUNT_EMAIL,
private_key: process.env.GOOGLE_PRIVATE_KEY,
});
await doc.loadInfo();
console.log('title', doc.title);
My next.config.js:
const nextConfig = {
reactStrictMode: true,
webpack: (config, { isServer }) => {
if (!isServer) {
config.resolve.fallback.fs = false
config.resolve.fallback.tls = false
config.resolve.fallback.net = false
config.resolve.fallback.child_process = false
}
return config
},
future: {
webpack5: true,
},
fallback: {
fs: false,
tls: false,
net: false,
child_process: false
},
}
module.exports = nextConfig;
Took inspiration/fix from here

Found this answer due to a similar issue. I later learned for next.js, with some of these api libraries, you must call call this type of code (serverside) in two contexts getStaticProps or getServerSideProps. See this and this for more details.

Try changing the import statement to:
const { GoogleSpreadsheet } = require('google-spreadsheet');
Source: https://www.npmjs.com/package/google-spreadsheet

The reason is that the library you require uses some nodejs native modules, like path, fs or child_process.
As part of the build process nextjs will create js bundles for your client and server separately. The issue is that your client build cannot resolve those nodejs modules. As a workaround you can tell nextjs to ignore these modules for the client build only.
next.config.js
const nextConfig = {
webpack: (config, { isServer }) => {
if (!isServer) {
config.resolve.fallback = {
fs: false,
path: false,
}
}
return config
}
}
module.exports = nextConfig;

the library does not support ES6 feature yet
if you look to the module export you will find somthing like this :
module.exports = {
GoogleSpreadsheet,
GoogleSpreadsheetWorksheet,
GoogleSpreadsheetRow,
GoogleSpreadsheetFormulaError,
};
https://github.com/theoephraim/node-google-spreadsheet/blob/master/index.js
change the import statement to commonjs modules like this :
const { GoogleSpreadsheet } = require('google-spreadsheet');

How to run Node packages from Puppeteer client environment for testing purposes

I'm using Puppeteer to test a client function within a react environment - the function itself doesn't use React, but is meant to be imported in es6 react modules and run inside a end user DOM environment. I need Puppeteer since this function relies on properties such as innerText, that aren't available in jsdom.
This function takes a DOM element as an argument, however I am having trouble writing test files for it. Here is a sample of my code:
import path from 'path';
import puppeteer from 'puppeteer';
import {getSelectionRange, setSelectionRange} from './selection';
describe(
'getSelection should match setSelection',
() => {
let browser;
let page;
beforeAll(async done => {
try {
browser = await puppeteer.launch();
page = await browser.newPage();
await page.goto(
`file://${path.join(process.env.ROOT,
'testFiles/selection_range_test.html')}`
);
await page.exposeFunction(
'setSelectionRange',
(el, start, end) => setSelectionRange(el, start, end)
);
await page.exposeFunction(
'getSelectionRange',
el => getSelectionRange(el)
);
} catch(error) {
console.error(error);
}
done();
});
afterAll(async done => {
await browser.close();
done();
});
it('should match on a node with only one text node children', async () => {
const {selection, element, argEl} = await page.evaluate(async () => {
const stn = document.getElementById('single-text-node');
// Since console.log will output in the Puppeteer browser and not in node console,
// I added a line inside the selectionRange function to return the element it receives
// as an argument.
const argEl = await window.setSelectionRange(stn, 1, 10);
const selectionRange = await window.getSelectionRange(stn);
return {selection: selectionRange, element: stn, argEl};
});
// Outputs <div id="single-text-node">...</div>
// (the content is long so I skipped it, but it displays the correct value here)
console.log(element.outerHTML);
// Outputs {}
console.log(argEl);
});
}
);
As described in the comments, the element that is directly returned from page.evaluate() is correct, but when passed as an argument, the function receives an empty object. I suspect a scope issue but I am totally out of solutions here.

Sadly I couldn't find any solution that wouldn't invoke transpiling my files, but hopefully I managed to make it work correctly.
The key point was to create a second transpile configuration that will generate a code directly usable by a web browser, using UMD format. Since I use rollup, here is my rollup,config.js file:
import babel from 'rollup-plugin-babel';
import commonjs from 'rollup-plugin-commonjs';
import resolve from 'rollup-plugin-node-resolve';
import pkg from './package.json';
// The content that is actually exported to be used within a React or Node application.
const libConfig = [
{
inlineDynamicImports: true,
input: './src/index.js',
output: [
{
file: './lib/index.js',
format: 'cjs'
},
],
external: [...Object.keys(pkg.dependencies || {})],
plugins: [
commonjs(),
resolve(),
babel({exclude: 'node_modules/**'})
]
}
];
// Used to generate a bundle that is directly executable within a browser environment, for E2E testing.
const testConfig = [
{
inlineDynamicImports: true,
input: './src/index.js',
output: [
{
file: './dist/index.js',
format: 'umd',
name: 'tachyon'
},
],
external: [...Object.keys(pkg.dependencies || {})],
plugins: [
commonjs(),
resolve(),
babel({runtimeHelpers: true})
]
}
];
const config = process.env.NODE_ENV === 'test' ? testConfig : libConfig;
export default config;
I then rewrote my scripts a bit so my test bundle is generated on each test run.
package.json
{
"scripts": {
"build:test": "NODE_ENV=test rollup -c && NODE_ENV=",
"build": "rollup -c",
"test": "yarn build:test && jest"
},
}
Finally, I added the transpiled script to my selection_test.html file:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Selection range test</title>
<script src="../dist/index.js"></script>
</head>
...
Which lets me write my test file like this:
import path from 'path';
import puppeteer from 'puppeteer';
import {describe, beforeAll, afterAll, it} from '#jest/globals';
describe(
'getSelection should match setSelection',
() => {
let browser;
let page;
beforeAll(async done => {
try {
browser = await puppeteer.launch({
headless: true,
args: ['--disable-web-security', '--disable-features=SameSiteByDefaultCookies,CookiesWithoutSameSiteMustBeSecure'],
});
page = await browser.newPage();
await page.goto(`file://${path.join(process.env.ROOT, 'tests/selection_test.html')}`, {waitUntil: 'networkidle0'});
await page.setBypassCSP(true);
} catch(error) {
console.error(error);
}
done();
});
afterAll(async done => {
await browser.close();
done();
});
it('should match on a node with only one text node children', async () => {
const data = await page.evaluate(() => {
// Fix eslint warnings.
window.tachyon = window.tachyon || null;
if (window.tachyon == null) {
return new Error(`cannot find tachyon module`);
}
const stn = document.getElementById('single-text-node');
const witnessRange = tachyon.setRange(stn, 1, 10);
const selectionRange = tachyon.getRange(stn);
return {witnessRange, selectionRange, element: stn.outerHTML};
});
console.log(data); // Outputs the correct values
/*
{
witnessRange: { start: 1, end: 10 },
selectionRange: {
absolute: { start: 1, end: 10 },
start: { container: {}, offset: 1 },
end: { container: {}, offset: 10 }
},
element: '<div id="single-text-node">Lorem ... sem.</div>'
}
*/
});
}
);
The only remaining issue is that start.container and end.container within the results of getRange are undefined, but it seems more likely an issue from puppeteer that cannot handle the Range startContainer and endContainer properties - I was able to pass DOM references between the content of page.evaluate and my module function without any issues, so it doesn't look like the problem anymore.

How to use puppeteer-core with electron?

I got this code from another Stackoverflow Question:
import electron from "electron";
import puppeteer from "puppeteer-core";
const delay = (ms: number) =>
new Promise(resolve => {
setTimeout(() => {
resolve();
}, ms);
});
(async () => {
try {
const app = await puppeteer.launch({
executablePath: electron,
args: ["."],
headless: false,
});
const pages = await app.pages();
const [page] = pages;
await page.setViewport({ width: 1200, height: 700 });
await delay(5000);
const image = await page.screenshot();
console.log(image);
await page.close();
await delay(2000);
await app.close();
} catch (error) {
console.error(error);
}
})();
Typescript compiler complains about executablePath property of launch method options object cause it needs to be of type string and not Electron. So how to pass electron chromium executable path to puppeteer?

You cannot use electron executable with Puppeteer directly without some workarounds and flag changes. They have tons of differences in the API. Specially electron doesn't have all of the chrome.* API which is needed for chromium browser to work properly, many flags still doesn't have proper replacements such as the headless flag.
Below you will see two ways to do it. However you need to make sure of two points,
Make sure the puppeteer is connected before the app is initiated.
Make sure you get the correct version puppeteer or puppeteer-core for the version of Chrome that is running in Electron!
Use puppeteer-in-electron
There are lots of workarounds, but most recently there is a puppeteer-in-electron package which allows you to run puppeteer within electron app using the electron.
First, install the dependencies,
npm install puppeteer-in-electron puppeteer-core electron
Then run it.
import {BrowserWindow, app} from "electron";
import pie from "puppeteer-in-electron";
import puppeteer from "puppeteer-core";
const main = async () => {
const browser = await pie.connect(app, puppeteer);
const window = new BrowserWindow();
const url = "https://example.com/";
await window.loadURL(url);
const page = await pie.getPage(browser, window);
console.log(page.url());
window.destroy();
};
main();
Get the debugging port and connect to it
The another way is to get the remote-debugging-port of the electron app and connect to it. This solution is shared by trusktr on electron forum.
import {app, BrowserWindow, ...} from "electron"
import fetch from 'node-fetch'
import * as puppeteer from 'puppeteer'
app.commandLine.appendSwitch('remote-debugging-port', '8315')
async function test() {
const response = await fetch(`http://localhost:8315/json/versions/list?t=${Math.random()}`)
const debugEndpoints = await response.json()
let webSocketDebuggerUrl = debugEndpoints['webSocketDebuggerUrl ']
const browser = await puppeteer.connect({
browserWSEndpoint: webSocketDebuggerUrl
})
// use puppeteer APIs now!
}
// ... make your window, etc, the usual, and then: ...
// wait for the window to open/load, then connect Puppeteer to it:
mainWindow.webContents.on("did-finish-load", () => {
test()
})
Both solution above uses webSocketDebuggerUrl to resolve the issue.
Extra
Adding this note because most people uses electron to bundle the app.
If you want to build the puppeteer-core and puppeteer-in-electron, you need to use hazardous and electron-builder to make sure get-port-cli works.
Add hazardous on top of main.js
// main.js
require ('hazardous');
Make sure the get-port-cli script is unpacked, add the following on package.json
"build": {
"asarUnpack": "node_modules/get-port-cli"
}
Result after building:

the toppest answer dones't work for me use electron 11 and puppeteer-core 8.
but start puppeteer in main process other then in the renderer process works for me.you can use ipcMain and ipcRenderer to comunicate each other.the code below
main.ts(main process code)
import { app, BrowserWindow, ipcMain } from 'electron';
import puppeteer from 'puppeteer-core';
async function newGrabBrowser({ url }) {
const browser = await puppeteer.launch({
headless: false,
executablePath:
'/Applications/Google Chrome.app/Contents/MacOS/Google Chrome',
});
const page = await browser.newPage();
page.goto(url);
}
ipcMain.on('grab', (event, props) => {
newGrabBrowser(JSON.parse(props));
});
home.ts (renderer process code)
const { ipcRenderer } = require('electron');
ipcRenderer.send('grab',JSON.stringify({url: 'https://www.google.com'}));

There is also another option, which works for electron 5.x.y and up (currently up to 7.x.y, I did not test it on 8.x.y beta yet):
// const assert = require("assert");
const electron = require("electron");
const kill = require("tree-kill");
const puppeteer = require("puppeteer-core");
const { spawn } = require("child_process");
let pid;
const run = async () => {
const port = 9200; // Debugging port
const startTime = Date.now();
const timeout = 20000; // Timeout in miliseconds
let app;
// Start Electron with custom debugging port
pid = spawn(electron, [".", `--remote-debugging-port=${port}`], {
shell: true
}).pid;
// Wait for Puppeteer to connect
while (!app) {
try {
app = await puppeteer.connect({
browserURL: `http://localhost:${port}`,
defaultViewport: { width: 1000, height: 600 } // Optional I think
});
} catch (error) {
if (Date.now() > startTime + timeout) {
throw error;
}
}
}
// Do something, e.g.:
// const [page] = await app.pages();
// await page.waitForSelector("#someid")//
// const text = await page.$eval("#someid", element => element.innerText);
// assert(text === "Your expected text");
// await page.close();
};
run()
.then(() => {
// Do something
})
.catch(error => {
// Do something
kill(pid, () => {
process.exit(1);
});
});
Getting the pid and using kill is optional. For running the script on some CI platform it does not matter, but for local environment you would have to close the electron app manually after each failed try.
Please see this sample repo.