Google Cloud SQL advertises that it's only $0.0150 per hour for the smallest machine type, and I'm being charged for every hour, not just hours that I'm connected. Is this because I'm using a pool? How do I setup my backend so that it queries the cloud db only when needed so I don't get charged for every hour of the day?
const mysql = require('mysql');
const pool = mysql.createPool({
host : process.env.SQL_IP,
user : 'root',
password : process.env.SQL_PASS,
database : 'mydb',
ssl : {
[redacted]
}
});
function query(queryStatement, cB){
pool.getConnection(function(err, connection) {
// Use the connection
connection.query(queryStatement, function (error, results, fields) {
// And done with the connection.
connection.destroy();
// Callback
cB(error,results,fields);
});
});
}
This is not so much about the pool as it is about the nature of Cloud SQL. Unlike App Engine, Cloud SQL instances are always up. I learned this the hard way one Saturday morning when I'd been away from the project for a week. :)
There's no way to spin them down when they're not being used, unless you explicitly go stop the service.
There's no way to schedule a service stop, at least within the GCP SDK. You could alway write a cron job, or something like that, that runs a little gcloud sql instances patch [INSTANCE_NAME] --activation-policy NEVER command at, for example, 6pm local time, M-F. I was too lazy to do that, so I just set a calendar reminder for myself to shut down my instance at the end of my workday.
Here's the MySQL Instance start/stop/restart page for the current SDK's docs:
https://cloud.google.com/sql/docs/mysql/start-stop-restart-instance
On an additional note, there is an ongoing 'Feature Request' in the GCP Platform to start/stop the Cloud SQL (2nd Gen), according to the traffic as well. You can also visit the link and provide your valuable suggestions/comments there as well.
I took the idea from #ingernet and created a cloud function which starts/stops the CloudSQL instance when needed. It can be triggered via a scheduled job so you can define when the instance goes up or down.
The details are here in this Github Gist (inspiration taken from here). Disclaimer: I'm not a python developer so there might be issues in the code, but at the end it works.
Basically you need to follow these steps:
Create a pub/sub topic which will be used to trigger the cloud function.
Create the cloud function and copy in the code below.
Make sure to set the correct project ID in line 8.
Set the trigger to Pub/Sub and choose the topic created in step 1.
Create a cloud scheduler job to trigger the cloud function on a regular basis.
Choose the frequency when you want the cloud function to be triggered.
Set the target to Pub/Sub and define the topic created in step 1.
The payload should be set to start [CloudSQL instance name] or stop [CloudSQL instance name] to start or stop the specified instance (e.g. start my_cloudsql_instance will start the CloudSQL instance with the name my_cloudsql_instance)
Main.py:
from googleapiclient import discovery
from oauth2client.client import GoogleCredentials
import base64
from pprint import pprint
credentials = GoogleCredentials.get_application_default()
service = discovery.build('sqladmin', 'v1beta4', credentials=credentials, cache_discovery=False)
project = 'INSERT PROJECT_ID HERE'
def start_stop(event, context):
print(event)
pubsub_message = base64.b64decode(event['data']).decode('utf-8')
print(pubsub_message)
command, instance_name = pubsub_message.split(' ', 1)
if command == 'start':
start(instance_name)
elif command == 'stop':
stop(instance_name)
else:
print("unknown command " + command)
def start(instance_name):
print("starting " + instance_name)
patch(instance_name, "ALWAYS")
def stop(instance_name):
print("stopping " + instance_name)
patch(instance_name, "NEVER")
def patch(instance, activation_policy):
request = service.instances().get(project=project, instance=instance)
response = request.execute()
dbinstancebody = {
"settings": {
"settingsVersion": response["settings"]["settingsVersion"],
"activationPolicy": activation_policy
}
}
request = service.instances().patch(
project=project,
instance=instance,
body=dbinstancebody)
response = request.execute()
pprint(response)
Requirements.txt
google-api-python-client==1.10.0
google-auth-httplib2==0.0.4
google-auth==1.19.2
oauth2client==4.1.3
I am creating database records from nodejs using sequalizejs . I am passing array of data and this is being added into the database table.
The problem is the data which is passed is not saved as it is . The order is different . Please suggest how to maintain order while doing bulk create in sequalizejs? . The issue is seen only with LINUX environment. In Windows server , it is working fine.
data - ["A","B","C","D"]
table row order - "B" , "D" , "C" , "A"
Every run the order varies .
models.MyTable.bulkCreate(req.body, { individualHooks: true }).then(function (mydata) {
result.status(200).json(mydata);
});
When you use { individualHooks: true } then sequalize will create multiple SQL queries for insert.
All this queries will run in parallel ( like Promise.all([insertQueries]).Most likely you use connection pool for DB connection - this mean that all your queries will run simultaneously ( limited by settings options.pool.max = default 5 ).
See https://github.com/sequelize/sequelize/blob/v3/lib/model.js#L2169
If order of query is important for you, that use next
var mapSeries = require('bluebird').mapSeries;
mapSeries(req.body, (item) => models.MyTable.create(item))
.then(function (mydata) {
result.status(200).json(mydata);
});
I am creating android application, using the PhoneGap platform. I have tested the use of resources by the application and I noticed that uses a lot of free 3G. In comparison to an application such as whatsapp, using too much connection.
My app is a chat, and I use javascript to make requests to refresh messages. Following the javascript that takes care of this:
var refreshShout = setInterval(app.ajaxFunction, 1000);
[...]
ajaxFunction: function () {
var shoutbase = $('.shoutbox');
if (shoutbase) {
$.get('http://edonetwork.altervista.org/Magnitude/log.php', function (data) {})
.done(function(data) {
$(shoutbase).html(data);
})
.fail(function() {
$(shoutbase).html('<div class="alert-internet">Ops... Occorre una connessione ad internet per usufruire dell\'App!</div>');
});
}
}
In log.php page there are all existing messages taken with MYSQL query from the database of altervista.
Below is the query that is executed:
$dati=mysql_query(" SELECT * FROM `Shoutbox` ORDER BY `Shoutbox`.`ID` DESC LIMIT 0 , 30 ");
How can I limit the use of the internet? As currently is excessive! But I think all chat app continue to make requests but why are more performant?
Thank you.
I am a bit confused with SQLite at the moment, as this is the first time I'm ever using a database. I got sqlite3 from here: https://github.com/developmentseed/node-sqlite3.
I'm looking at that example there, some things I do understand, while others I do not. Most of those database commands that are wrapped in .run(), .prepare() and the like give me a hard time.
This is the example:
var usersDB = new sqlite3.Database("databases/users.db");
usersDB.serialize(function() {
usersDB.run("CREATE TABLE lorem (info TEXT)");
var stmt = usersDB.prepare("INSERT INTO lorem VALUES (?)");
for (var i = 0; i < 10; i++) {
stmt.run("Ipsum " + i);
}
stmt.finalize();
usersDB.each("SELECT rowid AS id, info FROM lorem", function(err, row) {
console.log(row.id + ": " + row.info);
});
});
usersDB.close();
Also, how do I store simple things such as usernames, passwords (do I have to hash them myself?) and emails in the SQLite database on Node.js?
There are two distinct things to learn: sqlite the database program, and node-sqlite3 the nodejs module that provides access to the sqlite db services. Your database questions will be best answered by learning about sqlite, the database program first.
I would recommend getting and installing sqlite from: http://www.sqlite.org/. The site has good documentation that will help you learn to store user names and passwords. You can create tables from the command line, add data and get as sense of what is going on. After that if you understand the concepts of node.js then node-sqlite3 will make much more sense to you. Otherwise, spend some time with the node.js site.
Maybe you can try node-sqlite from grumdrig. He has a very nice "example-driven" documentation.
I am new to Web SQL database and I use it to save data in a local database in a web page.
I can create a database by
var db = openDatabase('database', '1.0', 'my database', 2 * 1024 * 1024);
and I can create a table by doing this
db.transaction(function (tx) {
tx.executeSql('CREATE TABLE IF NOT EXISTS mytable (blah,blah)');
});
I can delete the table by
db.transaction(function (tx) {
tx.executeSql('DROP TABLE mytable');
});
but is there a way to delete the database programmatically?
Using PersistenceJS there is a persistence.reset API which will wipe the database clean.
PersistenceJS Site
For developing / testing purposes, you can view content and delete webSQL, IndexedDB, cookies, etc by searching for your domain name at this URL in Chrome:
chrome://settings/cookies
There, you can delete all the storage for a domain or just certain local storage entities. Yes, the URL implies just 'cookies', but the interface at this URL includes all types of offline storage.
It would be great I think if the Chrome developer tools interface had the ability to right-click and delete a data storage entity in the Resources tab along with inspecting the content. But for now, all I know of is the settings/cookies URL.
Spec says:
4.1 Databases
Each origin has an associated set of databases. Each database has a name and a current version. There is no way to enumerate or delete the databases available for an origin from this API.
I am developing a phonegap+jquery-mobile+KO app with offline storage using web sql via persistencejs, and jasmine js for BDD.
I'm working on some sort of "database cleaner" to be executed after each spec. When I was searching on how to drop a web sql database I read the reply https://stackoverflow.com/a/10929725/667598 (in this thread/question), and went to see what's in that directory (Mac OS X).
cd ~/Library/Application\ Support/Google/Chrome/Default/databases
Inside you will see a Databases.db SQLite3 database, and directories for each origin. These directories are named with the pattern protocol_host_somenumber (I don't know what that number is). So for example, in my case, since my apps are just files I open in Google Chrome with the file:/// … protocol, I can see a file__0 directory. And for twitter and I can also see a http_twitter.com_0 and a https_twitter.com_0.
Inside this directories all file names are just numbers. For example inside file__0 I found a file named 8 and another named 9. In my case, these files are websql database. I don't know if there also Indexed DB databases in chrome's Default/databases dir.
With this names it is a little hard to guess what database is what. You can open the database and you'll have to infer the app or site via its tables and data.
Luckily, the Databases.db I mentioned before is a mapping between those files named with numbers and the databases.
You can open the Databases.db and any other web sql file with the sqlite3 command
sqlite3 Databases.db
Obviously, once inside the sqlite3 shell, is handy to have some SQL knowledge. Anyway, it is also always handy some help, which is available via the command
.help
With the command .tables you can list tables in the database. Inside this Databases.db we can find the tables Databases and meta. The important one is Databases, so with a
select * from Databases;
we can see the mapping between the databases and their files. For example
7|http_jquerymobile.com_0|testdb|html5 test db|200000
8|file__0|elfaro_dev|Base de datos de ElFaro para desarrollo|734003200
The first column is the id of the table which is the number used for db file names, the second is the origin (the directory) the other columns are the db name, the db description and the estimated size used when creating the db from the Javascript API.
So to actually delete a database what I did was to delete it from this table, for example:
delete from Databases where id = 8
And then delete the actual file from the filesystem (outside sqlite3 shell)
rm file__0/8
And that's it.
PS: I know this is a too long answer for a simple subject but I just needed to flush this from my system and back it up somewhere like SO or a blog.
The developer options
There is no way to enumerate or delete the databases programmatically (yet).
Chrome developers can navigate to chrome://settings/cookies search and delete any database
Opera developers can navigate to opera://settings/cookies
The only way to truly delete a database (and everything else)
A new Spec says this might be possible in the feature with both response header and javascript.
The disadvantages is that you can't control what is being deleted, So you would need to create a backup first of everything else unless you want to clear everything
2.1.3. The storage parameter
The storage parameter indicates that the server wishes to remove locally stored data associated with the origin of a particular response’s url. This includes storage mechansims such as (localStorage, sessionStorage, [INDEXEDDB], [WEBDATABASE], etc), as well as tangentially related mechainsm such as service worker registrations.
Js:
navigator.storage.clear({
types: [ "storage" ],
includeSubdomains: true // false by default
});
Response header:
res.header("Clear-Site-Data", "storage; includeSubdomains");
But this is not avalible to any browser yet...
Best solution for clients (not the developers)
/* This will fetch all tables from sqlite_master
* except some few we can't delete.
* It will then drop (delete) all tables.
* as a final touch, it is going to change the database
* version to "", which is the same thing you would get if
* you would check if it the database were just created
*
* #param name [string] - the database to delete
* #param cb [function] - the callback when it's done
*/
function dropDatabase(name, cb){
// empty string means: I do not care what version, desc, size the db is
var db = openDatabase(name, "", "", "");
function error(tx, err){
console.log(err);
}
db.transaction(ts => {
// query all tabels from sqlite_master that we have created and can modify
var query = "SELECT * FROM sqlite_master WHERE name NOT LIKE 'sqlite\\_%' escape '\\' AND name NOT LIKE '\\_%' escape '\\'";
var args = [];
var success = (tx, result) => {
var rows, i, n, name;
rows = result.rows;
n = i = rows.length;
// invokes cb once it’s called n times
function after(){
if (--n < 0) {
// Change the database version back to empty string
// (same as when we compear new database creations)
db.changeVersion(db.version, "", function(){}, error, cb);
}
}
while(i--){
// drop all tabels and calls after() each time
name = JSON.stringify(rows.item(i).name);
tx.executeSql('DROP TABLE ' + name, [], after, error);
}
// call it just 1 more extra time incase we didn't get any tabels
after();
};
ts.executeSql(query, args, success, error);
});
}
Usage
dropDatabase("database", function(){
console.log("done")
});
The localdatabase files are stored in your Windows user settings under Application Data > Google > Chrome > User Data > Default > databases.
So manually deleting them is theoretically possible. This is only useful while testing / developing on your own computer, since when another user opens your app/site, it is unlikely to have file system access.
However, even though you can find the files and delete them, the data sticks around. I've tried it with Chrome both open and closed and all chrome processes ended, and yet the browser inspector keeps showing me my old database with all the unwanted fields and data in it.
This is answered in HTML5 database storage (SQL lite) - few questions.
To summarize:
Currently no way to drop a WebSQL database.
Probably use Indexed DB or localStorage instead.
In my library implementation, I just delete all tables. Which, indeed, delete the database. List of tables are select * from sqlite_master.
Please note that if you use multiple
tx.executeSql('DROP TABLE mytable');
statements in the same transaction callback then make sure that they all exist or consider using DROP TABLE IF EXISTS syntax instead. If even one table doesn't exist when you try to drop it will result in the entire transaction failing. This failure results in a rollback of the transaction and means that the data will stay in your database even when you thought that it should have been deleted. There is no error reported unless you're specifically listening for it in the executeSql's 4th argument which is an error callback. This is intended behavior but is, in my experience, confusing.
No method to delete the existing database in websql it will clear when the cache is cleared or
The browser is closed. If you want to create a database with the same name Just use openDatabase Method It will first check for the existence of the database with the same name. If not exists it will create one otherwise it will open the existing one
please follow this link http://html5doctor.com/introducing-web-sql-databases/