PostgreSQL extensibility is awesome. With things like extensions or custom worker backgrounds, there are many ways for a PostgreSQL developer to create modules without having to touch a single line of the core code at all. Among those tools, PostgreSQL contains a set of hooks that can be used to plug customized code at certain points of the server processing. Hooks are not documented at all, so if you want to know more about them you need either to have a look directly at the PostgreSQL code or to read the slides of the presentation about hooks given by Guillaume Lelarge at PGcon 2012.
Personally I recommend the latter, Guillaume’s presentation being really good.

You can do many things with hooks, like creating a custom planner, outputting a custom EXPLAIN, running some personalized versions of utilities or control query processing at execution level. Hooks can be loaded as shared libraries using shared_preload_libraries or with a simple LOAD command for current session. In this post, I am going to show a restriction on DROP DATABASE using the hook for utility processing. In my case a given database “foodb” can only be dropped by a given user “foo”, our godly super-superuser.

When implementing a hook, here is how it should look for the basics.
#include "postgres.h"
#include "miscadmin.h"
#include "tcop/utility.h"
 
PG_MODULE_MAGIC;
 
void _PG_init(void);
void _PG_fini(void);
 
static char *undroppabledb = "foodb";
static char *supersuperuser = "foo";
static ProcessUtility_hook_type prev_utility_hook = NULL;
 
static void dbrestrict_utility(Node *parsetree,
  const char *queryString,
  ParamListInfo params,
  DestReceiver *dest,
  char *completionTag,
  ProcessUtilityContext context);

_PG_init and _PG_fini are respectively executed when the library is loaded and unloaded. PG_MODULE_MAGIC is necessary to define a PostgreSQL module in the case where it is loaded by server. dbrestrict_utility will be the function used instead of standard_ProcessUtility in utility.c.

Here is what you need to do with _PG_init and _PG_fini to install and uninstall correctly the hook.
void
_PG_init(void)
{
 prev_utility_hook = ProcessUtility_hook;
 ProcessUtility_hook = dbrestrict_utility;
}
void
_PG_fini(void)
{
 ProcessUtility_hook = prev_utility_hook;
}

The previous hook pointer is saved in a static variable to avoid any conflicts once the library is unloaded.

Then here is dbrestrict_utility, which performs the block on DROP DATABASE.
static
void dbrestrict_utility(Node *parsetree,
  const char *queryString,
  ParamListInfo params,
  DestReceiver *dest,
  char *completionTag,
  ProcessUtilityContext context)
{
 /* Do our custom process on drop database */
 switch (nodeTag(parsetree))
 {
  case T_DropdbStmt:
  {
   DropdbStmt *stmt = (DropdbStmt *) parsetree;
   char *username = GetUserNameFromId(GetUserId());
 
   /*
    * Check that only the authorized superuser foo can
    * drop the database undroppable_foodb.
    */
   if (strcmp(stmt->dbname, undroppabledb) == 0 &&
     strcmp(username, supersuperuser) != 0)
    ereport(ERROR,
     (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
      errmsg("Only super-superuser \"%s\" can drop database \"%s\"",
      supersuperuser, undroppabledb)));
   break;
  }
  default:
   break;
 }
 
 /* Fallback to normal process */
 standard_ProcessUtility(parsetree, queryString, params, dest,
  completionTag, context);
}

An important thing you should do as much as possible: always provide a safe exit by calling the function hook replaces at the end or the beginning of the new function to avoid weird behaviors. In the case of my example not calling standard_ProcessUtility would have resulted in blocking all the utilities… Note that this is of course not mandatory, just be sure about what you do as a hook not correctly coded can break easily a server.

Finally define a Makefile like this one and install the library (the file containing source code is called dbrestrict.c).
MODULES = dbrestrict
PG_CONFIG = pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
include $(PGXS)

OK, now let’s test this feature with a couple of superusers.
postgres=# CREATE ROLE foo SUPERUSER LOGIN;
CREATE ROLE
postgres=# CREATE ROLE foo2 SUPERUSER LOGIN;
CREATE ROLE
postgres=# \c postgres foo2
You are now connected to database "postgres" as user "foo2".
postgres=# CREATE DATABASE foodb; -- before loading restriction
CREATE DATABASE
postgres=# DROP DATABASE foodb;
DROP DATABASE
postgres=# LOAD 'dbrestrict.so';
LOAD
postgres=# CREATE DATABASE foodb; -- after loading restriction
CREATE DATABASE
postgres=# LOAD 'dbrestrict.so';
LOAD
postgres=# DROP DATABASE foodb;
ERROR: Only super-superuser "foo" can drop database "foodb"

Note that superuser “foo2″ is not able to drop the database “foodb” once restriction has been loaded.

However user “foo” can drop it freely.
postgres=# \c postgres foo
You are now connected to database "postgres" as user "foo".
postgres=# LOAD 'dbrestrict.so';
LOAD
postgres=# DROP DATABASE foodb;
DROP DATABASE

As LOAD command is session-based, I had to reload the restriction library each time a reconnection to server was done, but you can make this change permanent by setting shared_preload_libraries appropriately in postgresql.conf.

Feel free to play with the code, it is attached to this post as dbrestrict.tar.gz.
See ya~

pg_buffercache is a PostgreSQL contrib module allowing to get an instant relation-based view of the shared buffer usage by querying the wanted server.

This can be pretty useful for performance analysis of queries on a given relation as it allows to have a look at how much a relation is cached. In the case of data cached for a given relation, you do not need to access data directly on disk to retrieve the data and can directly rely on the cache, so the data fetching is simply faster, by a factor of the order of 1000 (Shared memory/disk speed difference). Take care however that a shared lock is taken when analyzing the shared buffer content, so it can impact concurrent queries.

In order to install pg_buffercache from source code, you need to perform the following commands to install its related files.
cd contrib/pg_buffercache
make install

Depending on your environment and the Postgres packages you installed, you might not need to do that of course. Once this is done the following files are installed in $INSTALL_FOLDER/share/extension.
ls $INSTALL_FOLDER/share/extension
pg_buffercache--1.0.sql
pg_buffercache.control
pg_buffercache--unpackaged--1.0.sql

Then connect to your Postgres server and finish pg_buffercache installation with CREATE EXTENSION command.
postgres=# CREATE EXTENSION pg_buffercache;
CREATE EXTENSION
postgres=# \dx pg_buffercache
List of installed extensions
Name | Version | Schema | Description
----------------+---------+--------+---------------------------------
pg_buffercache | 1.0 | public | examine the shared buffer cache
(1 row)

The view created after installation of the extension called pg_buffercache has several columns.

  • bufferid, the block ID in the server buffer cache
  • relfilenode, which is the folder name where data is located for relation
  • reltablespace, Oid of the tablespace relation uses
  • reldatabase, Oid of database where location is located
  • relforknumber, fork number within the relation
  • relblocknumber, age number within the relation
  • isdirty, true if the page is dirty
  • usagecount, page LRU (least-recently used) count

The buffer ID corresponds (surprisingly!) to the number of the buffer used by the relation. The total number of buffers available is defined by two things:

  • Size of a buffer block, this is defined by the option –with-blocksize when running configure. Default value if 8kB, which is sufficient in most of the situations, but you can go up to 32kB or down to 1kB depending on the situations. In order to change this value, it is necessary to recompile the code and rebuild a database server from the initdb step.
  • Number of shared buffer allocated for the system defined by shared_buffers in postgresql.conf. This can be changed at will by restarting the server.

For example, by using 128MB of shared_buffers with 8kB of block size, there are 16,384 buffers, so pg_buffercache has the same number of 16,384 rows.
With shared_buffers set at 256MB and block-size at 1kB, there are 262,144 buffers.

Let’s have a quick look at the feature with a pgbench database that has been already used with a 5-minute test. This simple query (given by the documentation) provides the number of buffers used by each relation of the current database.
postgres=# SELECT c.relname, count(*) AS buffers
postgres=# FROM pg_buffercache b INNER JOIN pg_class c
postgres=# ON b.relfilenode = pg_relation_filenode(c.oid) AND
postgres=# b.reldatabase IN (0, (SELECT oid FROM pg_database
postgres=# WHERE datname = current_database()))
postgres=# GROUP BY c.relname
postgres=# ORDER BY 2 DESC
postgres=# LIMIT 10;
relname | buffers
-----------------------+---------
pgbench_history | 2515
pgbench_accounts | 1818
pgbench_accounts_pkey | 276
pgbench_tellers | 61
pgbench_branches | 61
pg_attribute | 22
pg_statistic | 11
pg_proc | 10
pg_class | 8
pg_proc_oid_index | 8
(10 rows)

Not only relation data, but also indexes are included in the image given. The server of this example used an amount of shared_buffer of 128MB, so all the relation data was completely cached.
The usage count of each buffer page is a good indication of how many times a buffer is used after being created. In case it is low, it means that the buffers do not survive a long time and that the cache hit ratio is low.

hstore is a PostgreSQL contrib module in core code for a pretty long time. Its code is located in contrib/hstore in source folder. It is particularly useful to store sets of key/value in a single table column.

Since Postgres 9.1, its installation needs to be done in two phases.
First install the hstore library, here done from the source code. Please note that your postgres package normally already contains it.
So after downloading the source code and installing the core, do the following commands.
cd $PG_SOURCE_ROOT
cd contrib/hstore
make install

At this point all the libraries and files related to hstore are installaed in $INSTALL_FOLDER/share/extension.
ls $INSTALL_FOLDER/share/extension
hstore--1.0--1.1.sql hstore--1.1.sql hstore--unpackaged--1.0.sql hstore.control

Then connect to your Postgres server and finish hstore installation with CREATE EXTENSION command.
postgres=# CREATE EXTENSION hstore;
CREATE EXTENSION
postgres=# \dx hstore
List of installed extensions
Name | Version | Schema | Description
--------+---------+--------+--------------------------------------------------
hstore | 1.1 | public | data type for storing sets of (key, value) pairs
(1 row)

With a psql client, ‘\dx’ allows to check the list of extensions already installed on your server.

A new column type called hstore has been added. This is the column used to store the list of key/value pairs for the table.
Let’s take a table referencing a list of products as an example.
postgres=# create table products (id serial, characs hstore);
CREATE TABLE

The insertion of data can be done with several methods. Here are some of them.
postgres=# -- common insertion
postgres=# INSERT INTO products(characs) VALUES ('author=>Dave, date=>"Dec 2012", price=>"500", currency=>"dollar"');
INSERT 0 1
postgres=# -- array-based insertion
postgres=# INSERT INTO products (characs) VALUES (hstore(array['author','date','stock'],array['Mike','Nov 2012','200']));
INSERT 0 1
postgres=# -- single-pair insertion
postgres=# INSERT INTO products (characs) VALUES (hstore('author','Kim')); -- single-element
INSERT 0 1
postgres=# SELECT * FROM products;
id | characs
----+----------------------------------------------------------------------------
1 | "date"=>"Dec 2012", "price"=>"500", "author"=>"Dave", "currency"=>"dollar"
2 | "date"=>"Nov 2012", "stock"=>"200", "author"=>"Mike"
3 | "author"=>"Kim"
(3 rows)

Based on the existing fields, it is also possible to add or update values for a given key using a concatenate-based method.
Here is the update of a key “author”.
postgres=# UPDATE products SET characs = characs || 'author=>Sarah'::hstore where id = 1;
UPDATE 1
postgres=# select * from products where id = 1;
id | characs
----+-----------------------------------------------------------------------------
1 | "date"=>"Dec 2012", "price"=>"500", "author"=>"Sarah", "currency"=>"dollar"
(1 row)

If the key updated is not present in existing list, it is simply added as a new element.

You can also delete single elements.
postgres=# UPDATE products SET characs = delete(characs,'price') where id = 1;
UPDATE 1
postgres=# select * from products where id = 1;
id | characs
----+-------------------------------------------------------------
1 | "date"=>"Dec 2012", "author"=>"Sarah", "currency"=>"dollar"
(1 row)

SELECT query can also use key-based scan. Here for instance this query looks for the products with less than 300 stocks.
postgres=# SELECT id FROM products WHERE (characs->'stock')::int <= 300;
id
----
2
(1 row)

(OK, not the best way of doing for your application but this is just a scholar example!).

On top of that, hstore also supports gin and gist indexes for the operators @>, ?, ?& and ?, as well as btree and hash indexes for '='.
postgres=# create index products_index on products(characs);
CREATE INDEX

Hope this gives a good introduction to hstore.

Here is a short script/memo to find strings inside given file.
The script is assumed to be called strfind. It is written in bash.

Here is the spec of this script.
michael@boheme:~/bin $ strfind ?
Usage: strfind [-i] [filename] [string]
Exemple: strfind "[hc]" text

You can then find strings with commands like:
strfind *.c $TEXT_SEARCH
It is also possible to ignore case distinctions.
strfind -i *.c $TEXT_SEARCH

So here is the script.
#!/bin/bash
#Find string strings in select file extension
 
#Expected base arguments
EXPECTED_ARGS=2
IFLAG=0
 
while getopts 'i' OPTION
do
 case $OPTION in
 i) #Track in repo all untracked files
   IFLAG=1
   #+1 base argument
   EXPECTED_ARGS=$(($EXPECTED_ARGS + 1))
   ;;
 ?) echo "Usage: `basename $0` [-i] [filename] [string]"
   echo "Exemple: `basename $0` \"[hc]\" text"
   exit 0
   ;;
 esac
done
 
if [ $# -ne $EXPECTED_ARGS ]
then
 echo "Usage: `basename $0` [-i] [filename] [string]"
 echo "Exemple: `basename $0` \"[hc]\" text"
 exit 1
fi
 
#Have only 2 or 3 arguments
if [ "$EXPECTED_ARGS" = "2" ]
then
 FILENAME=$1
 TXTSTRING=$2
else
 FILENAME=$2
 TXTSTRING=$3
fi
 
#Print file name and line number
OPTIONS="-Hn"
 
#Don't care about large characters
if [ "$IFLAG" = "1" ]
then
 OPTIONS=$OPTIONS"i"
fi
 
#Execute command
echo find . -name "$FILENAME" -exec grep $OPTIONS $TXTSTRING {} \;
find . -name "$FILENAME" -exec grep $OPTIONS $TXTSTRING {} \;
exit 0;

Here is a short script to replace strings with sed easily written in bash.
#!/bin/bash
#Replace string in file of given extension
#argument 1, extension type
#argument 2, old string
#argument 3, new string
EXPECTED_ARGS=3
 
if [ $# -ne $EXPECTED_ARGS ]
then
  echo "Usage: `basename $0` [extension] [old_str] [new_str]"
  echo "Exemple: `basename $0` php old_text new_text"
  exit 1
fi
 
EXTENSION=$1
OLDSTR=$2
NEWSTR=$3
 
#Simply replace string with sed and erase old file
for file in `find . -name "*.$EXTENSION"`
do
  sed -i "s/$OLDSTR/$NEWSTR/g" $file
done
exit $?

The user can specify an extension, as well as the strings to be replaced and to replace.
This is just a memo, nothing serious…

©2010-2013 Michael Paquier All content is ©Copyright of Otacoo.com 2010-2013. Privacy Policy - Terms of Use