pg_xlogdump is a new contrib module introduced in PostgreSQL 9.3 by this commit.
commit 639ed4e84b7493594860f56b78b25fd113e78fd7
Author: Alvaro Herrera
Date: Fri Feb 22 16:46:24 2013 -0300
 
Add pg_xlogdump contrib program
 
This program relies on rm_desc backend routines and the xlogreader
infrastructure to emit human-readable rendering of WAL records.
 
Author: Andres Freund, with many reworks by Alvaro
Reviewed (in a much earlier version) by Peter Eisentraut

Mainly useful for educational and debugging purposes, pg_xlogdump can be used to understand the internals of PostgreSQL by dumping the WAL (Write-ahead log, which is the basic mechanism used by the server for transaction replay during recovery) into a shape humanly readable.

Just a little bit more information about WAL… Its information is stored in files located in pg_xlog of $PGDATA whose name respect a format name subdivided into 3 sequences of 8 hexa digits defining:

  • Timeline ID
  • Block ID
  • Segment ID

The counter for blocks is incremented once segments are filled.

Postgres includes a couple of functions that can help you to determine in which file is located a given WAL record (pg_xlogfile_name) or what is the current WAL position (pg_current_xlog_location).
michael=# select pg_current_xlog_location();
pg_current_xlog_location
--------------------------
4A/1799988
(1 row)
michael=# select pg_xlogfile_name('4A/1799988');
pg_xlogfile_name
--------------------------
000000010000004A00000001
(1 row)

Here the server is currently on timeline 1, with a Block ID of 4A and a segment ID of 1. The composition of Block ID + segment ID is a LSN or log sequence number, for example ’4A/1799988′. The XLOG files are located in the folder pg_xlog of $PGDATA. Each file has a size of 16MB, and server switches to a new file once this maximum size is reached or once no new file has been written since a time of archive_timeout, parameter of postgresql.conf.

After this digression, let’s have a look at what this utility can do.
The only mandatory option is to specify a start from where the dump will be taken, by either specifying a start LSN with –start or a start WAL file with commands similar to that.
pg_xlogdump --start 0/010EA4D0
pg_xlogdump 000000010000000000000001

If no path is specified to scan the segment files in a given directory, the default is to look if there is a folder called pg_xlog in current directory and get results from it.
In a dump you will get information like that for each WAL record:
rmgr: Heap len (rec/tot): 143/ 175, tx: 688, lsn: 0/02010BE8, prev 0/02010BA0, bkp: 0000, desc: insert: rel 1663/16384/11782; tid 41/54
Each field being in more details:

  • rmgr, the resource manager involved, can be filtered with option -r/–rmgr
  • len, about the length of the record
  • tx, ID of transaction involved with this record, can be filtered with option -x/–xid
  • lsn, log sequence number, including the previous and current lsn, can be filtered with –start and –end.
  • bkp, for the backup block
  • desc, for the description of the action record is doing, and some information related to the relation and page item with which interacts the action

WAL records are divided by resource managers, like database, tablespace, sequence, heap, etc. Based on that you can filter the dump depending on resource manager you want to see. To get a complete list of the resource managers available, simply do that:
pg_xlogdump --rmgr=list

For example, let’s create a sequence on server.
postgres=# create sequence aas;
CREATE SEQUENCE

Here is what you dump when filtering for the resource manager Sequence.
$ pg_xlogdump --start 0/02000000 -r Sequence
rmgr: Sequence len (rec/tot): 158/ 190, tx: 688, lsn: 0/02013C18, prev 0/02013B60, bkp: 0000, desc: log: rel 1663/16384/16390

You can then for example dump all the WAL records for this transaction by running a command like that:
pg_xlogdump --start 0/02000000 -x 688

Only a short introduction of what you can do this module is provided in this post, so for fore details feel free to have a look at the documentation about the available options and what you can do (and cannot do) with pg_xlogdump. Honestly I think it is a good tool to understand the internals of Postgres for newcomers.

This article is made using Eclipse 3.7 and its famous plug-in dedicated to GIT called Egit.

When working on a java project, eclipse is a nice way to implement, code and debug. It is even nicer when it can be used with a version controller like GIT. Since Eclipse 3.5, GIT has its own plug-in called EGit. Let’s be honest. It rocks. The interface is well-thought, and you can manage easily your GIT repository through friendly interface. In case you get crazy with some error messages, you can still easily fallback to the good-old command terminal, and everything made will still be visible in Eclipse. That is really a nice tool

However, when starting a new Java project from an existing GIT repository, you can easily import your project by doing Files -> Import. Then in the import window, select Git -> Projects from GIT.
And in a couple of clicks, you are able to clone existing repositories, and add new projects. The problem is that you generally need to import the code as a general project, and under Eclipse it will be recognized as a Java project.

There is a workaround for this problem after importing a GIT code as a general project. You first need to modify the .project file in the project repository and modify it as below to make it a Java project.
<projectDescription>
 <name>my_project</name>
 <comment></comment>
 <projects>
 </projects>
 <buildSpec>
  <buildCommand>
   <name>org.eclipse.jdt.core.javabuilder</name>
   <arguments>
   </arguments>
  </buildCommand>
   </buildSpec>
   <natures>
    <nature>org.eclipse.jdt.core.javanature</nature>
   </natures>
</projectDescription>

Then, restart eclipse. So now the project became a Java project, but there are still no JRE libraries.
In order to do that, modify .classpath as follows:
<classpath>
  <classpathentry kind="con" path="org.eclipse.jdt.launching.JRE_CONTAINER"/>
</classpath>

What remains to do, depending on the project is to add at least a source folder. This can be done by right-clicking on the project, then New -> Source Folder.
And you are done.

Developers are sometimes looking for cheap solutions to have their own private repositories. There are multiple solutions for open source software such as source forge or GitHub that can provide wide and secured functionalities. However, in the case of the 1st solution it is not possible to create private repositories, and in the second case private repositories are possible but this solution is not worth the money for independant programmers.

The cheapest solution remains in having its own hosting service (buying a domain, creating a free domain, etc). Google that will for sure lead you to free services with dedicated domain names for example.

Most of the time such hosting services are shared-hosting based. This means that multiple users are using the same server for their websites. In this case normal users do not have root rights (well, normal!), so it is impossible to make fine settings of the configuration files of apache, like httpd.conf.

GIT supports http protocol for its repositories for a long time, but the original protocol uses WebDAV and is really heavy and slow. Roughly, you needed to send to remote server entire files and not diffs. However, since version 1.6.6, GIT supports smart HTTP protocol, this has speed up http repositories and you do not even need WebDAV. An important point, WebDAV can be activated in httpd.conf of your apache server with the keywords “DAV On” but this creates an error, that’s why the solution presented here only uses smart http.

So, to set your GIT repositories, what is needed first is GIT installed on your server.
git --version
git version 1.7.0.4

The important point is to have support of the command git-http-backend.

This done, you also need the apache modules mod_cgi, mod_alias, and mod_env to be activated.

Now, let’s go through the whole setting process. In the case of this tutorial, our goal is to create a private repository for a project called foo-project. The private repositories that will be set are protected in read and write by apache group management.

From the root repository of your domain http://www.example.com/, if you have a connection through ssh, go to the root repository that should be called public_html. Then type the following command:
htpasswd -c .htpasswd user-name
You can also do that in another folder or in a subdomain of course.

You will be requested to write a password. This command will create a file called .htpasswd containing data like this:
user-name: $encrypted-passwd
“user-name” can be the name you want. It is an apache-level security group, so if you want you don’t need to use the same user name as your Linux session. This file contains user and password data for the access to private repositories.

It is possible to add new users to this file with commands like:
htpasswd .htpasswd new-user-name

Then create a file called .htgroup. It contains the following data:
foo_write: user-name new-user-name
This file will be used to control the group data of apache. You can create for each private repository a group with a list of users. One line has to be used for each group. Keep in mind that it is easier to maintain the group list in a common file. However you can set group file in different files if you wish. Just don’t forget to list those files in appropriate .htaccess files.

Then it is time to create the access control to private repositories. Create a folder called git in the root folder public_html and move in it:
mkdir git
cd git

There you need to create a new CGI script that will be used to rewrite requested URL for private GIT repositories. With an editor, create a file called git-http-backend.cgi with the following data in it.
#!/bin/sh
#first we export the GIT_PROJECT_ROOT
export GIT_PROJECT_ROOT=/to/site/folder/public_html/git/
 
if [ -z "$REMOTE_USER" ]
then
 export REMOTE_USER=$REDIRECT_REMOTE_USER
fi
 
#and run your git-http-backend
/usr/bin/git-http-backend

GIT_PROJECT_ROOT is an environment variable pointing to the root folder of your GIT repositories. A mistake here may lead to an error 500…

Depending on the server of your shared hosting service, git-http-backend may not be in /usr/bin/ but in /usr/lib/git-core/ or whatever. Be sure to check where it is with the command:
which git-http-backend

Then create an .htaccess file in git to control URL rewrite. It contains the data:
Options +ExecCgi
 
#This is used for group/user access control
AuthName "Private Git Access"
AuthType Basic
AuthUserFile /to/site/folder/public_html/.htpasswd
AuthGroupFile /to/site/folder/public_html/.htgroup
Require valid-user
 
#This is the rewrite algorithm
RewriteEngine on
RewriteBase /git
SetHandler cgi-script
RewriteRule ^([a-zA-Z0-9._]*\.git/(HEAD|info/refs|objects/(info/[^/]+|[0-9a-f]{2}/[0-9a-f]{38}|pack/pack-[0-9a-f]{40}\.(pack|idx))|git-(upload|receive)-pack))$ /git/git-http-backend.cgi/$1

Then it is time to create the GIT repository of foo-project and move in it.
mkdir foo-project.git
cd foo-project.git

Now you should be in folder /to/site/folder/public_html/git/foo-project.git.

Then initialize your GIT repository with the following commands.
git --bare init
git --bare update-server-info
cp hooks/post-update.sample hooks/post-update
chmod a+x hooks/post-update
touch git-daemon-export-ok

This basically makes all the necessary settings to allow your folder to use smart http mode. If you don’t care about GIT details, just copy/paste that!

What finally remains is to create an .htaccess file in public_html/git/foo-project.git to control access to this repository.
Allow from all
Order allow,deny
#foo_write is the group is .htgroup. All the users of this group will be authorized to access this repository at will.
Require group foo_write

The setting on remote side is done. So now, here is how to access to the remote from your local machine.
You may either clone the new git repository.
git clone http://www.example.com/git/foo-project.git

Or add a remote URL.
mkdir myproject
cd myproject
git init
git remote add myproj http://www.example.com/git/foo-project.git
git fetch myproj

It may be necessary to install the library curl and set the file called .netrc in your home repository (accessible with $HOME/.netrc) like this:
machine www.example.com
login user-name
password $mypasswd

If you don’t want to use .netrc file you can directly add you user name in the remote URL.
http://www.example.com/git/foo-project.git
Becomes
http://user-name@example.com/git/foo-project.git
In this case you will be requested a password each time you interact with the remote folder. This is annoying so you should stick with curl.

Then, manage your folder as you always do. First begin by pushing your first commits to your newly-made repository. Here is an example:
echo "My first commit" > README
git add README
git commit -a
git push origin master

When pushing to your repository, you may find upload package errors. A common message is:
error: unpack failed: unpack-objects abnormal exit
Don’t panic. You made it well. It should not occur normally but it may happen in certain environments. This is a write permission issue. Be sure to have the repository “objects” set with correct write permissions to allow a push to be written correctly in remote repository.

An additional tip…
There are also a nice pure php solution to allow you to have a gitweb-like service in pure PHP.
GitPHP is a web frontend for git repositories. This is extremely handy in a shared hosting environment as you do not need to set httpd.conf and you don’t need root rights on your server.

©2010-2013 Michael Paquier All content is ©Copyright of Otacoo.com 2010-2013. Privacy Policy - Terms of Use