Thursday, June 24, 2021

Monitoring the real progress in Linux while copying file to a slow USB drive

For tracking file copy process in Linux systems, one option is using Pipe Viewer (pv). It is a terminal-based tool for monitoring the progress of data through a pipeline. 

$ pv source-file > destionation-file
1,36GiB 0:05:24 [4,28MiB/s] [==================================================>] 100% 

But when it comes to copy files to slow USB drives, the progress may reach to 100% immediately and then stay hanged without giving any clue about when the copying will be finished. This is because of that the Linux kernel first makes disk writes into cache, and later flushes them to disk asynchronously. To get a real idea during the process about when it will completed, we may set this cache to a specific size (10 MB was enough in our case) like below:

$ sudo sysctl vm.dirty_bytes=104857600

For more information:




Wednesday, May 19, 2010

Mirroring of Subversion Repositories

[Turkce]

In this article, I want to explain creating and maintaining a read-only subversion mirror, which may be used for backup purposes in addition to mirroring.

Let's think a scenario that there are two server machines which run Debian GNU/Linux for O.S. One of them act as the main svn repository, and the other acts as the mirror of it. Thus we talk about two repository:
  • Main or source repository. In our example we assign "main-site.x.y.z" to it as domain name.
  • Mirror or target repository. In our example we assign "mirror-site.x.y.z" to it as domain name.
In the main repository, we should define a mirror user (in our example, it is svn) and give read-only access to this user. So, in the main repository directory (in our examle, it is /home/svn/main-repo), contents of "conf/authz" file must contain an entry for this user like below:
[/]
svn = r
And, "conf/svnserve.conf" must give at least read access to authorized logins like below:
auth-access = read
At the mirror site, first, install subversion server:
apt-get install subversion
Configure it to run via inetd superserver, by adding following entry to /etc/inetd.conf:
svn  stream  tcp  nowait  svn  /usr/bin/svnserve svnserve -i -r /home/svn
Restart inetd superserver:
/etc/init.d/inetutils-inetd restart
or
/etc/init.d/openbsd-inetd restart
Create a mirror repository:
su - svn
svnadmin create mirror-repo
cd mirror-repo
Create a hook script to handle syncronization operations:
touch hooks/pre-revprop-change
chmod u+x hooks/pre-revprop-change
Put following content into the hook script. In this content, "svn" denotes the mirror user which we configured with read-only access at main repository site:
#!/bin/sh
if [ "$3" = "svn" ]; then exit 0; fi
echo "Only the - svn - user may edit revision properties through svnsync" >&2
exit 1
Between main and mirror sites, especially if they both are not in a local secure network, we should prefer using secure connection. For this aim, at the mirror site, we create public and private keys:
su - svn
cd ~/.ssh
ssh-keygen -t dsa -b 1024 -f mirror-site.key
In the .ssh directory, this will create the public key named "mirror-site.key.pub". Add this public key into the .ssh/authorized_keys file of the svn user at the main site.

At the mirror site, login as svn user:
su - svn
Set the SVN_SSH environment variable like this:
export SVN_SSH="ssh -i /home/svn/.ssh/mirror-site.key"
Initialize the mirror repository:
svnsync init file:///home/svn/mirror-repo svn+ssh://svn@main-site.x.y.z/home/svn/main-repo
Start first synchronization:
svnsync sync file:///home/svn/mirror-repo
Edit crontab to schedule synchronization operations:
crontab -e
Add following entry into crontab (in this example, for every hour):
0 * * * *     (export SVN_SSH="ssh -i /home/svn/.ssh/mirror-site.key"; svnsync sync file:///home/svn/mirror-repo)
This automation keeps mirror repository synchronized with main repository at most with one hour latency. Sometimes errors occurs because of unfinished synchronization operations. Then following command may help cleaning locks at mirror site:
svn propdel svn:sync-lock --revprop -r 0 file:///home/svn/mirror-repo
That is it.