Cloning a B-Translator Server
Table of Contents
- 1. Installation
- 2. Setting the domain name
- 3. Replacing NGINX with Apache2
- 4. Enable other drupal features
- 5. Start ssh
- 6. Reset the password of drupal admin
- 7. Transfer content
- 8. Fix path aliases and menus
- 9. Transfer drupal private settings
- 10. Get and import PO files
- 11. Sync vocabulary data
- 12. Sync users and contributions
- 13. Switching to the new server
Why do I need to clone a B-Translator server? An obvious reason is to have a backup server. In case that something goes wrong with the main server I can quickly switch to the backup server, until I find and fix the problem.
A clone can also be used for testing. Before applying something on the main server I can test it first on a clone/backup server.
Another reason for cloning can be load-balancing. It is possible to keep several clone servers synchronised with each-other, and then the work can be shared between them with a load balancer. I haven't tried this yet, but I think that it should work.
It can be possible and even simple to make identical clones just by copying the physical disk or the filesystem. However I prefer to build a new server from scratch and copy only the relevant data from the old server.
1 Installation
First, I build a new server from scratch:
mkdir /var/chroot cd /var/chroot/ git clone https://github.com/dashohoxha/B-Translator.git nohup nice B-Translator/install/install.sh btr-1 & tail -f nohup.out
Then configure it:
chroot btr-1/ /tmp/install/config.sh
And then reboot the host:
reboot ## it is advisable to reboot the host after this installation
The last step of configuration is about setting the primary language of translation (sq in my case), and a list of auxiliary languages. This list should include at least fr, since it is used instead of POT (PO template) files, when they are missing. But the features related to auxiliary languages are not implemented yet, so it is not much useful to include other languages on that list, and I leave just fr.
2 Setting the domain name
If I can assign a domain name to this copy of B-Translator (for
example test.l10n.fs.al
), but during the configuration it is not
given properly, then I can correct/change it like this:
chroot /var/chroot/btr-1 cd /var/www/btranslator/profiles/btranslator install/config/domain.sh
If this is a local copy (installed on my local machine) and its
domain name is for example l10n.org.xy
, then I also add this
domain name on the file /etc/hosts
of my machine:
127.0.0.1 l10n.org.xy localhost
This way I can access it by typing https://l10n.org.xy
on the
browser location (not https://127.0.0.1
or https://localhost
).
3 Replacing NGINX with Apache2
I have used NGINX as the default web server because it is supposed
to be faster and more responsive under heavy load. Unfortunately it
has been working very slow, for some reasons that I have not been
able to find and fix. On the log file
/var/log/nginx/btranslator.error.log
I see messages like this:
2013/07/15 13:31:57 [info] 10294#0: *47 client closed prematurely connection while SSL handshaking, client: 127.0.0.1, server: l10n.org.xx 2013/07/15 13:31:57 [info] 10294#0: *48 client closed prematurely connection while SSL handshaking, client: 127.0.0.1, server: l10n.org.xx
Note: If find out what is wrong with NGINX and manage to fix it, please let me know.
So, I replace NGINX with Apache2 like this:
chroot /var/chroot/btr-1 cd /var/www/btranslator/profiles/btranslator dev/apache2.sh start
It will stop services nginx, php5-fpm and memcached, will
start apache2, will disable drupal module memcache, and will
modify properly settings.php
.
However there are also some changes outside chroot that should be
done. The script dev/apache2.sh start
cannot do them automatically
(because it runs inside the chroot), so I have to do them manually.
On the init script /etc/init.d/chroot-btr-1
I make this change:
#CHROOT_SERVICES="cron php5-fpm memcached mysql nginx" CHROOT_SERVICES="cron mysql apache2"
4 Enable other drupal features
After installation I also enable some features that are optional and are not enabled by default during installation:
drush features-list drush --yes pm-enable btranslator_simplenews drush --yes pm-enable btranslator_mass_contact ## drush --yes pm-enable btranslator_googleanalytics ## drush --yes pm-enable btranslator_drupalchat ## drush --yes pm-enable btranslator_janrain
4.1 Enable the feature btranslator_fb
Note: The feature btranslator_fb
is not yet finished and
tested properly.
Enable it like this:
## drush --yes pm-enable btranslator_fb
After installing btranslator_fb
, the configuration part related
to FB should be un-commented, at the end of the file
/var/www/btranslator/sites/default/settings.php
:
// /* fb config $conf['fb_api_file'] = 'profiles/btranslator/libraries/facebook-php-sdk/src/facebook.php'; include "profiles/btranslator/modules/contrib/fb/fb_url_rewrite.inc"; include "profiles/btranslator/modules/contrib/fb/fb_settings.inc"; if (!headers_sent()) { header('P3P: CP="We do not have a P3P policy."'); } // fb config */
If you forget to do it, you will notice performance degrade with the site.
5 Start ssh
If this copy of B-Translator is remote, then I install ssh as well for accessing it easily and for using remote drush commands:
chroot /var/chroot/btr-1 cd /var/www/btranslator/profiles/btranslator dev/install-sshd.sh
This script will also take care to change the ssh port to 2201, in order to avoid any conflicts with any existing daemon on the host environment, and also for increased security.
For drush remote access to work correctly, the public/private key ssh access should be set up and configured as well. For more detailed instructions on how to do it see: http://dashohoxha.blogspot.com/2012/08/how-to-secure-ubuntu-server.html
6 Reset the password of drupal admin
I almost always forget the password of admin (the primary user of Drupal) that I assign during installation. So, I have to reset it:
drush user-password admin --password="new-password"
By the way, on the file /etc/drush/drushrc.php
you can see this
drush setting:
<?php // by default use the B-Translator root directory $options['r'] = '/var/www/btranslator';
This means that the root directory of drush
is always
/var/www/btranslator
, no matter where we call it from.
7 Transfer content
On the master (main/live) server, I export all the content as a feature, with the help of the module node_export. But first I have to disable the existing btranslator_content feature, otherwise the feature export will fail.
drush --yes pm-disable btranslator_content drush --yes features-export --destination=/tmp btranslator_content_1 node_export_features tar --create --gzip --file=btranslator_content_1.tgz --directory=/tmp btranslator_content_1
Now I transfer btranslator_content_1.tgz
to the clone server and
replace the existing content with it:
cd /var/www/btranslator/profiles/btranslator cd modules/features/ tar --extract --gunzip --file=btranslator_content_1.tgz drush --yes pm-disable btranslator_content drush delete-all all ## delete all existing nodes drush --yes pm-enable btranslator_content_1
8 Fix path aliases and menus
Path aliases and some menus have to be fixed (recreated) manually. I couldn't find any modules, drush commands or scripts that can transfer them automatically. If you know any tricks to export/import them automatically, please let me know.
I transfer manually the configuration of the Homebox as well. I
open admin/structure/homebox
on both sites (the main and the
clone), export the configuration of 'dashboard' from the main, then
copy/paste and import it on the clone.
9 Transfer drupal private settings
Private settings are those variables that are site specific and
cannot be included in features, for example: disqus_domain
,
disqus_userapikey
, disqus_publickey
, disqus_secretkey
, etc.
We can transfer them like this:
- Save them on the main site:
cd /var/www/btranslator/profiles/btranslator modules/features/save-private-vars.sh
It will generate the file
restore-private-vars.php
. - Transfer
restore-private-vars.php
to the clone site and then apply it like this:drush php-script restore-private-vars.php
Note: If the clone site will be used for testing, you may consider
to edit manually the file restore-private-vars.php
, before
applying it, and change some values. For example, I usually change
email addresses from info@l10n.fs.al
to info+test@l10n.fs.al
.
I also enable email rerouting by changing these variables:
$variables['reroute_email_enable'] = 1; $variables['reroute_email_enable_message'] = 1;
10 Get and import PO files
The database of translations is almost empty (it has only the PO files that were imported for testing during installation). Downloading and importing all the PO files is easy (but it takes a long time).
10.1 Download (get) PO files
cd /var/www/btranslator_data nohup nice get/all.sh & tail -f nohup.out or cd get/ ./gnome.sh ./kde.sh ./firefox-os.sh ./drupal.sh ./office.sh ./mozilla.sh ./wordpress.sh ./ubuntu.sh
Note: These scripts get the data from some URL. They should be checked first, to make sure that the URL still works or that we are getting the latest version.
Note: Make sure that hostname
is listed on /etc/hosts
otherwise the command svn checkout
will not work (strange, but
that's how it is). For example if the output of the command
hostname
is dashamir
, then /etc/hosts
should look like this:
127.0.0.1 l10n.org.xx localhost dashamir
10.2 Import PO files
cd /var/www/btranslator_data nohup nice import/all.sh & tail -f nohup.out
11 Sync vocabulary data
Vocabulary is a pseudo-project, its PO file does not really belong to the translation of any program. I use it to collect interesting terms and translations that I encounter while translating the other projects. It can help as a reminder (in case that I forget the translation of a term). It is also useful for discussing translations of difficult terms with other people, and indirectly it helps to ensure consistency among the translations. Terms are added to vocabulary by the translators. In order to transfer them from one instance of B-Translator to another, it can be exported as a PO file on one system and imported to the other.
- Export
vocabulary-sq.po
:cd /var/www/btranslator_data export/export.sh misc vocabulary sq $(pwd)/tmp mv tmp/vocabulary/vocabulary-sq.po . rm -rf tmp
- Transfer
vocabulary-sq.po
to the other system and them import it:cd /var/www/btranslator_data mv vocabulary-sq.po vocabulary/ import/vocabulary.sh
12 Sync users and contributions
Now the cloned site is almost identical with the primary site in terms of Drupal settings and configuration and in terms of translation data (projects that are imported, strings and their translations, etc.). What is still missing is the list of users that are registered on the primary site, as well as their contributions (votes and suggested translations).
We can transfer them like this:
- Export them on the primary site:
cd /var/www/btranslator_data db/export-users.sh db/export-contributions.sh
- Transfer
*.sql.gz
files to the clone site and import them:cd /var/www/btranslator_data db/import-users.sh users-20130717.sql.gz db/import-contributions.sh contributions-00000000-20130717.sql.gz
Note: Once you have transferred all the users, the clone site will
send them daily a string for review, in addition to the one that is
sent by the primary site. We can prevent this by disabling the
cron. Edit the file /etc/cron.d/drupal7
and comment the line that
starts the cron.
13 Switching to the new server
Suppose that I want to make the cloned server primary. In this case there are some steps that should be done:
- Transfer GoogleApps verification files. I use GoogleApps for email
accounts etc. (it offers 10 email accounts for free). To verify
that I own this domain, GoogleApps requests me to put a certain
file on the root of my webserver. This file looks like
google9350a51ac2d503bf.html
and I place it on/var/www/btranslator
. - Transfer SSL certificates. I have obtained a free SSL certificate
for my site (see:
http://arstechnica.com/security/2009/12/how-to-get-set-with-a-secure-sertificate-for-free/).
The configuration on
/etc/nginx/sites-available/default
looks like this:ssl_certificate /etc/ssl/certs/ssl-cert-l10n_fs_al.pem; ssl_certificate_key /etc/ssl/private/ssl-cert-l10n_fs_al.key;
The corresponding configuration on
/etc/apache2/sites-available/default-ssl
looks like this:SSLCertificateFile /etc/ssl/certs/ssl-cert-l10n_fs_al.pem SSLCertificateKeyFile /etc/ssl/private/ssl-cert-l10n_fs_al.key
The files .pem and .key need to be transferred to the new server and the configuration files of nginx and apache2 should be modified properly.
- Enable cron.
Since I have disabled cron (on a test site), I have to enable it again
by un-commenting it on
/etc/cron.d/drupal7
. - Replace test settings with live settings. Export drupal setting on the
main site with
modules/features/save-private-vars.sh
and then import them on the new site withdrush php-script restore-private-vars.php
. - On the DNS server I change the record of
l10n.fs.al
to point to the new IP. But the DNS change may take about 2 days to be propagated worldwide. So, after 2-3 days I do again a transfer of users and contributions from the old server to the new one. These transfer operations are designed to be idempotent, which means that the result will be the same even if they are applied many times.