Setting up database replication on MySQL

Database replication is, as the americans would say, quite awesome. It provides, for one, an always fresh database backup. But most of the time you won't do it for backups. The greatest thing about database replication is that you can use it for load balancing.

Just replicate the database onto another server, and you have two atabase servers for SELECTs and you can leave the master for INSERTs, DELETEs and UPDATEs. The only bad thing about this is that you will need different connections for read and write (nothing that Dr. Nic's magic can't resolve), and you must have a fast connection between the two servers or you may have records that don't show up instantly when you store them (that's bad for a web application).

First of all, I'm supposing you already have a production database running in a server. If you start from scratch with several database servers, then it's much easier, as you don't have to dump/restore databases.

Go to your master server (let's say her name is scherie), and edit /etc/mysql/my.cnf:

Make sure you have set server-id to 1:

server-id = 1

You need to uncomment the bind-address line, as we need port 3306 open for our slave:

#bind-address = 127.0.0.1

By default, Debian for instance, already saves the transaction log. Uncomment or add it if you don't have it:

log_bin  = /var/log/mysql/mysql-bin.log

Add a line for the database that will be dumped to the binary log:

binlog-do-db=database_to_be_replicated

You can now restart the database:

scherie:~# /etc/init.d/mysql restart

Open a mysql client, and create a new user called replication_user (change this to whatever you want):

scherie:~# mysql -u root -p

mysql> grant replication slave on *.* to 'replication_user'@'%' identified by 'password';

Now we have to lock temporarily the database while we proceed to dump it, so we have the correct binary log offset (Position).

mysql> flush tables with read lock;
Query OK, 0 rows affected (0.00 sec)

mysql> show master status\G;
*************************** 1. row ***************************
File: mysql-bin.00002
Position: 230
Binlog_Do_DB: database_to_be_replicated
Binlog_Ignore_DB:

1 row in set (0.00 sec)


Don't close the mysql client
. Leave the terminal open, otherwise, the database will be unlocked. Write down the File and Position values, we will need them later.

Dump the database (you could use load data from master, but it's deprecated and will be removed in the future).

scherie:~# mysqldump -u root -p database_to_be_replicated > database_to_be_replicated.sql
Enter password:

Once dumped return to the open terminal, and unlock tables:

mysql> unlock tables;
Query OK, 0 rows affected (0.00 sec)

mysql> quit;
Bye

It would be a good idea to compress the database dump before transferring it:

scherie:~# bzip2 database_to_be_replicated.sql
scherie:~# scp database_to_be_replicated.sql daniel@athena:

You should also allow access from the slave to the master server, but block it otherwise (basic iptables rules):

-A INPUT -s ip_of_slave_server -p tcp -m tcp --dport 3306 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 3306 -j REJECT --reject-with icmp-port-unreachable

You're done with the master configuration. Let's open a terminal to our slave server (let's say she is called athena):

Check whether we have access to the master:

athena:~# mysql -u replication_user -p -h scherie
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or g.
Your MySQL connection id is 38
Server version: 5.0.45-Debian_1-log Debian etch distribution

We can restore now the database dump from the master:

athena:~# bzip2 -d insolitus.sql.bz2
athena:~# mysql -u root -p insolitus < insolitus.sql
Enter password:

Add the following data to the slave's my.cnf, under the [mysqld] section:

server-id                   = 2
master_host             = ip_of_master_server
master_user             = replication_user
master_password    = password
replicate_do_db       = database_to_be_replicated

Restart mysql:

athena:~# /etc/init.d/mysql restart
Stopping MySQL database server: mysqld.
Starting MySQL database server: mysqld.

Connect as root to our slave server:

athena:~# mysql -u root -p

Stop the slave:

mysql> slave stop;
Query OK, 0 rows affected (0.00 sec)

Now, we need those File and Position parameters we got from the master, change the values on master_log_file and master_log_pos for those you wrote down previously:

mysql> change master to master_host='ip_of_master_server', master_user='replication_user', master_password='password',master_log_file='mysql-bin.000002', master_log_pos=230;
Query OK, 0 rows affected (0.01 sec)

We start the slave:

mysql> slave start;
Query OK, 0 rows affected (0.00 sec)

Now, if we issue a show slave status, you should see the following variables to 'Yes':

mysql> show slave statusG;

          Slave_IO_Running: Yes
          Slave_SQL_Running: Yes

If you have Slave_IO_Running set to No, you haven't probably locked correctly the database prior to dumping it. You will have to repeat the dumping process locking correctly the database.

If everything is correct, your database will be automatically replicated. Doing a simple show processlist; in either the master or the slave will show whether the communication is working.

You can also take a look at the full guide to replication at the MySQL official site.



 

Testing Nexenta (GNUSolaris, Elatte) in Parallels

Nexenta is a GNU/Solaris distribution, built on top of the OpenSolaris kernel, licensed by Sun. The distribution is based in Debian, and already worked quite well when I tested it more than a year ago, using the LiveCD. It has improved since, and now has ZFS support, so I decided to give it another try.

The installation does not work using an ISO image in Parallels. It will crash after the Loading Nexenta message, restarting the installation constantly.



I recorded it to a rewritable DVD and voilà!




The installation is similar to a non-graphical Linux one, like Debian. If you don't want any special partition layout, you can pretty much let it install by default.





You can play with the extremely difficult bundled Tetris while the installation finishes. Note the message at the right: Won't give you this one. Damn!




After installation, the X Window System starts but the resolution is somehow wrong, and you can't see a thing. And you can't change to non-graphical interface either.

I solved this by doing a nmap sweep of the network assigned by Parallels. In my case:

$ nmap -v -v 10.211.55.0/24

Discovered 2 hosts. One is the Mac host, and the other is Nexenta.


Scanning 2 hosts [1697 ports/host]
Host 10.211.55.2 appears to be up ... good.
Host 10.211.55.4 appears to be up ... good.

Luckily Nexenta enabled sshd by default. And a Samba server?

Interesting ports on 10.211.55.4:
Not shown: 1690 closed ports
PORT STATE SERVICE
22/tcp open ssh
25/tcp open smtp
111/tcp open rpcbind
139/tcp open netbios-ssn
445/tcp open microsoft-ds
587/tcp open submission
4045/tcp open lockd

Log in using the user created during the installation.

$ ssh daniel@10.211.55.4

daniel@solaris:~$ uname -a
SunOS solaris 5.11 NexentaOS_20070402 i86pc i386 i86pc Solaris

Edit /etc/X11/xorg.conf to fix the X Window Server:

Remove wacom drivers (those in InputDevice). And be sure to remove the includes at the bottom (ServerLayout).

For the mouse to work, remove the Protocol line, and the CorePointer option, if present. Leave it this way:

Section "InputDevice"
Identifier "Configured Mouse"
Driver "mouse"
Option "Device" "/dev/mouse"
EndSection

Vesa works nicely in Parallels. Leave Monitor default.

Section "Device"
Identifier "Generic Video Card"
Driver "vesa"
BusID "PCI:0:2:0"
EndSection

Screen looks like this:

Section "Screen"
Identifier "Default Screen"
Device "Generic Video Card"
Monitor "Generic Monitor"
DefaultDepth 16
SubSection "Display"
Depth 16
Modes "1024x768" "800x600" "640x480"
EndSubSection
SubSection "Display"
Depth 24
Modes "1024x768" "800x600" "640x480"
EndSubSection
EndSection

This is what ServerLayout should contain:

 
Section "ServerLayout"
Identifier "Default Layout"
Screen "Default Screen"
InputDevice "Generic Keyboard"
InputDevice "Configured Mouse"
EndSection


Kill gdm to restart:

root@solaris:/etc/X11# ps -Af |grep gdmgreeter

    root  3256   673   0 23:32:47 pts/1        0:00 grep gdmgreeter
    gdm  3246  3236   2 23:32:00 ?            0:02 /usr/lib/gdm/gdmgreeter

root@solaris:/etc/X11# kill -9 3246

And we are in.




ZFS. Yay!

root@solaris:/export/home/daniel# zfs list
NAME USED AVAIL REFER MOUNTPOINT
home 36.3M 22.4G 36.2M /export/home
root@solaris:/export/home/daniel# df -h
Filesystem size used avail capacity Mounted on
/dev/dsk/c0d0s0 7.9G 1.8G 6.0G 24% /
/devices 0K 0K 0K 0% /devices
/dev 0K 0K 0K 0% /dev
ctfs 0K 0K 0K 0% /system/contract
proc 0K 0K 0K 0% /proc
mnttab 0K 0K 0K 0% /etc/mnttab
swap 409M 588K 408M 1% /etc/svc/volatile
objfs 0K 0K 0K 0% /system/object
/usr/lib/libc/libc_hwcap1.so.1
7.9G 1.8G 6.0G 24% /lib/libc.so.1
fd 0K 0K 0K 0% /dev/fd
swap 408M 12K 408M 1% /tmp
swap 408M 164K 408M 1% /var/run
home 22G 36M 22G 1% /export/home



 

Configuring an NFS Server

NFS (Network File System) is a protocol that allows to export a filesystem to other remote machines, where it can be mounted like a local disk. While Samba is probably better for domestic use (more available clients, printer sharing), NFS is better for sharing partitions across servers. Actually, you will find NFS easier to configure than Samba, and if you don't need to share printers, and your computers are Unix based (Mac or Linux, basically), I think it's better than Samba even for domestic use.

First of all, we need to install portmap if it's not installed already (if you have a window manager, such as GNOME, it will be already installed).
europa:~# apt-get install portmap

The next step is checking whether it accepts connections other than from 127.0.0.1. To do so, we have to check /etc/default/portmap, commenting the following line if it exists:
#OPTIONS="-i 127.0.0.1"

Later on, we will firewall portmap. Now, we can restart the service as usual.

europa:~# /etc/init.d/portmap restart
Stopping portmap daemon....
Starting portmap daemon....

The following command will install both NFS server and client in deb-based systems:
europa:~# apt-get install nfs-kernel-server nfs-common

Once installed, we need to provide the NFS server with the shares we want. The file we need to edit is /etc/exports. Here's an example:

/home/user                       192.168.1.3(rw,sync,no_subtree_check)
/directory_to_export     192.168.1.2(ro,sync,no_subtree_check) 192.168.1.5(rw,sync,no_subtree_check)


The first line will export the directory /home/user, allowing access to 192.168.1.3, with read and write permissions.
The second line will export the directory /directory_to_export, allowing read only access to 192.168.1.2 and read and write access to 192.168.1.5.

Once we have configured our exports, we need to run exportfs to update the configuration:
europa:~# exportfs -a

Mounting a share from a client (supposing our server is 192.168.1.1, and that we are connecting to /home/user) is as easy as:
scherie:~# mount -t nfs 192.168.1.1:/home/user /mnt/home

Or we can add it to /etc/fstab as a regular partition:
192.168.1.1:/home/user /mnt/home nfs defaults 0 0

The last we should do is reject access to portmap and the NFS server (even if nobody will be able to export a directory since they are not on the exports file).

This will allow localhost and 192.168.1.2 to connect to portmap, and reject everyone else. You can obviously use a range in the source to allow an entire LAN.

europa:~# iptables -t filter -A INPUT -s 127.0.0.1 -p tcp -m tcp --dport 111 -j ACCEPT
europa:~# iptables -t filter -A INPUT -s 127.0.0.1 -p udp -m udp --dport 111 -j ACCEPT
europa:~# iptables -t filter -A INPUT -s 192.168.1.2 -p tcp m tcp --dport 111 -j ACCEPT
europa:~# iptables -t filter -A INPUT -s 192.168.1.2 -p udp -m udp --dport 111 -j ACCEPT
europa:~# iptables -t filter -A INPUT -p tcp -m tcp --dport 111 -j REJECT --reject-with icmp-port-unreachable
europa:~# iptables -t filter -A INPUT -p udp -m udp --dport 111 -j REJECT --reject-with icmp-port-unreachable

This will do the same thing with the NFS server.

europa:~# iptables -t filter -A INPUT -s 127.0.0.1 -p tcp -m tcp --dport 2049 -j ACCEPT
europa:~# iptables -t filter -A INPUT -s 127.0.0.1 -p udp -m udp --dport 2049 j ACCEPT
europa:~# iptables -t filter -A INPUT -s 192.168.1.2 -p tcp m tcp --dport 2049 -j ACCEPT
europa:~# iptables -t filter -A INPUT -s 192.168.1.2 -p udp -m udp --dport 2049 -j ACCEPT
europa:~# iptables -t filter -A INPUT -p tcp -m tcp --dport 2049 -j REJECT --reject-with icmp-port-unreachable
europa:~# iptables -t filter -A INPUT -p udp -m udp --dport 2049 -j REJECT --reject-with icmp-port-unreachable



 

Using FastCGI with Apache2 on Debian AMD64

I have been using Apache2 with FastCGI to run in.solit.us. Some time ago, I moved to a new AMD64 server, and apparently, there is no Debian support for FastCGI on Apache2 for the amd64 platform. There is, however, a package for Apache 1.3.

For some time, I have been using libapache2-mod-fcgid, which is a replacement for libapache2-mod-fastcgi, but there are some things that are not working with fcgid, such as the upload progress status stuff in rails, and, for example, HTTP Authentication using the browser headers. And fastcgi seems to perform better, too.

Fortunately, the Ubuntu distribution has a deb package for libapache2-mod-fastcgi for the amd64 platform, and I was able to use it on my Debian installation. You only need to have apache2 (it will fail with apache2.2) installed, and it will install flawlessly.

If you are using testing, and therefore you had apache2.2 installed, you can downgrade it changing sources.list, removing apache2 and reinstalling it again.




 

Evitar hotlinking o robo de imágenes en servidor web

Imagino que a los que tengáis blogs, o webs en general os habrá pasado que al mirar los logs, una parte nada despreciable del ancho de banda consumido por el servidor web corresponde al "robo de imágenes" o hotlinking por parte de otras web donde insertan un "<img src>" con un link a una de nuestras imagenes.

Últimamente me pasa con foros y sobre todo, con eBay. Gente que vende un Mac de segunda mano, por poner un ejemplo real, imagino que buscando imagenes en google, encuentra una foto de algún post que he hecho y la inserta directamente.



 

Instalar Ruby on Rails en Linux

Después de leer multitud de alabanzas hacia Ruby on Rails en varios blogs y de boca de cierto individuo, me he decidido a instalarlo y comprobar si es tan bueno como dicen.

He aprovechado para hacer un pequeño tutorial sobre el proceso de instalación en Debian (o en cualquier otro Linux), que lo cierto es que es bastante complejo. También incluyo un mini-ejemplo de aplicación web en Ruby on Rails.

Os adelantaré que el proceso de deploy de la aplicación web me ha recordado muchísimo al deploy de servlets en Tomcat, y de hecho, según dicen, Ruby on Rails pretende ser una suerte de PHP pero orientado a objetos, y además, elegante y divertido de programar. En sí, Ruby es el lenguaje de programación, que ya existía previamente, y Rails es un framework para Ruby para desarrollar aplicaciones web.



 

Eurocastellanizar GNU/Linux Debian.

Instalaremos las locales.

host:~# apt-get install locales language-env user-es euro-support

host:~# echo es_ES@euro ISO-8859-15 > /etc/locale.gen

host:~# echo export LANG=es_ES@euro >> /etc/bash.bashrc

host:~# echo export LC_ALL=es_ES@euro >> /etc/bash.bashrc

Regeneraremos nuestras locales.

host:~# locale-gen




 

SSH - SHFS y transferencia de archivos

En este articulo explicaré como sacarle el mayor partido a nuestro servidor SSH.

Además de un sustituto seguro de Telnet, SSH nos permite montar directorios, transferir archivos, o simplemente usarlo como un servidor FTP. Puede ser un buen sustituto de Samba o NFS si sólo queremos usarlo para compartir una partición con varias máquinas, o de nuestro servidor inseguro FTP.



 

Configurar DHCP en Linux Debian

En este artículo se explica cómo configurar a un servidor en Linux para que asigne automáticamente IPs a las máquinas que se conecten a nuestra red de área local mediante un servidor DHCP (Dynamic Host Configuration Protocol).

Esto es especialmente útil para ofrecer conexión a una red inalámbrica abierta sin necesidad de que los clientes tengan que configurar sus equipos, o en redes locales dónde la configuración manual sea dificultosa debido al gran número de estaciones.



 

Dividir avi (con cualquier compresión) en varios archivos

A veces, nos interesa dividir algun archivo de vídeo de que disponemos, en varias partes para así poder grabarlo en CDs. Este es el caso por ejemplo, de películas que ocupen 1.4GB.

El programa que nos lo permite hacer es avisplit ( apt-get install avidemux ).

Funciona de la siguiente manera:

avisplit -i La.Interprete.avi -o La_Interprete -s 700

Esto nos creará tantos archivos de 700MB de tamaño como sean necesarios, con el siguiente formato:

[La_Interprete-0000.avi] (000000-098600), size 699.4 MB. (V/A) (3944040/3944060)ms
[La_Interprete-0001.avi] (098602-184581), size 700.4 MB. vid=7383280.00 ms aud=7383264.00 ms



 

Search



About

newton.gra2.com is a blog about technology, opinion and random thoughts written by Daniel Alvarez, a computer engineer currently living in Zurich, Switzerland.

Topics

News (20/0)
Manuals (24/0)
Security (7/0)
Music (3/0)
Weeklog (1/0)
Personal (34/0)
Photos (3/0)
Opinion (14/0)
Windows (5/0)

Blogroll

Pros i contres (Jordi)
Entrepa de fusta (Oriol)
Spaghetti Code (Isaac)
Made in net (Eric)
Nogare (Juan)
Blog de Isaac Jimenez
Web d'en Jaume Benet
Montcada Wireless (Fran)
Blog d'en Ricard Forniol
Angela Fabregues
in.solit.us

Libertad Digital
FOX News
The Wall Street Journal
The Washington Times
The Jerusalem Post

Michelle Malkin
Eurabian News
Nihil Obstat
Barcepundit
Expose the left
Davids Medienkritik
Johan Norberg
Ayaan Hirsi Ali

User Functions

:

:


Lost your password?

Latest posts

Stories

No new stories

Comments last 2 days

No new comments

Trackbacks last 2 days

No new trackback comments

Links last 2 weeks

No recent new links