5.2. Hosted domain data

When our service first launched all configuration data was hosted in an MySQL database. When we had only two machines this worked well, but we soon realised that if we had a significant number of satellite MX machines that replication would become difficult.

Rather than using a database to store configuration data we slowly migrated to storing details relating to each hosted domain as as a tree of directories and configuration files. This layout proved to be significantly easier for all parts of our service to deal with - and it had the advantage that the satellite MX machines didn't need a continuous link to the master host to be present.

Although we were able to choose our own storage location we decided that we'd place everything relating to our hosted domains beneath the single prefix directory of /srv, which is vaguely FHS compliant.[1]

Using a hierarchy of directories and flat-files to contain configuration data was a significant evolution of our service and it imposed some new problems of its own, most notably we could no longer rely upon database replication instead we had to come up with our own method of propagating changes made via the control panel to each satellite MX machines.

Our solution was to use rsync to push any changes from the master control panel machine to the satellite machines, after first recording them in a local Mercurial repository.

Whenever the master machine recorded a change to the configuration for a particular domain it would create the file /srv/dirty. A simple cronjob running every minute upon the master machine would test for the presence of this file - and when it was found it would trigger the synchronisation of the local /srv hierarchy to each of the satellite machines.

In addition to pushing to the satellite machines the /srv/dirty marker would also be used to trigger an update to our revision control system as the entire contents of the /srv hierarchy were actually maintained under the mercurial revision control system, so that changes could be recorded.

Any changes made to the /srv were also sent to our administrative mailing list. Such an email message might look like this:

Example 5-3. Example mercurial notice.


Added: hosted.org/whitelisted/senders/hostmaster@nic.uk
pushing to /var/backups/srv
searching for changes
adding changesets
adding manifests
..
 

It was anticipated that in the future we'd actually make more use of the revision control system to propagate changes: Rather than having the master push the changes to the satellite machines via rsync, each satellite machine would regularly poll the central Mercurial repository to fetch updates and apply them every few minutes.

Notes

[1]

Filesystem Hierarchy Standard.