Free MySQL backup with Dropbox

Recently, I've been working on creating an automatic service for backing up our RedmineMySQL database on the Debian server.

So I've decide to share there how I made a free backup service for the MySQL database.

I set several goals that should be attempted in order to consider that backup service is useful

1. Only free & open source applications.
2. Remote disk storage also for free.
3. Backup must be automatic.
4. Backup encryption.

To accomplish these requirements finally I have selected the next tools:

1. AutoMysqlBackup - http://sourceforge.net/projects/automysqlbackup - the core of theservice.
2. DropBox free account (2 Gb for free) - www.dropbox.com
3. crontab - a standard Linux job automation tool
4. openssl - for encryptionHow does this it works together.A little description.

First of all you will be need a DropBox account - there your MySQL backups will arrive and willbe stored. You have to download the DropBox client application or a DropBox daemon. I suggest you to use a dropboxd (daemon) for the server use:

https://www.dropbox.com/install?os=lnx - install DropBox as a daemon

Downlaod & install AutoMysqlBackup tool. After the installation you have to modify the script /etc/automysqlbackup/automysqlbackup.conf

You should uncomment the line you need and put the appropriate values. Each line in this script is well documented and you will find all the options you need. Just a brief description of what can you configure with it : mysql server address, mysql credentials for the command mysqldump, location where the dump will be stored, the regularity of the backuping, etc.

Next step, is creating your own bash script that will be called by cron, make a backup with autoMysqlBackup, copy the backup files into the local dropBox folder.

Download my example : http://dl.dropbox.com/u/3839545/backup.sh

Make sure that your own backup.sh is runnable (don't forget chmod u+x backup.sh)

The next step is to configure your crontab. In the crontab I've defined the period of time the script backup.sh will be called. I set up the backup everyday at 3:00 AM.

Make sure you have the root privileges and run the following command :

crontab -e

and at the end of the file add the line :

00 03 * * * /root/backup.sh

And the last step is about the encryption. The openSSL encryption is made by the AutoMysqlBackup, for this you have to uncomment in your /etc/automysqlbackup/automysqlbackup.conf the following line:

CONFIG_encrypt='yes'

don't forget to set the password to encrypt!

That's all, make a test that prove that your backup is working fine before moving all this stuff in the production environment.

Git killer features

After 3 weeks of working with Git (the DCVS written by Linus Torwalds), I definitely can say that it is a great technology.

If you are asking yourself what you need to learn right now? What is the next major revolution in the IT world? Maybe, the answer will be Git.

What are the killer features of Git for me:

• Distributed – I can develop normally even without Internet and access to the central repository. It’s a great feature to make small and concise commits while developing at home, during the weekend.
• Rich history revision mechanism – the search within the log-history is wonderful. With

  git log –sSecurityManager

  I can find, for example, all commits that introduce or remove the SecurityManager class. Attention, the search is performed on the source code and not on the commit log. It’s a best tool for making the code live analysis.
• Git bisect – automatically find a failing commit using the binary search and a set of unit tests. If you don’t use the CI server (shame on you :)), you can nevertheless quickly find a bad commit. Git bisect command will run the tests against the project snapshots. Each snapshot is associated with a particular commit. I did not try to use this command yet. But, the concept is great.
• CMD – git is a command line based tool. You can script the commands, make aliasing etc. CMD speed up your productivity. Of course, for making the merge it’s not so trivial, but the guys from Eclipse will soon release a completely functional version of the plugin eGit.
• git cherry-pick - reintegate an existing commit to the current branch. How many times you have forgot to switch to the correct branch before making a commit? I have been confronted with this painfull mistake quite often. Now you don't have to report your commit manually to the correct branch. The magic cherry-pick command will make it four you!
  

I have worked with git on Windows and Mac. Git for Mac is a stable and I have not noticed any problem. It is not the case for Windows...

Git remote tags problems

For running Git on Windows box, you have to install msysgit (I used 1.7.4.msysgit.0). The one problem I encountered with msysgit, it was that I could not push my tag to the remote repo. The command

git push --tags origin master

did not create a tag remotely. The same command executed on Mac did the job correctly.

Git - pdf book about git
I can recommend to download a free PDF book "Git Community Book" about git from http://book.git-scm.com

As long as I learning Git, as more I understand its powerfull.

Tools : free screenshot capture

The tool of the day is gadwin printScreen (only for Windows users).

A really powerful and free screenshot capture tool that I used on my Windows box.

You can download it here

Log4j : Customize LoggingEvent for SMTPAppender

Hi,

Recently I was confronted to the need of customizing the log4j message generation. When an exception occurs, the application should sent the email to the development team in the following format:

The error message should be enclosed by the corporation information represented by the header and footer. The problem is the application URL (marked by the red ellipse). This URL can be different depending on the environment where the application is running and throwing the exception. In the complex, industrial projects where exist many environments, this can be very helpful to determine which environment has a problem.

First of all, how set the footer and header? It’s quite simple, you have to use org.apache.log4j.EnhancedPatternLayout, in the parameter ConversionPattern, you put the header, footer and the message error with some additional information (date, class name which generated the exception, etc). The standard log4j distribution(log4j.jar file) does not contain the class EnhancedPatternLayout, you have to download the additional extras companion (here)

Pay particular attention to do not use org.apache.log4j.PatternLayout which contains some issues with synchronization (see the first paragraphe of the associated JavaDoc). PatternLayout class is still widely used and all exemples on Internet use it, but for make your code robust you don’t have to use it.

And what about the environment dependent URL ? How can I put the http://myapp.production.com or http://myapp.integration-test.com in the log message? I found 2 solutions:

1. Use ant or maven to replace a specific symbol in your log4j.xml (log4j.properties) when you are building the app. You can use filter resource maven plugin fot that. But is not the most elegant solution, because you will have a strong dependency with the build tool. For production system, it is not a choice.
2. Use org.apache.log4j.EnhancedPatternLayout and %properties{key} mechanism

So first of all, you have to customize the SMTPAppender in order to put the specific information into the each message that will be sent. For that you have to subclass SMTPAppender and override the method append. Take a look on my test implementation:

The code is self documented. I put the URL to prod environment behind the key named "app".I hardcoded the value in order to make the explanation simpler. But in your production code you may get the correposnding value from properties file, from environment variable or from the database.

Next, the key “app” in the event property becomes available for Log4J runtime. You may see how I use it in my log4j.xml file (the 2nd screenshot).

The setProperty method is an extension point and the excellent example of a good designed API.

Updated : actually, there is a simpler method to inject the environment name into the log4j logging message - simply, change the ConversionPattern for each log4.propeperties file.

JavaOne 2011 in Moscow. Review.

La semaine dernière j’étais en Russie, à Moscou, à la conférence JavaOne organisé par Oracle.

J’ai eu deux jours du plaisir technique. Entouré par les gens passionnés par la technique, j’ai pris une bonne dose de motivation.

La conférence se déroulait au centre de Moscou, au bâtiment épique de l’académie de science de la fédération russe (Российская Академия Наук - РАН) connu aussi sous le nom « Les Cerveaux » :)

Quelques points/événements clés que j’ai retenus à l’issue de la conférence sont présentés ci-dessous suivis de photos de ma collection.

Les nouveautés dans Java 7 présenté par une personne charismatique - Danny COWARD: join/fork mécanisme pour les traitements parallèles (map-reduce) ; le sucre syntaxique avec l’opérateur « Diamond » (<>) qui va faciliter la lecture du code qui utilise les génériques; le support de String dans l’opérateur switch ; les clauses catch avec la possibilité de préciser multiples exceptions.

Les standard Java EE 6 : avec l’API Servlet 3.0 (les appels asynchrones, configuration avec les annotations, le fichier web.xml est optionnel, l’utilisation du fichier web-fragment.xml pour les composants auto-configurable dans le contaneur, etc.), le support native de getsion de dépendence (DI) avec l’arrivé de la notation @Inject.

La présentation de Vladimir IVANOV (Oracle, Saint-Petersburg) sur la diagnostique et tuning du GC. J’ai noté un outil visualvm pour le profiling de performance des applications Java. Après la présentation, j’ai parlé avec cet ingénieur d’Oracle sur l’utilisation de Soft, Weak et Phantom références dans Java.

Une petite salle de repos pour les geeks :)

Et bien sûr, La Place Rouge avec le Kreml’.

Pour conclure, Oracle est motivé pour supporter le monde Java. L’organisation de la conférence est excellente, rien à reprocher. Merci Oracle pour ce JavaOne 2011.

Integration tests with JUnit and Spring

Récemment dans le framework Spring j'ai trouvé une classe très utile pour faire des tests d'integration avec la base de données - AbstractTransactionalJUnit4SpringContextTests.

Cette classe contient plusieurs méthodes utilitaires pour tester le code qui interagit avec la BD. Voici la liste des méthodes que j'ai utilisé pour créer les tests d'intégration très élégante - faciles à implémenter et lire:

Je vais décrire la stratégie la plus facile à écrire une classe de test d'intégration est la suivante:

1. Ecrire une classe de tests qui hérite de AbstractTransactionalJUnit4SpringContextTests
2. Définir une méthode setUp avec l'annotation @BeforeTest. Cette méthode va utiliser executeSqlScript pour préparer la base de données en y injectant les données qui sont nécessaires pour l'exécution de votre méthode de test. Attention, à ne pas utiliser les ";" à la fin des lignes avec SQL. Les lignes qui se terminent par le ";" seront ignorés par Spring.
3. La méthode de test qui va faire des "asserts" sur le nombre des lignes insérées dans les différentes tables impactés par les opérations DAOs. Pour faire ces asserts la méthode countRowsInTable est la plus appropriées.
4. Pour que la base de données reste dans son état initial, il faut supprimer maintenant toutes les données qui ont y été ajoutées par deux étapes précédentes. Pour cela, il suffit de créer une méthode tearDown annotée par @AfterTest, dans cette méthode vous pouvez appeler la méthode généreusement offerte par Spring - deleteFromTables.

Regardez les fonctionnalités complètes de cette classe sur le site officiel de Spring ici.