As web developers, we strive for a constant improvement in our day to day work efficiency and productivity. Either through improving the tools we are working with (editors,workstations etc.), or improving the established development workflows (actions you’re usually performing when dealing with some particular development tasks). That could mean managing servers, connecting to them – FTP, SSH, editing files, backups, uploads, downloads, etc. Read about building an elegant Joomla development workflow using Akeeba UNiTE.
One important aspect of every good development workflow is also the development vs production environment setup, and building some sane deployment system that will connect both of these environments. That way you get a flexible and supportive working configuration that will be less complicated and less error-prone to work with.
Recently, our team was faced with a challenge to build such a deployment system, and what I will do here is basically outline the general steps of this process, as well as the approach we took to achieve that.
The situation
As with most development projects, there was an underlying situation or problem built up that required some immediate action from us. Our difficult situation was multifaceted:
- Multitude of Joomla sites: 20+, with multiple users/developers working on them.
- Direct production (live) site changes. No DEV – PROD workflow or system was existing at that time.
- Limited control of the production server. Apart from a simple FTP access, we didn’t have any other way to change or modify any setting whatsoever on the server. The production server was in full control of our client.
- Client requested that we minimise live site edits.
- Client implemented aggressive front-end caching.
- We struggled to debug efficiently.
- We lost time creating DEV sites manually.
So we needed some system that will (semi) automatically create and update development sites for us, that will allow us a more or less worry-free environment to work on, and simple way to deploy the changes back to the production sites.
Main functionalities
As a starting point we defined main functionalities for our system/tool as follows:
- Clone
- Create copy of PROD site on DEV server
- Overwrite existing DEV site
- Deploy
- Copy DEV site to PROD server
- Overwrite existing PROD site
- Lock
- Lock DEV site to prevent overwrite
- Used during dev on DEV 🙂
- Freeze
- Freeze PROD site to prevent backend changes
- Block access to /administrator
The development process
Since these were all Joomla sites, and we were all experienced with Akeeba Backup extension, we decided to make use of it for our tool. We already knew about the command line framework the Akeeba guys produced called Akeeba UNiTE and we decided to give it a try.
We started creating the Development server as a replica to the production, with all of the sites present on both environments. The setup looked like the scheme below:
| Development | Production |
|---|---|
| cl–site1-com.dev.server.com | site1.com |
| cl–site2-com.dev.server.com | site2.com |
| cl–site3-com.dev.server.com | site3.com |
| cl–site4-com.dev.server.com | site4.com |
| deploy.dev.server.com | deploy.site.com |
As you probably noticed we also created 1 separate domain in each environment named deploy. That’s where our deployment application instances were placed at. We were going for a two-part app approach, where instances communicate between each other with some simple HTTP requests.
Naming of the sites
We agreed upon naming convention for the development sites. They would get cl– prefix as well as replacing all dots in the production site folder names with a dash on the development server. That way we can automatically retrieve the live site root folder names as well as the domain names.
Akeeba UNiTE turned out more than ideal for our purpose. It allowed completely unattended site restorations, as well as remote backups at the same time.
Creating remote backups
To allow UNiTE to create remote backups we had to go to the Akeeba Backup backend configuration on each production site and turn on the: “Enable front-end and remote backup” setting. Also, we had to put some Secret word there.
The only thing that UNiTE requires to do remote backup + restoration to a specified folder/server is a simple XML file with some required config data:
<?xml version="1.0" encoding="UTF-8"?>
<unite scripting="02_angie">
<remote>
<host>http://www.example.com</host>
<secret>test</secret>
<component>com_akeeba</component>
<profile>1</profile>
<downloadmode>http</downloadmode> <dlurl>ftp://user:pass@ftp.example.com/administrator/components/com_akeeba/backups</dlurl>
<delete>1</delete>
</remote>
<siteInfo>
<package from="remote"></package>
<deletePackage>0</deletePackage>
<localLog>test.log</localLog>
<emailSysop>0</emailSysop>
<name>My Shiny Restored Site</name>
<email>nicholas@dionysopoulos.me</email>
<absolutepath>/Users/nicholas/Sites/restored</absolutepath>
<homeurl>http://www.example.net</homeurl>
<livesite>http://www.example.net</livesite>
</siteInfo>
<databaseInfo>
<database name="site">
<changecollation>0</changecollation>
<dbdriver>mysqli</dbdriver>
<dbhost>127.0.0.1</dbhost>
<dbuser>mydbuser</dbuser>
<dbpass>password</dbpass>
<dbname>example</dbname>
<dbprefix>exmp_</dbprefix>
</database>
</databaseInfo>
</unite>Sample UNiTE XML file
The <remote> section represents the important production site data, things like host, the remote backup secret word, the backup profile ID, the download mode and the actual download FTP url.
<siteinfo> section contains one very important info and that is the <absolutepath>. Here you add your development site root folder path. Also in the <package> you tell UNiTE that you want the package to come from a remote backup of the live site (remote section above)
<databaseinfo> section contains all of the development site database connection data, so UNiTE knows where to restore the previously backed up database.
You can read more about UNiTE XML settings in the UNiTE documentation.
The application flow
The development part of the app had an interface like the following:
Creating the site clone
The “NEW CLONE” button initiates the cloning process. The application first dynamically builds the UNiTE XML file, then runs the UNiTE unite.phar executable php archive file with some PHP exec call:
<?php $command = "php -f unite.phar clone.xml 2>&1"; exec($command, $output, $return);That’s basically all you have to do. With this call UNiTE reads the XML file, makes a real-time remote backup of the production site and then restores it under the development path. All of that behind the scenes and without any user interaction whatsoever.
Cloning in reverse
DEV TO PROD button starts a reverse process. First the app makes a development site backup with some CLI akeeba backup call. Sends the backup archive to the production server temp folder and with some HTTP call to the production deploy app, it initiates the same UNiTE restore process. For example, dev app sends this post request:
$http = JHttpFactory::getHttp();
$data = [ 'token' => self::TOKEN, 'task' => 'restore', 'jpa' => $jpa, 'dir' => $dir ];
$res = $http->post(self::REMOTE_URL, $data);The production deploy app then takes the dir and jpa value from the request, and starts to create the UNiTE XML file. This time by directly reading the live site configuration.php database data. Then it calls the same UNiTE unite.phar file like before, and it restores the production site instantly.
Freezing functions
The other two functions: Freeze Clone and Freeze Live are actually doing a simple locking on the development, respectively production sites, so the application is not allowed to do any cloning or restoring processes while these are active.
Simple comments
We also implemented a simple commenting feature. So whenever the user is updating the development site he/she can add some comment, describing why the site was updated and on which tasks he/she is working on.
The outcome
When the tool was put into production, it immediately started to positively influence our productivity as workflow improved significantly.
Developers had more freedom when testing new code or debugging on development sites. Errors were less likely to happen on the production sites, or were completely avoided.
The developers always had the latest content from the production sites and they could freely add and test new additions to it. Then copy all of that back to the production sites quickly and easily.
The sheer impact of such a setup for the team was of tremendous value. After the initial “getting used to” phase, almost all of the team members expressed appreciation for such a change in their workflow. Ultimately, we were all puzzled how we even managed to do our work without such a tool at our disposal.
Potential pitfalls
Since our tool is running through a web browser, with a web server backend (apache), and not directly executing UNiTE like a php cli script, there are a couple of things that one should be aware of when building such a system.
- Availability of the PHP system call functions like exec(), shell_exec, system() etc. On some hosting environments, it is a common approach to disable these functions because of their potential security impact they have, when not used properly.
- php.ini’s max_execution_time setting. One should be able to modify the default setting here which will definitely not be enough for these backup/restore functions. From our experience, on some sites the cloning process was easily taking more than 2 minutes.
If some of these things are not met then building such a similar system would be challenging, right from the start.
Will this be useful for me?
This whole process and the tool as a product out of it was just our response to the needs that occurred at some point in our development process. But I’m sure the other developers could also benefit from such a system. Particularly (but not ultimately) if they happen to manage multitude of Joomla instances. Using the Akeeba UNiTE’s powers, with some trivial effort at start, a nice and smart deployment environment could be structured and put in place. And that will surely generate positive trends in your workflow outcome and efficiency.
Want to read more? Stay tuned for our article on Centralising resources across Joomla sites.
