Deploying a Lucee application to an Amazon Elastic Beanstalk Environment

If there was a piece of interwebs kit I would like to buy flowers, it would without doubt be Amazon’s Elastic Beanstalk with autoscaling.. Couple it with the Lucee scripting language and I’d happily take it home to meet my parents..

After much experimentation and many iterations (failed attempts and hair-pulling), I’ve managed to deploy Lucee applications (with my fave framework.. CFWheels) to the AWS Elastic Beanstalk platform and have it running reliably in production for a number of months.

My setup is a bit more complex than described below, but hopefully these bare bones will get you started on the path to automated utopia.. Those bare bones will result in a finished .war file that you can upload to EB using their deploy by file upload feature. You could also use the AWS command line tools, AWS toolkit for Eclipse, a Jenkins plugin or one of the many SaaS deployment platforms around.

I personally use Jenkins (on an AWS EC2 instance) to pull, build, test and deploy. I prefer this type of setup as there is much less margin for human error, and the resulting 60Mb+ war file is transferred within your Virtual Private Cloud rather than travelling across the interwebs if you were to manually upload it for every deployment.. oh, and it’s open source.

At a high level, I will outline how to download the Lucee server war file, combine your application’s code, and use some AWS sorcery to configure your Lucee application. There are some pre-requisites:

  • The settings that you would normally configure in Lucee admin, should be in Application.cfc (See the Export feature in Lucee admin)
  • The commands to create the finished war file are written for Linux (If you’re Windows inclined , you could either ‘translate’ them to Windows commands, try Cygwin or a Virtual Machine)
  • You’ll need a src/ directory in your application containing the files outlined below
  • An AWS account with an empty Tomcat 8 Elastic Beanstalk environment

In a nutshell, what happens is the Lucee jars, your code and some config files are all copied to the “build” directory, then zipped into a sexy war file ready for upload/deploy to the Elastic Beanstalk environment.

src directory
src/eb-tomcat/web.xml
src/eb-tomcat/urlrewritefilter-4.0.3.jar
src/eb-tomcat/urlrewritefilter.xml
src/eb-tomcat/.ebextensions
src/eb-tomcat/.ebextensions/lucee-server.xml
src/eb-tomcat/.ebextensions/01-copy-lucee-server-xml.config

src-directory-structure

These are configuration files.. for a production application, you’ll need to provide your own lucee-server.xml and most likely your own rewrite rules in urlrewrite.xml. If you don’t need url rewriting, then remove all reference to them. The .ebextensions execute EB container commands.. the one provided simply copies your lucee-server.xml file into place. To create your finished war file, cd to your application’s root directory then run these commands. I’d suggest creating a shell script.. and if you feel adventurous, allow an ‘environment’ option that will create different environment builds for your application’s deployment pipeline

# some vars..
DOWNLOADS_PATH="/tmp/downloads/"
LUCEE_VERSION="4.5.1.000"
LUCEE_WAR="lucee-$LUCEE_VERSION.war"
LUCEE_WAR_PATH="$DOWNLOADS_PATH$LUCEE_WAR"
FINISHED_WAR_PATH="/var/www/myapp/war/"

# ensure the required packages are installed
sudo apt-get install unzip wget
# sudo yum install unzip wget # (or using yum)

# create required directories
mkdir build
mkdir -p $DOWNLOADS_PATH
mkdir -p $FINISHED_WAR_PATH
# only download the lucee war file once
if ! [ -e "$LUCEE_WAR_PATH" ]
then
  wget -O $LUCEE_WAR_PATH http://bitbucket.org/lucee/lucee/downloads/$LUCEE_WAR
fi

# unzip the lucee war file into the build utility directory
unzip $LUCEE_WAR_PATH -d build
rm -rf build/assets

# explicitly copy required code (this is cfwheels specific.. but just plonk in your own directory names)
cp -r config controllers customtags events images javascripts models plugins  stylesheets tests views wheels build
cp -r Application.cfc index.cfm rewrite.cfm root.cfm build
# remove any peksy files that you don't want deployed to production
rm -r build/favicon.ico build/License.txt
# copy config files
cp -r src/eb-tomcat/.ebextensions build
cp -f src/eb-tomcat/web.xml build/WEB-INF/web.xml
# copy tuckey url rewrite jar if required
cp src/eb-tomcat/urlrewritefilter-4.0.3.jar build/WEB-INF/lib/urlrewritefilter-4.0.3.jar
cp src/eb-tomcat/urlrewrite.xml build/WEB-INF/urlrewrite.xml

# this is how I force my app into 'production' mode.. again very cfwheels-centric
rm build/config/environment.cfm
echo 'set(environment="production");' > build/config/environment.cfm
# create le war file
cd build && zip -r -q $FINISHED_WAR_PATH"app-production-master.war" . && cd ..
# cleanup
rm -r build

This should give you a production ready war file located at: build/app-production-master.war. Now you can manually upload your app-production-master.war and be on your way..

Some of my recommendations:

  • Build in the cloud using Jenkins (or your choice of CI server)
  • Host the Lucee war file in your own S3 bucket.. you’re less at the mercy of 3rd parties connectivity issues.
  • You can easily incorporate additional jar files as per the tuckey rewrite tool.. just copy them into build/WEB-INF/lib/

Any suggestions, improvements & feedback welcome.

Enforcing the use of routes in CFWheels

If you really like routing (you’d find that really funny if you’re Australian) and want to enforce their use in all linkTo(), urlFor() and startFormTag() function calls.. Do as I do..

Put the function below into your /controllers/Controller.cfc

<!--- /controllers/Controller.cfc --->
<cffunction name="urlFor" access="public" output="false" hint="ensure routes are used for all urls">
    <cfif ! StructKeyExists(arguments, "noroute") && ! ( (StructKeyExists(url, "controller") && url.controller == "wheels") || StructKeyExists(arguments, "url") )>
      <cfif ! (StructKeyExists(arguments, "route") && Len(arguments.route) gt 0)>
        <cfthrow message="Please use a route rather than an action.. Thanks!">
      </cfif>
    </cfif>
    <cfreturn core.urlFor(argumentCollection=arguments)>
  </cffunction>
</code>

This overrides, the core urlFor() method, checking for the existence of a route argument.. UNLESS, the controller is “wheels” which probably means you’re using the test framework or managing a plugin. There is also a way to bypass the route nazi.. by using the noroute=true arguments.

Happy routing.. Teehee!

CFWheels Calculated Properties on Steroids using an afterFind Callback

One limitation with using calculated properties in your CFWheels models is that one is limited to what can be achieved using SQL. CFML has almost infinite possibilities for formatting strings which simply cannot be done easily (or at all) in SQL.

A technique that I use to get around this limitation is to create new properties using an afterFind callback for both objects AND queries in a single callback.

In the code below I create two new properties, createdAtShort and createdAtLong which are just different date masks applied to the trusty createdat property. It’s a fairly simple example, but highlights the huge potential of keeping this type of logic in your model and out of your views. Hopefully this will get your brain moist.

<!--- /models/User.cfc --->

<cfcomponent extends="Model" output="false">

  <cffunction name="init">
    <cfset afterFind("$$setStuffAfterFind")>
  </cffunction>

  <cffunction name="$$setStuffAfterFind" access="private">
    <cfif StructKeyExists(arguments, "createdAt")>
      <cfset arguments.createdAtShort = DateFormat(arguments.createdAt, "d/m/yy")>
      <cfset arguments.createdAtLong = DateFormat(arguments.createdAt, "dddd, dd mmmm yyyy") & " " & TimeFormat(arguments.createdAt, "hh:mmtt")>
      <cfreturn arguments>
    </cfif>
  </cffunction>

</cfcomponent>

Now you can call your model and the result should contain your new properties..

<!--- /controllers/Users.cfc --->

<cfset user = model("User").findOne()>
<cfset users = model("User").findAll(maxrows=5)>

<cfdump var="#user.properties()#">
<cfdump var="#users#">

Notes:

  1. Avoid performing further database operations inside the callback when using findAll.. y’know the whole querying within a recordset loop thing..
  2. afterFind callbacks are not called on included models. So.. model("Company").findAll(include="Users") won’t contain your new properties.

UPDATE: Chris Peters has since shown me the error of my ways.. it turns out that the arguments scope can be used for both queries and objects. This has simplified the entire concept considerably (and made ‘my’ code look eerily similar to the official CFWheels docs! The code and description above have been modified to use arguments scope.

Run your CFWheels database migrations automatically on application start

It’s no secret that I’m a massive fan of the Coldfusion on Wheels framework.. and there are a couple of tools that I use in my deployment process that are invaluable to me.

One of those is the DBMigrate plugin. I recently came up with a way to automatically migrate to the latest database version when the application first starts.

This can be useful/essential when:

  • Your deployment process doesn’t use post-deploy hooks
  • You want to make your application more portable
  • You want to simplify your deployment script

Essentially, it calls a couple of the plugin’s methods whilst doing a little array-fu.

/events/onapplicationstart.cfm

<cfscript>
// migrate database
_dbm = {}
// create a pointer to the dbmigrate plugin
_dbm.plugin = application.wheels.plugins.dbmigrate
// create an array of available migrations sorted by version in descending order
_dbm.available = []
for(item in _dbm.plugin.getAvailableMigrations()){
_dbm.available.Append(item.version)
}
ArraySort(_dbm.available, "numeric", "desc")
// migrate to the most recent version
_dbm.plugin.migrateTo(_dbm.available[1])
</cfscript>

There are a few caveats:

  • It will execute every time the application loads OR is manually reloaded via the reload=true parameter (though it’s fairy lightweight when there are no migrations to execute)
  • It will execute on every server in your cluster
  • There is (currently) no error handling
  • It could disable your application if your migrations fail

Install Python 3.4, Django 1.7 on Ubuntu 14.04 in a venv Virtual Environment

Python’s venv module provides support for creating lightweight “virtual environments” with their own site directories, isolated from system site directories. Each virtual environment has its own Python binary (allowing creation of environments with various Python versions) and can have its own independent set of installed Python packages in its site directories. – python.org

It’s well worth the trouble of setting up your Django projects this way, as system updates (eg: a system Python upgrade) won’t break your “virtual” projects. The venv concept has a big advantage over a traditional virtual machine, as there are no extra resources required to run another operating system. Win!

Run the following commands at the terminal to install Python 3.4 in a “virtual environment” and install Django inside the venv.

Install python3 and curl

sudo apt-get install python3 curl -y

Go to your home directory

cd ~

Create a virtual environment called “myvenv”

pyvenv-3.4 ~/myvenv --without-pip

Activate your new virtual environment

source ~/myvenv/bin/activate

Go to your new virtual environment directory

cd ~/myvenv

Create a directory and extract the python setup tools into it

mkdir pypioffline
cd pypioffline
curl -O https://pypi.python.org/packages/source/s/setuptools/setuptools-6.1.tar.gz
tar xvzf setuptools-6.1.tar.gz

Run the setup tools

cd setuptools-6.1
python ez_setup.py

Install pip

easy_install pip

Install django

pip install django

Run the Django web server

python manage.py runserver

Now visit http://127.0.0.1:8000/ to (hopefully) see your “Welcome to Django” page!

Other handy commands you’ll use…

Deactivate your new virtual environment

deactivate

Reactivate your new virtual environment

source ~/myvenv/bin/activate

Credits: Code Ghar

Passing a list of strings to CFWheels’ findAll() where argument

Use the ListQualify() function to pass a list of strings to the where argument in CFWheels ORM finders. ListQualify() encloses each list element in with the string specified (in this case, a single quote). Without it, CFWheels interprets the list as a single value.

<cfscript>
names = "Foo,Bar";
users = model("User").findAll(where="firstname IN ('#names#')");
/*
 * Generates SQL Query.. Not so good
SELECT firstname, lastname FROM users WHERE firstname IN ('Foo,Bar')
*/

users = model("User").findAll(where="firstname IN (#ListQualify(names, "'")#)");
/* 
 * Generates SQL Query. Great success!
SELECT firstname, lastname FROM users WHERE firstname IN ('Foo','Bar')
*/
</cfscript>

Great success!

Ensuring Full Unit Test Coverage In Your CFWheels App

Ensuring that you have unit tested all (or most) of your code in your CFWheels app is an important part of any test suite. As you hopefully know by now, the CFWheels framework features an in-built unit test framework.

The code below is a test itself, that will scan your controllers, models and views folders and make sure that you have a corresponding “test” cfc for each. If it discovers that a test is missing, it will fail.. so your app will not pass unless it has full test coverage.

To get it working, just copy the code below and save it into a file in /tests/TestsExist.cfc (or anywhere under the tests folder actually). You can retro-fit it to any app, though you may need to re-organise your test files somewhat.

There are some conventions that you need to follow to have it work properly, but I consider this being well organised (and you should be used to conventions by now). Your folder structure under /myapp/tests/ should look like this... Say you have a Login.cfc controller, your controller test cfcs themselves should be named like this TestLogin.cfc.. but you could also have multiple test files for your Login.cfc, just as long as the file BEGINS with “TestLogin”, eg: TestLoginRedirects.cfc, TestLoginBusinessLogic.cfc.. you get the idea.

The same goes for models and views. There is much one could do to improve/re-factor/customise this code, but it will hopefully catch anything you may have forgotten to test. Feel free to contribute to this project on GitHub.

So to summarise:

  • Re-organise your folders as per the example above
  • Re-name your test cfc files using Test{YourControllerName}.cfc
  • Copy the code into a TestsExist.cfc file under the /tests/ folder
  • Run your unit tests
<cfcomponent extends="wheelsMapping.Test">
	
	<cffunction name="setup">
		<cfset testsPath = ExpandPath("tests")>
		<cfset controllersPath = ExpandPath("controllers")>
		<cfset modelsPath = ExpandPath("models")>
	</cffunction>
	
	<cffunction name="test_00_setup_and_teardown">
		<cfset assert("true")>
	</cffunction>

	<cffunction name="test_01_all_controller_tests_exist">
		<cfdirectory action="list" directory="#controllersPath#" filter="*.cfc" name="controllers" />
		<cfdirectory action="list" directory="#testsPath & '/' & 'controllers'#" filter="*.cfc" name="controllerTests" />
		<cfset assert("controllerTests.recordCount gt 0")>
		<cfloop query="controllers">
			<cfif ListFindNoCase("Controller.cfc,Wheels.cfc", controllers.name) eq 0>
				<cfset abbrev = ListFirst(controllers.name, ".")>
				<cfset isTestFileFound = false>
				<cfloop query="controllerTests">
					<cfif FindNoCase("Test" & abbrev, controllerTests.name) gt 0>
						<cfset isTestFileFound = true>
						<cfbreak>
					</cfif>
				</cfloop>
				<cfset assert("isTestFileFound", "controllers.name")>
			</cfif>
		</cfloop>
	</cffunction>
	
	<cffunction name="test_02_all_model_tests_exist">
		<cfdirectory action="list" directory="#modelsPath#" filter="*.cfc" name="models" />
		<cfdirectory action="list" directory="#testsPath & '/' & 'models'#" filter="*.cfc" name="modelTests" />
		<cfset assert("modelTests.recordCount gt 0")>
		<cfloop query="models">
			<cfif ListFindNoCase("Model.cfc,Wheels.cfc", models.name) eq 0>
				<cfset abbrev = ListFirst(models.name, ".")>
				<cfset isTestFileFound = false>
				<cfloop query="modelTests">
					<cfif FindNoCase("Test" & abbrev, modelTests.name) gt 0>
						<cfset isTestFileFound = true>
						<cfbreak>
					</cfif>
				</cfloop>
				<cfset assert("isTestFileFound", "models.name")>
			</cfif>
		</cfloop>
	</cffunction>

	<cffunction name="test_03_all_view_tests_exist">
		<cfdirectory action="list" directory="#controllersPath#" filter="*.cfc" name="controllers" />
		<cfdirectory action="list" directory="#testsPath & '/' & 'views'#" filter="*.cfm" name="viewTests" />
		<cfset assert("viewTests.recordCount gt 0")>
		<cfloop query="controllers">
			<cfif ListFindNoCase("Controller.cfc,Wheels.cfc", controllers.name) eq 0>
				<cfset abbrev = ListFirst(controllers.name, ".")>
				<cfset isTestFileFound = false>
				<cfloop query="viewTests">
					<cfif FindNoCase("Test" & abbrev, viewTests.name) gt 0>
						<cfset isTestFileFound = true>
						<cfbreak>
					</cfif>
				</cfloop>
				<cfset assert("isTestFileFound", "controllers.name")>
			</cfif>
		</cfloop>
	</cffunction>

	<cffunction name="teardown">
	</cffunction>

</cfcomponent>