Dependency Injection in Scala and Play! using the cake pattern

It is my first time using Scala and I wanted a simple way to do DI so I could unit test my controller easily. The ability to use traits really seems to have a positive impact on code design, and enables things like DI to be quite easy without a runtime framework.

Real work scala dependency injection was a great introduction to implementing the Cake pattern in Scala so I won’t repeat the specifics of it here.

In case you haven't come across it the Play! framework is a complete web framework for JVM languages.

Personally I did not want to override plays controller instantiation so things stayed simple and I kept a clean routes table.

# Routes
# This file defines all application routes (Higher priority routes first)
# ~~~~

POST    /:topic/:key          controllers.Application.send(topic: String, key: String)
GET     /:topic/:key          controllers.Application.get(topic: String, key: String, groupId: String)

The standard play pattern changes slightly with the controller which is split between two definitions. The first is the Application object which works with the play route and inherites the ApplicationController trait. It defines a message consumer to a real implementation which is going to connect to a Kafka cluster and it is this that we will want to mock when running our unit tests.

object Application extends ApplicationController {
  val messageConsumer = new KafkaMessageConsumer
}

Now we have the ApplicationController itself which contains the controller behaviour and inherits from a MessageConsumerComponent which is where it has access to a message consumer. The controller does not care how this is created and just uses it.


trait ApplicationController extends Controller with MessageConsumerComponent {

  def get(topic: String, key: String, groupId: String) = Action {
    this.messageConsumer.get(topic, key, groupId)
    Ok("done")
  }

  def send(topic: String, key: String) = Action(parse.text) {
    request =>

      val body: String = request.body

      this.messageConsumer.send(topic, key, body)

      Ok("received: " + topic + ", " + key + "," + body)
  }
}

The message consumer is implemented as a pair of traits, the first as the interface which can be mocked and the second which is used in the cake pattern.

trait MessageConsumer {
  def get(topic: String, key: String, groupId: String)
  def send(topic: String, key: String, message: String)
}

trait MessageConsumerComponent {

  val messageConsumer: MessageConsumer

  class KafkaMessageConsumer extends MessageConsumer  {
    def get(topic: String, key: String, groupId: String) { ... }
    def send(topic: String, key: String, message: String) { ... }
  }
}

This MessageConsumerComponent defines a messageConsumer field which will be accessible by any classes which inherit from the trait and can be set to either the KafkaMessageConsumer or a mock object. That's all there is to this implementation, so let's take a look at how this is done in a unit test.

trait TestEnvironment extends MessageConsumerComponent with ApplicationController with Mockito
{
  val messageConsumer = mock[KafkaMessageConsumer]
}

@RunWith(classOf[JUnitRunner])
class ApplicationSpec extends Specification with TestEnvironment {

  "getting by topic" should {

    "return list of messages" in {
      val result = this.get("topic","key","group")(FakeRequest())
      there was one(messageConsumer).get("topic", "key", "group")
      status(result) must beEqualTo(OK)
    }
  }
}

I've followed Jonas' example in using a TestEnvironment object, but to avoid creating an extra component for the Application my system under test is the TestEnvironment. It might be worth changing this later but to keep things simple I've chosen this way.

There were a further two implementations I came across which are here and here. You might prefer one of those implementations and they are worth a read.

I am new to Scala so if there are errors, design issues or a Scala faux pas please let me know!

@naeemkhedarun

Excluding jmxtools with SBT

If you are using Scala and SBT you might experience this problem. In particular I was referencing Apache Kafka and found it trying to resolve the jmxtools which has been removed from the maven repository.

It took me a while to find the syntax to exclude this from my build.sbt file, but here it is:

libraryDependencies ++= Seq(
  "org.apache.kafka" % "kafka_2.10" % "0.8.0" excludeAll(ExclusionRule(organization = "com.sun.jdmk"),
                                                         ExclusionRule(organization = "com.sun.jmx"))
)

For more information you can check out the SBT documentation here.

How we used the chrome dev tools to optimise our html5 game

We wanted to experiment and see whether we could build a simple cross platform game using the latest web tech. The idea was that it would run across the mobile platforms and be easy to develop using our existing skills and tools. So for some background we are using cocoonjs to hardware accelerate canvas drawing and melonjs which is an OSS game engine which is easy to run on cocoon.

Our initial attempt was running very smoothly at 60fps on our powerful desktop browsers, however I was getting half that on my Galaxy Nexus. Given how simple the game was we were concerned and looked to find out why. We are developing using Chrome Canary which has the latest developer tools within.

CPU Profile

This was the first place we looked to see what was happening.

image

The trace is telling us we spend a majority of our time rendering rather than executing game logic. The sprites we are drawing are basic and we’ve made them as small as possible so this was the first surprise.

image

The flame chart puts into perspective how much idle time we have on a desktop machine and that each frame is speedily rendered in approximately 5ms. Initially that sounded good but given the lack of complexity in our graphics this performance is extremely disappointing and is a good indicator as to why the a mobile device might be struggling.

It wasn’t enough to know what to fix yet and for that we used the Canvas Debugger.

Canvas Debugger

This is an experimental chrome feature which means you will need to manually enable it. I used the learningthreejs blog which has a good video explanation but if you prefer something more textual you can follow the guide at html5rocks.

With the ability to inspect each canvas call both at the API level and visually we could track down where we were losing the performance. Below is a gif animation of cycling through each draw call shown in the debugger:

before

And with that visual representation it has become quite obvious where the extra draw calls are coming from, the background gradient is a 1px wide repeating image! Ironically we chose to do it this way for performance thinking loading a smaller image would be lighter on resources.

We were able to easily fix this in the map editor and it resulted in a big reduction in draw calls.

after

The gif animation also highlights the draw call for each tile which is imported from the map editor and this could be a further avenue to investigate if we want to target even lower performance devices.

image

Each frame now takes between 2 and 3ms to complete and more importantly the draw portion of that has been greatly reduced. It only takes 1ms to render on the desktop and the game code is now visible in the profile.

image

These changes were not only enough to run the game at 60fps on our mobiles, but has allowed us to increase the animation and visual fidelity while keeping the frame rate smooth. If you are working with canvasses either for game development or visualisations like d3 I recommend you grab the latest chrome tools and give them a go.

@naeemkhedarun

Getting started with Redis on Windows and a look at performance for logging

There’s a few scenarios in a few different projects where I need a fast dumping ground for data. In my deployments I need somewhere quick to pump logging messages to. These messages do not need to be written to disk or read out immediately so can be processed shortly after the fact out of process. I have the same requirements when rendering machine performance on a UI; I need a fast and non-intrusive way to collect the data from different machines and processes which can be read quickly later.

Redis is something which has been coming up frequently when talking linux and much has been said about its performance, so I wanted to take a quick look at performance in the context of logging. But first let’s get it up and running. I’ll be using PowerShell, chocolately and npm, so get those set up first.

Let’s begin by using chocolately to install redis.

cinst redis

The package should have installed a windows service running redis, let’s double check using powershell.

C:\> get-service *redis*

Status   Name               DisplayName
------   ----               -----------
Running  redis              Redis Server

Great, now time to get a UI to make using redis a little easier, this time the package is on npm.

C:\> npm install –g redis-commander;
C:\> redis-commander
No config found.
Using default configuration.
path.existsSync is now called `fs.existsSync`.
listening on  8081
Redis Connection 127.0.0.1:6379 Using Redis DB #0

The manager runs on port 8081 by default, let's take a look in a browser (use another shell as redis-commander will block your current one).

start http://localhost:8081

If you can see the UI then you are all set up and its time to push some data in and run some performance tests. Our build and deployment framework logs to the filesystem. This worked great for builds and simple deployments, but for scaled out environments and production which runs on many machines this is problematic. To avoid deadlocks and waits we manage a log file per machine and cat the tail into the main log when we can. We could aggregate these files post-deployment but its time to rethink how we do our logging. For now I just want to check raw speed and basic features of a few different technologies, today its redis.

We’ll use measure-command to find out how quickly we can log a bunch of messages to a file and then to redis. Using our framework to deploy to an environment for performance testing generates around 100K lines of logs across 123 log files and 27 machines. These get aggregated onto the deployment server for convenience but its still a lot of logs which are written during the deployment.

measure-command { 1..100000 | %{ Add-Content -Path C:\log1.txt -Value $_ } }
TotalSeconds : 319.9256509

This command is analgous to what we do in our PowerShell framework, since we do not keep a handle open on any one file we have to open it before writing every time. It takes 5 minutes to write that many lines of logs to the file system using Add-Content. This is a long time to spend logging and as we only write numbers in the benchmark involves less IO then deployment messages. We will benchmark redis for comparison and use the PowerRedis module.

We will make a single connection to redis as cmdlet takes care of the global state of the connection.

Connect-RedisServer

And the benchmark:

measure-command {1..100000 | %{ Add-RedisListItem -Name "deployment:log" -ListItem $_ } }
TotalSeconds      : 14.0009325

A fantastic improvement and the ability to log from multiple machines at the same time makes it a good option to consider. I can't help but wonder how fast we can log to a file if we did it more responsibly though.

C:\> $writer = New-Object System.IO.StreamWriter C:\log3.log
C:\> measure-command { 1..100000 | %{ $writer.WriteLine($_); } }
TotalSeconds      : 2.5816333
C:\> $writer.close()

So there are a few lessons to learn here. Avoid add-content if you’re writing more than a few messages. Do always measure your code so you do not waste minutes(!) logging to files. Redis is certainly fast enough for logging.

Newer Posts Older Posts