‘Software Development’

« Older Entries

Building COMET applications with Redis PubSub, Atmosphere and Akka

Tuesday, August 3rd, 2010

Atmosphere is a brilliant framework when you need to make your JVM web application COMET-enabled. It’s simple, lightweight and written in a very good manner (wise OO and OSGi bundles that makes it easy to extend). However, if you want to build a durable application of a really high load with thousands of clients simultaneously flooding the network with the tones of messages, you’ll probably want have to more control over the COMETed broadcast of messages. Simplifying the state of things, Atmosphere broadcaster is based on two in-memory queues: the queue of messages and the queue of references to the waiting clients. The aim is to replace those queues with a MQ solution that will allow managing active topics and clients waiting for messages.

Since recently, Redis has a strong story to tell about messaging – Redis PubSub is a nice performant alternative to the messaging mastodons like ActiveMQ. Its API is extremely simple though it offers more than enough, if you just want to take message queues away from the JVM. Bonus and the source of additional motivation (at least for me) is that Redis PubSub has a Scala actor-based client API and integration with Akka Framework

So, the following components will make the core of the COMETed system based on Redis:

  1. Akka. Offers Scala idiomatic integration with Redis and capabilities for building fault-tolerant publishers/subscribers;
  2. Atmosphere. One of the most attractive Atmosphere framework features is that it knows how to enable NIO capabilities at the most popular JVM application servers. Whilst client works with a high-level abstract API (say, suspends HTTP connection), an appropriate server NIO implementation is used on the background (Continuations mechanism, if the server is Jetty, or CometProcessor for Tomat). Moreover, in the latest versions the same API can also be used to transfer data over Websockets;
  3. Redis PubSub as a provider and background of messaging (gives ability to post/control topics through Redis admin utilities; watch queues, etc.);

(more…)

Tags: , , , , , , , ,
Posted in Software Development | Comments Off

SBT: falling in love again

Saturday, June 5th, 2010



    The thing that’s true about me and Scala (the language, tools, approaches, frameworks, etc.) – this is a real love, but it has never been a love at first sight. I remember, when I met Scala the first time, it appeared horrific to me – offensive syntax, functional programming, banging actors, and all the other things that can easily scare away a Java-developer, who was grown on imperative languages. But I always knew that Scala has a great potential (thanks “Programming Scala” – excellent reading for those, who wants to get hands dirty with Scala, but doesn’t know what to start with), and since I gave it another try 6 months ago it’s getting better and better every day (not kidding!). The same story was with Lift Framework – overcomplicated paradigm and HTML snippets and templates (what are they doing in the 21st century web-framework?!) first frightened me off, whereas now I’m actively using a lot of Lift components having fun.

    Getting into Scala requires patience.

    I wish I would remember that, when several months ago I tried SBT, thought to myself “why should I go there leaving old good Maven”, and got back hacking POMs and adding another plug-in to enable basic Scala features. Guess what has happened? Right, all the last week I’m actively using SBT, converting my old Maven’s POM into SBT’s project definitions :) You ask me why?

  1. SBT is designed for Scala. Do you remember your first (positive) experience with Maven? It worked like a charm – all you needed to do, is to pick an appropriate archetype, and voila, you are ready to make the most of the workflows you need (compiling, testing, building, etc.). It would be great, if Maven worked so smooth with Scala, but unfortunately, that truth is that defining a complex Scala project in Maven turns into a pluginism. With SBT I feeling myself like in the old good days – you literally need seconds to get the basic Scala project compiling and Specs (which is important!) running.
  2. SBT killer-feature (for me) – continuous whatever (compiling, deployment, testing, etc.). All you need to do is to put ~ sign before SBT command, so that, when the code of the project is changed, the action will immediately happen. E.g., ~jetty-run literally turns Scala into a kind of dynamic language – every update of a source file is instantly recompiled and deployed to Jetty by SBT. The same with ~compile – I wouldn’t say, you no longer need an IDE (with prompt highlights of errors) for hacking Scala, but when you aren’t able to run a graphical environment, immediate compilation of the file you’re editing makes life times and times easier.
  3. You can write build scripts in Scala (as far as I remember, Maven 3 is got to get support of Groovy and other languages for defining a project model, as well). Once you’ve tried writing a build script in a language you like, you will never be happy getting back to XML (after all these years of Spring, Hibernate, iBatis, etc. XML-nightmare). Writing build script with the language that enables OOP and functional programming is a brand new experience (for me, at least) and gives a lot of fun.

    So far so good, but frankly speaking, there 2 small thing that may spoil your experience with SBT slightly:

  1. IDE support for SBT is very limited. As far as I know, SBT is not on the feature list for Netbeans so far, so if you are sent into panic by console, SBT is not for you, sorry.
  2. Out of the box, SBT has a basic integration with Maven and Ivy – SBT understands simple pom.xml/ivy.xml and can use repositories/dependencies specified in them. However, it’s not possible to use dependencies specified in POM and SBT project definition simultaneously (pom.xml is simply ignored, if there’s an in-line dependency in project definition). I don’t know, what causes this limitation, but in case you have a complex Maven-based project it can delay you transition to SBT.

    Yesterday I’ve started a thread at StackOverflow to find out, what possibilities exist to translate POM into project definition. A basic solution was proposed by @retronym – it finds dependencies in POM and turns them into Scala code. I’ve implemented a bit more general script, that creates a new project definition of a correct type (based on specified packaging style) from POM with repositories and dependencies – works fine for simple projects (if you deal with more sophisticated ones, you can enhance my script or simply write your own – with Scala XML it’s extremely easy).

Tags: ,
Posted in Software Development | Comments Off

Kandash Project v 0.3 is now available!

Friday, May 28th, 2010


     As you might remember, friends, Kandash Project is my fun-project to learn Scala and Mongo a free, open-source implementation of the on-line board with sticky notes that is used in Kanban (and SCRUM), with some basic reporting and analysis capabilities. It’s extremely lightweight, but has almost everything you need for synchronizing your work with colleagues – interactive board (that tends to replace a physical one) with COMET support to provide a rich collaboration through the browser, charts to follow the trends of your team and catch the smells, and reports to get a deeper insight.

    New in this version:

  • Reporting and analysis capabilities
  • Boosted performance. Basic Akka features are used for it: Actors implementation, configurable work-stealing dispatchers, fault-tolerance, etc.
  • A lot of bugfixes :)

    You can find the code of Kandash Project at Google Code hosting.

Tags: , , , , , ,
Posted in Software Development | Comments Off

Flexible load balancing with Akka in Scala

Monday, May 10th, 2010



    Scala concurrency is based on Actors paradigm. Actors communicate with each other sending messages and doesn’t share global state, which guarantees safe concurrency. Every actor has own mailbox the messages are added to, and processes one message at a time.
    Akka framework introduces simplifications and richer capabilities compared with the native actor model. Configurable message dispatchers and fault tolerance make Akka’s Actors implementation an enterprise-scale alternative. 

    Dispatcher   

    Work stealing dispatcher, like any other Akka’s dispatcher, delivers messages to actors’ mailboxes. Message delivery is backed by Java’s ExecutorService (with manageable thread pool). Unique feature of the work stealing dispatcher is that if the message was initially sent to the busy actor, dispatcher finds another actor (“thief”) that can process the message, and puts it to the actor’s mailbox.
    Below we create a new instance of work stealing dispatcher that will be used by the pool of actors. Dispather has its own thread pool that is used for effective asynchronous distribution of messages. Thread pool size equals to the count of actors that will be load balanced:

(more…)

Tags: , ,
Posted in Software Development | 2 Comments »

Effective Stored JavaScript in MongoDB

Friday, April 16th, 2010



    MongoDB is shipped with a huge number of dynamic ad hoc queries that may be used in a wide range of trivial scenarios. But when it comes to building really complex analytical queries, you have to write all the logic in JavaScritpt manually (as you may know, MongoDB uses JavaScript as an interface language for querying, updating, data processing and many other tasks).
    The easiest way to execute server-side JavaScript query is to call eval() function in your favourite Mongo driver (personally I use Scamongo built on top of Java Mongo Driver). The code should look more or less like:

    MongoDB.use(DefaultMongoIdentifier) ( db => {
        db.eval(“”" function() {
                    …
                    })
                }”"”)
      })

   Code of the in-lined function is evaluated on the server-side and executed on-the-fly.
   This approach has two obvious drawbacks:

  1. Performance. Evaluated functions are not persisted between connects, therefore should be re-evaluated, which takes a lot of time (especially, if you deal with complex calls);
  2. Testability and readability. In-lined code is untestable and looks ugly.

    Solution

    Elegant solution is based on stored JavaScript (that is someway similar to stored functions in SQL-databases). In general, you can store you function in a special system collection,  from where you will call it with eval() function.
    Example below is based on the open source code of Kandash Project.
(more…)

Tags: , , , , ,
Posted in Software Development | Comments Off

Kandash pre-alfa (version 0.1) is available

Wednesday, April 14th, 2010

    Nearly a month ago I’ve decided to suspend IdeaBox project, and start another one that I’m of a bigger demand – Kanban Dashboard (short “Kandash”).
    The reason is obvious – anyone who uses Kanban for maintenance projects can hardly find any other automation tool better, than a set of Excel spreadsheets with scripts. Sure, if the  team is located in one place (meaning, in one room) with all the job always being done in the office (no one works from home) you should be more or less satisfied with a physical board, marker and stickers. But, if your team is distributed across different locations, or you don’t want to synchronize the physical board with your electronic datastorage, when you need to build a report in order to analyze the state of things, something more capable and available is required.
    Kanban is growing in popularity last years, and people are finally coming to conclusion that there could be no universal tool for Agile development (“No tool is complete, no tool is perfect”, Kanban and Scrum. Making the most of both) – sometimes you need a fork, sometimes a knife, sometimes both. Nevertheless, the most of Kanban practitioners are either have to use Scrum tool-set or invent their own approach. Moreover, there are still not many tools that can visualize the board even for Scrum (whilst it’s the main component of the both Scrum and Kanban that makes the development process transparent) and decrease the amount of dirty manual job – the most of Scrum-tools on the market more look like as a colorized spreadsheets with the basic analysis functions and ability to build a burndown chart.


Kandash board screenshot

    In Kandash I make accent on the board with rich user experience. You can do with this board anything you do with the old-fashioned board, marker and stickers, plus having an ability to restrict work in progress, collapse/expand sections, persist and export the board model, control access and many more. Statistics is gathered when you place the stickers on the board, drag’n’ drop them between tiers (IOW, states of the workflow), create new markup, etc. Basic analytics is included as well – Kandash board model can be exported into Excel and PDF to make a weekly (or whatever else) regular reports with cumulative flow chart and tables.
    You can download the sources of the current version at gcode project hosting. You can also find a table with 0.1 features and a small sceenshot-based guide (but don’t be afraid – everything is absolutely intuitive).
    If you find this idea interesting, please contact me – let’s make the things happen together!

Tags: , , , ,
Posted in Software Development | Comments Off

Rules of falling interest

Monday, April 5th, 2010



    Have you ever asked yourself, why your day-time job in the big company is so simple and boring, and it’s not getting better, when you’re moving from one project to another (or, even, changing the organizations you work for)? Be a realist – it’s not because you’re so much smarter than the other people (but you’re definitely smarter than 9-to-17 office plankton, which is totally satisfied with the state of things, and, obviously, treats programming only as a well-paid way to kill time between breakfast and supper). It’s all about human’s natural tendency to fight entropy and disorder, which turns out into two simple rules:

1. All the projects become boring over time

    This rule is based on Plauger’s “falutin’ index” (index of complexity) that can be applied to projects, people, problems and solutions. Ideal balance is reached, when all the components of equation are harmonized, meaning high-falutin’ (smart) people should find ingenious solutions for complex problems, when low-falutin’ people (or new to the area) create “good-enough” solutions to well-know problems. The first case (complex unstructured (new) problems) may require empirical approach (a number of fast “plan-prototype-learn” cycles), whilst the second (without regards to the project size) can be decomposed to the level, where formalized approaches become applicable. Every time the balance is broken, and high-falutin’ developers are solving simple problems, resulting solution is either unnecessarily complex, or has acceptable complexity, but gives and zero satisfaction.
    Relative complexity of the problem to developer (and developer’s interest) is conditioned on the problem’s novelty (besides the area/domain complexity), which decreases over time, when developer is moving along the learning curve (and gaining the critical mass of knowledge). Applied to iterative development process, development productivity and interest doesn’t reach the extremum in every iteration, but decreases during the entire development lifecycle (local maximums on the graphic are getting lower compared with predecessors every next iteration):

Development speed, productivity and interest to the project during 4 consecutive iterations

2. Big companies prefer commit to simple (boring) problems. The bigger they get, the simpler problems they choose (though overall project/communication/information complexity may grow)

    The Gresham’s Law of Planning says:

Daily routines drive out planning and even innovation when people and organizations are faced both with highly programmed (structured) and highly unprogrammed (unstructured) tasks, the former tend to take precedence over the latter even in the absence of strong over-all time pressure.

    In other words, the preference is given to those problems that can be solved with the known arsenal.
    That is, I suppose, the second biggest reason (after the corporate culture), why some people prefer small companies, or become self-employed. Big players studiously avoid complex unstructured problems that may require empirical approach, and decrease risks and maintenance costs using the technologies and methodologies they know well, even if they don’t fit the particular situation (turning organizations into paper factories and Spring-plus-Hibernate-anywhere zombies).
    All this ends up with committing only to those problems that fit highly formalized predictable processes, bringing down the enthusiasm of developers that are forced to do the same things served up with different dressings over and over again.

Tags: , , , ,
Posted in Software Development | Comments Off

Rapid development with Lombok

Monday, March 15th, 2010

    Being an old NetBeans lover I’ve almost started feeling depressed second time for the last 10 days. My first disappointment was Spring Roo, which is going to be supported only by Eclipse – NetBeans and IntelliJ users are outboard.

Basically, the problem is that Spring Roo is heavily based on AspectJ, which is not fully supported by the both platforms. Roo followers may argue that the best IDE for Roo is the console app (with is really really good and well-developed by SpringSource team), but as a typical enterprise Java developer, I’m tied to Windows platform (which is a common corporate standard), so in the most cases I prefer to leverage the advantages of modern graphical IDEs.

    The second disappointmend could have become Project Lombok (that was highlighted on the last JavaPosse roundup). Founded by two enthusiasts, Project Lombok was integrated only with Eclipse until recently (though project contributors were actively looking for support in NetBeans and IntelliJ communities to help with making pligins for these IDEs). But in the last beta version, rich support for NetBeans was added.
    So what’s the idea behind Lombok. Simply putting lombok.jar to your classpath you can remove a lot of boilerplate code from your old Java classes. For example, @Data annotation allows you to make a kind of Scala’s case class of your bean.    
(more…)

Tags: , ,
Posted in Software Development | Comments Off

Implementing Jersey REST-services in Scala

Thursday, March 4th, 2010



    In this article I’m going to continue exploring enterprisy (thanks JavaPosse#298 for this expression) development with Scala and sharing my experience with those Java developers, who slowly start to recognize power and usability of this beautiful language, combined with ability to stay in the comfort area of both known and new high-grade Java frameworks. Previously, I briefly described, how iBatis 3.0 framework can be used together with Scala in order to build a robust persistence layer. In what follows I will show, how easy it is to build REST endpoints with JAX-RS and SJSON library in Scala.

    Jersey is the free production-ready implementation of jsr-311. It’s one of the simplest JAX-RS frameworks to use due to POJO (and also POSO) support with a wide range of annotations, and a very small additional configuration to be done in web.xml (no web.xml alteration at all is required if you use Grizzly (Simple/lightweight) HTTP web server, but I’m going to deploy my services on Tomcat). SJSON is the library developed by Debasish Ghosh that allows users to serialize/deserialize Scala classes into JSON with a very small effort (literlly, 1-2 annotations per class).

(more…)

Tags: , , , , , , , ,
Posted in Software Development | 1 Comment »

Using iBatis 3.0 with Scala

Thursday, February 25th, 2010



iBatis has been my ORM of choice for the last several years. From the Ruby-programmer’s point of view, iBatis is overloaded with XML boiler-plate code, but when it comes to high performance, iBatis is out of competition – explicit mapping of queries in XML configuration files (you have to write all the queries you’re going to use manually) has benefits of transparency and ease of maintenance (at any moment of time you can give your XML with mappings, based on standard SQL, to DBA that should not have a framework-specific knowledge to understand and fine tune your queries).

Since recently I’m tending to use Scala, modern elegant JVM-based language, for the back-end areas that require high performance. Though Scala and Java are 99% backwards compatible (you can call Scala code from Java, and vise verse), iBatis configuration under Scala was quite complicated (and ugly) as of late (until the beta 7 version of Scala 3.0 released, which contains support for non-Java classes: Scala objects, etc.).

Besides the non-Java classes support, iBatis 3.0 has some noticeable surprises (both good and bad) for old iBatis users (I’ve been using iBatis 2.3 in production, before):

  • iBatis 3.0 is absolutely incompatible with older pre 3.0 versions of iBatis: package structure, class names, XML schemes and approach, in general, are different;
  • It’s now possible to map beans via constructors (not only setter methods) – that’s the coolest feature in the light of Scala’s case classes;
  • Mapping can be configured with annotations (an obvious attempt to follow the fashion). At the first glance, not very useful – instead of active record approach (when mapping annotations are added directly to the bean to be mapped), iBatis requires to create one mapping class per domain bean;

In the rest of the article I’m going to give you a small overview that will help to use iBatis 3.0 and Scala together. The example below shows some details of the back-end for IdeaBox project that I mentioned in one of my previous articles.

First of all, you need to create a domain class that will be mapped to query results. For domain classes in Scala you may use case class concept that provides lots of useful features (I’m particularly interested in the field’s immutability and handy default implementation of toString method):

trait DomainClass{
   def isNotEmptyString(str: String) = str != null && str.length > 0
}
 
case class Topic(id: String, name: String) extends DomainClass{
   assume(isNotEmptyString(id))
   assume(isNotEmptyString(name))
}

(more…)

Tags: , , , , ,
Posted in Software Development | 5 Comments »

« Older Entries