Building COMET applications with Redis PubSub, Atmosphere and Akka

August 3rd, 2010

Atmosphere is a brilliant framework when you need to make your JVM web application COMET-enabled. It’s simple, lightweight and written in a very good manner (wise OO and OSGi bundles that makes it easy to extend). However, if you want to build a durable application of a really high load with thousands of clients simultaneously flooding the network with the tones of messages, you’ll probably want have to more control over the COMETed broadcast of messages. Simplifying the state of things, Atmosphere broadcaster is based on two in-memory queues: the queue of messages and the queue of references to the waiting clients. The aim is to replace those queues with a MQ solution that will allow managing active topics and clients waiting for messages.

Since recently, Redis has a strong story to tell about messaging – Redis PubSub is a nice performant alternative to the messaging mastodons like ActiveMQ. Its API is extremely simple though it offers more than enough, if you just want to take message queues away from the JVM. Bonus and the source of additional motivation (at least for me) is that Redis PubSub has a Scala actor-based client API and integration with Akka Framework

So, the following components will make the core of the COMETed system based on Redis:

  1. Akka. Offers Scala idiomatic integration with Redis and capabilities for building fault-tolerant publishers/subscribers;
  2. Atmosphere. One of the most attractive Atmosphere framework features is that it knows how to enable NIO capabilities at the most popular JVM application servers. Whilst client works with a high-level abstract API (say, suspends HTTP connection), an appropriate server NIO implementation is used on the background (Continuations mechanism, if the server is Jetty, or CometProcessor for Tomat). Moreover, in the latest versions the same API can also be used to transfer data over Websockets;
  3. Redis PubSub as a provider and background of messaging (gives ability to post/control topics through Redis admin utilities; watch queues, etc.);

Read the rest of this entry »

Tags: , , , , , , , ,
Posted in Software Development | Comments Off

Time-management. Rly?

June 18th, 2010

Watched interview with Gleb Archangelsky – time-management guru from Russia… [It sounds sooo funny - time-management guru. For me, it something as laughy as urinotherapy guru. The both religions are based on a blind truth in a miracle, and ignorance of rational aspects]

Feels like the only thing time-management enthusiasts can do – extremely effectively waste time on talking about time-management. However, everything they say is so-so-so-so straightforward – I would be ashamed of saying such banal things to people. Time-management strategies they sell ("pomodoro" technique, "pond with frogs" strategy, etc.) can help only spineless creatures, who have to train themselves like dogs: has made something unpleasant? – take a sugar, good boy!

They're all of one nature: time-management for lazybones; pills for fat-asses, who want to lose weight eating hamburgers and drinking beer; success-stories of the house wives, who earned millions of dollars spending just 2 hours per day on the Internet, etc. It is for naive fairy-tail lovers, who believe in a mystery knowledge that can make anyone a mega-successful super-human with a small effort. That's crap, don't you think so?

If you want something, it's not enough just to dream about it and believe that one day it will be yours. You should work. Even worse, you will probably have to work 15-16 hours per day, without weekends, sleeping 4-5 hours. It's the painful truth, though masses still want to believe that working 4 hours per day they can achieve something.

Be yourself and follow your own inner rhythms. Time is priceless – don't waste it on learning time-management techniques.

Tags: , ,
Posted in opinion | Comments Off

SBT: falling in love again

June 5th, 2010



    The thing that’s true about me and Scala (the language, tools, approaches, frameworks, etc.) – this is a real love, but it has never been a love at first sight. I remember, when I met Scala the first time, it appeared horrific to me – offensive syntax, functional programming, banging actors, and all the other things that can easily scare away a Java-developer, who was grown on imperative languages. But I always knew that Scala has a great potential (thanks “Programming Scala” – excellent reading for those, who wants to get hands dirty with Scala, but doesn’t know what to start with), and since I gave it another try 6 months ago it’s getting better and better every day (not kidding!). The same story was with Lift Framework – overcomplicated paradigm and HTML snippets and templates (what are they doing in the 21st century web-framework?!) first frightened me off, whereas now I’m actively using a lot of Lift components having fun.

    Getting into Scala requires patience.

    I wish I would remember that, when several months ago I tried SBT, thought to myself “why should I go there leaving old good Maven”, and got back hacking POMs and adding another plug-in to enable basic Scala features. Guess what has happened? Right, all the last week I’m actively using SBT, converting my old Maven’s POM into SBT’s project definitions :) You ask me why?

  1. SBT is designed for Scala. Do you remember your first (positive) experience with Maven? It worked like a charm – all you needed to do, is to pick an appropriate archetype, and voila, you are ready to make the most of the workflows you need (compiling, testing, building, etc.). It would be great, if Maven worked so smooth with Scala, but unfortunately, that truth is that defining a complex Scala project in Maven turns into a pluginism. With SBT I feeling myself like in the old good days – you literally need seconds to get the basic Scala project compiling and Specs (which is important!) running.
  2. SBT killer-feature (for me) – continuous whatever (compiling, deployment, testing, etc.). All you need to do is to put ~ sign before SBT command, so that, when the code of the project is changed, the action will immediately happen. E.g., ~jetty-run literally turns Scala into a kind of dynamic language – every update of a source file is instantly recompiled and deployed to Jetty by SBT. The same with ~compile – I wouldn’t say, you no longer need an IDE (with prompt highlights of errors) for hacking Scala, but when you aren’t able to run a graphical environment, immediate compilation of the file you’re editing makes life times and times easier.
  3. You can write build scripts in Scala (as far as I remember, Maven 3 is got to get support of Groovy and other languages for defining a project model, as well). Once you’ve tried writing a build script in a language you like, you will never be happy getting back to XML (after all these years of Spring, Hibernate, iBatis, etc. XML-nightmare). Writing build script with the language that enables OOP and functional programming is a brand new experience (for me, at least) and gives a lot of fun.

    So far so good, but frankly speaking, there 2 small thing that may spoil your experience with SBT slightly:

  1. IDE support for SBT is very limited. As far as I know, SBT is not on the feature list for Netbeans so far, so if you are sent into panic by console, SBT is not for you, sorry.
  2. Out of the box, SBT has a basic integration with Maven and Ivy – SBT understands simple pom.xml/ivy.xml and can use repositories/dependencies specified in them. However, it’s not possible to use dependencies specified in POM and SBT project definition simultaneously (pom.xml is simply ignored, if there’s an in-line dependency in project definition). I don’t know, what causes this limitation, but in case you have a complex Maven-based project it can delay you transition to SBT.

    Yesterday I’ve started a thread at StackOverflow to find out, what possibilities exist to translate POM into project definition. A basic solution was proposed by @retronym – it finds dependencies in POM and turns them into Scala code. I’ve implemented a bit more general script, that creates a new project definition of a correct type (based on specified packaging style) from POM with repositories and dependencies – works fine for simple projects (if you deal with more sophisticated ones, you can enhance my script or simply write your own – with Scala XML it’s extremely easy).

Tags: ,
Posted in Software Development | Comments Off

Kandash Project v 0.3 is now available!

May 28th, 2010


     As you might remember, friends, Kandash Project is my fun-project to learn Scala and Mongo a free, open-source implementation of the on-line board with sticky notes that is used in Kanban (and SCRUM), with some basic reporting and analysis capabilities. It’s extremely lightweight, but has almost everything you need for synchronizing your work with colleagues – interactive board (that tends to replace a physical one) with COMET support to provide a rich collaboration through the browser, charts to follow the trends of your team and catch the smells, and reports to get a deeper insight.

    New in this version:

  • Reporting and analysis capabilities
  • Boosted performance. Basic Akka features are used for it: Actors implementation, configurable work-stealing dispatchers, fault-tolerance, etc.
  • A lot of bugfixes :)

    You can find the code of Kandash Project at Google Code hosting.

Tags: , , , , , ,
Posted in Software Development | Comments Off

Flexible load balancing with Akka in Scala

May 10th, 2010



    Scala concurrency is based on Actors paradigm. Actors communicate with each other sending messages and doesn’t share global state, which guarantees safe concurrency. Every actor has own mailbox the messages are added to, and processes one message at a time.
    Akka framework introduces simplifications and richer capabilities compared with the native actor model. Configurable message dispatchers and fault tolerance make Akka’s Actors implementation an enterprise-scale alternative. 

    Dispatcher   

    Work stealing dispatcher, like any other Akka’s dispatcher, delivers messages to actors’ mailboxes. Message delivery is backed by Java’s ExecutorService (with manageable thread pool). Unique feature of the work stealing dispatcher is that if the message was initially sent to the busy actor, dispatcher finds another actor (“thief”) that can process the message, and puts it to the actor’s mailbox.
    Below we create a new instance of work stealing dispatcher that will be used by the pool of actors. Dispather has its own thread pool that is used for effective asynchronous distribution of messages. Thread pool size equals to the count of actors that will be load balanced:

Read the rest of this entry »

Tags: , ,
Posted in Software Development | 2 Comments »

Scorpions (GER) @ Minsk Arena, Minsk

May 5th, 2010


Read the rest of this entry »

Tags: , ,
Posted in Concert Photography | Comments Off

Effective Stored JavaScript in MongoDB

April 16th, 2010



    MongoDB is shipped with a huge number of dynamic ad hoc queries that may be used in a wide range of trivial scenarios. But when it comes to building really complex analytical queries, you have to write all the logic in JavaScritpt manually (as you may know, MongoDB uses JavaScript as an interface language for querying, updating, data processing and many other tasks).
    The easiest way to execute server-side JavaScript query is to call eval() function in your favourite Mongo driver (personally I use Scamongo built on top of Java Mongo Driver). The code should look more or less like:

    MongoDB.use(DefaultMongoIdentifier) ( db => {
        db.eval(“”" function() {
                    …
                    })
                }”"”)
      })

   Code of the in-lined function is evaluated on the server-side and executed on-the-fly.
   This approach has two obvious drawbacks:

  1. Performance. Evaluated functions are not persisted between connects, therefore should be re-evaluated, which takes a lot of time (especially, if you deal with complex calls);
  2. Testability and readability. In-lined code is untestable and looks ugly.

    Solution

    Elegant solution is based on stored JavaScript (that is someway similar to stored functions in SQL-databases). In general, you can store you function in a special system collection,  from where you will call it with eval() function.
    Example below is based on the open source code of Kandash Project.
Read the rest of this entry »

Tags: , , , , ,
Posted in Software Development | Comments Off

Kandash pre-alfa (version 0.1) is available

April 14th, 2010

    Nearly a month ago I’ve decided to suspend IdeaBox project, and start another one that I’m of a bigger demand – Kanban Dashboard (short “Kandash”).
    The reason is obvious – anyone who uses Kanban for maintenance projects can hardly find any other automation tool better, than a set of Excel spreadsheets with scripts. Sure, if the  team is located in one place (meaning, in one room) with all the job always being done in the office (no one works from home) you should be more or less satisfied with a physical board, marker and stickers. But, if your team is distributed across different locations, or you don’t want to synchronize the physical board with your electronic datastorage, when you need to build a report in order to analyze the state of things, something more capable and available is required.
    Kanban is growing in popularity last years, and people are finally coming to conclusion that there could be no universal tool for Agile development (“No tool is complete, no tool is perfect”, Kanban and Scrum. Making the most of both) – sometimes you need a fork, sometimes a knife, sometimes both. Nevertheless, the most of Kanban practitioners are either have to use Scrum tool-set or invent their own approach. Moreover, there are still not many tools that can visualize the board even for Scrum (whilst it’s the main component of the both Scrum and Kanban that makes the development process transparent) and decrease the amount of dirty manual job – the most of Scrum-tools on the market more look like as a colorized spreadsheets with the basic analysis functions and ability to build a burndown chart.


Kandash board screenshot

    In Kandash I make accent on the board with rich user experience. You can do with this board anything you do with the old-fashioned board, marker and stickers, plus having an ability to restrict work in progress, collapse/expand sections, persist and export the board model, control access and many more. Statistics is gathered when you place the stickers on the board, drag’n’ drop them between tiers (IOW, states of the workflow), create new markup, etc. Basic analytics is included as well – Kandash board model can be exported into Excel and PDF to make a weekly (or whatever else) regular reports with cumulative flow chart and tables.
    You can download the sources of the current version at gcode project hosting. You can also find a table with 0.1 features and a small sceenshot-based guide (but don’t be afraid – everything is absolutely intuitive).
    If you find this idea interesting, please contact me – let’s make the things happen together!

Tags: , , , ,
Posted in Software Development | Comments Off

Apocalyptica @ Palace of the Republic, Minsk

April 6th, 2010

Read the rest of this entry »

Tags: , ,
Posted in Concert Photography | 2 Comments »

Rules of falling interest

April 5th, 2010



    Have you ever asked yourself, why your day-time job in the big company is so simple and boring, and it’s not getting better, when you’re moving from one project to another (or, even, changing the organizations you work for)? Be a realist – it’s not because you’re so much smarter than the other people (but you’re definitely smarter than 9-to-17 office plankton, which is totally satisfied with the state of things, and, obviously, treats programming only as a well-paid way to kill time between breakfast and supper). It’s all about human’s natural tendency to fight entropy and disorder, which turns out into two simple rules:

1. All the projects become boring over time

    This rule is based on Plauger’s “falutin’ index” (index of complexity) that can be applied to projects, people, problems and solutions. Ideal balance is reached, when all the components of equation are harmonized, meaning high-falutin’ (smart) people should find ingenious solutions for complex problems, when low-falutin’ people (or new to the area) create “good-enough” solutions to well-know problems. The first case (complex unstructured (new) problems) may require empirical approach (a number of fast “plan-prototype-learn” cycles), whilst the second (without regards to the project size) can be decomposed to the level, where formalized approaches become applicable. Every time the balance is broken, and high-falutin’ developers are solving simple problems, resulting solution is either unnecessarily complex, or has acceptable complexity, but gives and zero satisfaction.
    Relative complexity of the problem to developer (and developer’s interest) is conditioned on the problem’s novelty (besides the area/domain complexity), which decreases over time, when developer is moving along the learning curve (and gaining the critical mass of knowledge). Applied to iterative development process, development productivity and interest doesn’t reach the extremum in every iteration, but decreases during the entire development lifecycle (local maximums on the graphic are getting lower compared with predecessors every next iteration):

Development speed, productivity and interest to the project during 4 consecutive iterations

2. Big companies prefer commit to simple (boring) problems. The bigger they get, the simpler problems they choose (though overall project/communication/information complexity may grow)

    The Gresham’s Law of Planning says:

Daily routines drive out planning and even innovation when people and organizations are faced both with highly programmed (structured) and highly unprogrammed (unstructured) tasks, the former tend to take precedence over the latter even in the absence of strong over-all time pressure.

    In other words, the preference is given to those problems that can be solved with the known arsenal.
    That is, I suppose, the second biggest reason (after the corporate culture), why some people prefer small companies, or become self-employed. Big players studiously avoid complex unstructured problems that may require empirical approach, and decrease risks and maintenance costs using the technologies and methodologies they know well, even if they don’t fit the particular situation (turning organizations into paper factories and Spring-plus-Hibernate-anywhere zombies).
    All this ends up with committing only to those problems that fit highly formalized predictable processes, bringing down the enthusiasm of developers that are forced to do the same things served up with different dressings over and over again.

Tags: , , , ,
Posted in Software Development | Comments Off

« Older Entries