observu

It has been a while since our last update. In this time, we’ve been working closely with our first customers to determine and implement various essential features. We’ve also applied our experience and research on hosting in the cloud to their projects. This has led up to a major milestone last week: observu.com now hosts the latest beta and a more descriptive website. It is still very private, but if you get on our mailinglist, we can let you in soon.

In terms of development we’ve made a lot of progress on properly organizing the API and mobile website code, to share 100% of the codebase with the main website. (proper MVC with only difference being the View). We are a big fan of Redis, which we’ve used extensively for various queues and rate limiting solutions, that would be challenging to get right otherwise.

What we are working on now is usability and extended reporting. Of course we also have a heap of features in mind, but we would love to have your feedback first, to know what really matters.

As Observu is all about improving uptime and removing bottlenecks, we strongly believe that we can’t do with an ad-hoc infrastructure either. Especially as the exact time you need Observu is often in case of emergency, we feel strongly about the ability to recover from outages quickly.

We’ve selected Amazon for hosting because it is both flexible, is available in multiple parts of the world and has an excellent network quality. However, earlier this year is has been shown multiple times that no datacenter has 100% uptime and that if failure occurs, it is big. Therefore we are currently working on our architecture to be able to quickly overcome such events. The actual details warrant a separate post.

Our obsession with reliability also touched an other area of development: Our initial implementation of SMS notifications proved unreliable. Therefore we changed to Nexmo as our partner. It provides us with actual delivery confirmations, allowing us to monitor delivery.

Our Growing Development Stack

I personally always like to know what people are using to create their product, therefore a listing of almost everything we use:

• Ubuntu on Amazon EC2
• MySQL
• Redis
• PHP
• Perl
• jQuery
• RaphaelJS
• boto
• Fabric
• chef-solo

In the area of 3rd party services we rely on:

• Amazon AWS
• Nexmo
• Tropo
• Sendgrid
• Github
• Uservoice

These services allow us to focus on the things that really matter: gaining insight in all parts of your deployment and staying on top of the events that will occur. To improve that insight, we are currently working with the first customers to implement monitoring as part of their stack. A great example is the need to monitor logfiles centrally as soon as there are multiple servers handling your front-end.

An important milestone has been met: we’ve implemented all core features that make up our monitoring system. The last important hurdle was the completion of a flexible, yet easy to understand, system to create and configure event rules.

Of course it is far from the system we imagine. Nevertheless, we feel confident that what we have now is a very useful product.

In addition to this technical progress, I can also present you the new observu.com logo:

Current efforts are focussed around documentation and workflow, to make sure the first users get the experience they deserve. Furthermore, the API is being finalized and the production environment architected.

We are very excited about the coming weeks when the first testers will finally enter the system.

We are getting closer and closer each day, we are almost ready for the first bunch of testers. Our attention is shifting more towards the user interface, to create the best experience possible.

An important part of that user interface are the graphs, that start to look better and better. An example is this stacked CPU usage graph:

This also shows an other major point of progress: we now have a fairly easy to install daemon script to collect this data on linux servers. Currently it collects load, cpu, memory, disk and network statistics. We are planning on collecting a whole lot more soon.

The final part of this months effort is a mobile website. It provides with a simple and clear way to check your site and server status on-the-go. For example, when you receive a text notification, you can instantly check what is actually going on.

We feel confident that we can allow the first testers access next month and are really looking forward to their feedback. To make sure, you can be one of those testers, sign up for our mailing list!

It has been a while since I last updated you about our progress. We are still continuing our work on various reports. An important part of this is creating informative graphs. Although we liked Open Flash Chart, it felt a bit sluggish and we decided to go for a solution based on the Raphael library: an SVG abstraction with fallback for IE. It comes with a limited graphing library: g.raphael, but it was not mature enough for our needs. Another Raphael based charting library is Grafico, it’s able to display a few great graphs. However, we choose to create our own, mostly because Grafico depends on Prototype, which we do not use and because we would need to extend it for our own graph types. Although the library is well coded, we did not feel confident about customizing it to our needs. We will open-source our own library as soon as it is in a usable state.

At the same time, we’ve started to work on a very basic mobile website, which allows you to check your status on-the-go. We hope to slowly add more functionality and at the same time keep things really quick and simple.

Another major part of development involves the collection agent. We’ve chosen to use Perl, to maximize portability and reduced dependencies. Additional advantages include easy customization and the ability to verify that it does not contain harmful code. The next challenge in this area is to create an install script that works across distributions.

On the front-end we’ve introduced a new splash page for observu.com, it contains the first iteration of our new logo, which still needs a bit of work. We also got a very nice mascot designed, but we will keep that a secret for now.

If you visit the new splash page, you will also notice that we’ve selected UserVoice for feedback and support. I’ll write a separate blog post about the selection process soon.

We are also talking to potential users about their monitoring needs, if you feel you could contribute by telling us about your problems, please feel free to contact me.