Just to refresh our memories about what Vert.x

Vert.x as we know is a lightweight framework for creating distributed microservices. It can sale up and scale out depending on your needs. It also takes away all your pain of dealing with complexity of heavily multithreaded environments, race conditions etc. etc.

Primary unit of work in Vert.x is a verticle. Verticles are thread safe and they can run locally or remotely. One Verticle interacts with other verticle using Events which carry data with them.

Let’s take a day to day scenario.

We are getting lot of requests. Each request is independent of each other but we are unable to process all these requests on the commodity hardware that we have. How to serve all these requests coming to our cool high traffic website?

Well one answer is serve each request in a new “thread” and keep increasing the CPU Cores (Scale Up) and hope it will work. This is what your webserver does. Problem is you can only increase no of cores to a limit (How high you can go?).

Once you reach that limit you will add more such machines and leave it to load balancer to divide all these requests equally between all machines. Sounds familiar?

Well, you will have problem relying on load balancer when every service in the system faces same issue. Every time you will have to scale these services and keep re-configuring load balancer. What if this was possible in application layer dynamically. What if we could scale up and out without any pain of load balancer.  Good news is it can be achieved using Vert.x except load balancing happens inside your application layer. There can be lot of benefits of this approach which I will discuss some other time but for now let’s just focus on how can this be achieved using Vert.x.

So this problem has 2 major challenges:

1. How to divide the work between different machines so that we can keep up with this load.
2. How to combine the result from all this processing so that we can return this result to client (master) who needs answers from all workers before proceeding further (Can this be achieved by load balancer?).

So Master is like Management whose only job is to distribute all the work to developers (like you and me) and when work is done. Combine all the statuses, create a report and notify the boss and hopefully get a fat pay hike (sounds familiar?).

In terms of Vert.x We have a master Verticle which gets lot of work to do. But Master does not want to do any work.Why? Because its a “Master”. So master wants to assign all this work to Workers. Worker are also verticles in Vert.x. But then problem arises that master needs to know if all work is completed so that it can make right decision about what to do next.

So here is high level architecture we are going to flow.

Ok..so in order to simulate this first lets create lot of work.

So worker does the work and sends an event on event bus with result.

Now master needs to combine all these results. This is way cool features introduced in Vert.x 3…Composable futures It makes this so easy.

That’s all !!.  We hope this will be useful in some of your scenario.

Source code is available at https://github.com/singhmarut/vertx-master-worker-simulator

Happy coding !!

Thank you for reading. For continued insights and in-depth discussions, please follow our blogs at Ezeiatech.

The post Master Worker Architecture Using Vert.x first appeared on Ezeiatech.

These tools can be built on various technologies and integration between different component becomes an issue when people start writing code for individual component and do not target integration as one cohesive process.

Result is cluttered tightly coupled development based on many assumptions. Necessity is to take a comprehensive uniform approach to tackle this problem.

Various open source ESB and commercial solutions are available for this but in my experience I have found that using an ESB right away is a difficult thing for developers to accept as every ESB faces a certain style of development.

So approach should be slowly introduce the concept involved in an ESB during development process.

One such core ingredient in ESB is called  Enterprise Integration Patterns into core modules and let people learn the concept and admire them.

Enterprise Integration Patterns (EIP) are patterns just like GOF patterns but context is “integration” between disparate systems.

These patterns are well documented in the book by Gregor Hohpe and Bobby Woolf.

If you want to have a quick look at what these patterns are then go to http://camel.apache.org/enterprise-integration-patterns.html

Apache Camel is an out of the box solution which implements all patterns described in EIP and provides integration with many more components which can help developers by avoiding pain of writing boilerplate code. It also gives many useful implementations which will bring a consistency in your large code base that everyone will be able to easily understand.

Patterns are a proven way of building robust scalable and easy maintenance software and following them religiously can make a developer’s life easier and make organization save huge amount of money in long term.

Camel not only provides implementation of these EIP but more importantly it’s syntax is so fluent that it brings a paradigm shift the way one writes integration code.

It is an important framework to learn for a Java developer and I do not find any direct alternative in Microsoft world.

Just to give you an idea consider a simple scenario. You want to integrate to a third party API like a social networking, Courier Service, Text to Speech service or something else. It may so happen that while calling this API network glitch occurs and your call does not succeed. What do you want to do?

You guessed it right..All we can do is retry. So probably you will write a while loop which will check the HTTP status of this API and then if it’s not 200 then just retry. OK you can do that but problem is very common to be faced by every developer in your team so all of them are writing same type of code.

If you are a well managed team then you will probably create a shared library and publish it to your local maven repo and inform about it.

Great..you did that. Let’s add another twist to the tale..You want to retry but after some time. What do you do? Put sleep somewhere in your while loop. Again enhance your library and roll out a new version. Over the period you will come up with many such problems and end up writing same code which is there in camel (only better implemented).

Camel on the other hand provides a nice fluent interface which promotes consistent coding style across the board and you have a tool in your hand which solves lot of your problems out of the box.

In the next article we will try to develop a small application for building a fault tolerant application using Apache Camel.

The post Apache Camel – Why You Should Learn? first appeared on Ezeiatech.

Anyways, coming to the point. Talk started with introduction to No SQL, what it is and what kind of use cases it might be fit. As expected lot of people were from RDBMS background so it was very hard for them to initially understand the concept of No SQL.

Fortunately that was not the case with me as I have been exploring these technologies for last couple of years and I have delivered some successful projects using Neo4J and MongoDB.

So I would like to put my thought process forward.

No SQL means that people in SQL world should look out for alternative persistence technologies when need arises. Lot of times when data needs to be stored SQL does not provide a natural way of storing it.

Take for example hierarchical data and unstructured data. Many to Many relationships are not a pretty sight anyways.

I found SQL to be very limited in features and capabilties when it comes to storing hierachical data.

All you can do is create child parent relationship and do recursive queries. As we all know 7 out of 10 times in a big application database is the first culprit and you need real experts to fine tune SQL Queries when you start feeling that application is not behaving upto  expectations and users are drifting away.

No SQL can be divided primarily in four categories:

1. Document Based (Mongo, Couch)
2. Key Value (Redis, Memcache, Dynamo, Riak)
3. Columnar Database (Cassandra, HBase)
4. Graph (InfiniteDB, neo4J)

Out of these 4 graph database have a unique place and easier to decide at least in my experience. Whenever data is hierarchical and relations can not be modeled using RDBMS easily one can go for neo4J. Hierarchical data may require deep traversals and RDBMS definitely does not rock at this.

Document databases are easiest to use and MongoDB is a sheer pleasure to work with. It gets up and running very easily and have most features compared to any other database when it comes to querying.

So I will divide this post into some headings.

When to use No SQL

My answer is always. Hardly there is any application today which does not have unstructured data. Everybody wants to grow so it is most likely that sooner or later you are going to generate data that will be large. Be it from social media, your own click stream capture. Storing Web logs or whatever. You want lot of users to come to your site.  More the merrier so yes you will generate lot of data.

So having a polyglot persistence built in right from the beginning in application is gonna help you at later stage.

It’s easier to define what kind of use cases No SQL is not a good fit rather than finding good use cases (except big data).

When you need strong ACID support (Financial information specifically). Payments, User registration then I will never think about storing these in a No SQL. Risk is just too great.

Some people argue like one gentleman at the conference that amazon is using Dynamo for storing user cart information. May be it can be used. But I will not agree with this 100%. Reason is simple. All NoSQL databases are eventual consistent. That means due to replication there is a delay in syncing the data on multiple machines.

So when you run a  query you do not know which copy of data will be returned whether that’s latest information or old information. So if you use Mongo may be there is a chance in theory that user will see his old cart and not latest one and next time he looks at his cart he might be seeing latest one. I would not want this. So consider this use case out of scope for mongodb.

Some NoSQL dbs like Riak provide vector clocks but they have their own problems.

http://docs.basho.com/riak/latest/references/appendices/concepts/Vector-Clocks/

So one has to be very careful in such scenarios.

Take another use of promotional campaigns. Lot of companies do promotional campaigns and they need to store these huge emails and they even track their performances then it is a definite use case for a NoSQL. Data is huge..it does not have to be transactional and if we loose some data due to some node failure we will not loose our job.

In No SQL world two principles are very prevalent.

1. Prefer redundancy over normalisation. Disk is cheap theortecally infinite and No SQL due to in built horizontal scalability have no problem handling data. So when you have to optimize your query do not change your schema but you can store redundant data in separate table suitable for this query only.
   Design schema for your queries. Write down use cases and design your data storage accordingly. Do not try to do otherwise as in SQL world.
2. Design your app for Consistency .relations/rules/data quality are all handled in application as NoSQL does not guarantee this. There are no joins and locks are at row level.

Let’s look at some of the use cases for each database.

MongoDB

It should be first choice by default..more so when you do not have much experience with No SQL world. Most close to RDBMS supported by excellent client drivers and easily integrated with any stack PHP, Java, Python, Node JS you name it.

It can be used as a general purpose database. Supports secondary indexes. Shards easily

Only problem I find with MongoDB is versioning. I never know what version of data is going to be returned to me.

Neo4J

Most suitable for Social Graphs. Deep traversals. Recommendations. Implemented a subject hierarchy using this and traversals were damn fast. Provides excellent Apis in Java..supports REST. No other fully open source graphDB comes to close to this one.

Hypergraph is comparable but lets down in Apis compare to Neo4J especially traversals.

Column Oriented

Cassandra and HBase

Both are column based with minor differences here and there. Cassandra was developed by Facebook and became an Apache incubator later on. HBase sits on top of hadoop.

Their use case I see is only one. When you have lots of data. Hundreds of TB to PB and you just want to do Map Reduce though Cassandra provides CQL. You will know when you have that much data.

Key Value Pair

e.g Memcache/Redis – They are damn good at what they do. Primarily used in caching layer they can server data to your clients insanely fast. Can shard on hundreds of servers easily and redis even provides many useful Data Structures in built.

So here in short I just provided my experience with No SQL.

Comments are welcome.

The post NoSQL and It’s Importance first appeared on Ezeiatech.

There are 4 stacks that are very popular these days for an internet application to be built in.

1. Java world – Spring/Grails ( I have worked on both)
2. .Net world – Asp.Net/C# Combination
3. Ruby World – Ruby on Rails
4. PHP – LAMP Stack (Zend/Symfony)

Ruby on Rails, PHP frameworks and Grails are built around same philosophy i.e. Rapid Application Development. They follow an approach called Convention over configuration.

They all follow same development style and are supported by dynamic languages Grooy/PHP/Ruby. e.g. MVC is the universally accepted pattern for developing Web Applications now.

An internet application would typically have three layers.

1. Presentation Layer – All frameworks provide a rendering engine to easily create web pages. Some even provide pluggable engines so you get a choice.
2. Business Layer – Usually is written in one primary language but often interacts with external services.
3. DB Layer – ORM like hibernate/GORM/ActiveRecord/Doctrine.

Almost all stacks provide an easy to use ORM layer which conceptually works in more or less same manner.

I do not see how someone experienced in GORM would have any problem working with Symfony ORM or Active Record.

Consider search which is a very important factor for a website. You will integrate a tool like Solr/Elastic Search in your application. It hardly matters what language you use for accessing Solr, you will still be doing similar calls.

As far as MySQL is concerned it is Database and is totally independent of language being used though there might be some differences in syntax of ORM layers. I have worked on C++/.Net/Java and Grails and I find myself equally at ease because underlying remains same.

Yes it is true that you can do faster development if you have someone who is experienced only in the stack you are using but things change when it comes to scalability.

Having said that largest of the applications today follow an approach called “right tool for the right job”. e.g. FB uses PHP as their Front End application but in the backed they have whole lot of languages including Java/C++/Erlang. That is how they achieve the scalability.

Facebook long before created a project called Thrift just to achieve seamless integration between different languages and is widely used today. I have used it myself to call Neo4J API in Java from .Net as .Net apis were not available for this graph database.

Even twitter story is very famous and they all solve their scalability problems on JVM and not Ruby on Rails which has been their primary stack for a long time.

PHP by design is not a multithreaded language and I am sure that people when they look for scalability they more often than not try to go to JVM and apache projects. This is called Polyglot style of programming where you do not stick just to one stack but do the right thing.

It is tremendous tool for creating sites quickly and gives many features/plugins out of the box but it can not solve all the problems a successful website will face in future.

I know some of the e-commerce company who quickly created a website by using Magento but now find it extremely difficult to scale due to it’s design of database which follows EAV model.

Though scalability is again subjective and it depends on what you are looking for. One may argue that many sites will never reach the scale of Facebook and Twitter but my argument is simple that if using right tool is the approach then it will not harm you. It is like Test Driven Development. Some people find it wastage of time but gives you robustness undoubtedly.

Take another scenario of NoSQL. I have used couple of NoSQL solutions and for a start up it is important to save cash, they do not try to Scale UP as bigger machines are quite expensive but scale out using commodity hardware. Hence NoSQL. NoSQL is analogous with Big data these days.

Again this is called polyglot persistence and is heavily in use today. It is easier to keep using MySQL but there are problems where MySQL does not scale well hence you look to add a new stack in your development process.

I have introduced new technologies time to time very systematically and that is how I have been able to solve scaling problems. These technologies at the same time have saved considerable time of the team. e.g. Using MongoDB we were able to solve long standing problems of sending and storing bulk emails for campaigns.

So I am of the opinion for different jobs different tools are required and they should be used systematically in a timely fashion. If right strategy is adopted they can give tremendous results in productivity and scalability.

As far as my experience with PHP is concerned then apart from fixing some bugs in an existing PHP application.

As part of my job I would not mind using or learning any technology including PHP but my effort would always be in the direction of doing the right thing keeping in mind time and budget constraints. Having worked in different stacks in my career I think this is something I really like about myself.

The post My Opinion About PHP first appeared on Ezeiatech.

While this sounds good at the same time I feel we are unable to utilize core strengths of technologies used.

Their usage in architecture will always be dependent on person to person and their experience and comfort level. But some simple things if adopted early stage can give immense benefits.

One such technology is our simple cache which is being used everywhere but alas not in the right way.

It’s called redis. Yes the kind of benefit this humble software can bring to create a highly scalable websites is awesome.

I have seen many startups are using Elastic Search as a “Cache”. Yes, I can not imagine someone doing this until you are total nuts. ES is an indexing technology and not a “Cache”. It will never update in real time which is what is expected from cache. It will never be in sync with underlying storage and hence almost always have stale data.

Using it as primary data source for anything is simply insane and shows the immaturity of the architect who chose it in the first place. It will not work except for most trivial projects.

Anyways ES has its niche but it’s not a cache but Redis is. It provides many advanced data structures and can do hell lot of thing in caching layer itself.

One needs to take a hard look at various data structures provided and how they can be used to create a “real” caching layer on top of it and get same output which one was hoping to get from ES out of the box.

It has sets/hashes/sorted sets/lists etc.

Keep putting data at the rate your site might never experience and retrieve it at blazingly fast speed.

Most importantly its single thread model brings the consistency one might need for creating distributed counters.

Think about it and if you are confused how it can be used for your use case just like always drop an email..who knows I may like your use case interesting enough and help you.

The post Web Application Scalability – ElasticSearch VS Redis first appeared on Ezeiatech.