Saturday, July 5, 2008

Yet an another EC2 Experience - Cybernet Slash Support runs Payroll on EC2

The objective of this exercise is to determine the effectiveness of utility computing over enterprise method of computation. To ascertain the facts on a real-time scenario, we decided to try out a payroll system as it involves complex and time consuming processes.

We were amazed by the outcome of this exercise as our Payroll system thrived on EC2, and in a nutshell the initial finding is that this is a right step towards reducing cost /time and improving productivity. We are pleased with the outcome and would like to take this opportunity to congratulate Amazon for this great service. Infact, we are pretty confident that based on our testimony of this great service from Amazon, many more enterprises would contemplate and eventually move their payroll system to Amazon web services environment.

Our payroll system takes about 5 to 10 seconds to calculate a pay check per employee and it takes approximately 6 hours for 4500 employees. Our current infrastructure has two dedicated servers to process the payroll and we could achieve each payroll cycle in about 3 hours.

Our challenges were to,

1. Reduce the processing time
2. Optimize the usage of available resources
3. To come up with an approach to withstand the increasing head counts.

Let’s take a look at our approach towards solving the above listed problems.

Our Approach
We looked at various cloud computing offerings and finally decided on AWS. This is mainly because of its cost effectiveness and composite service offerings like EC2, S3 and Simple Message Queue. This unique infrastructure provides flexibility for us to store and operate independently.

Our payroll software is well architected based on distributed computing. We could easily plug out the Payroll engine and make our first AMI. The core components of this AMI are Fedora, Java 1.5 and the Payroll engine. We have also created a My-Sql AMI to host our payroll database.

On the other hand, the transfer of entire database to S3 may pose problems due to its volume. So we decided to extract the necessary information using Kettle (It’s an amazing tool for data extraction needs) and upload to S3.

We launched four different instances of our Payroll AMI and one instance of My-Sql AMI, importing the database from S3. The db server was running with a binary log option switched on. The “payroll engine” configurations were modified in accordance with the My-Sql database and launched the payroll execution. The entire payroll process is completed in about 2 hours, which includes delivering the pay slip PDF to our employees email box.

Upon completion of the payroll process, we took a differential backup on the payroll database and stored it in S3. The differential database is pushed back to the original payroll database using kettle again.

It was a great experience running our payroll on AWS. This is not only because of the fact that it enabled us to reduce the processing time considerably (from 5 hours to approximately 1 hour), but we have also got a scalable solution to withstand our increasing head count day by day.

As we discussed, AWS has rich set of offerings for utility computing like EC2, S3, Simple Message Queue...etc. This would definitely change the enterprise computing to a new dimension soon and our payroll exercise is a classic beginning for that.

The biggest advantage of using EC2 is ‘Power Costs associated with air conditioning in India’ A 200 square feet data center consumes close to $1500 in power costs each month. We plan to embrace Amazon’s cloud computing for several applications that are not constrained by latencies of (250 to 250 ms) on the cloud.


Anonymous said...

Good stuff. Thanks for sharing.

Botchagalupe said...

Great stuff... However, I am trying to better understand the ROI here. Initially you say the payroll takes 6 hours to complete.. then you say..

"two dedicated servers to process the payroll and we could achieve each payroll cycle in about 3 hours."

Later you say it takes two hours then you end with this statement...

"enabled us to reduce the processing time considerably (from 5 hours to approximately 1 hour)"

I am not trying to be a jerk but I just trying to understand the numbers. Here are a few questions I have:

1) What was the original configuration of the dedicated servers (e.g., one MySQL and one Payroll server)?

2) When you went to 4 Payroll servers did how did you achieve the parallel processing effect (splitting the data up, SQS, ...)?

3) Was their any latency uploading and downloading the S3 data?


CSS Labs said...

If you read the blog carefully, 2 hours is for completing the entier process including delivery of our pay check PDF file.

I mentioned abour one hour time period for completing the payroll.

The point here is not about the numbers, its about building scalable solution using cloud computing.

To answer to your questions,

Our local setup has clustered database servers used for various purposes not only for keeping payroll related information.

We have two dedicated servers for processing our payroll.

As i mentioned in my post, our payroll engine has been built for distributed computing. I believe it uses some messaging infrastrucutre to share their state. We have used only EC2 and S3for running our payroll.

We found some latency issues uplaoding directly from india to the AWS S3 servers. But we managed with the help of our US servers.

Payroll services said...

Great stuff...
good information...

Payroll services

Term Papers said...

I have been visiting various blogs for my term papers writing research. I have found your blog to be quite useful. Keep updating your blog with valuable information... Regards

treating ed said...

Thanks. I think amazon is the best shop

Anonymous said...

My friend just started his business last year and he wants to try outsourcing the company's payroll processing. He tried to look for payroll Colorado providers in the Internet. Prior to that, he studied the advantages and disadvantages of outsourcing payroll processing service for the company.

louis vuitton waterproof keepall said...

I think... I agree with you but I disagree this..
louis vuitton have an outlet
louis vuitton ankle boots

lvbagmall said...

Your site is amazing.I am very impressed to see this,i want to come back for visiting your site.Keep doing Good as well as you can..

Anonymous said...

Your blog is really helps for my search and i really like it.. Thanks a lot..:)
Term Papers Writing Services

youlacka said...

That is really very good article.kızlarla chat,I am glad to know.mersin chat,Thanks for sharing !avrupa chat

inflatable bouncer said...

I have read your article, and I think it is great, thanks for your sharing.

data recovery boston said...

Very interesting article,all the information is so beneficial and valuable for us.presentation of article is attractive, that i am surfing around your posts.

DKM Online said...

To efficiently manage the entire payroll work the requirement of payroll processing services is needed.
Payroll Processing Servicess

DKM Online said...
This comment has been removed by the author.
yetiken software said...

The information which you have provided is really nice and awesome. Provides some very useful insights of how things actually work. I want to share something more about this as well.

software development company

Bill Cheng said...

For good payroll systems please contact us from the links below

The outsource payroll company are something that has been best enough in the market as well as it has successfully grabbed the attention of many users who are efficiently looking for financial help as well as management services. The best of payroll services Epsom are gaining popularity and there is where such services are making the best profit for all users.

Sarah Hall said...

Thank you for sharing this report with us! Keep in mind that our team includes professionals exceptionally holding higher degrees in the majority of academic disciplines; thus it is possible to cooperate with confidently!