Karan Singh

Where there's a Cloud , there's a way !!

Working With NUMA/CPU Pinning

| Comments


The term CPU pinning / process affinity / NUMA generally boils down to the same idea that In a multi socket system, application best performance can be achieved by allowing application threads to get execute on the CPU core which is as close as to its memory bank. In most of the cases Linux process scheduler is intelligent enough to do this , however if you do this manually by yourself , it’s most likely that you will enjoy luxury of increased application performance. Here are some of my notes describing steps required for process affinity setup

Verify how application (radosgw in my case) threads being executed currently. The 5th column psr which represents processor core.

for i in $(pgrep radosgw); do ps -mo pid,tid,fname,user,psr -p $i;done

Don't Underestimate the Power of Ceph Placement Groups

| Comments

Ceph PG

Today i would like to share findings from one of my curious testing, which came from this basic question.

How Placement Group count affects Ceph performance?

If you are reading this blog then i assume you know what Ceph is and how Ceph Placement Groups (PG) works.

It all started when me and my colleague Sir Kyle Bader were discussing around how to get more performance out of our Ceph cluster with the following environment details

How Application IO's Are Treated by IO Scheduler

| Comments


Recently i have been doing FIO benchmarking and i found that IOPS reported by FIO != IOPS reported by iostat . Which made me think Why The Heck ?

So here is my FIO job with bs=4M and seq write

$ fio --filename=/dev/sdb --name=write-4M --rw=write --ioengine=libaio --bs=4M --numjobs=1 --direct=1 --randrepeat=0  --iodepth=1 --runtime=100 --ramp_time=5 --size=100G --group_reporting

write-4M: (g=0): rw=write, bs=4M-4M/4M-4M/4M-4M, ioengine=libaio, iodepth=1
Starting 1 process
Jobs: 1 (f=1): [W(1)] [100.0% done] [0KB/220.3MB/0KB /s] [0/55/0 iops] [eta 00m:00s]
write-4M: (groupid=0, jobs=1): err= 0: pid=424038: Tue Jun  7 23:48:32 2016
  write: io=22332MB, bw=228677KB/s, iops=55, runt=100001msec

As you can see fio reported 55 iops

Deploy COSBench Using Ansible

| Comments


In my previous blog about COSBench , i gave an introduction to this tool and have explained how you can install it and get it working.

Recently i automated COSBench installation and configuration using ansible and have created an ansible role. With this blog i will demonstrate how you can get COSBench up and running with minimal steps … because Ansible Rocks 8-)

FIO Tip: Use Genfio to Quickly Generate FIO Job Files

| Comments


FIO is one of the most popular benchmarking tool out there and its my favourite too. It’s a feature rich and provides pretty useful supporting utilities. One of the utilities it provides is genfio . As the name suggest it a tool that generates FIO job file based on arguments you provides to it.

Recently i have been try to benchmarking my server containing 35 disks such that each operation should run in parallel on all disk’s and i should get aggregated results for IOPS and Bandwidth. So i used genfio to generate FIO job file and then run fio command line using the job file

COSBench: Cloud Object Storage Benchmarking Tool

| Comments

cosbench Benchmarking storage systems is a pretty interesting job and these days i am playing hard with it . Having said that object storage benchmarking is currently under my radar.

Block and file storage is there since ancient times and so as their benchmarking tools. Object storage is way too different and the traditional benchmarking tools don’t talk object. There are not very many tools out there for benchmarking object storage, the one which is getting popularity is COSBench aka Cloud Storage Benchmarking tool.

COSbench is developed and open sourced by Intel. It’s based on classic client / server model where client node aka COSBench Driver node ( COSBench client ) accepts the job from server aka COSBench controller node ( COSBench server) executes the job against storage system gathers the result and send them back to COSBench controller node. Here are some good to know things about COSBench

Introducing Ceph Cookbook

| Comments

ceph cookbook

Why do we care

We are a part of a digital world that is producing an enormous amount of data each second. The data growth is unimaginable and it’s predicted that humankind will possess 40 Zettabytes of data by 2020. Well that’s not too much, but how about 2050? Should we guesstimate a Yottabyte? The obvious question arises: do we have any way to store this gigantic data, or are we prepared for the future? …. Software Defined Something ….. Well it’s a great saying that “Software is eating the world”. This appears to be true. However, from another angle, software is the feasible way to go for various computing needs, such as computing weather, networking, storage, datacenters, and burgers, ummm…well, not burgers currently. As you already know, the idea behind a software-defined solution is to build all the intelligence in software itself and use commodity hardware to solve your greatest problem. And the greatest minds in the industry thinks that, the software-defined approach should be the answer to the future’s computing problems.

Ceph is a true open source, software-defined storage solution, purposely built to handle unprecedented data growth with linear performance improvement. It provides a unified storage experience for file, object, and block storage interfaces from the same system. The beauty of Ceph is its distributed, scalable nature, and performance; reliability and robustness come along with these attributes. And furthermore, it is pocket friendly, that is, economical, providing you more value for each dollar you spent.

Effectively Monitoring a Ceph Cluster

| Comments

Effectively Monitoring a Ceph cluster

Hello guys, during my system house keeping i found this video , which i had recorded some time ago, thought to share with you.

This video is about how we do monitoring of our Production Ceph clusters. In our company we prefer open source software for service monitoring. We have been using the following open source projects all-together for monitoring.

  • Opsview : Its based on Nagios. Until March 4 , 2015 OpsView was available under GPL license , however its opensource version is now discontinued.
  • Collectd : collectd gathers statistics about the system it is running on and stores this information
  • Graphite : Is a tool for monitoring and graphing the performance of computer system in real time.
  • Grafana : Is Graph and dashboard builder for visualising time series data

Opsview Monitoring for Ceph cluster

Live Demo : OpenStack and Ceph

| Comments

openstack and ceph live demonstration

Hey Guys , with this blog post i would like to share live demonstration screen cast that i have prepared using my openstack and Ceph environment.

What’s expected from this Demo

  • Creating OpenStack glance image stored on Ceph RBD
  • Creating OpenStack instance with Ephermal disk on Ceph RBD ( storing instance on Ceph )
  • Creating OpenStack Cinder volume on Ceph RBD
    • Attaching this volume to OpenStack instance
    • Mounting this volume and generating some load
  • OpenStack Instance LIVE MIGRATION across hypervisors