Category Archives: Infrastructure

Networking Primer – Part 5.1: Network Layer – IP Addressing

Posted on by

Previous: Networking Primer – Part 4: Transport Layer, TCP and UDP

The clue might be in the title, but the Network Layer is one of the more important layers in the network stack.  So far we have defined high-level identities, application services and data formats. We have also established our protocols, chopped up data into smaller more manageable pieces and tagged them with a sequence number. It is at this point we need to start defining more details on where we will be sending the data and moving closer to establishing a unique address for both source and destination. We also need network components to be in place to allow us to transmit and route the data appropriately.

Image of 3D IP Address

The two core concepts at work in this layer are addressing and routing.

Addressing: Internet Protocol (IP)

The most ubiquitous protocol used for network addressing and routing is the Internet Protocol, IP (i.e. the IP from TCP/IP). As with most protocols, there have historically been alternatives such as IPX/SPX Novell Netware’s protocol, but these have mostly disappeared as IP has become the defacto standard for data communication. Again, the clue might be in the name here, but it is this protocol that underpins the internet and fundamentally enables everything we do on the internet.

The first thing we need to consider in IP, is addressing. For each node in a network (note: nodes are sometimes referred to as Hosts), we need to provide a unique IP address.  As of today, the bulk of internet communications use IP version 4 (IPv4). IPv4 defines an address as four numbers between 0 and 255, separated by periods  (For example: 135.168.0.23). The format is a little strange and this is due to the underlying need for the systems to convert the more human readable IP address to/from the more computer usable binary representation of the address.

This works very much like a postal address. In a postal address we have a house number, a street name, a city, a region, a country and a post code (USA: Zip Code).  An IP address is much simpler. That four part address is actually  made up of a Host ID (i.e. Node ID) and a Network ID. So really all we need in IP is a house number and a Post Code.  We can split the above address into two parts as follows:

Network ID: 135.168.0
Host ID: 23

For the purposes of this primer, we will keep it simple. In reality, due to the nature of the mapping between binary bits and IP address components. Splitting the address into a Host and Network ID can become much more complex and may straddle the two numbers on each side of a period. You can, for instance, have a node that sits on 135.168.10.2 and another node that sits on 135.168.10.244. One might assume that these are both on the same network (135.168.10) but they could be on different networks. For each node, we apply what’s known as a subnet mask and it is this mask that dictates which Host IDs sit in which networks. There are some moderately involved calculations that happen in the background to decide where the Host to Network split occurs in an address. If you want to do this, my best advice here is to go find a subnet mask calculator.

Networks are also defined by their size, Class A (large: up to 16,777,216 addresses), Class B (medium: up to 65,536 addresses) and Class C (small: up to 256 addresses). There is also a Class D & E but these are not related to size.

A typical subnet mask for a Class C network, looks something like this:

Subnet: 255.255.255.0

IP Network Scope: Private vs Public and NATing

It is worth understanding a little bit about scope here. A network’s scope may be restricted in an isolated room with no connectivity to the outside world. In which case, all the nodes in that room must have unique network addresses. Let’s call it Room 1. If we have another similar room (Room 2) of nodes all isolated from the outside world, those nodes could have the same addresses as those in Room 1. There is no requirement for uniqueness as the isolation of the two networks means there is never any confusion over which node has which address as they never communicate between rooms. This is most commonly referred to as a private network.

On the internet, all addresses must be unique but due to the exponential growth of the internet, public IPv4 addresses have become scarce as there simply aren’t enough of them to hand out. This has led to organisations implementing architectures where they can minimise the use of public IP addresses. An organisation might only have one public IP address, but many hundred’s of nodes inside their network. So how do all those nodes talk to the outside world? This is done through a process called NATing (NAT – Network Address Translation).  The analogy with the postal service here, is that an office building may have a front door number (Public IP Address), but lots of room numbers inside (Private IP Addresses). Different buildings will have different front door numbers, but inside their is nothing stopping them having the same room numbers (Room 1, Room 2, etc).

The NATing process is responsible for handling the conversion of private IP addresses to public ones and back again. You can think of this as a mail manager sitting at the front door. When someone from Room 1 sends something out of the building, the mail manager logs it’s destination in a table and the fact it came from Room 1. In the destination building, the receiver doesnt necessarily know it’s from Room 1, they just know that it came from that building. If they respond, the mail manager checks his table sees that the originating communication came from Room 1 and sends the reply back up to Room 1.

Public and Private IP Example

In fact, even today it is very likely that your own home uses this concept of public and private IP addressing. If you have broadband at home, the router that your ISP provided will manage a pool of internal private IP addresses that are handed out to your devices (Computers, IPads, TVs, etc) and a single public IP address that the provider uses to identify you on the internet.

Note: There is a new IP protocol which has been available for some year, IPv6. This resolves the “running out of addresses” problem, but isn’t widely deployed today.

Let’s take a look at the IP configuration on your system. Note: PC/Laptop only:

  • Go to your Start Menu and search for and run “Windows Powershell”. This may be in a slightly different location depending on your version of windows. At the Powershell prompt: Type “IPCONFIG” and see the results. You should see something like this:

    IPCONFIG Powershell Screenshot

    Some of the items listed will be recognisable, such as IPv4 address and Subnet Mask. There are also some other items like Default Gateway which we’ll cover in later posts.

  • Open up a terminal window and type IFCONFIG. On a Mac you will see a result that looks like this:

    IFCONFIG Screenshot

    I’ve highlighted some of the same information that you could also see on the Windows tab. The output here is a little more complex and has much more content. We can see our IPv4 address “inet 192.168.0.2” and also our subnet mask “netmask 0xffffff00”. You’ll notice that the subnet mask is much less recognisable and this is because it’s displayed in hexadecimal format, instead of standard IP or binary. If you convert the hexidecimal to IP, you get 255.255.255.0 which is a standard Class C network subnet mask.

    Linux produces a similar format to this with some slightly different details, but ultimately following the same approach. As you might expect Linux and Mac systems have many similarities due to their sibling nature and UNIX ancestry.

IP addresses are clearly an important factor in the network layer, but even though we have a unique network identity for both our source and destination nodes, they wouldn’t be of much use if we didn’t have a way to direct the data packets across the network from source to destination. That’s where routing comes in.

The Pirate Ship: Let’s use some of the similarities with the postal service in our analogy. Instead of an IP address we’ll have:

My node address as:  Room 62, Building 34, Legofiend Street, Manchester.
Rich’s node address as: Room 78, Building 2, Zoo Street, London.

We’re going to assume that all addresses are public and that there’s is no NATing process occuring between public and private. To align more closely with IP Addressing, let’s split our addresses into a Node and Network ID. So, in both cases the “Room” is the node and the “Building, Street, City” part of the address makes up the network ID.

To pick up where we left our transport manager earlier in the series, we need to apply more information to our packet. It currently has a TCP label and sequence number.  We’re now going to take that packet and put it inside another one, a slightly larger box. On the new package, we’re going to write the destination node address (Rich’s) and the source node address (Mine).

Next: Networking Primer – Part 5.2: Network Layer – DNS and DHCP

Networking Primer – Part 4: Transport Layer, TCP and UDP

Posted on by

Previous: Networking Primer – Part 3: Application, Presentation and Session Layers

The transport layer is responsible for providing reliable data transfer services to the upper layers of the OSI stack. It is at this stage that we start to consider actually sending the data. This layer also involves the segmentation/desegmentation of data into smaller chunks. It is very rare that a network will be configured to take a large payload from one source node to a destination node. This is why we will segment the overall data payload into smaller pieces. This is also the first place in the stack where we start to apply some addressing so the destination node understands which listening receiver the data is intended to reach. This address comes in the form of a port number. The destination node may be simultaneously listening for (or have the ability to connect to) different network services and the way we distinguish between these services is by giving them a different port number. For example, web server usually listen on port 80 where email servers will listen on port 25.

Connection-Orientated vs Connection-less Protocols

There are essentially two flavours of protocols that exist in this layer.  Connection-orientated protocols will involve communications that must be received without data loss with the data arriving in a certain sequence. Transmission of legal or financial documents would fit into this category. The applications at both sending and receiving end have an expectation that the documents will be received in full and with the correct format. An example of a connection orientated protocol is the TCP (Transmission Control Protocol) protocol which makes up the TCP out of the TCP/IP suite.  The TCP sender will break down data into TCP segments.  It will send each of these across what may possibly be an unreliable network. Due to the nature of packet switching networks, although each of these packets may have been sent in sequence it is possible that they may get routed differently and arrive out of sequence.  At the receiving end, the TCP receiver will take the data re-sequence it if the packets have been received in the wrong order and also send a request back to the source, if any of the packets are missing.

Connection-less protocols, such as the UDP (User Datagram Protocol) don’t have the same level of error-checking and control. In a connection-less transmission, packets are sent in sequence but they are sent with the best effort approach.  There is no checking to see if the packets were received and no resequencing of packets at the destination node.  So why use connection-less? Well, not all applications need data transmissions to be without loss. For example, if you are streaming online video and the packets for that live video are received slightly out of sequence, or the occasional packet is dropped; this doesn’t make much difference to the overall viewing experience at the destination node. A huge benefit of this approach is that the protocol is much less chatty, therefore can provide better performance with less bandwidth.

The Pirate Ship: To continue our analogy, let’s imagine that I have a transport manager in my office and Rich has a transport manager in his office. In this case my transport manager is going to use the connection-orientated TCP protocol. This is because he needs to break down the Lego bricks into segments (individual blocks) and send them reliably across to the destination.  All the blocks must be received at the other end and they must be reassembled in the right sequence in order to reconstruct the ship. To do this he takes each individual block and puts it in a small parcel, he then writes on the parcel the protocol (TCP), a port number and sequence number.  This is where the transport manager’s responsibility ends, he will  listen to find out if any segments were not received and then resend these, but for the actual movement of the parcels other entities further down the stack will take over.

Next: Networking Primer – Part 5.1: Network Layer – IP Addressing

Networking Primer – Part 3: Application, Presentation and Session Layers

Posted on by

Previous: Networking Primer – Part 2: Defining Networking with OSI and TCP/IP Suite

I’ve decided to group the top three layers together into one post. This is because these are more related to the data to be transmitted across the network, rather than the underlying transport mechanisms themselves. These three layers deal with the semantics of the communication, such as who the data will be sent to, the format of the data and the etiquette to be adhered too between the communicating nodes.

Lego Pirate Ship

The Pirate Ship: As with most technical concepts, analogies can help us understand the underpinning processes which are happening as part of the communication.  For this series, I’m going to use the following analogy: I work in an office in Manchester and I’d like to send a pirate ship made of Lego to a friend, Rich, who works in an office in London.  In our day to day lives, that’s a pretty simple concept that requires a couple of addresses and a postal service. Communicating data across a network can occur in much the same way. Let’s step through the network stack to see how.

Application Layer (OSI Layer 7)

The application layer is the piece of the puzzle that is going to sit closest to our end user application. It is worth mentioning here that when we are referring to the services in this layer we are not referring to the actual application being used by the end-user. To expand on this, an example application being used might be the AnyCo ERP solution. That ERP solution may provide the capability to send reports via “email”. So it’s actually the email service which fits into the application layer, not AnyCo ERP. AnyCo ERP would sit outside of the OSI stack in an upper out of scope layer.  Other application services you might find in the application layer, might be “File Transfer”, “Web Access” or “Network Management” services.

This layer is primarily responsible for determining suitable communication partner nodes and their identities. It  also responsible for ensuring that the relevant resources are available to send the transmission. It’s in this layer that the aforementioned X.400 protocol exists. Synchronisation of communication is also dealt with at this level.

The Pirate Ship: In our scenario, The Layer 7 service we want to use is Lego Sending. I have established that Rich is a suitable communication partner as he has advertised that he likes Lego and can accept that type of toy.  I’ve also established that the postal services have capacity and is suitable for sending the pirate ship.

 The Presentation Layer (OSI Layer 6)

Now that we have established a suitable place to send my data and that the relevant network resources are in place to do that. We need to look at what exactly we are going to send. The Presentation layer deals with the format of the data, it is there to abstract the meaning of the data as the application sees it into a standardised format that can be used by the underlying network layers. Where an application may be providing freeform text, the network needs a way of encoding it. An example of a protocol working at this level is XML.  Encryption may also happen at this level.

The Pirate Ship: Let’s think of the presentation of our pirate ship as a set of Lego bricks stuck together in a specific arrangement. The bricks are of standard sizes, colours and shapes. It’s those attributes that make up the format of the data.

The Session Layer (OSI Layer 5)

This is the layer responsible for setting up and tearing down the connection that will be used to transmit the data. It should be thought of as something that is more persistent than a single transmission of data. It is not responsible for actually sending the data. It simply executes the steps required to set up and maintain a connection. These steps might be simple requests for resources or handshakes between devices to be traversed. During a session, you might for example authenticate with a website and create a session. From there you may download lots of different files using the same session. For our purposes, we’ll keep it simple.

The Pirate Ship: I call my postal service to tell them I’m going to send a package to my friend. They verify my account number and then book slots for the package on all the the vehicles which will be traversed between my office and Rich’s.

Summary

So far, via analogy, we have established the objective of our application and the Layer 7 application service (Lego Sending). We have found the identities of our destination communication partner, ensure that the relevant resources are in place to send our lego and called the postal service to set-up the relevant connections to start the communication. Next well see what happens when we actually start sending the Lego.

Next: Networking Primer – Part 4: Transport Layer, TCP and UDP

Networking Primer – Part 2: Defining Networking with OSI and TCP/IP Suite

Posted on by

Previous: Networking Primer – Part 1: Introduction

What is a network?

This may sound like a very basic question, but I’ll assume the lowest common denominator here and define this briefly.

A network is set of two or more computing entities (nodes) that are configured to communicate with each other by passing information across an interconnecting media.

Ok, now we got that out of the way, we can talk about how exactly those nodes communicate with each other in a way that makes sense and achieves our objective of passing information between them.

A Little History

It is easy to imagine how the first baby steps of networking which occurred in the mid-20th century. As with all technology we start with the simplest goal and see how we can use what tools we available to use to achieve that goal. I’m not going to go into sending signals down telegraph wires, etc. but we can assume that the first step was to send a simple signal across a wire between to locally sited computers. Beyond this, more milestones were surpassed to enable us to send over greater distances and with more complex topologies. This resulted in the creation of a number of protocols, which specified things like how two nodes would set up a communication session and what format the data sent should be in. As technologies are researched, more often than not they diverge into multiple streams of activity with different camps developing different ideas on how to progress. This ultimately results in a bunch of disparate and non-compatible technologies.  Whereas in other areas this might be palatable, workable and have little impact.. clearly in networking this is not viable. The whole point of networking is that the entities involved can speak the same language.  There must be standardisation and the first and overarching grandfather of network standardised is the Open Systems Interconnection (OSI) model.

The OSI model

The OSI model is a set of specifications, rules, guidelines, instructions and protocols that describe how networking should work. It is important to understand how this is used today. As you might guess from the above description, the model is large, complex and in some ways all encompassing. Back in the early 80’s and 90’s many companies implemented technologies that strictly adhered to the OSI standards. In fact, I worked directly with one of those related to email messaging (the X.400 protocol suite) in a previous role.  The OSI specifications are quite complex and difficult to implement. It takes a lot of specialist knowledge and effort and as a result the detailed elements of the model were soon ditched in favour of more agile standards which could be delivered quickly with ease. For example, SMTP is now the defacto standard for email messaging and X.400 is only used in some specialist areas in the military and other area. (Read more about SMTP for military email here:  Command Email Whitepaper).

That being said, the OSI model is still widely used today. Although the detailed implementations have been ditched, the model is used at a conceptual level in day to day networking conversations. It breaks down the elements of network communication into seven logical layers, and by understanding these it is very easy for network engineers to gain a common frame of reference to quickly isolate the crux of an issue during a discussion.

The Seven OSI Layers

The seven layers of the OSI model are as follows:

OSI Stack

As we can see these are stacked one on top of another, which is why we commonly refer to the multiple layers as a “network stack”.  All of the nodes in a network will have similar stacks.  A common method to aide in remembering the (Application, Presentation, Session, Transport, Network, Data Link and Physical) sequence of the layers is to use a mnemonic. “All People Seem To Need Data Processing” is  good example of one of these, but there are many and you could make up your own.

OSITwoNode

During a network communication, we start at the top of the stack with application level semantics and gradually process down through the layers. Through each layer we use the mechanism of encapsulation, until we reach the physical layer, which is responsible for sending the actual bits and bytes from the source node across the interconnecting media to the destination node(s). At the destination node(s) the physical bits are then pushed up through the stack using decapsulation at each layer until the destination application or service receives it’s intended information.  This can be thought of like nested Russian dolls, with a different doll representing a different layer of the stack but instead of Russian dolls, it might be easier to visualise by thinking of a envelope, within an envelope, within an envelope and so on.  Well discuss these with a practical example in upcoming blogs.

The TCP/IP Suite

I’m not going to cover the TCP/IP suite in depth, but it is worth understanding what it is and how it relates to the OSI model.

TCP/IP Suite to OSI Mapping

As we saw above, although we still use the OSI model as a conceptual frame of reference. We no longer use it’s detailed implementation specifications. The TCP/IP suite was loosely developed as an alternative to the OSI model, with a view to creating a simplified four layer model and implementation mechanism. The TCP/IP suite contains a much smaller set of protocols and is actually used in the bulk of network implementations today.  We do however need to be careful in the use of our terminology. If we’re referring to layer 4 in a discussion, it is most likely that we are talking about the Transport Layer of the OSI stack and not the Application Layer of the TCP/IP suite.

Next:  Networking Primer – Part 3: Application, Presentation and Session Layers

Networking Primer – Part 1: Introduction

Posted on by

The world of networking has been fairly static for many years now. It’s been historically characterised by static infrastructures that require infrequent changes.  These configuration changes were performed via command line interfaces by network engineers, usually sitting with a laptop and a cable plugged directly into a piece of networking hardware. Activities were manual, repeated for every individual device and extremely error prone due to the non-human readable nature of network configuration information.

The workloads running in the modern datacenter have most definitely changed in recent years. It has become apparent that the capabilities of current networking devices and operational approaches simply cannot keep up with the pace of change.  In the modern datacenter, the rapid and overwhelming success of server virtualisation has fundamentally changed the way applications consume resources and the network has become somewhat of a bottleneck in providing agile, reliable and cost effective means of delivering new applications.   In addition to the shortcomings of existing technology, operational processes and a tendency to silo server, storage and networking departments has also become a major blocker for any significant progress in dealing with these challenges.

In the last 2-3 years, there has been industry recognition that these challenges need to be addressed and there has been a marked shift in strategy.  There has been a wide realisation that the boundaries need to break down and the silo’d teams need to converge into singular, collaborative and multi-skilled teams, delivering IT in a more integrated manner.  The technology also needed to change and the Software Defined Networking (SDN) movement is one that has been central to this shift.

Some time ago I worked in the military messaging field and have had wide exposure to networking, as it relates to battlefield communications protocols. The concepts and NATO protocols that underpin military messaging and not so different to those used in our datacenters and I have been working on understanding the datacenter networking space in the last 6 months or so. I’d like to share what I’ve learned and hopefully provide a reasonable learning resource for those administrators who are preparing themselves for the new converged infrastructure world.  I’ll be taking things right back to basics. Explaining at a beginner level what networking fundamentally is and working through to how we are addressing the key challenges that are being faced by organisations today.

Next: Networking Primer – Part 2: Defining Networking with OSI and TCP/IP Suite

 

Software Defined X – Automation: Job Stealer or Job Enabler?

Posted on by

I’ve had many conversations in recent weeks about the commoditization of the data center with many being concerned about the effect of the diminishing need for specialist hardware and greater automation through software. More specifically, how that might affect the job prospects of administrators and other technical roles in the modern IT environment.

We are in an era of rapid evolutionary change and this can be unsettling for many as change often is.  There seems to be a wide variety of reactions to these changes. At one end there is the complete denial and a desire to retain the status quo, with an expectation that these industry changes may never occur. In the middle, we have those that tip their hat in recognition of the general direction of the trends, but expect things to happen more gradually and then there are those that embrace it with an expectation of gaining some level of competitive advantage by being a first mover.  If there is one thing that is certain, if you find yourself at the wrong end of that spectrum, you will most definitely find yourself in difficulty.

No Change Around Here

The change is happening and happening more quickly than most expect.  The automation of data center operations and a focus on innovation is a key objective for most organisations at the moment. “Keeping the lights on” tasks are becoming less relevant in this world.

Casting Off the Shackles of Hardware

Development of custom hardware based intelligence is complex. This often involves the research and production of custom chipsets for these devices.  Due to the research, prototyping and production requirements of this type of operation.  We are usually working to a 2-3 year development and release cycle. In fact, most organisations have been used to using this kind of procurement cycle, executing a hardware refresh every 3-5 years.

This has worked historically, but today there are new kids on the block and they are eating the market with a new approach to developing and delivering services. Pioneers like Facebook, Google and Netflix have fundamentally changed how service delivery works. These operations have decoupled their software intelligence from hardware and deliver their services based on commodity inexpensive hardware. This not only reduces their capital outlay, it also provides them with a platform to rapidly deliver agile software services. In these types of environments, it is not uncommon to see software releases move from a 18-24 month cycle to a daily or weekly cycle. Strategically they can pivot at a moments notice and they can easily scale or contract operations at a very low-cost. As you might imagine, this kind of agility has become very challenging from a competitive stand point for companies like Microsoft who have had 3-4 year major release cycles baked into the fibre of their operational approach (e.g. Exchange, Windows Server, etc).

What About Automation?

The more we move towards software controlled infrastructures, the more easily they can be automated. Most solutions today are built with some kind of API (application programming interface) to enable other applications to programmatically control or manage them in someway. In this decade, the industry has moved firmly away from proprietary API technologies, towards standardised ones. More often not based on the RESTful API architecture. Alongside this we are starting to see the rise of DevOps tools such as Puppet and Chef, which help bridge the gap between IT operations and the developers actually creating the applications that organisations rely on.

So What Does This Mean For the Modern IT Professional?

As the development of these tools and API interoperability progresses, undoubtedly, IT operations roles will also have to evolve.  This does not mean that there will be fewer jobs in IT.  In fact, IT skills have become more relevant than ever, but those skills have to change a little.  It is time to start moving up the stack by putting more focus on innovation in the area of application and service, rather than keeping the lights on down in the bits and bytes of the infrastructure. By doing this, these industry changes should become a massive job and career enabler, not a cause of suspicion and concern for job security.

I had a chat with a family member this week which summed this up really well for me.  We were discussing the Luddites, a 19th century movement in my home region of the North of England. The Luddites, were a group of textile workers who protested against the mechanisation of the production of garments. They did this violently under the auspices of “those machines are taking our jobs, we’ll have nothing to do and we’ll all starve”. A couple of hundred years on, we can see that starvation didn’t happen and those same people survived by finding new ways to innovate. On a sidenote, I once received a letter from a CBE calling me a Luddite who had seen me on TV discussing an environmental issue. I found this most amusing given the industry I work in and my lust of technological progress. In the same conversation with the family member, I mentioned that I was looking forward to the introduction of robot-taxis (e.g. Self-driving Google Cars) due to the efficiencies and cost of car sharing. They replied “but that could be 20,000 taxi drivers losing their jobs in Manchester alone”. I replied “Yes, but that’s also 20,000 people who could alternatively be working on curing cancer, pioneering space travel or solving the world’s energy problems”.

Conclusion – Software Defined X – Automation: Job Stealer or Job Enabler?

For me I see it as a Job Enabler. My advice… embrace the change, relish the opportunity to innovate and change the world for the better.. one step at a time.

Whitepaper: Virtual Backup Strategies: Using Storage Snapshots for Backups

Posted on by

Introduction
Effective data protection is a mandatory element in the modern IT environment. Historically, backup strategies were confined to the last few chapters in an administrator’s manual and treated like an afterthought. Now they sit firmly at the forefront of every CIO’s mind. The ability to continue business operations after a system failure and the need to fulfil stringent compliance requirements have made backup a necessity—not only for business continuity, but also for
business survival. The question organizations need to ask about data protection is not whether to backup their data, but how to backup their data.

IT systems are prone to rapid evolution and present a constantly shifting landscape and the techniques used to protect those systems need to evolve as well. Perhaps one of the most significant changes in recent years has been the advent of virtualization. In the virtual world, legacy backup systems have become unfit for their purpose, causing backup windows to increase beyond a manageable scope. While this paradigm presents new challenges, new opportunities to improve efficiency, cut costs and reduce risks are also created.

This paper will examine the use of storage snapshots as backups for virtual environments. We will evaluate the relative benefits and limitations while also considering where they fit into a holistic backup strategy when compared to a virtual disk-to-disk backup solution such as Veeam® Backup & Replication™

Background
Pre-virtualization backup strategies were underpinned by operating system (OS) and application-level features. The typical implementation would involve installing a backup agent into an OS and the agent would be responsible for putting applications into a consistent state for backup; copying backup data across the network to a backup server and subsequently monitoring any ongoing changes.

While this worked well in the physical world, virtualization changed everything as operating systems began to share the same physical hardware. Instead of having one backup agent consuming resources from a physical host, there was an agent for each virtual machine (VM) on that host. This meant that ten agents (based on a 10:1 consolidation ratio) or even more could be contending for the host’s CPU, RAM and disk resources. This contention was not only with each other, but also with the applications they were installed to protect. In addition, volumes of data increased to a level where it was no longer feasible to use standard transports to move it across the production network to the backup server. This situation clearly could not continue as virtualization has become the standard practice of datacenters worldwide.

Virtualized Layers

Where virtualization presented new challenges, it also presented new opportunities. The physical world consisted solely of the application/OS layer. The virtual world, Continue reading →

PernixData Unbuttons Trench Coat at SFD3 and Reveals..

Posted on by

Flash Virtualization Platform.

SFD3_FVP

It seems in Storage circles there is much discussion around using new technologies to cache data in faster more accessible media. Flash is everywhere.. but there are many choices for where and how you can deploy flash technology in order to alleviate strain on storage systems, whose current SAS/SATA based disk drives are struggling to keep up with the day to day IOPS requirements of many organisations.

Pernix Data believe they have the solution for VMware environments in the form of their Flash Virtualisation Platform (FVP). They certainly have the credentials to be making such claims with team members coming from many leading companies. This includes, Satyam Vaghani, their CTO who came from VMware and was responsible for the creation of VMware’s VMFS filesystem.

FVP is a hardware agnostic server-side flash virtualization solution.  It virtualizes PCIe flash cards and local SSD at the server side. What I found particularly impressive about this solution is that it is a software only solution, that looks very easy to implement. It sits seamlessly between hypervisor and SAN without requiring any configuration changes to the either the datastores at the hypervisor or LUNs at the SAN. It’s just a extension that is easily implemented on vSphere.

The clustered nature of the product also overcomes some of the current server-side flash device challenges. When flash caching is being used in a server, a footprint of hot, commonly accessed data is built up for the workload running. If a VM (and its associated workload) migrate to another host due to vMotion or some other reason, the footprint needs to be recreated from scratch. FVP resolves this by replicating copies of the footprint data to other host in the cluster, making it easy for a VM to pickup it’s cached footprint if it moves. There are also, obviously data protection benefits to keeping multiple copies of the data in the event a server dies along with it’s cache.

In addition, to easy implementation the product also provides solid easy to read stats on what results it’s achieved. A sure fire way to build a solid business case around IOPS saved and the reduced requirement to scale up your SAN to deal with load.

What these new caching capabilities amount to is an entirely new storage tier between RAM & SAN. This new tier (or layer) will definitely come with challenges. One such challenge would be ensuring consistent copies of data at the SAN for things like Backup processes. If FVP caches the data at the server, some reads/writes never actually reach the SAN. So if your backing up from SAN you need a way to flush the data through in a consistent state. FVP does include a “write-through” mode which should flush changes to disk and stop caching (i.e. write-back mode). In order to achieve consistency there will need to be careful orchestration from VSS (or prefreeze/postthaw scripts on Linux) to FVP to VMware Snapshot and beyond. The product will have a PowerShell interface which could be used to switch between write modes for such an operation, but users should be aware that this is a requirement.

All in all, FVP looks like a great product that hist a lot of my hot buttons.. easy-to-install, easy-to-use, transparent but powerful and solid reporting on results. Although not publicly available as yet, it will be interesting to see the licensing model, TCO and ROI information Pernix provide. If they get that right, they could be very successful.

Here is the Intro from Storage Filed Day 3:

 

Webinar: Disaster Recovery for Virtual Environments, One Simple Solution for Five Common SAN Replication Challenges

Posted on by

This is a replay of webinar, I ran last year.. the associated Whitepaper is linked below:

Whitepaper Available here: http://wp.me/p2ZZG3-fG

A new sister webinar/whitepaper focusing on using SAN snapshots in a holistic data protection strategy to be posted shortly.