Real World Application of GRE and VXLAN Encapsulation between Mixed Hypervisors on Disjoint Networks PART-1.

Happy Thanksgiving! I thought about putting this all into one post. I quickly realized it wouldn’t be possible as there is just to much!

Contrary to what this first post’s format this will be a technical how to. This is just the introduction to what we are going to be doing. The following parts will be posted in the next several days.

 

For some time, I have been heavily experimenting with virtual networking between different hypervisors, bare metal workloads and virtual to physical gateways or “vtep”s for virtual tunnel endpoints.  It quickly dawned on me that there exists a split in what we as the consumers of these different data center technologies want and what providers of these technologies are willing to give us.

For example, in my lab, and, I’ll go out on a limb here and say that in most other labs and production environments as well, there are a variety of hypervisors such as VMware esxi, Citrix Xenserver, KVM  and bare metal workloads as well as orchestration platforms such as Openstack. What does not exist is a way to easily create a virtual layer 2 network between all of these objects. Yes, sure there are vlans but who wants to configure 30 switches every time you or one of your tenants wants to create a new network?  Providers of these technologies have no desire to allow you to easily inter-operate their product with that of another manufacturer and for good reason. It makes no sense for one tech provider to spend their time and money into developing their product to play nice with the competitors product (this is a whole other topic which is out of the scope of this discussion).  My search for such a solution came up empty except for few very expensive solutions that are not even in the same universe as something that would ever be considered for anyone except the largest corporations. So I started to build my own.

This guide is a work in progress. It is here to document my steps as they happen and provide a reference for me to look back on. I thought that I would make this public so that it can also help others who are trying to do something similar.

One thing that has really impressed since I first jumped into *nux several years ago was the whole idea of open source and the way it enforces community and fosters people helping other people.  I haven’t had much chance to give anything back even though I want to, mostly because I was just beginning and still consider myself a novice.  What follows are the steps I took and that worked for me, a very opinionated beginner who does not claim that anything below is the correct or most secure way to do things.

The Goal. I want to be able to easily create and delete virtual networks between virtual machines running on VMWare esxi 5.5, Citrix Xenserver and instances running on KVM in a Openstack based cloud. As well I want to create a software gateway that will allow me to connect a physical server otherwise known as a bare metal workload to any of these virtual networks.

 

Encapsulation is very intriguing. Explaining it to someone who is not technical is not very easy so we will start there.

Lets say you have a business that is setup to use only Fedex for all of its deliveries, your entire system is setup to interface with Fedex only, your ordering system, your receiving system your dock all speak the “Fedex” language only. The system has been working great and you get all your packages sent and received, thousands of them every week like a well oiled machine. Now your business is doing so well you decide to open up a new branch in a remote location. Your remote location is setup to receive Fedex packages like your main location, and you expect the new location to send many packages both to and from your main office plus to and from all of your clients that are near the new branch. However when you call Fedex to setup the new service at the branch location they tell you that that your new branch is outside of their service area and they can not help you! Ouch, your entire infrastructure only speaks Fedex! United Parcel Service delivers there but your system at the new branch doesn’t know the UPS language and even if it did, that would not help because your main office, everything else is Fedex based.

So being the clever business man you are, you leverage Encapsulation! With minimal modification you get your packages ready to be shipped by Fedex once your system processes all the orders for Fedex and the boxes are on the Fedex truck, your modified system calls UPS and says “Hey, UPS! I have a package for you! Your package is this entire Fedex Truck! Please take this Fedex truck and ship it to my new branch. Once you get to the new branch open the doors to your UPS truck, unload the Fedex truck and leave it at the door. My system will know what to do with the Fedex truck from there on!

After failing to have you committed,  your board of directors and mainly your CFO will try to reason with you and say things like, do you have any idea how much overhead this process will incur? The gas alone to carry an entire Fedex truck inside a UPS truck is enough to bankrupt us!

This is when you calm him down and explain to him that no, actually the entire process of encapsulating a Fedex truck inside a UPS truck will be taken care of in software… Wait what?

Ok, so there is some slow down and some overhead to encapsulating, it is not anything close to what it would cost to carry a physical delivery vessel inside of another yet larger physical delivery vessel however. Don’t worry.

To apply this to ip packets… You have your home network, it starts at the cable modem and your single public ip address get’s NAT’d by your rinky dinky home router into your local subnet which is 192.168.1.0/24 <– this means you have 254 ip addresses on your local network. The first one is 192.168.1.1 and the last one is 192.168.1.254. So your laptop could be 192.168.1.10, your printer is 192.168.1.11 your ipad is 192.168.1.12 and so on. On your home network you can easily share pictures and music from your ipad to your laptop because they are on the same layer 2 domain, in other words they are both directly on the 192.168.1.0/24 subnet so when you try to print a picture from your ipad who’s ip address is 192.168.1.12 it has no problem talking to your printer who is in the same state, county, zipcode, area code, etc.. in other words your ipads MAC address has direct communication to your printers MAC address which is a layer 2 construct and is necessary for proper communication. so now lets say you go to your summer home 100 miles away and your ipad comes with you. In your summer home your internet connection is very similar to your home connection there is a cable modem and a rinky dinky router that NAT’s all the traffic to your local subnet that happens to be 192.168.2.0/24 but can just aswell be 192.168.1.0/24 (I’m changing it slightly so as not to consfuse the two places) now when you log in from you ipad it gets a ip address (WiFi) from your router of 192.168.2.14 and you want to print a picture to your printer that is back in your main home on a seperate dijointed network. well if you had a vpn connection setup between your two homes then that vpn connection would utilize packet encapsulation in order to allow you to print accross the internet. What would happen is the following series of events to the packet flowing from your ipad to the printer. The packet would be addressed to the MAC address of the printer 100 miles away, your VPN connection would realize that this mac address is not at your current location and it will know the real path that the ip packed has to take to get from your ipad to the printer so it will modify the packet from your ipad by encapsulating it with a new ip header, so now your ip packet has a inner header and a outer header the inner header has a to and from address label that only makes sense to your local network back home. However the new outside header masks the inside header and contains the real to and from address to get that packet out to the public internet across countless public routers and eventually to the single public address of your main home 100 miles away. In your home 100 miles away the other end of your VPN system receives the encapsulated packet and strips the outer header that is no longer needed. What is left is the inner header (the Fedex Truck!) now that is the original ip packet that your ipad sent out its’ antenna and it has made it to your home network 100 miles away and is on the local subnet that the original to and from address makes sense to. So now your vpn sends that packet to you main homes rinky dinky router and that router has no issue realizing that that  mac address contained inside the to label is for the printer and it sends it on its way or forwards it and your printer prints!