Secure Mesh VPN Network Interconnect for LXC containers in Multiple IaaS Clouds

5 March, 2021 by

Administrator

| No comments yet

https://peervpn.net/

Problem Statement

On any of the existing IaaS Cloud providers like AWS, Digital Ocean etc you can easily create virtual machine “instances” of running Linux servers.

Note that some IaaS clouds (Azure, AWS as examples) let you also create Windows virtual machines also. But again tthis is about Linux & LXC not Windows.

Although you can create and run a Linux server in those Clouds you cannot “nest” other linux servers inside of those Cloud server “instances” … by “nesting” I am referring to using KVM or VirtualBox etc inside of an AWS Ubuntu server instance to create other virtual machines (vm inside a vm). There may be some IaaS providers that permit nested VMs but I am not aware of any. AWS for instance does not allow this.

The reason is that those clouds do not permit nested hardware virtualization.

NOTE: On your home linux pc/server you can nest KVM hw virtualized instances.

LXC containers are:

much more light-weight than using full HW virtualization like KVM, vmware, VirtualBox etc. This means LXC is faster and use less “host” server resources (memory, cpu etc). Canonical (ubuntu, lxc, juju etc) just published some performance test results of LXD (LXD utilizes LXC !) versus KVM instances. LXD/LXC in terms of both scalability & performance far surpasses KVM. On a server where you may be limited to running ~20 HW virtualized VMs you may be able to run 80-100+ LXC containers.
- LXC containers can be “nested” within a HW virtualized Linux on AWS, Digital Ocean etc
the LXC containers also all share the same kernel as the host machine so they are able to take advantage of the “host” security, networking, file system management etc.
extremely fast to start up & shut-down… almost instantaneous.
flexible because you can use say an Ubuntu host and have LXC containers that are other linux distro’s such as Debian or Centos.

A benefit of LXC is that you can use it to create create full container based servers in IaaS Clouds like AWS, Digital Ocean etc and you can also “nest” LXC containers (containers inside a container) on those cloud “instances”.

LXC has some characteristics which are “default”. These can be modified/changed but I will not be going to cover that in this document.

LXC containers are by default created/started/running behind a NAT’d bridge interface called “lxcbr0” which is created when you install LXC on a server.

lxcbr0 is by default given a 10.0.3.x network/subnet

NOTE: you can change this if you want/need to

Each LXC container you create on the “host” will be assigned an IP address in that 10.0.3.x subnet (examples: 10.0.3.123, 10.0.3.18 etc).

NOTE: Your Cloud “instance” (re VM) will be assigned an IP address at the time you create the cloud “instance” by whoever the Cloud IaaS provider is. Actually there are usually 2 ip addresses assigned, one private to the cloud and one “public” so the Cloud instance can be reached from the Internet.

The LXC containers you create & run on any Cloud instance (the “instance” will from now on be referred to as the LXC “host”) can by default reach anything on the Internet which the “host” can reach. Again, that is configurable.

By default, all LXC containers running on any one “host” can also reach each other.

But what if you wanted LXC containers running on a host on say AWS to interact with LXC containers running on a host on Digital Ocean’s cloud? No you can’t… not without some network configuration magic the LXC containers running on one host cannot talk to containers running on another host because all will be running behind their own hosts lxcbr0 NAT’d interface.

Also, LXC containers running on AWS cannot reach LXC containers running on another host also on AWS (ditto for other Clouds).

So the problem becomes… what if you wanted to do this though?

What if you wanted your LXC containers on a host somewhere (cloud or elsewhere) to be able to reach & interact with LXC containers running on any other host anywhere (assuming firewalls etc don’t prevent it).

Also, how could you make this secure so not just anyone could do this?

A Solution Approach I Utilized

Virtual Private Networks (VPNs) are commonly used in the normal networking world to securely interconnect remote sites & servers. Think of a VPN as a “tunnel”.

VPNs encrypt the data links utilized for this interconnect to keep the VPN and any data traversing it “private”. So a VPN is an encrypted “tunnel”.

Most common VPN are peer-to-peer (P2P). P2P VPN usually require configuration of each server that you want to connect to. If you have 100 servers or sites then that means configuring each individual site for 99 different connections (1 for each “peer” site/server).

That solution if used beyond a few servers can be both complicated & messy to maintain.

The solution to this is to use what is called a Mesh VPN. A “mesh” VPN means that every host configured as part of the VPN can connect to every other host connected to that VPN without necessarily being configured specifically to do so.

In Open Source there are quite a few Mesh VPN choices and some offer more or less features and are more or less complicated to setup. Some mesh vpn solutions are more complicated to configure than others.

Some Mesh VPN utilize a concept of a “super-node” which is used to keep a stateful database of all “member” servers/hosts that are part of the VPN.

Other Mesh VPN have been designed so as to not require a “super-node” at all! This reduces overhead traffic to/from the “super-node” and any subsequent delays that traffic can cause.

In this blog post I am describing how to utilize one such Open Source Mesh VPN named PeerVPN (http://www.peervpn.net/) which is the work of Tobias Volk.

Key PeerVPN Features include:

Ethernet tunneling support using TAP devices.
IPv6 support.
Full mesh network topology.
Automatically builds tunnels through firewalls and NATs without any further setup (for example, port forwarding).
Shared key encryption and authentication support.
Open Source (GPLv3)

PeerVPN uses UDP exclusively and PeerVPN sends UDP packets that are larger than the MTU.
Tobias Volk, the author of PeerVPN, has indicated that PeerVPN fragments/reassembles packets itself to enable this MTU capability.

PeerVPN is both simple to setup and it does create a full mesh VPN and it does not require a “super-node”.

You can define multiple separate VPN’s on each host(s)! To define additional VPN networks just create additional copies of your peervpn.conf using a new unique name for each.

edit each new configuration file (call the new config file anything you want)
change the networkname variable to be a unique name of the additional VPN
change the port variable to be unique for each new VPN
generate a different PSK encryption/authentication key for each additonal VPN and add that PSK key to the appropriate VPN’s .conf file after the PSK variable in that file.

NOTE: All servers that you want to be part of the same VPN must use the same config file values (exceptions: “interface” & “ifconfig4/ifconfig6” values)

The minimum PeerVPN configuration file requires only 9-11 items to be configured depending on whether you are using IPv4 and IPv6 or not:

port 7000                                   # your specified Port number to be used by any individual/distinct PeerVPN (other VPN’s require a different Port #)
networkname VPNnet1      # your name for each unique VPN network deployed (other VPN’s require a different “networkname”)
psk MyCryptoSeedPassword                  # PSK is an encryption/authentication “password” upto 512 characters (other VPN PSKs should be unique). For PeerVPN, the PSK that you enter in the config is just a seed password used to generate the “real” crypto keys. The “real” crypto keys are always AES keys with 256 bit length, which are generated individually for each VPN link.
enabletunneling <yes|no>                    # Default is YES. Enables tunneling interface (refer to config documentation link below)
enableipv4 <yes|no>              # Default is YES
enableipv6 <yes|no>                              # Default is YES
interface peervpn0                                 # name you want to give the local VPN Tunnel End Point (TEP) on a “host” (name it whatever you like)
ifconfig4 10.8.0.1/24                             # IP address of “this” hosts TEP. Next Host TEP may be 10.8.0.2/24 etc.
ifconfig6 <configure>                              # this is the node’s IPv6 address that should be assigned to the tunneled interface (i.e the encrypted tunnel).
initpeers 10.8.0.2 7000 10.8.0.3 7000 # For HA you should ID at least several Peer Node Public IPv4 addresses that “this” node should try to initially connect or reconnect to if connection is lost
initpeers 2001:DB8:1337::1 7000         # For HA you should ID at least several Peer Node Public IPv6 addresses that “this” node should try to initially connect or reconnect to if connection is lost
enablendpcache <yes|no>                    # Default is NO. If using IPv6 set to YES. Enables caching of tunneled IPv6 NDP messages to improve performance by reducing the amount of NDP multicast messages that need to be sent between peers

For a basic PeerVPN configuration file that’s it! Pretty simple to implement I think compared to other mesh VPN solutions I have seen!

To start peervpn use the following command:

usage: ./peervpn <path to peervpn config file>

IMPORTANT NOTE: For complete PeerVPN configuration options and descriptions see: https://github.com/peervpn/peervpn/blob/master/peervpn.conf

The 10,000 ft view of the overall process to setup and use PeerVPN are:

Create ubuntu server instance (re host) on each cloud
On each Cloud Instance/host open port 7000 which is used by PeerVPN.
Install peervpn on each cloud instance. Copy the .zip file and unzip it in a subdirectory of your choosing.
Create a peervpn.conf configuration file. Refer to: http://www.peervpn.net/tutorial/
Generate a PSK encryption password “seed”. I used “psktool”… and set the “psk” variable in your peervpn.conf file to that key
- note:: use the same PSK on all VPN “member” hosts within the same VPN
Follow the instructions on the above PeerVPN link in regards to adding more server/hosts to the VPN. You can add as many as you can support from a traffic perspective.

Start a new instance of the peervpn and point to the additional .conf configuration file

examples:

./peervpn ./vpn-network-A.conf
./peervpn ./vpn-network-B.conf
etc

If you do this, each VPN will be separate & isolated from every other VPN not of the same “networkname”.

How to Install & Use PSKTOOL to generate your PSK encryption password

An important part of any VPN is the encryption of the data traversing the VPN tunnel. This is especially true for data crossing the Internet. To insure the security of the data you send through your VPN tunnel PeerVPN’s configuration file (peervpn.conf) allows you to specify a PSK encryption password. The PSK you enter into the peervpn.conf file is used as a “seed” to generate the actual 256bit AES keys used to encrypt the VPN link.

Pre-Shared Keys (PSK) can be used to provide both authentication & encryption. Pre-Shared Keys is the most common authentication method used today.

I used psktool for my experiment and it is included in the gnutls package(s).

On Ubuntu the following will install what is required for you to use psktool:

$ sudo apt-fast install gnutls-bin gnutls26-doc guile-gnutls -y

Usage : psktool [options]
-u, – -username username
specify username (username not important for our use-case here with peervpn but tool requires one)
-p, – -passwd FILE specify a password file.
-s, – -keysize SIZE specify the key size in bytes! NOTE: the max keysize is 64 bytes (ie 512 bits)
-v, – -version prints the program’s version number
-h, – -help shows this help text

then… to generate a 512 bit PSK for “any” username and save it to some file (example = ./mypsk):

example: $ psktool -u bmullan -p ./mypsk -s 64

Edit the mypsk file and copy everything after the name you used (the name will be the only readable text in that file) and add that copied PSK password key into your peervpn.conf file after the variable “PSK”

How to use the PeerVPN “mesh” VPN with LXC

The big question is… how does this help interconnect LXC containers running on possibly many remote and independent server/hosts

All it takes is a small networking change…

If on each host you configured & started PeerVPN properly, then on each host if you execute “ifconfig” you will see one or more VPN Tunnel End Point (TEP) “interfaces” created by PeerVPN.

NOTE: The TEP will be named the same “name” as you entered in the PeerVPN configuration for the variable called “interface” … refer to the above PeerVPN tutorial.

To connect LXC containers running on any PeerVPN configured host you attach the “peervpn0” interface to the lxcbr0 bridge that lxc uses on that host.

NOTE: Depending on your peervpn.conf file configuration you are the one that defines the PeerVPN TEP interface IP address. In the PeerVPN Tutorial example the peervpn0 interface is given a 10.8.x.x address

When you installed LXC on a host (sudo apt-get install lxc) a default LXC bridge will be created and given a 10.0.3.x IP address. Also, any lxc containers created using the lxc-create command on that host will by default get a 10.0.3.x IP address.

While logged into each of your servers you should now be able to ping the 10.8.0.x IP address of the other PeerVPN member servers.

Our next step is to connect our TEP to the LXCBR0 bridge to enable containers attached to that bridge to pass data over the VPN tunnel.

Since the PeerVPN TEP interface (“peervpn0” in the Tutorial example) is just like any other Linux ethernet interface we can use the “ip link” command to connect the peervpn0 interface to the LXC lxcbr0 bridge.

$ sudo ip link set dev peervpn0 master lxcbr0

NOTE: After executing this command on EACH Host… you will find that you can no longer PING the 10.8.0.x IP addresses of the other PeerVPN member servers!

This is expected and is OK because if you still have the terminal up where you executed the command to start the PeerVPN (ie sudo peervpn … you should still see your “peers connected” !

Next create an LXC container on each “host”

example:

$ sudo lxc-create -t download -n my-container — -d ubuntu -r trusty -a amd64

Note: this will create a new LXC container named “my-container” using Ubuntu and the Trusty release (re v14.04) and also make it a 64Bit OS in that container.

Next… start the container your created on each host and then get access into the LXC container “my-container”

$ sudo lxc-start -n my-container -d

$ sudo lxc-attach -n my-container

If you look closely at the Terminal window you are using you will see that the “prompt” has now changed to show that you are logged into the container “my-container” and that you are logged in as root.

Note: root in a container is NOT the same as root in the “host”

On each host get the IP address of each host’s container that you created and write it down.

You can get those IP addresses using the following LXC command on both Host A and Host B

$ sudo lxc-ls -f

Or if you are logged into the Container on each host just do:

$ ifconfig

NOTE: (your container IP addresses will be different but for our example here let’s say

eth0 of Host A’s container has IP address 10.0.3.136
eth0 of Host B’s container has IP address 10.0.3.15

While logged into the Container on Host A, try to ping the Container IP address on Host B

Using our example IP addresses from above (again your own Container IP addresses will be different:

$ ping 10.0.3.15

This should now work and Containers on Host A can reach Containers on Host B via the PeerVPN Tunnel you created.

Important Note

For our proof-of-concept trial here you need to understand that we have left LXC on each host node to utilize the default LXC configuration. So each host will have its own LXC lxcbr0 bridge … and … the lxcbr0 bridge on each host will all have the same 10.0.3.x subnet & and ip address from that 10.0.3.x subnet defined for lxcbr0.

Furthermore, the LXC containers created and running on the individual “hosts” will all also have been assigned a 10.0.3.x ip address by the local lxcbr0 dnsmasq.

Even though LXC “by default” creates & assigns “unique” IP addresses to each LXC container created inside a particular “host”… LXC running on separate “hosts” is NOT by default aware of IP addresses used by LXC on any other host.

For our “proof-of-concept” here, that means there is the potential for a “duplicate” 10.0.3.x IP address to be assigned to a container on one or more “hosts”.

For a small proof-of-concept this is probably unlikely to occur and so for this blog write-up we will ignore that fact. But for a production environment you will want to look into using a centralized IPAM (ip address management) solution which will probably involve other linux tools such as DNSMASQ, DHCP, DNS. However, that is beyond the purpose of this proof-of-concept article/blog post.

Finally Step — repeat this process for each cloud instance/host if you’d like to test beyond just a couple servers. However, remember there is a “remote” possibility of some LXC container getting a duplicate IP address in your own proof-of-concept trial. it is remote but it is possible.

NOTE: you can configure LXC on each host to use a different bridge you create (say br0) and then on 1 host create & add a DNSMASQ and attach it to the br0 bridge. After doing so, all LXC containers on any host anywhere that is part of the same PeerVPN tunnel will get their IP assigned by a single dnsmasq and you will not have to worry about IP duplication.

Now each lxc container on each cloud instance should be able to ping the 10.8.x.x address of any other lxc container on any other PeerVPN host you have setup anywhere as well as ping any other LXC container on any of those Hosts.

Also, for any Production use it might be advantageous to utilize Non-Privileged LXC containers. All of this blog post has just talked about “privileged” LXC containers.

Use & Implementation of IPv6 as a Production Solution

The introduction & increasing use of IPv6 instead of IPv4 will greatly simplify this overall PeerVPN solution in regards to IPAM because IPv6 was designed to allow local IPv6 address assignments which are guaranteed to be unique even between separate and remote host compute nodes/containers. Google “ipv6” and read up to become more familiar with it as the “internet of things” or IOT as its popularly called will require the vast number of available IPV6 addresses in order to connect the future world’s billions of inter-connected internet devices (phones, tv’s, cars, tablets, laptops etc).

ARIN announced in June 2015 that it has exhausted ALL IPv4 addresses !

So no more new IPv4 is available. For this reason, its important to start learning, testing, deploying IPv6 where you can. In the U.S. almost all ISP’s (Cable, ATT, Mobile etc) now support IPv6 !

NOTE: The main advantage of IPv6 over IPv4 is its larger address space. The length of an IPv6 address is 128 bits, compared with 32 bits in IPv4. The address space therefore has 2¹²⁸ or approximately 7038340000000000000♠3.4×10³⁸ addresses

General IPv6 Configuration for LXC

Searching the web I found a good write-up describing the configuration of IPv6 for LXC container use.

Although this article does not address anything about VPNs I think it provides a great background to understand the critical steps & considerations to configure IPv6 for LXC and the LXC Host machine.

Refer to: LXC Host featuring IPv6 connectivity

Unique Local IPv6 Generator

There is a great online tool to help you generate a unique “local” IPv6 address to utilize with your mesh network or simply to use IPv6 with LXC or Linux configurations. See: http://unique-local-ipv6.com/