OpenStack infrastructure automation with Terraform – Part 2

TL;DR: Second of a two post series looking at automation of an openstack project with Terraform, using the new Terraform OpenStack Provider.

With the Openstack provider for Terraform being close to accepted into the Terraform release, it’s time to unleash it’s power on the Cisco Openstack-based Cloud..

In this post, we will:

  • Write a terraform ‘.TF’ file to describe our desired deployment state including;
    • Neutron networks/subnets
    • Neutron gateways
    • Keypairs and Security Groups
    • Virtual machines and Volumes
    • Virtual IP’s
    • Load balancers (LBaaS).
  • Have terraform deploy, modify and rip down our infrastructure.

If you don’t have the terraform openstack beta provider available, you’ll want to read Part 1 of this series.

Terraform Intro

Terraform “provides a common configuration to launch infrastructure“. From IaaS instances and virtual networks to DNS entries and e-mail configuration.

The idea being that a single Terraform deployment file can leverage multiple providers to describe your entire application infrastructure in one deployment tool; even if your DNS, LB and Compute resources come from three different providers.

Support for different infrastructure types is supported by provider modules, it’s the Openstack provider we’re focused on testing here.

If you’re not sure why you want to use Terraform, you’re probably best getting off here and having a look around Terraform.io first!

Terraform Configuration Files

Terraform configuration files describe your desired infrastructure state, built up of multiple resources, using one or more providers.

Configuration files are a custom, but easy to read format with a .TF extension. (They can also be written in JSON for machine generated content.)

Generally, a configuration file will hold necessary parameters for any providers needed, followed by a number of resources from those providers.

Below is a simple example with one provider (Openstack) and one resource (an SSH public key to be uploaded to our Openstack tenant)

Save the above as  demo1.tf and replace the following placeholders with your own Openstack environment login details.

Now run $terraform plan  in the same directory as your demo1.tf  file. Terraform will tell you what it’s going to do (add/remove/update resources), based on checking the current state of the infrastructure:

Terraform checks, the keypair doesn’t already exist on our openstack provider, so a new resource is going to be created if we apply our infrastructure… good!

Terraform Apply!

Success! At this point you can check Openstack to confirm our new keypair exists in the IaaS:

 

Terraform State

Future deployments of this infrastructure will check the state first, running $terraform plan  again shows no changes, as our single resource already exists in Openstack.

That’s basic terraform deployment covered using the openstack provider.

Adding More Resources

The resource we deployed above was ‘ openstack_compute_keypair_v2 ‘. Resource types are named by the author of a given plugin! not centrally by terraform (which means TF config files are not re-usable between differing provider infrastructures).

Realistically this just means you need to read the doc of the provider(s) you choose to use.

Here are some openstack provider resource types we’ll use for the next demo:

“openstack_compute_keypair_v2″
“openstack_compute_secgroup_v2″
“openstack_networking_network_v2″ 
“openstack_networking_subnet_v2″
“openstack_networking_router_v2″
“openstack_networking_router_interface_v2″
“openstack_compute_floatingip_v2″
“openstack_compute_instance_v2″
“openstack_lb_monitor_v1″
“openstack_lb_pool_v1″
“openstack_lb_vip_v1″

If you are familiar with Openstack, then their purpose should be clear!

The following Terraform configuration will build on our existing configuration to:

  • Upload a keypair
  • Create a security group
    • SSH and HTTPS in, plus all TCP in from other VM’s in same group.
  • Create a new Quantum network and Subnet
  • Create a new Quantum router with an external gateway
  • Assign the network to the router (router interface)
  • Request two floating IP’s into our Openstack project
  • Spin up three instances of CentOS7 based on an existing image in glance
    • With sample metadata provided in our .tf configuration file
    • Assigned to the security group terraform created
    • Using the keypair terraform created
    • Assigned to the network terraform created
      • Assigned static IP’s 100-103
    • The first two instances will be bound to the two floating IP’s
  • Create a Load Balancer Pool, Monitor and VIP.

Before we go ahead and $terraform plan ; $terraform apply  this configuration.. A couple of notes.

Terraform Instance References / Variables

This configuration introduces a lot of resources, each resource may have a set of required and optional fields.

Some of these fields require the UUID/ID of other openstack resources, but as we haven’t created any of the infrastructure yet via  $terraform apply , we can’t be expected to know the UUID of objects that don’t yet exist.

Terraform allows you to reference other resources in the configuration file by their terraform resource name, terraform will then order the creation of resources and dynamically fill in the required information when needed.

For example. In the following resource section, we need the ID of an Openstack Neutron network in order to create a subnet under it. The ID of the network is not known, as it doesn’t yet exist. So instead a reference to our named instance of the the openstack_network_v2 resource,   tf_network  is used and from that resource we want the ID passing to the subnet resource hence the .id  at the end.

Regions

You will notice each resource has a region=""  field. This is a required field in the openstack terraform provider module for every resource (try deleting it, $terraform plan  will error).

If your openstack target is not region aware/enabled, then you must set the region to null in this way.

Environment specific knowledge

Even with dynamic referencing of ID’s explained above, you are still not going to be able to copy, paste, save and $terraform apply , as there are references in the configuration specific to my openstack environment, just like username, password and openstack API URL in demo1, in demo2 you will need to provide the following in your copy of the configuration:

  • Your own keypair public key
  • The ID of your environment’s ‘external gateway’ network for binding your Neutron router too.
  • The pool name(s) to request floating IP’s from.
  • The Name/ID of a glance image to boot the instances from.
  • The Flavour name(s) of your environment’s instances.

I have placed a sanitised version of the configuration file in a gist, with these locations clearly marked by <<USER_INPUT_NEEDED>> to make the above items easier to find/edit.

http://goo.gl/B3x1o4

Creating the Infrastructure 

With your edits to the configuration done:

Terraform Apply! (for the final time in this post!)

Enjoy your new infrastructure!

We can also confirm these items really do exist in openstack:

Destroying Infrastructure

$terraform destroy  will destroy your infrastructure. I find this often needs running twice, as certain objects (subnets, security groups etc) are still in use when terraform tries to delete them.

This could simply be our terraform API calls being quicker than the state update within openstack, there is a bug open with the openstack terraform provider.

First Run:

Second Run: Remaining resources are now removed.

Thats all for now boys and girls!

Enjoy your weekend.

 

 

 

OpenStack infrastructure automation with Terraform – Part 1

Update: The Openstack provider has been merged into terraform. It comes with the terraform default download as of 0.4.0.

Get it HERE: https://terraform.io/downloads.html

Then proceed directly to the second part of this series to get up and running with Terraform on Openstack quickly!

Or.. read more below for the original post.

Continue reading OpenStack infrastructure automation with Terraform – Part 1

UCS vMedia Configuration and Boot Order

Just a quick note on Cisco UCS vMedia.

If you have configured a remote CD/DVD from a remote ISO and UCS manager is showing the image is ‘mounted’ but your server is still stuck in a PXE/Netboot loop…

It may be helpful to know that your regular boot order policy in your service profile doesn’t apply here.

AKA. If you have ‘CD/DVD’ in your Boot Order.
This still wont automatically boot into a vMedia CD/DVD.

UCS System manager boot priority list
UCS System manager boot priority list

 

Solution
You’ll need to F6 from the KVM console on server boot, there you will see an option for booting from CIMC vMedia DVD.

B Series UCS F6 Boot Options
B Series UCS F6 Boot Options

This will get you where you need to be!

Also. For those that don’t know. You can check the status of your vMedia mount under Equipment > Server > Inventory > CIMC.

Scroll down and you’ll see something like below.

UCS System Manager vMedia inventory

 

Matt

ZFS on Linux resilver & scrub performance tuning

Improving Scrub and Resilver performance with ZFS on Linux.

I’ve been a longtime user of ZFS, since the internal Sun beta’s of Solaris Nevada (OpenSolaris).
However, for over a year i’ve been running a single box at home to provide file storage (ZFS) and VM’s and as I work with Linux day to day, chose to do this on CentOS, using the native port of ZFS for linux.

I had a disk die last week on a 2 disk RAID-0 mirror.
Replacement was easy, however reslivering was way to slow!

After hunting for some performance tuning ideas, I came across this excellent post for Solaris/IllumOS ZFS systems and wanted to translate it for Linux ZFS users. http://broken.net/uncategorized/zfs-performance-tuning-for-scrubs-and-resilvers/

The post covers the tunable parameter names and why we are changing them, so I won’t repeat/shamelessly steal. What I will do is show that they can be set under linux just like regular kernel module parameters:

[root@ZFS ~]# ls /etc/modprobe.d/
anaconda.conf blacklist.conf blacklist-kvm.conf dist-alsa.conf dist.conf dist-oss.conf openfwwf.conf zfs.conf

[root@ZFS ~]# cat /etc/modprobe.d/zfs.conf
options zfs zfs_arc_max=2147483648 zfs_top_maxinflight=64 zfs_resilver_min_time_ms=5000 zfs_resilver_delay=0

Here you can see I have set the zfs IO limit to 64 from 32, the resilver time from 5 sec from 3 and the delay to zero. Parameters can be checked after a reboot:

cat /sys/module/zfs/parameters/zfs_resilver_min_time_ms

Result: After a reboot, my resilver speed increased from ~400KB/s to around 6.5MB/s.

I didn’t tweak anymore, it was good enough for me and had other things to get on with.

One day i’ll revisit these to see what other performance I can get out of it. (I’m aware on my box, the RAM limitation is causing me less than ‘blazing fast’ ZFS usage anyway)

Happy Pools!

[root@ZFS ~]# zpool status
pool: F43Protected state: ONLINE
scan: resilvered 134G in 2h21m with 0 errors on Tue Jun 24 01:07:12 2014

pool: F75Volatile
state: ONLINE scan: scrub repaired 0 in 5h41m with 0 errors on Tue Feb 4 03:23:39 2014

 

Matt

CF Push and case insensitive clients

So here’s a weird one that may save someone some time…..

Trying to perform a cf push with a .jar file. Get’s the following strange error!

 

Lots of head scratching;

Looks like (and is) a local error, unpacking the JAR, we haven’t even touched the PaaS/cloudfoundry yet!

Also, unpacking the JAR manually seems to complain also, hmm.

You’re running MacOSX? … (Or maybe Windows?)

Looks like the issue is because the JAR has been built on a CASE-SENSITIVE OS (in our case, Jenkins on Linux)

… and you’re trying to run cf push on a CASE-INSENSITIVE OS (in our case, a 2013 Retina MacBookPro).

Workaround..

Run the same cf push from a linux box and it works fine.

This link put me onto the issue (as the error is more than a bit confusing!);

https://groups.google.com/forum/#!topic/selenium-users/f8OMertwzOY

Hope this helps someone!

Openstack and PaaS; Love a good geek rift!

I’ve been in the bay area, CA for a couple of weeks now (excluding my cheeky jaunt to vegas!) and even though i’m now on vacation, it’s been the perfect place to watch the OpenStack Havana Drama unfold; Mostly stemming from this catalyst;

http://www.mirantis.com/blog/openstack-havanas-stern-warning-open-source-or-die/

Well (for me, anyway) especially this bit;

Too many times we see our customers exploring OpenShift or Cloud Foundry for a while, and then electing instead to use a combination of Heat for orchestration, Trove for the database, LBaaS for elasticity, then glue it all together with scripts and Python code and have a native and supported solution for provisioning their apps.

Hell no! Was my initial reaction, and while there has been a definite retraction from the tone of the whole post… I still think a hell no is where I stand on this.

And i’ll tell you why, but firstly;

  • I like Openstack, as an IaaS. I like it’s modularity for networking and the innovation taking place to provide a rock-solid IaaS layer.
  • It was a much needed alternative to VMWare for a lot of people and it’s growth into stability is something i’ve enjoyed watching (competition is never a bad thing right! 😉 ).

That said, here’s why I’ll take my PaaS served right now, with a sprinkling of CloudFoundry;

  • People tying things together themselves with chunks of internally-written scripts/python (i’d argue even puppet/chef as we strive for more portability across public/private boundaries) is exactly the kind of production environment we want to move away from as an industry;
    • Non-portable.
    • Siloed to that particular company (or more likely, project/team.)
    • Often badly maintained due to knowledge attrition.

.. and into the future;

  • Defined, separated layers with nothing connecting them but an externally facing API was, in my mind, the very POINT of IaaS/PaaS/XaaS and their clear boundaries.
  • These boundaries allow for portability.
    • Between private IaaS providers and the PaaS/SaaS stack.
    • Between public/private cloud-burt style scenarios.
    • For complex HA setups requiring active/active service across multiple, underlying provider technologies.
      • think ‘defence in depth’ for IaaS.
      • This may sound far fetched, but actually is and has already been used to back SLA’s and protect against downtime without requiring different tooling in each location. 
    • I just don’t see how a 1:1 mapping of PaaS and IaaS inside OpenStack is a good thing for people trying to consume the cloud in a flexible and ‘unlocked’ mannor.

It could easily be argued that if we are only talking about private and not public IaaS consumption, i’d have less points to make above; Sure, but I guess it depends on if you really believe the future will be thousands of per-enterprise, siloed, private IaaS/PaaS installations, each with their own specifics.

As an aside, another concern I have with Openstack in general right now is the providers implementing OpenStack. Yes there is an OpenStack API, but it’s amazing how many variations on this there are (maybe i’ll do the maths some day);

  • API versions
  • Custom additions (i’m looking at you, Rackspace!)
  • Full vs. Custom implementation of all/some OpenStack components.

Translate this to the future idea of PaaS and IaaS being offered within OpenStack, and i see conflicting requirements;

From an IaaS I’d want;

  • Easy to move between/consume IaaS providers.
  • Not all IaaS providers necessarily need the same API, but it would be nice if it was one per ‘type’ to make libraries/wrappers/Fog etc easier.

From a PaaS i’d want;

  • Ease of use for Developers
  • Abstracted service integration
    • IaaS / PaaS providers may not be my best option for certain data storage
    • I don’t want to be constrained to the development speed of a monolithic (P+I)aaS stack to test out new Key-Value-Store-X
  • Above all, PORTABILITY

This seems directly against the above for IaaS…

Ie, I don’t mind having to abstract my PaaS installation/management from multiple IaaS API’s so that I can support multiple clouds,(Especially if my PaaS management/installation system can handle that for me!); however i DON’T want lots of potential differences in the presentation in my PaaS API causing issues for the ‘ease of use, just care about your code’ aspect for developers.

I’m not sure where this post stopped becoming a nice short piece and more of a ramble, but i’ll take this as a good place to stop. PaaS vendors are not going anywhere imho and marketing-through-bold-statements on the web is very much still alive 😉

Matt

 

 

 

 

Pivotal Cloud Foundry Conference and Multi-Site PaaS

So I recently got back from Pivotal’s first Cloud Foundry conference;

pivotalcfconf

 

as I’m not a developer, I guess by the power of deduction I’ll settle for cloud leader.

While there, this newly appointed cloud leader, erm, lead a pretty popular discussion on multi-site PaaS (with a focus on Cloud Foundry, but a lot of the ideas translate) with the intention of stirring up enough technical and business intrigue to move the conversation into something of substance and action.

I’m about to kick off further discussions on the VCAP-DEV mailing list (https://groups.google.com/a/cloudfoundry.org/forum/#!forum/vcap-dev), but draft run here can’t hurt

  • Having multiple separate PaaS instances is cheating (and a pain for everyone involved)
  • My definition of multi-site PaaS is currently to support application deployment and horizontal scalability in two or more datacentres concurrently, providing resilience scalability beyond what is currently available.
  • The multi-site PaaS should have a single point of end user/customer interaction.
  • Key advantages should be simplified management, visibility and control for end customers.
  • Multi-Site PaaS should be architected in such a way as to prevent a SPOF or inter-DC dependencies

All well and good saying WHAT i want, how about how?

  • A good starting point would be something that sits above the current cloud foundry components (ie, something new that all api calls hit first) that could perform extensions to existing functions;
    • Extend API functionality for multi-site operations
      • cf scale –-sites 3 <app>
      • cf apps –-site <name>
      • etc
    • Build up a map of PaaS-wide routes/DNS A records to aid in steering requests to the correct sites ‘Gorouters’
    • This may also be a good place to implement SSL with support for per-application SSL certificates as these will need to be available in any site the application is spanned to also.
  • Further thoughts that i need to pin down
    • I’d like to see this getting health state from each DC and redirecting new requests if necessary to best utilize capacity (taking into account latency to the client of the request)

Where this layer will sit physically, and how it will become this unification layer without becoming the SPOF point itself is still playing around my mind.
Current thoughts include;

  • Calling the layer Grouter (global-router) for convenience.
  • This layer will be made up of multiple instances of GRouters
  • Passive listening to NATS messages WITHIN the G-Router instances site for all DNS names/apps used at that site.
  • Distributed protocol for then sharing these names to all other Grouters (NOT A SHARED DB/SPOF).
    • No power to overwrite another G-Routers DB
    • Maybe something can be taken from the world of IP routing where local routes outrank that of remote updates, but can be used if need be.
    • Loss of DB/updates causes Grouter instance to fail open and forward all requests to local GoRouters (failure mode still allows local site independent operation).
    • DNS should only be steering real PaaS app requests or API calls to the GRouter layer although may be for different DC, Depending on use of Anycast DNS *needs idea development*
    • Grouters In normal mode can send redirects for requests for Apps that are In other DC’s to allow a global any-cast DNS infrastructure
  • Multiple instances per DC just like GoRouters so they scale.

Other points which will need discussion are;

  • How does a global ‘cf apps’ work
  • Droplets will need saving to multi-DC blob stores for re-use in scale/outage scenarios
  • We will need a way to cope with applications that have to stay within a specific geographic region.
  • Blue/Green zero downtime deployments?
    • One site at a time
    • Two app instances within each DC and re-mapping routes
    • Second option would prevent GRouter layer needing to be involved in orchestration, reducing complexity.

My next steps are to map out use cases and matching data flows embodying my ideas above.

Just FYI, all this was written on a SJL>LHR flight were awake-ness was clinging to me like the plague, so expect revisions!

Matt

 

Google Nexus 4 Smartphone

I was going to write about the new Nexus 4 i’ve finally managed to get my hands on, and why, after many years of my mobile phones all having names beginning with ‘i’, I’m actually finding this new android device hard to fault…

But this guy pretty much 100% summaries my thoughts for me, right down to why previous attempts for me running android have failed.. and therefore saves me the trouble! Worth a read, whichever side of the fence you are on!

http://gizmodo.com/5973073/an-iphone-lovers-confession-i-switched-to-the-nexus-4-completely

 

It used to just be paperwork that took time to work out!

So the delivery of a new corporate laptop this week got me thinking;

It’s much more powerful, portable and generally nicer than anything I own and the fact of the matter is; if I’m near a computer for five out of the seven days every week, it’s going to be this one.

This ties into my last post about BYOD and staff ‘bringing their own data’. This works backwards to corporate-supplied devices too;

BYOD is currently, lets be honest, only really bringing your own ‘addon’ devices, and not your full digital working requirement.

BYODv2 in my mind will be where users fully bring every bit of local compute power they need to work, maybe minus the clutter/heavyness of standard peripherals such as screens and keyboards, instead have a uniform, wireless dock interface.

So for as long as there are corporate laptops, which their users are more likely to have on them than any personal laptop for a massive percentage of their weekly lives, there will always be remnants of their own personal data from lunchtimes, weekends, corporate travel, after works etc.

I digress, back to MY new shiny corporate laptop;

  • I needed a kick to get all my data, which is currently spread everywhere into some semblance of order and peace of mind over backups etc.
  • I hate always being on the wrong device for what I’m looking for (ooh, I took that photo on my iPhone, not my laptop/tablet/Nexus 4/Tomorrows device number seventy six and a half).
  • I don’t like having to swap between personal/corp equipment, especially now the corp laptop is actually much more powerful than my own!

Turns out, this isn’t something I should have thought about, as while the cloud solves some of these data-everywhere questions, it also opens up a lot more choice, ie a lot more questions on howto go about all this, it’s amazing just how many branches you end up with in your mind when you try and tackle that which, on the face of it, is a simple ‘get to my data on my devices’ question.

Even without going into all the corporate access stuff, my personal digital life and (more to the point) access to it, created this monstrosity, can you do any worse? Have you had the same thoughts and come up with a solution that works well? Do you want to talk to me about cool ideas for BYODv2 above? Comment or e-mail.

-Matt

Mind Map of how to get to my data on all my devices
Oh dear, it’s 3AM again

With ‘Bring your Own Device’ will come ‘Bring your Own Data’

A friend put me on to http://owncloud.org/, which despite it’s awful naming (oooh cloud! That’s new and good isnt it? snore!) is an open source implementation of some of Apples iCloud featureset (from what I can see) which can be hosted anywhere you want.

At first I was about to hit download, as a techy, running my own things in my own VM’s or somewhere I control the data security, backups etc, makes me feel a bit better; but then I realized something which I think a lot of people will realize, techy or not;

I’m already using my own Phone/Tablet for both work and for personal. My corporate dropbox/exchange/data stores will be much better backed up (one would hope) than anything I’m going to run locally and probably better than any very cheap consumer-level ‘cloud’ data stores; and even if not it doesn’t cost me anything.

… I’ll just backup/store my contacts/calendar/photo’s on there.

If you’re encouraging people to Bring their Own Devices to into your corporate infrastructure, don’t be surprised if they bring some personal data too. Expect ‘Personal’ or ‘Private’ or ‘NonWork’ directories to start appearing in users dropboxes, expect groups of contacts named ‘personal’ so they can filter them on their phone when they are not working. Have you capacity planned for this when thinking about BYOD?

For the record, I don’t think it’s such a bad thing and I think taking a hardline stance against this will slow down BYOD adoption; the company is making savings on client endpoints, the users therefore must feel it is ‘worth’ tying their own devices into a possibly restrictive corporate policy.

Just my 10p
Keep the change!