My thoughts on the Intel NUC5i3RYK as a vSphere host

IMG_1529 copyRecently I have been expanding my lab, which included two vSphere hosts.  At the time, my hosts which each had 16GB, were substantial lab nodes and their Nehalem architecture was on top.  Now of days they are modest at best.  I expanded them out each with 32GB (previously they had 16GB), but still needed a few more cores and memory in order to deploy Pivotal Cloud Foundry.  My thought was to add a host that would serve to run vCenter Server Appliance, as well as be able to offer a bit more cores and memory to the available resource pool.

I quickly started looking at the Intel NUC line of computers.  After looking at the best value for my application, I settled on the NUC5i3RYK which has the following attractive qualities:

I quickly added the following components to my base NUC5i3RYK

16GB is sufficient for a vSphere environment.  My personal rule of thumb is 8GB per core, which works out nicely with the 2C provided by the NUC.  I would of course much prefer a 32GB option, and of course 4C would be eve better.  The Intel i3-5010 is said to support 16GB, but Intelligent Memory has figured out a way of stacking chips to allow it to see 32GB.  This however is not a cheap solution at about $700 for 32GB!

The transcend 128GB M.2 SSD is very nice.  Its amazing what you can fit inside of a little NUC!  Why have an SSD in a vSphere lab environment?  Personally I like to use it for the host scratch space, Host Cache files, and VM swap files.  This alone is worth it to add a relatively cheap SSD into your vSphere hosts. Speeds things up very nicely and cuts down on the amount of iSCSI traffic.  I also chose to park my vCenter Server Appliance on the SSD as well……why not, there is plenty of room.

The Kingston Data Traveler Micro is an awesome device that looks like it was built just for the NUC!  Its only USB 2.0 but that is sufficient.  ESXi fits nicely into 8GB.

Does it run vCenter Server Appliance well?

Yes!  It runs VCSA just great.  Its fast, responsive and all is well.  I think these are perfect hosts to add for this function.  Many people have a few ESXi hosts, like myself.  But where do you put vCenter?  Of course you can run it on both and link them, or you can vMotion it around, but I prefer to just have a stand alone host.  I also prefer to not run Windows, so rather than installing Windows on the NUC’s bare metal and then installing vCenter on it, I chose to install ESXi 5.5 and then just install VCSA on top of that.

What could be improved?

The NUC5i3RYK does not have vPro technology in it which allows for VNC based KVM remote control.  There are NUCs that have this, and I am sure more to come, such as the NUC Kit DC53427HYE.  But of course these are more expensive.  I just really like to have some sort of remote KVM in my vSphere hosts.

The NUCs are limited to 16GB and 2C.  There are currently no NUCs that support more than 16GB (without something ridiculously expensive like the aforementioned Intelligent Memory solution), nor more than 2C.  This is sufficient for the price point, but its nice to have quad core and 32 GB, so you just get extra density.   At the price however (< $300 for a NUC5i3RYK), its hard to be too critical of this point.

The NUCs have only a single 10/100/1000 port, and no way to expand beyond this.  This is what really relegates the NUC to tasks like vCenter vs. running more of the serious VM’s I need in my lab.  If I can’t have a dedicated port for the storage network, the speeds wills suffer.  Of course, I just make a trunk and the NUC is able to communicate on its single physical port using multiple vKernel’s for management and storage.  I like to have multiple physical ports to run iSCSI on however, simply because I find storage to be the bottleneck in most labs and real networks.  I like to be able to run 2x1GB for my iSCSI back to my Synology DS-1010.  Obviously the NUC is limited to at best communicating at 1GB.

How to run vSphere on the NUC5i3RYK?

Everything you need to know was already written very well by Florian Grehl at Virten.net.  You can find his article on How to Install ESXi on 5th Gen Intel NUC.

Final Thoughts

I will buy a NUC again. In fact, I hope to see the product evolve to at least 2 NIC ports, vPro on more models, more memory.  If that happens I may replace my entire lab with these cool little devices.  They run extremely efficient, take up hardly any room and they are dead quiet.  With a cheap investment I was able to expand my lab and not have vCenter take up resources on my key ESXi hosts.  I did reduce the memory of VCSA to just 4GB, which is adequate for a small deployment, this leaves at least 10GB free for vSphere to consume.

 

 

Posted in Intel | Tagged , , | Leave a comment

Properly shutting down and firing back up Pivotal Cloud Foundry

featured-pcfBecause my Pivotal Cloud Foundry lab runs on my modest vSphere environment, every now an then I need to power something down, and I don’t always have the resources to vMotion everything to the remaining servers.  So I needed a method to be able to power down the PCF lab and bring it back up.  Pivotal tech support responded with these recommended instructions:

Normally, it is not necessary to power off PCF because it is providing continuous cloud service. However, if you have to power off the system or some VMs, here are the tested steps.

1) follow the backup procedures outlined at [1] to backup databases and NFS servers, this is for disaster recovery purpose.
2) confirm vSphere HA won’t automatically try to bring back VMs you shut down.
3) shutdown Ops Manager Director VM first, because it detects offline VMs and recreates them automatically.
4) shutdown Cloud Controller VM the next, this action will terminate any futher Cloud Foundry operation.
5) shutdown other VMs.
6) those VMs have VMtools installed, so you can use vSphere “Shut Down Guest” to shutdown VMs gracefully.

When you need to bring those VMs back, just follow the reverse steps above.
1) power on VMs prior to Cloud Controller and Ops Manager Director
2) power Cloud Controller
3) Ops Manager Director
CLI help for details.

Posted in Cloud Foundry, Pivotal | Tagged | Leave a comment

Enabling self service user and org creation in Pivotal Cloud Foundry

featured-pcfPivotal Cloud Foundry has a web page where users can goto and login and manage their environment.  Depending on the privileges assigned to the user this may mean the ability to invite other users, create orgs, create spaces, manage apps, etc.

PCF has a self-service ability to allow users to create their own accounts.  By default this it turned off.

For instance, my PCF domain is cf.lab.local.  I can access the user portal (Formerly referred to as the developer console or just console, but now referred to as apps_manager) at: https://login.cf.lab.local

Pivotal

The link shows to Create Account.  Clicking on it will give the following 403 Forbidden error:

Pivotal_CF

To allow for users to create accounts via self-service, you must set the environment variable ENABLE_NON_ADMIN_USER_MANAGEMENT, and additionally so that they can have their own org, you should enable ENABLE_NON_ADMIN_ORG_CREATION.

From a terminal window, run cf api URL and target your Apps Manager URL. For example:

nettles:~ bfeeny$ cf api api.cf.lab.local --skip-ssl-validation

Setting api endpoint to api.cf.lab.local...
OK

API endpoint:   https://api.cf.lab.local (API version: 2.23.0)   
Not logged in. Use 'cf login' to log in.

Run cf login and provide your UAA Administrator user credentials:

nettles:~ bfeeny$ cf login
API endpoint: https://api.cf.lab.local

Email> admin

Password> 
Authenticating...
OK

Select the system org and the console space:

Targeted org system

Select a space (or press enter to skip):
1. apps_manager
2. app-usage-service
3. autoscaling
4. notifications-with-ui

Space> 1
Targeted space apps_manager

API endpoint:   https://api.cf.lab.local (API version: 2.23.0)   
User:           admin   
Org:            system   
Space:          apps_manager   

Set the ENABLE_NON_ADMIN_ORG_CREATION environment variable to true:

nettles:~ bfeeny$ cf set-env apps_manager ENABLE_NON_ADMIN_USER_MANAGEMENT true
Setting env variable 'ENABLE_NON_ADMIN_USER_MANAGEMENT' to 'true' for app apps_manager in org system / space apps_manager as admin...
OK
TIP: Use 'cf restage' to ensure your env variable changes take effect

Set the ENABLE_NON_ADMIN_ORG_CREATION environment variable to true:

nettles:~ bfeeny$ cf set-env apps_manager ENABLE_NON_ADMIN_ORG_CREATION true
Setting env variable 'ENABLE_NON_ADMIN_ORG_CREATION' to 'true' for app apps_manager in org system / space apps_manager as admin...

OK
TIP: Use 'cf restage' to ensure your env variable changes take effect

Even though it says to run cf restage we can just do a cf restart:

nettles:~ bfeeny$ cf restart apps_manager
Stopping app apps_manager in org system / space apps_manager as admin...
OK

Starting app apps_manager in org system / space apps_manager as admin...

0 of 1 instances running, 1 starting
0 of 1 instances running, 1 starting
0 of 1 instances running, 1 starting
1 of 1 instances running

App started

OK

App apps_manager was started using this command `bundle exec rake server:start_command --trace`

Showing health and status for app apps_manager in org system / space apps_manager as admin...
OK

requested state: started
instances: 1/1
usage: 1G x 1 instances
urls: console.cf.lab.local
last uploaded: Fri Apr 17 00:21:01 UTC 2015

     state     since                    cpu    memory         disk           details   
#0   running   2015-04-16 10:38:25 PM   0.0%   209.3M of 1G   285.1M of 1G

Now that this environment variable is properly set we can goto the login page again:

Pivotal

And this time when we click Create Account we the page we would expect:

Pivotal_CF

We will add a user and click send and are given a positive result:

Pivotal_CF_and_bfeeny_—_tempest_pivotal-ops-manager____—_bash_—_198×47

And email should arrive which looks like below:

Inbox_—_Harvard_FAS_IMAP__75_messages_

Click on Verify your email address and you will be sent to a page to enter your password:

Pivotal_CF_and_Inbox_—_Harvard_FAS_IMAP__75_messages_

 

Once this is done, you now are asked to create an Org:

Pivotal_CF

Now finally you have access to your own version of Apps Manager, and can have an environment of your own:

Pivotal_CF_and_Inbox_—_Harvard_FAS_IMAP__75_messages_

Posted in Cloud Foundry, Pivotal, Uncategorized | Tagged | Leave a comment

smoke-tests fail in Pivotal Cloud Foundry 1.3 (Solution)

featured-pcfRecently I built a Pivotal Cloud Foundry lab. I used the following builds:

Ops Manager 1.3.4.0
Elastic Runtime 1.3.5.0
Ops Metrics 1.3.3.0

I fired up PCF in my lab and have been playing with it. One thing that bothered me is that the smoke-tests errand would fail on Elastic Runtime. I tried both Elastic Runtime 1.3.4.0 and 1.3.5.0, and my fix was just to uncheck it so it did not run as a Post Install Errand. But I was not happy with that. The exact errors given by the installer looks like so:

cf api https://api.cf.lab.local −skip−ssl−validation
Setting api endpoint to https://api.cf.lab.local...
FAILED
i/o timeout

You will see that basically what is happening, is that while running the smoke-tests it fails to establish a connection to the API. I tested using cf from my laptop however, and it works fine, and is quick.

So I simply finished the install and unchecked the smoke-tests errand. Installation completed just fine.

Determined to troubleshoot the issue, I manually kicked off the errand from the director using bosh run errand smoke−tests.

So you have your bearings, my environment is:

PCF Infrastructure network: 172.16.200.0/24

PCF Deployment network: 172.16.201.0/24

 

I then logged into the VM where the smoke-tests was running, and just tried the API commands that were failing:

vcap@a1e1b7bf−ae1a−4e29−9cfb−34cd1a62be07:~$ /var/vcap/packages/cli/bin/cf api https://api.cf.lab.local −skip−ssl−validation 
Setting api endpoint to https://api.cf.lab.local...
FAILED
i/o timeout
vcap@a1e1b7bf−ae1a−4e29−9cfb−34cd1a62be07:~$ 
vcap@a1e1b7bf−ae1a−4e29−9cfb−34cd1a62be07:~$ /var/vcap/packages/cli/bin/cf api https://api.cf.lab.local −skip−ssl−validation 
Setting api endpoint to https://api.cf.lab.local...
OK
API endpoint: https://api.cf.lab.local (API version: 2.13.0) 
Not logged in. Use 'cf login' to log in.

 

Sure enough, you can see it fails. But then, when you run it again it succeeds! So I went off troubleshooting. It turns out in trying three times it fails, succeeds, and succeeds. I was drawn to there possibly being a DNS issue.

So I inspected my DNS and found its working properly. I have wildcard DNS setup for *.cf.lab.local, and its returning the HA Proxy no problem. So I look at the DNS on the smoke-tests machine, and thats where I see the issue.

 

When I log into my smoke-tests VM, here is what the /etc/resolv.conf looks like:

vcap@497b8af0−dc12−49e4−a702−ad59c6348d59:~$ cat /etc/resolv.conf 
nameserver 172.16.200.2
nameserver 172.16.5.30
nameserver 172.16.201.4

172.16.200.2 is the Infrastructure Network address for Ops Manager Director

172.16.5.30 is my internal DNS server, the one I configured whenever asked for DNS (ova deployment, Ops Mgr install, ER install, etc)

172.16.201.4 is the Deployment Network address for Ops Manager Director

Obviously the host will attempt to use the first address listed in /etc/resolv.conf to
resolve names. It may then alternate name servers with subsequent requests. This explains why it fails the first time. It tries the first name server. But then if you run the API call again, it succeeds, and again it succeeds. Then it fails again. Below I test resolving against all three hosts listed in resolv.conf:

vcap@497b8af0−dc12−49e4−a702−ad59c6348d59:~$ nslookup api.cf.lab.local 172.16.200.2
;; reply from unexpected source: 172.16.201.4#53, expected 172.16.200.2#53
Server: 172.16.200.2
Address: 172.16.200.2#53
 
Name: api.cf.lab.local
Address: 172.16.201.5
 
vcap@497b8af0−dc12−49e4−a702−ad59c6348d59:~$ nslookup api.cf.lab.local 172.16.5.30
Server: 172.16.5.30
Address: 172.16.5.30#53
 
Name: api.cf.lab.local
Address: 172.16.201.5
 
vcap@497b8af0−dc12−49e4−a702−ad59c6348d59:~$ nslookup api.cf.lab.local 172.16.201.4
Server: 172.16.201.4
Address: 172.16.201.4#53
 
Name: api.cf.lab.local
Address: 172.16.201.5

When attempting to resolve against 172.16.200.2, you see it outputs “reply from unexpected source“.

There is also a considerable delay, enough to cause a time out. Resolutions against the other sources are instantaneous, with no delay or any strange output.

I test again against 172.16.200.2 for good measure:

vcap@497b8af0−dc12−49e4−a702−ad59c6348d59:~$ nslookup api.cf.lab.local 172.16.200.2
;; reply from unexpected source: 172.16.201.4#53, expected 172.16.200.2#53
Server: 172.16.200.2
Address: 172.16.200.2#53
 
Name: api.cf.lab.local
Address: 172.16.201.5

We can see the routing table on the Ops Manager Director below:

vcap@bm−f80e2644−c1d2−4c30−af89−4885bacf1a98:~$ netstat −rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 172.16.200.1 0.0.0.0 UG 0 0 0 eth0
172.16.200.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
172.16.201.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1

 

It is dual homed on both the Deployment and Infrastructure networks as expected.  It’s default gateway is on the Infrastructure network. The Ops Mgr should be able to communicate on either network, using either address. Obviously if the Ops Mgr were to receive traffic from a network not local to it, it would need to respond using its Infrastructure address 172.16.200.2, since that is what shares a network with the default gateway.

It see’s a packet coming to it from the smoke-test VM at 172.16.201.24, and destined to its 172.16.200.2 address. The Ops Mgr responds from its 172.16.201.4 address as its local to the requester. But this does not seem correct, as the requester is not prepared to see a reply from a different address than it requested. This asymmetric routing creates an issue.

It turns out, the fact that Ops Mgr’s Infrastructure network IP address being listed in /etc/resolv.conf on the smoke-tests VM is a bug. You will not see this issue if you just deploy one single network. But if you split Infrastructure and Deployment networks, then Ops Mgr is multi-homed and you will see this bug.

 

This is fixed in Pivotal Cloud Foundry 1.4. Thanks to Pivotal support who replied with a fix below:

To change this behaviour in Pivotal CF v1.3.x, on the Ops Manager VM, change /home/tempest−web/tempest/app/models/tempest/manifests/network_section.rb

Line 20: "dns" => [microbosh_dns_ip] + network.parsed_dns,

to "dns" => network.parsed_dns,

 

Now you can re-enable the smoke-tests errand and re-apply changes and all will be well!

Posted in Cloud Foundry, Pivotal | Tagged | Leave a comment

How to plot decision boundaries for Logistic Regression in Matplotlib

I recently answered the following question on StackOverflow How do I plot the decision boundary of a regression using matplotlib?

I am just going to link here to the post, and post the picture below.

LR-boundary

Posted in Data Analytics | Tagged , , | Leave a comment