Deconstructing Cisco NX-OS Part 1: Exploding Kickstart

I have worked for quite some time with SAN-OS, which later became the basis for NX-OS. Being the tinkerer that I am, I really like to get inside things and figure how they work. My past experience with UNIX based OS’s made it ideal for me to be able to further explore the innards of NX-OS. During this discovery, I found multiple security vulnerabilities, which were reported to Cisco via their PSIRT process and has hopefully made NX-OS a more secure and better product. The angle of this post is more from an engineering perspective, not security. Having access to the NX-OS filesystem is not really a security risk, but it is certainly interesting. There are many ways to achieve this. Like with IOS-XR, it is possible to gain access to the “shell” of the NX-OS system via its normal CLI interface, however we will go about it in a much more straight forward approach.

In this tutorial I am working with CentOS 5.5, but really just about any version of linux should work fine for these tasks.

NX-OS is based on MontaVista Linux (HardHat Linux). Cisco distributes the kickstart and system binary files which are used to run NX-OS. Kickstart is actually a composite file that contains multiple different sections and is used to get the system up and running. The system binary contains the majority of the actual daemons and processes that run the system.  You will obviously have to have access to NX-OS images to do this.  I have used SAN-OS images in the past as well.  Below we are working with the latest release of the Nexus 7000 code:

[root@cluebox nx-os]# ls -al
total 173412
drwxr-xr-x 2 root root    4096 Mar 24 00:48 .
drwxrwxr-x 7 95528 named   4096 Mar 24 01:11 ..
-rw-r--r-- 1 root root 146701191 Mar 24 00:48 n7000-s1-dk9.5.1.3.bin
-rw-r--r-- 1 root root  30674944 Mar 24 00:47 n7000-s1-kickstart.5.1.3.bin

 

The kickstart image contains a header of variable length. The header maps out a lot of data, notablythings like the version, image offset, image type, etc. The kickstart image is a NBI image.


[root@cluebox nx-os]# file n7000-s1-kickstart.5.1.3.bin
n7000-s1-kickstart.5.1.3.bin: Netboot image, mode 2

 

Ok, so we have established this is a standard NBI mode 2 file (Network Bootable Image). This is a standard and you can read about NBI images here Draft Net Boot Image Proposal 0.3
We can take a look at what segments we have in our kickstart file:


[root@cluebox nx-os]# disnbi n7000-s1-kickstart.5.1.3.bin
Type: NBI
Header location:        9440:0000
Start address:          9280:0000
Flags:
Vendor data:            mknbi-linux-1.2-6
Segment number 1
Load address:           00092800
Image length:           4608
Memory length:          6144
Position:               Absolute
Vendor tag:             16
Segment number 2
Load address:           00094000
Image length:           512
Memory length:          1024
Position:               Absolute
Vendor tag:             17
Segment number 3
Load address:           00090000
Image length:           512
Memory length:          512
Position:               Absolute
Vendor tag:             18
Segment number 4
Load address:           00090200
Image length:           6144
Memory length:          9728
Position:               Absolute
Vendor tag:             19
Segment number 5
Load address:           00100000
Image length:           2178048
Memory length:          4194304
Position:               Absolute
Vendor tag:             20
Segment number 6
Load address:           00714000
Image length:           28483584
Memory length:          67108864
Position:               Absolute
Vendor tag:             21
Vendor data:            “”
Vendor data in hex:     00 00 00 00
Segment number 7
Load address:           02500000
Image length:           0
Memory length:          940
Position:               Absolute
Vendor tag:             22

 

So we have six basic segments to this file. Segment 7 doesn’t really count, as its Image Length is 0, and we can’t have that now can we :). So we explode these out into our filesystem:

 

[root@cluebox nx-os]# cd kickstart/
[root@cluebox kickstart]# disnbi -e ../n7000-s1-kickstart.5.1.3.bin
Dumping directory to `nbidir’…
binmode() on unopened filehandle 0 at /usr/bin/disnbi line 156.
Extracting segment 0 to `segment0’…
Extracting segment 1 to `segment1’…
Extracting segment 2 to `segment2’…
Extracting segment 3 to `segment3’…
Extracting segment 4 to `segment4’…
Extracting segment 5 to `segment5’…
Extracting segment 6 to `segment6’…

 

You can safely ignore the error at the end, of course it can’t read the data, if its 0 in size that’s rather impossible now isn’t it? Note that when you do this, what was called “Segment 1” in the disnbi output gets saved as segment0, and what was called “Segment 2” gets saved as segment1, and so forth. The actual nbi header gets saved as nbidir. It contains all the information about each file such as the Load Address, Image Length, Memory Length, Position, and Vendor tag. So for example, looking at the nbidir file:

The segments break down as follows:
     nbidir:         directory
     segment0:       "first" segment
     segment1:       kernel parameters
     segment2:       floppy boot sector
     segment3:       setup segment
     segment4:       kernel
     segment5:       ramdisk (optional)

So we can get back to our original kernel as well as initrd with the following:

[root@cluebox kickstart]# cat segment[234] > vmlinuz

[root@cluebox kickstart]# ls -al vmlinuz
-rw-r–r– 1 root root 2184704 Mar 24 03:14 vmlinuz
[root@cluebox kickstart]# file vmlinuz
vmlinuz: Linux kernel x86 boot executable RO-rootFS, root_dev 0x6801, swap_dev 0x2, Normal VGA

So we are making progress. Hopefully this doesn’t seem to complex, I can imagine coming from the network side of things, all this linux jargon can be mind blowing.are getting close to some cool stuff though.

[root@cluebox kickstart]# file segment5

segment5: gzip compressed data, was “loopback”, from Unix, last modified: Fri Mar 11 02:43:05 2011, max compression

This is basically an initrd image (Initial Ramdisk) and its compressed, we can rename it correctly and take care of it:

[root@cluebox kickstart]# mv segment5 initrd.gz

[root@cluebox kickstart]# gunzip initrd.gz
[root@cluebox kickstart]# file initrd
initrd: Linux rev 1.0 ext2 filesystem data
[root@cluebox kickstart]# mkdir kickstart-filesystem
[root@cluebox kickstart]# mount -t ext2 -o loop initrd kickstart-filesystem/
[root@cluebox kickstart]# cd kickstart-filesystem/
[root@cluebox kickstart-filesystem]# ls
bin        dev   isanboot  logflash    modflash  root     srv  var
boot       etc   lib       lost+found  nfsroot   sbin     sys  volatile
bootflash  home  linuxrc   media       proc      selinux  tmp
debug      isan  log       mnt         README    slot0    usr

Ok, so we are inside the kickstart filesystem!!!!  The binaries are executable:

[root@cluebox kickstart-filesystem]# cd bin

[root@cluebox bin]# file tar
tar: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.6.21, dynamically linked (uses shared libs), for GNU/Linux 2.6.21, stripped
[root@cluebox bin]# ./tar
./tar: You must specify one of the `-Acdtrux’ options
Try `./tar –help’ or `./tar –usage’ for more information.

You may run into things you can’t run, because you have to set your libraries up properly with LD_LIBRARY_PATH, but basically you can do some really cool stuff.  You can even use the NBI tools to “repackage” your own NX-OS with modifications :).

[root@cluebox kickstart-filesystem]# cd isanboot/bin
[root@cluebox bin]# ./imghdr show /tmp/nx-os/n7000-s1-kickstart.5.1.3.bin
**********************************************
HEADER CONTENTS
**********************************************
Header version: 1.0
Len: 940 byte
Image length:30674944 byte
Magic number: 21326
Platform type: 7
Verification type: 1
Software family: 2
Image type: 1
Debug attribute: 1
Hardware type: 0
Compression type: 2
Run time location: 1
Packaged by: 0
Memsize: 256
Timestamp: 1299829376
Version string: 5.1(3)
Interim version string: 5.1(3)
Image full name: n7000-s1-kickstart.5.1.3.bin
Features:
Build ID: S39
**********************************************
Cisco NX-OS(tm) n7000, Software (n7000-s1-kickstart), Version 5.1(3), RELEASE SOFTWARE Copyright (c) 2002-2010 by Cisco Systems, Inc.

So there is much to look at!  This was just a splash dive into the kickstart, there is still much to discuss on how it actually boots, switches to the system image, and the role it plays.  The “system” image is handled a bit differently than the kickstart.  We will explore that in our next segment and in fact use some of the tools in the kickstart image to allow us to do this.

This entry was posted in Cisco, Nexus, NX-OS and tagged , , . Bookmark the permalink.

5 Responses to Deconstructing Cisco NX-OS Part 1: Exploding Kickstart

  1. brian says:

    Also, some may notice the README in the root directory of the kickstart filesystem. There is no valid password for “root” on NX-OS, its starred out:

    [root@cluebox kickstart-filesystem]# cat README
    $Id: README,v 1.1 2006/02/22 22:27:42 polk Exp $

    The default root password is “root”.

    [root@cluebox kickstart-filesystem]# cat etc/passwd
    root:*:0:0:root:/root:/isanboot/bin/nobash
    bin:*:1:1:bin:/bin:
    daemon:*:2:2:daemon:/usr/sbin:
    sys:*:3:3:sys:/dev:
    ftp:*:15:14:ftp:/var/ftp:/isanboot/bin/nobash
    ftpuser:UvdRSOzORvz9o:99:14:ftpuser:/var/ftp:/isanboot/bin/nobash
    __eemuser:*:101:100:eemuser:/var/home/__eemuser:/isanboot/bin/nobash
    nobody:*:65534:65534:nobody:/home:/bin/sh
    admin:x:2002:503::/var/home/admin:/isan/bin/vsh_perm
    [root@cluebox kickstart-filesystem]# openssl passwd -crypt -salt Uv nbv123
    UvdRSOzORvz9o

    As you can see, there is a password for ftpuser. This is used internally with NX-OS as a way of moving files between modules/supervisors.

  2. Pingback: Technology Short Take #12: Networking Edition - blog.scottlowe.org - The weblog of an IT pro specializing in virtualization, storage, and servers

  3. Krunal says:

    Great post………….. I tried with Ubuntu 10 and works just fine.

  4. simfox3 says:

    5 years later, these instructions are still valid (even though the segments are different structure). The netboot tools for CentOs can be obtained like this:

    # git clone http://git.etherboot.org/mknbi.git
    # yum install -y glibc-devel.i686 nasm
    # make
    # cp disnbi /usr/bin

Leave a Reply