Tricks & Tips - installing in racks - preinstallation stage

Summary: Tricks and tips that I learnt during my career.

A set of rules that I follow before my installations that I found myself repeating to junior colleagues. 

1- Check the installation space 

Apart of the obvious "check if there is enough space for the equipment", make sure have the required U(s) and install the equipment in the right U(s). Believe me, in this job, there are few things more frustrating than going for an installation and you need to rearrange some equipment's because someone didn't install theirs correctly. 

Nowadays, most of the racks have the Us numbered, so before the installation, confirm witch U(s) are you going to use:

 (If you don't know what a U is, please check the link below "A typical section of 19-inch (482.6 mm) rack rail" : Rack_rail_dimensions )

2- Confirm the power outlets

 I'm from Portugal so we mostly use CEE outlets but you might found racks with C13 or C20 power outlets so check out if the equipment came we the right cable. Here's a nice article that sums it all nicely:  

FS.com community: "How Much Do You Know about Power Cord Types?" 
 
This is something that most non-tech people overlook so I strongly suggest that you confirm it personally.

 
3- Confirm if there are cage nuts for the equipment

Here's something that's vastly overlook by non-tech people. Confirm if the equipment has  cage nuts or if their are included in the list of equipment.
Use a cage nut tool to install them but since it's most likely you don't have such tool, have a least a "flat-blade" screwdriver at hand.

 

4- Check the cabling and the SFPs

Here's something that shows if the non-tech people involved on the project has any experience. Usually, the most experience accounts or pre-sales team or project managers are aware of the quantities and different types of SFPs and also the cabling that is required, but I urge you to confirm if the order has the right amount of cabling or if the customer will provide some or all of it.
Also, make sure that the cabling has the right length and the proper termination, specially the FO pairs. Nowadays, the vast majority of FO terminations are LC-LC but you should be aware that there are other kinds of FO terminations (SC or ST).

  

5- Stage your equipment

Staging is was separates the pros from the newbies.

You never know what would be the conditions on site at the day of the installation so everything that can be prepared prior is a must. I usually say to my junior colleagues to stage the equipments in such way that when they go on site, they only need to mount, cable and power it because:

• You don't know if there won't be any power issues during the installation.
  Sometimes customers schedule different teams to work during their maintenance window and you might end up waiting for the power to come up. However, if your equipment was staged properly, you know that you can mounted it and just need to plug the power cables after they finish their work. Also, make sure you can mount your equipment without having to connect a laptop to check the config or status. For instance, if you are mounting a switch stack, you should know (for instance; write some note at the equipment box) which switch is the master switch and preferably, the switches are mounted according to the order of the stack. Sure you can renumber it after but it's not efficient work.
  
  
  
  

  

• You don't know what will be physical the conditions on site.
  It's a lot better to be seated in a comfy chair, in a room with proper light and AC while you install the blank covers, the mounting brackets, the stack modules, the fans and network modules instead of crouching over the equipment, with AC blasting over (or below) you, under poor light or at a very noisy environment.

• You don't know if you will have a place to lay your laptop and configure the equipment.
  It's a lot better to calmly configure a equipment while you are seated on a comfy chair, in a room with AC, with time to check manuals or clarify configuration details instead off standing in an extremely cold (or hot if there isn't any AC yet) tech room, while holding a laptop with one arm and punching commands with the other hand and there's someone using a power drill in a room near by.

• You don't know if you have a DOA (dead-on-arrival) equipment or a faulty cable.
  Which it's not the installers fault but all in all, it's the installer who stands in front of the customer. No matter what he says the thought that will linger in the mind of the customer will be "well, if you have tested the equipment before this wouldn't end up being a waste of time for all of us..."

• You don't know what is the firmware it's on and if an upgrade is required.
  It's a lot better to calmly download the proper version while you are seated on a comfy chair with a stable internet connection than being on site, looking for place to activate your personal hotspot and download through a mobile connection a image that can be several hundreds of megabytes in size.

Here's a story I also like to tell my junior colleagues and that it sums this all. When you schedule a cable service for your house, what do you expect from the cable technicians?

That they should arrive on time, extend the cable/fibber to your house, plug and power up everything, call central to activate the service, test everything and it's all done. You can go on with your lives. But I'm sure you will file a complain if you heard anything like "Oh, sorry, we need to postpone the installation because..."

"... I don't have the right power cable."

"... I don't have the right tools."

"... the equipment it's dead."

"... it's the first time I see this equipment and I don't know how to configure it"

Next time, I will blog about tricks and tips that I use during my installations.

How about you? Any more tricks and tips that you would like to share? Please write them on the comments below. Thank you.

How to do a password recovery in Cisco 9500 IOS-XE version 16.9.3

What I've found out when following the manual vs reality

The official manual needs an update. It seems that someone copy+pasted the procedure of the 9200 and hastily added a note but there are still some differences:

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9500/software/release/16-9/configuration_guide/sys_mgmt/b_169_sys_mgmt_9500_cg/troubleshooting_the_software_configuration.html

...

 DETAILED STEPS

---

Step 1

Connect a terminal or PC to the switch. Connect a terminal or a PC with terminal-emulation software to the switch console port. Connect a PC to the Ethernet management port.

Step 2

Set the line speed on the emulation software to 9600 baud.

Step 3

Power off the standalone switch or the entire switch stack.

Step 4

For Cisco Catalyst 9500 Series Switches, reconnect the power cord to the switch or the active switch.   As soon as the System LED blinks, press and release the Mode button 2-3 times.  The switch enters the ROMMON mode.    --- # Not the right procedure, do the CTRL-C sequence Note  Cisco Catalyst 9500 Series Switches- High Performance do not have a Mode button. You can exit the configuration dialog at any prompt using Ctrl-C to kill the bootup sequence.  The following console messages are displayed during the reload: Initializing Hardware... System Bootstrap, Version 16.6.1r [FC1], RELEASE SOFTWARE (P) Compiled Sat 07/15/2017 8:31:57.39 by rel Current image running: Primary Rommon Image Last reset cause: SoftwareReload  C9500-12Q platform with 8388608 Kbytes of main memory attempting to boot from [flash:packages.conf] Located file packages.conf # ##################################################################### Unable to load cat9k-rpboot.16.06.02b.SPA.pkg Failed to boot file flash:user/packages.conf ERROR: failed to boot from flash:packages.conf (Aborted)<--- abort="" em="" will=""> Initializing Hardware... System Bootstrap, Version 16.8.1r [FC4], RELEASE SOFTWARE (P) Compiled 20-03-2018 15:12:03.01 by rel Current ROMMON image : Primary Rommon Image Last reset cause:PowerOn C9500-48Y4C platform with 16777216 Kbytes of main memory Preparing to autoboot. [Press Ctrl-C to interrupt]  Break sequence to be pressed to get to rommon   Proceed to the Procedure with Password Recovery Enabled section, and follow the steps.

Step 5

After recovering the password, reload the switch or the active switch. On a switch: Switch> reload Proceed with reload? [confirm] y

---

Procedure with Password Recovery Enabled

Procedure

---

Step 1

Ignore the startup configuration with the following command: Switch: SWITCH_IGNORE_STARTUP_CFG=1  ROMMON x > SWITCH_IGNORE_STARTUP_CFG=1

Step 2

Boot the switch with the packages.conf file from flash. Switch: boot flash:packages.conf  --# the 9500 was with factory version 16.9.3 and with no packages.conf file   ROMMON x > boot

Step 3

Terminate the initial configuration dialog by answering No. Would you like to enter the initial configuration dialog? [yes/no]: No

Step 4

At the switch prompt, enter privileged EXEC mode. Switch> enable Switch#

Step 5

Copy the startup configuration to running configuration. Switch# copy startup-config running-config Destination filename [running-config]? Press Return in response to the confirmation prompts. The configuration file is now reloaded, and you can change the password.

Step 6

Enter global configuration mode and change the enable password. Switch# configure terminal Switch(config)#

Step 7

Write the running configuration to the startup configuration file. Switch(config)# copy running-config startup-config

Step 8

Confirm that manual boot mode is enabled. Switch# show boot BOOT variable = flash:packages.conf; Manual Boot = yes Enable Break = yes

Step 9

Reload the device. Switch# reload

Step 10

Set the SWITCH_IGNORE_STARTUP_CFG parameter to 0. Switch(config)# no system ignore startupconfig switch all Switch(config)# end Switch# write memory

The account and the engineer - the tale of the two hunters

Several years ago I was at a company that was really struggling. To get everybody hyped, they organized a social event and they invited a "motivational speaker" to rally the troops. During his speech he told this story:

"Two hunters go into the savanna. They arrived at the site where they decided to mount their base and as soon as they unload their baggage, one of them simply goes off into the woods while the other one is left alone to prepare everything. Mounting camp, getting the weapons ready, checking the tools and traps, checking their supplies, etc, etc. He’s about to finish all those tasks when he finally sees his companion again, sprinting towards their camp, trying to be out of reach of a lion that's almost catching him. 

And as he's about to reach their camp he shouts "You take care of this one while I get another!"

My friends; the hunter that was running is the Account. They should focus on bringing the game within range of the weapons and tools of a skillful hunter, the Engineer."

It might looked like a witty anecdote but there are a few mistakes:

- If the other hunter was expecting smaller animal and didn't had the tools or skills to tackle a lion, he would end up dead.

- If the other hunter was in fact expecting a lion and his companion brought him an elephant, he wouldn't have the weapons to handle such beast and would end up trampled. 

- If the other hunter was expecting a lion and instead got a pack of hyenas, he would have the weapon to tackle a few of them but it will end up overwhelmed.

Well, you get the idea.

But we can take that anecdote and ask ourselves how could make things go much better. 

First of all, there should have been a clear understanding between the two hunters; What are we going to hunt? Do we have the skills for that? What resources do we have available? What are the dangers and our limitations?

Once that's settle they should set a proper plan of action. They must be aware that despite all the planning and preparation, there events that they can't control and that might have impact on their plan. They need to judge if those events are manageable or if they are should stop I change their way.

Also very important is they both need to have the willing to follow the plan. If the running hunter sits in the shadow of a nearby tree or if he runs really fast but he's just doing laps close to their base, only by sheer luck would they get a lion passing through their camp. On the other hand, if it's the technical hunter sitting in the shadow and he doesn't prepare his weapons and tools, he won't be fully ready when the time for action finally arrives and miss the limited opportunity. 

Another thing that we can analyze in this anecdote is the skills of the technical hunter. Handing a rifle to a someone and tell him, "right, you got your weapon, you got your gear, you know where the jungle is, basically you're all set. Off you go and if you need anything just ask around". The results might be catastrophic:

But, if he's given time and training to sharpen his skills, he will become a skillful hunter, ready to fulfill the tasks that everyone is expecting him to do. Furthermore, if he's willing to do all the hard work and has the dedication and passion to continuously improve his skills, he could even reach a state where he will be skillful enough to kill a bull with a karate chop. 

Dear customers, when reporting a WiFi issue please don't forget the 3 W's: "Who", "When", "Where"

Summary: "Dear customers, please help us helping you"

Time and time again, those who support a WiFi network get those reports:

"Hey, we had a complain about the WiFi network. Can you check it out?"


That's an equivalent to taking your car to a mechanic and tell him:


"Hey, sometimes my car makes some funny noises. Can you please solve it?"

Hmmm, where to start?...

At the very least, whoever is going to analyze the problem will spent a lot of time just to identify the real issue, which can even turn out to be not an issue at all, or something that has nothing to do with the WiFi network.

So, first things first

"Who?"
         - Who's been affected by the problem? One person? One department? Everyone that uses that SSID? All SSIDs?


          The vast majority of issues are users authentication. Either bad dot1x passwords or bad pre-shared keys. It's common to have a user complaining while having other users working fine at the same AP, using the same SSID and the same radio interface.

          Sometimes, specially with smarphones, the problem occurs after an system update to the phone.



          There are ways to track users status so we can check if it's an authentication issue or if it's someone doing naughty stuff to our users like a neighbor flooding deauth messages to our users.
"When?"
         - When did it happened? It's at a specific time of day? all day? every day?

         When the problem varies though time usually it can either be the result of interference, like 40 Mhz personal hotspots, bad configured neighbors, motion sensors, wireless video cameras, weather radars.
         Or it can also be abnormal network usage, like a network that was design a few years ago to support up to 15 e-mail and basic web browsing users per AP will struggle to serve 30 to 40 users per AP, running more demanding network usage.

Either identify the source of the interference, which might require a site survey, or, if it's abnormal usage, redesign the network to fulfill the new requirements, which my also require a new site survey not only to check the placement of new APs but also to confirm if the network infrastructure can support the additional equipments (number of switch ports, PoE capacity, cabling)



"Where?"
         - At a specific zone of the building? Throughout the site? At every site or one delegation? Was there any change to the building?

         Again, this can be a problem caused by interference (those motion sensors can be a real pain) but sometimes it can happen because of changes that were done to the building; more or fewer walls, windows that were changed from plain glass to mirrored glass, new mezzanines, much more users in the same spaces. In case of warehouses, it can be cause by changes to the rack orientation or composition. 

Sometimes occurs at places that, according to the plan agreed with the customer, it weren't suppose to have WiFi coverage.
"Hey, we don't have WiFi at this floor/at the stairs/at the elevators!"
"Yes, as you can see from the report, that zone wasn't included in the plan. We need to install more APs if you want to have that area cover. Would you like to install more APs?"

In conclusion, please be aware that as soon as we can pin-point the source of the problem, the faster we will solve it.

So, for instance, if instead of "Hey, we had a complain about the WiFi network. Can you check it out?" we got a "Hey, all the departments are working fine at the 5th floor but the marketing users can't register at the "Marketing" SSID there. However they do work fine at the other floors. Can you check it out? Oh, and by the way, we had a power outage during the weekend" This will ensure that we will quickly start checking for the more probable causes of that problem (vlan miss configuration because the switch configuration wasn't properly saved, AP groups, WLAN-VLAN mappings)

First time with a Riverbed Xirrus and what I think of it

A partner lend me a Riverbed Xirrus (XD2-230) for a couple of weeks. Unfortunately (or fortunately) it came at a time I was so overwhelmed with work that I only managed to test it during a couple of mornings and a Sunday afternoon

This is just my opinion taken during the very limited time I had to try the AP and tools. So, on that note here's my opinion of it:

- Dashboard initial configs, Floor Plan, AP location
     Initial configs, ok.
     Floor plan, ok.
     AP Location.... they need to revise this. Dragged the AP to it's exact location but it kept identifying the address of the building that's in front of our office, on the other side of the street. Tried to write the address manually and found out that it didn't recognize addresses in Portugal. Even downtown addresses like "Avenida da Liberdade, 130, Lisboa, Portugal" was translated to "Lisboa, Portugal" and then the portal suggested several addresses in Brazil. I know we're a small country but really?!... moving on

- Couple of PSK ssids; one with no VLAN and other with VLAN Tag enabled.
     Everything worked fine. Both with IPv4 and IPv6 addressing scheme.

- Several type of Portals; Guest portal, self register, voucher
     Easy to configure and deploy

- Content analysis and filtering
     No problems here too.

- Dot1x with ISE
     A lot of struggles here. Manage to have users authenticate by ISE 2.2 but for some reason, the authenticated users wouldn't get the DHCP OFFER packet. Unfortunately time was up and I didn't have the time to confirm if the DHCP REQUEST was reaching the dhcp server or if was the server that wasn't answering or if it was something missing. Maybe next time.

What I liked the most

The EasyPass portal. Very intuitive configuration and easy to deploy.

What I disliked the most

No management app for android or ipad. As a engineer that also supports customers, I've becoming an enthusiastic fan of a app provided by the competition. In seconds, one can have a quick glance at the network status.
Address location was appalling when compared to the competition.

What I would like to have tried but didn't had the time to:

- Coverage site-survey.
- Throughput and capacity test
- Kali tools

Maybe some other time.

Como recuperar a password de GUI expirada num ISE 2.2

O que fazer quando tentamos aceder ao GUI de um ISE e aparece esta mensagem?

1 - Aceder ao ISE por SSH

    Vai aparecer uma mensagem a informar que a password expirou e para colocarmos a nova password

2 - Aplicar o comando para reiniciar a password do GUI

    Aplicar o comando application reset-passwd <application> <user>
e redefinir a password: 

3 - Aceder ao ISE por GUI

    Voltar a abrir a sessão por https e colocar a nova password:

Feito!

My certification path - the beginning of a journey

The funny think about moving to a new house is, while packing and unpacking your stuff, you'll end up discovering things from a distant past, like you're running an archeological expedition to your own personal history. While unboxing yet another pile of stuff I came across with this peace of paper. My first technical certification:

At that time, I had a post-sales position in Nortel Network and we were required to have this certification after a few years on the job.I did my first attempt back in 1999 or early 2000 but I didn't pass. Actually, I went for my first attempt with no slightest idea of what was a certification exam. Now we have study communities, youtube videos, social media, forums, but at that time it was you, the equipment and thousands of pages of manuals and study guides.

I stormed away from the test center after my first attempt and called my boss immediately; "Hey! Sorry, I didn't pass but really... what the hell!? most questions it's just stuff that we find in the manuals. This is not a test to my technical skills, it's a test to my memory! A real exam should be a practical exam, not this!"

So I forgot this "useless certification not required to my daily job" stuff and went on with my life. But in 2002, the team was committed to have everyone at least nncss level (equivalent to ccna). This time I knew what to expect so, for two months, I took the training manuals but kept this thought in my mind:
"if you were making an exam on this topic, what questions would you ask?" and it worked.
I don't know what was hardest; answering the exam or when I saw the result and had to muffle a triumph shout in the room. Felt like I'd just scored the winning goal against a rival club.

But that was a really odd year because after the joy of getting the certification, 6 months later I got made redundant because Nortel was sinking hard and it was decided to terminate the post-sales support in Portugal. "That's alright! I have my nncss so I should get a new job easily".

I went through a few interviews but it would basically stopped at this conversation:
"I'm a certified specialist!"
 "yeaah... you see but we're looking for someone with a ccna..."
"True, but it's a different vendor but it's equivalent to ccna. The standards and protocols are the same for every vendor"
"...yeah... you see, it's just that our technical manager... is asking for ccna's... we're sorry..."

I eventually manage a job related with Nortel equipment but that experience made me realize certifications are also a reference for the HR folk.

It also made me realized that I shouldn't get my eggs all in one basket and I should diversify my knowledge. At that time I was working as an outsource engineer at a bank. The network was so stable and I got my work so well organized that I got to a point where I could do the work load of a month in one hour so that gave me a lot of time where I could study.

Got a few more Nortel certifications but I was really bored with my job. I kept asking to move to other roles or other customers but my manager kept telling that only roles that were available required skills with Cisco solutions. So, instead of waiting for a hypothetical opportunity, I decided to create my own space and start my way in the Cisco certifications.
I bought the 3rd edition of the cybex CCNA study guide and went on with the studies. I barely passed my CCNA at first attempt (think it was 830 with passing score at 800) but I passed and that was all that matter. Now I add the hard proof that would assure my peers that you can get the job done, no mater the vendor.

A few years later,I had certifications in all vendors represented by my company (Nortel, Cisco, Alcatel-Lucent, Enterasys). It seemed the more I had the easiest was to get the next one. Unfortunately that wasn't being reflected in my payroll. Whenever I mention that "yeah... you know... the economy... It's hard... but we'll see... we are aware of you situation". Tired of this talk I decided to advance to more advanced certifications.

I didn't knew where to start but I realized that my company had some CCNPs but no CCSP. "Well, I kind of like this PIXy thingy, let's do that!"
I eventually got my CCSP and had my payroll revised, but I was hungry for more. I felt that I should do the CCNP too. After that, came the CCNA-W and then the CCNP-W and so on.

17 years later, when I picked that paper and looked back to my journey, I smiled for a few moments, proud of what I have achieved. An ex-nortel that eventually got accepted in the Cisco Champions program. But I quickly got back to unboxing my stuff while my hearing regains focus on the technical video that was playing in the my ipad. It still have a hunger for more...

A few notes;
Will you make use of all the topics covered by a certification?
Depends on your job role but there are a lot of topics that you'll never see again as soon as you press the submit button at the end of your exam.
But more importantly, when you are studying for a cert, you'll cross with a certain topic and will go "oh! so that's why that thing happened!" or "oh! if only i knew this i would have done things differently"
Are certifications useless?
No, even if you take a certification that most of the topics are useless at your current job, that doesn't mean that it won't be useful someday. Unless you change to a different job role you'll sure find your self muttering from time to time:

Are certifications the ultimate proof of competence or skill?
No,that number or icon is just that, a number or a icon. Competence and skills are the result a ongoing study and practice. Always getting new resources to feed that constant hunger for knowledge.
If you got your cert I think, "Ok, I'm a master now! No more study for me!" sorry but you'll be outdated soon. Certs are just a mark on the journey, not the end.

There's a engineer that holds this certification so he must be an expert on that, right?
Well... it depends. If he as got it through the right way, the only way:
 

But it will show if "expert" got the cert through cheating or poor training, like these group of "black belts":

Here's some thoughts that I found during my certification path:

- it's the journey, not the end.
        you'll be ecstatic when passing but you'll soon grant more value to the hours of work that to that new entry on linkedin.


- be vendor neutral and multi-vendor
        Every vendor has strengths and weaknesses and nothing lasts forever.

- go wide and then go deep
        Broad your knowledge and pursue expert level when you find your favorite topic. And by favorite I mean some topic that you can study for hours and felt it was just for a few minutes.

- "knowledge is like manure. It's only good when spread"
        It's a small world. So many things and thoughts out there. Be a sponge, understand others views and methods. Share and be there.  

- don't be afraid to fail.
        Learn from your failures. They are great teachers. Failing at an exam isn't a proof of incompetence or that your are a hack. 

Cisco Prime - Upgrade de 3.x para a 3.2

Para fazer um upgrade de Prime de 3.X para a 3.2 vai ser necessário:

• opcional mas altamente aconselhável; guardar uma cópia do backup do Prime
• acesso ao Prime com credenciais de admin
• ter a imagem de upgrade PI-Upgrade-<versões>.tar.gz
• ter acesso ao Prime por consola ou por VMware Remote Console

1º Guardar o backup do Prime

Em princípio, se o Prime 3.1 estiver a operar de acordo com as boas práticas, deverá ter o processo de backup a correr automaticamente. De qualquer maneira, para confirmar se o processo está a correr corretamente, aceder ao CLI do Prime via SSH ou consola e confirmar se os ficheiros de backup estão a ser gerados fazendo o comando:

dir disk:/defaultRepo

Deverão estar uns ficheiros de backup cujo nome começa com o hostname do Prime, a data e hora a que foi feito o upgrade. Exemplo:

<hostname>/admin# dir disk:/defaultRepo

Directory of disk:/defaultRepo

<hostname>-<data>-<hora>__VER......tar.gpg
Transferir uma cópia do último backup para outro servidor:

admin# copy disk:/defaultRepo/<ficheiro de backup> <destination>


destination - ftp://<FTP Server ip>/<ficheiro de backup>


2º Fazer o download da imagem de PI-Upgrade-3.X_to_3.2.0.0.258.tar.gz


3º Transferir a imagem para o Prime. 
(No meu caso, utilizei o FileZilla no meu laptop para fazer de ftp server) Aplicar o seguinte comando:

admin# copy <source> disk:/defaultRepo

source - ftp://<FTP Server ip>/PI-Upgrade-3.X_to_3.2.0.0.258.tar.gz




4º Apagar ficheiros de backup que estejam na diretoria disk:/defaultRepo


admin# delete disk:/defaultRepo/<file>



5º 
VC(29/Maio/19) Adenda: Este passo já nem aparece no processo de migração para a versão 3.4 e 3.5 
(opcional se for um upgrade de 3.1 para 3.2) Parar os serviços do Prime
Aplicar o comando:

ncs stop



6º Aceder ao Prime por porta de consola ou por consola virtual
No meu caso foi necessário instalar o programa VMware Remote Console

7º Fazer o upgrade para a 3.2
Aplicar o comando:

admin# application upgrade PI-Upgrade-3.X_to_3.2.0.0.258.tar.gz defaultRepo

No meu caso, para um Prime que estava a gerir uma rede com 5 APs e entre 100 a 150 equipamentos, o upgrade demorou cerca de 1h30 a concluir.