The Value of Training
I was asked the other day why technical people and especially engineering always requests and gets more training dollars than other folks. Well let's take a look at what technical people and especially engineering folks are tasked with.

According to the roles and responsibilities of the UEN Engineering department Engineering is tasked with the following:
-Network Design
-Responsible for testing in the lab Hardware and solutions
-Documentation of the Network Design
-Coordinated Handoff of Engineered projects
-Technology Research and Development
-Tools Development
-New Tools Development
-Analyzing budgetary impact for current and future technology
-Project Management
-Definition of device configuration standards

So basically engineering is tasked with taking a new request for a technology that has never been implemented in our network before or in a new way, is expected to lab and prove that design (see previous blog on that subject), design it, set standards for it, teach others how to implement and run it, manage the project and last but not least develop the tools to help manage it.

So I ask the question. How are you supposed to do this without the time, tools and resources and with only about $2,000 a year in training for each engineer. I can't remember how much Harry Potter's magic wand cost him but maybe we can get some for about that ;)
5:08:35 PM     comment []

The Value of a Networking Lab
Over my many years of working with technology, networking and the like I have seen the value of having a lab to prove technologies and designs and to learn how to deal with technology problems.

When I was at IHC there were numerous examples of how a lab was essential to implementing a new technology or network without taking down the "production network". Granted the risks were higher at IHC, lives we often literally at stake when we messed with things. This is one of the many reasons that Jim and I get so crazy when things go wrong. We are used to an environment where the possibility of harm to a patient is very high if there is a network or system failure.

When I left IHC we had several labs (which I understand have now been combined into one with mixed results). We had a LAN lab, a WAN lab, a Emerging Technologies and Internet/Web lab (that was mine, maybe some day I will blog about the story of how we got the lab, I have related it to some and it is pretty funny, depending on your perspective). Prior to when we were moved to the new Lake Park facility we had to scrounge whatever space we could in the BLT (Beneficial Life Tower). When we were switching from bridges to routers on our WAN the lab was located on the 19th floor and our offices were on the 8th floor. Our data center was on the 7th floor. During our evaluation of routers and router vendors we were able to run a connection to the data center and mess with routers back to back and through simulated wan links (both direct and through our T1 networking gear). I wrote a couple simple programs and had some test files where we could see HTTP and FTP transfer speeds. I also had a baby AS/400 in the lab that we used to understand how to make APPN work over an Ethernet/IP network. I am convinced that not having this lab and the AS/400 (donated to the cause by IBM) I would have had a very hard time figuring out the network design and how it was going to work. At the time we already had an extensive 56K APPN/SDLC and Token Ring network already set up that I designed and built and getting time on the system to test was really hard because of the 24/7 nature of the health care business.

Later on at Lake Park in the ET lab we tested and proved our new internet system based on Sun servers and Nokia/Checkpoint firewalls. This was a fully redundant server and firewall system that I conceived in late 1998 and early 1999 to take care of the Y2K problem (that was the good thing about the Y2K bug, it allowed us to justify spending money in an area where we really needed it and the company had been reluctant to spend the bucks on the internet). Anyhow this is another project that would have failed if we tried to just turn it on and hope it worked. Moving things from BLT to LP was bad enough and nothing changed except for the location and the switch gear we were connecting to (which had problems with speed and parity and kept us down for a while as we tried to figure that out - again a good lab would have avoided that).

Then on to Phobos. When I first started there we had engineering in Sugarhouse and corporate in Murray. I spent most of my time at the Sugarhouse office since it was closer to home. We had a lab, but not a lab network so our engineers had to use the "production" or business network for testing of high powered switched, load balancers and the like. Our data center and servers were located in the Sugarhouse office and the business folks in Murray. Corporate was connected via. a T1 line. Here is a typical situation: I'm in my office working on whatever and get a call from Gary Wall (the CFO) saying that they can't get into our ERP system (Visual manufacturing). What would happen is the LAN at my location would be flooded with UDP test traffic and the CO could not get to the ERP server. The T1 was fine, but the intense traffic on the LAN was preventing anything getting through.

When we moved everyone into the new building on 45th south we built a separate engineering network and a production network. We also had a very nice lab for engineering and a very decent data center for the IT folks.

Fast forward to UEN present. I find it amazing that I work for the hands down largest most sophisticated network in the state and we don't have a decent networking test lab. We have an area called the "Shop" but it is just that a "shop" with a bunch of benches and some racks. There is limited protected and UPS power, not much HVAC (except for building HVAC) and is a wide open space. Besides that, generally the area looks like a disaster area with equipment all over the place. Granted it is a good space and is functional for what it was designed for, but not much was considered in the network lab aspect of the space. I cam e on board after most of the decisions were made about this space so I was unable to get my .02 cents in on this one.

We have many instances here at UEN that demonstrate the need for a good lab. Recently we implemented a policy based firewall solution for Box Elder district that in my opinion was a disaster. Now two weeks into it we continue to have problems. All of these issues could have been tried and tested in the lab, but since we don't have the equipment to test on we are forced to use our and Box Elders "production" network as our lab. The result 6-8 hours of down time instead of the anticipated 1-2. EdNet video down for over two days. After the EdNet video fixed, all the libraries down for 2-3 days (6 if you count the weekend).

There are many more examples, but I have already written too much.

Pete recently wrote about our experience in the "lab" here at UEN where we were trying our QoS mechanisms. It took four of us 2-3 hours just to set up the test and we ran out of time before we could do the tests we really needed to do.

One of my next blogs will be dedicated to what I think would be in a functional lab and the resources that would be available.
4:45:09 PM     comment []