Tuesday, March 4, 2008

The CareGroup Network Outage

On November 13, 2002 at 1:45pm, Beth Israel Deaconess Medical Center went from the hospital of 2002 to the hospital of 1972. Network traffic stopped. No clinician could order medications or labs electronically. No decision support was available. Luckily, no patients were harmed. Here's the story of what happened and our lessons learned.

In the years after the 1996 merger of the Beth Israel and Deaconess Hospitals, operating losses caused several years of capital starvation (the network budget in 2002 was $50,000 for the entire $2 billion dollar enterprise). We were not available to invest in infrastructure, so our network components were beyond their useful life.

However, it was not this infrastructure underinvestment that was the root cause of the problem, it was my lack of enterprise network infrastructure knowledge. I did not know what I did not know. Here are the details:

1. Our Network topology was perfectly architected for 1996. Back in the days when the internet was a friendly place where all internal and external collaborators could be trusted, a switched core (layer 2) that transmitted all packets from place to place was a reasonable design. After 1996, the likelihood of denial of service attacks, trojans, or other malware meant that networks should be routed and highly segmented, isolating any bad actors to a constrained local area. At the time of our outage, a data flood in one part of the network propagated to every other part of the network - a bit like living downstream from a dam that collapsed. A well meaning researcher created a napster-like application that began exchanging hundreds of gigabytes of data via multicast to multiple collaborators. The entire network was so saturated that we could not diagnose the root cause of the data flood. I did not know that a switched core was a point of failure.

2. Our Network team was manged by a very smart engineer who did not share all his knowledge with the network team. Much of our network configuration was poorly documented. With the knowledge of our network isolated to one person, we had a single point of human failure. I did not know that this engineer was unfamiliar with the best practices for routed/redundant network cores, routed distribution layers and switched access layers isolated into vlans with quality of service configurations to prevent monopolization of bandwidth by any one user or application. We brought in a Cisco partner, Callisma, to document the network, but the network failure occurred before they were finished.

3. I did not know about spanning tree algorithms, hot standby routing protocols (HSRP), and Open Shortest Path First (OSPF). During the outage, I approved configuration changes that actually made the situation worse by causing spanning tree propagations, flooding the network with even more traffic.

4. I did not establish a relationship with the vendor (Cisco) that enabled them to warn me about our vulnerabilities. A relationship with a vendor can take many forms, ranging from a sales driven vendor/client adversarial relationship to a collaborative partnership. In 2002, Cisco was just another vendor we purchased from. Today they are a collaborative partner with a seat at the table when we plan new infrastructure.

5. I did not know that we needed "out of band" tools to gain insight into the problems with the network. In effect, we required the network to be functional to diagnose problems with the network.

6. We did not have a robust, tested downtime plan for a total network collapse. When the outage occurred, we rapidly designed new processes to transport lab results, orders, and other data via runners from place to place.

7. We did not have a robust communication plan for responding to a total network collapse. Email, web-based paging, portals, and anything that used the data network for communication was down. Voice mail broadcasts using our PBX and regular phones (not IP phones) turned out to save the day.

8. When we diagnosed the problem, we explored many root causes and made many changes in the hope that we'd find the magic bullet to cure the problem. In the end, we ended up fixing many basic structural problems in the network, which took 2 days and eventually solved the problem. A more expedient solution would have been to reverse all changes we made in our attempts to fix the network once we had stopped the internal attack. When a crisis occurs, making changes on top of changes can make diagnosis and remediation even more difficult.

9. We did not have an enterprise wide change control process to ensure that every network configuration, server addition, and software enhancement was documented and assessed for impact on security, stability, and performance. Today we have a weekly change control board that include includes all IS departments, Cisco engineering services, and IS leadership to assess and communicate all configuration changes.

10. I was risk averse and did not want to replace the leadership of the network team for fear that terminating our single point of human failure would result in an outage. The price of keeping the leadership in place was a worse outage. I should have acted sooner to bolster leadership of the team.

Despite the pain and stress of the outage, there was a "lemons to lemonade" ending. Without this incident, the medical center would never have realized the importance of investing in basic IT infrastructure. If not for the "perfect storm", we may have limped along with a marginal network for years.

Today, we receive annual capital funding to support a regular refresh of our technology base, and we are asked to introduce change at a pace that is manageable. People in the medical center still remember the outage and are more accepting of a tightly managed infrastructure such as locked down workstations, security precautions, disaster recovery initiatives, and maintenance downtime windows.

During and immediately following the event, I presented this overview of the outage to senior management. Shortly after the outage, I worked with CIO Magazine to document the events and lessons learned.

I hope that my experience will help other organizations prevent network and other IT outages in the future.

Electronic Health Records for non-owned doctors -scalable infrastructure

This is my fifth entry about our Electronic Health Record project for non-owned doctors. As I've described, the scope of the project is to implement a highly reliable, secure, feature rich, well supported, but affordable electronic health record for private practices. Today's entry is about building the scalable centralized Software as a Service hosting infrastructure to meet these goals.

A key design requirement for the project is scalability. Our projected customer base is 300 clinicians and we have a fixed start up budget. However, we must design the infrastructure in a way that can cost effectively support the smallest amount of adopters as well as scale to thousands if our project is wildly successful. We debated two possibilities (metaphorically speaking)

a. Build a hotel, not knowing if anyone will ever check-in
b. Build a housing development, where the limits of expansion are only defined by available land

We decided on choice "b", starting with a robust foundation and adding new equipment and storage as we add clinicians. We standardized our central site equipment on products from HP EMC and Cisco, with guidance from our infrastructure partner, Concordant, and our equipment supplier CDW, ensuring it was easy to plug in additional hardware on demand. We invested a significant amount of time designing the central hosting facility, doing it right the first time. Over the years, I've seen CIOs rush through the design phase, only having to rebuild the infrastructure later when application performance did not scale. We partnered with our vendors to build something special, that if successful, could be a model for other medical centers and communities.

Considerations in designing our hosting infrastructure included:
* Supporting a user base that is remote, unmanaged, and diverse. We need to be able to identify any performance issues via end to end monitoring of all components
* Meet important security and privacy restrictions, as well as address liability issues (who is responsible for what)
* Understand infrastructure costs for a) start up b) additional capacity that occurs in bursts or steps, and c) variable requirements as practices go live.
* Provide connectivity to external parties (labs, claims, etc...) through interfaces which create additional security and performance complexities
* Address the limitations and performance of "last mile" connectivity through publicly available internet access (DSL, cable, etc.)

The infrastructure choices we made are:

Virtualized servers - VMware was the natural choice because of the scalability design goals. VM and V-Motion technologies also play an important part in redundancy, failure recovery and disaster recovery

Physical services - We debated rack mounted verses blade servers and elected to use powerful small footprint HP rack servers connected to fast multi-tiered storage. We computed the economics of blade servers verses rack mounted servers and the use of VMWare made small powerful rack servers the most cost effective solution.

Storage - We purchased a Clariion CX3-20 series SAN. We will go live with 11.1TB of total storage (2.1TB of fast, Tier 1 storage for database transactions and 9TB of secondary, Tier 2 storage for files). A single CX3-20 will allow us to expand in a modular fashion to accommodate up to 1200 practices. We'll also be leveraging a disk to disk backup strategy, using tape only for disaster recovery.

Network Infrastructure – We implemented a high speed network backbone with multiple paths for redundancy using
* Cisco Integrated Services Routers (ISR) 2811s for internet connectivity
* Adaptive Security Appliances (ASA) 5520 for Firewalling, Intrusion Protection and IPSec VPN Client termination
* Catalyst 4948 Switches for Server connectivity and layer 3 routing
* MDS 9000 Series Multilayer SAN Switches for SAN connectivity

Security – We incorporated physical, technical and administrative controls to protect confidentiality, integrity and availability.

SSL Accelerators - We are using Array Networks TMX-2000, the hardware recommended by eClinicalWorks to optimize web server performance.

Redundancy & Disaster Recovery - One of the real challenges to this project is the price sensitivity of our private clinicians. We needed to build a world class system at a price that all clinicians could afford. Redundancy and disaster recovery is like life insurance - it's a great investment only if you need it. We had to balance our infrastructure investment with total cost of ownership, given the fixed hospital contribution and physician frugalness. In the end we used the equation

Risk=likelihood of bad events * impact of bad events.

We believe that it is much more likely that a component will fail than an entire data center be destroyed, so we elected to build a highly redundant infrastructure in a single data center for now, expanding to a secondary data center once we have sufficient clinicians signed up to fund the new infrastructure. Networking gear, servers, power and cabling are duplicated within a commercial co-location facility. Storage is disk to disk redundant. Tapes are moved offsite nightly. Once the hardware is up, we'll work with Concordant, Cisco, EMC, HP, Array Networks, and the Co-Location facility to test physical hardware and operating system/database software redundancy. Then we'll install eClinicalWorks and run the redundancy tests again. We'll also engage Third Brigade at that time for intrusion/security testing.

We've written a comprehensive disaster recovery plan and if we lost the co-location facility due to disaster, we would recover the tapes from offsite storage and build a replica of the hosting environment (VMWare plays a key role here) and restore the data. The recovery point and recovery time objectives for this plan will be clearly communicated to all who sign up for service. The customer base for our Software as a Service solution is mainly small practices which operate Monday through Friday 8am-6pm. Our disaster recovery plan includes a solid practice/workflow specific contingency/downtime plan. We will also perform a mock downtime as part of each implementation.

By creating a highly redundant single data center with rack mounted servers, two tiered storage, virtualization and offsite tape backup, we believe we've balanced scalability, affordability, and maintainability. We go live this Summer and I'll let you know if we were right!

Monday, March 3, 2008

Women, Now You Can...PEE STANDING UP!

Thanks to Rich in Boston for alerting me to the fact that women can now pee ("urinate" for you purists) standing up. You might think it is gross to talk about, but for women who are tired of laying toilet paper on the seats or using their quads until they tremble with fatigue, this is a novel idea. I like the handy little cardboard. Hope it is biodegradable, just like pee.

Sunday, March 2, 2008

You're the Doctor - Answer

The patient in the photo below has Poison Oak (Rhus dermatitis). Don't feel bad if you missed it. She had 4 medical visits before it was successfully diagnosed.Poison Oak is a form of a contact dermatitis (good work, "erp") and she actually contracted it while hiking in Point Reyes, California. Once the urushiol oil from the plant sets on the skin, an inflammatory and allergic response keeps

Saturday, March 1, 2008

Running an effective meeting

Today marks a blogging milestone for me, my 100th daily post. Thanks for your support and comments along the way.

My 14 year old daughter and I recently talked about my jobs. I gave her the long explanation about what I do - strategy, structure, staffing, and process optimization. She asked for the elevator speech version and I said, "Basically, I run a lot of meetings."

Running an effective meeting takes a lot of energy. The meeting organizer is responsible for the logistics, bringing together the right people, ensuring all stakeholders are heard, managing the interactions, and documenting the results. Many meetings are not a good use of time. Here's columnist Dave Barry's analysis:

"Compare the modern corporate meeting to a funeral, in the sense that you have a gathering of people who are wearing uncomfortable clothing and would rather be somewhere else. The major differences are 1. Usually only one or two people get to talk at a funeral 2. Most funerals have a definite purpose (to say nice things about a dead person) and reach a definite conclusion (this person is put in the ground); whereas meetings generally drone on until the legs of the highest ranking person fall asleep."

Here's my guidance to running an effective meeting:

1. Organize the meeting in the most painless way possible
Outlook invitations work well for folks who have a lot of flexibility in the calendars or who are always working in a single location. For executives who attend meetings 12 hours a day and who travel between several corporate locations, automated invitations just do not work. I cannot be in Los Angeles for breakfast and Boston for lunch. Outlook invitations do not take into account travel time, location, and the general pace of the day. In my view, there are three effective ways to organize a meeting

a. Propose 4 or 5 possible dates/times and circulate them via email among the attendees/administrative assistants. Determine the "must have" attendees at the meeting. Based on the best fit of the "must haves", select a date/time. Just about all meetings can be organized this way.
b. If urgency is required, get all the admins on a conference call and do a real time reconciliation of calendars for the best fit. A use case for this is a meeting to discuss a strategic opportunity with time sensitivity.
c. If real urgency is required, just set a date and time and ask everyone to cancel their other commitments. A use case for this is a Joint Commission visit. Just drop everything you're doing.

Using endless "reply to all" emails among a large group of attendees to schedule a meeting generally does not work. For a senior leader, trying to organize a meeting yourself, without administrative coordination is very problematic, since there is a lot of communication required to find the best fit among many schedules.

2. Food and drink are real motivators to attend a very early, lunch time or very late meeting
Be kind to your attendees. They are giving their time to you and have very busy schedules. We are all under such time pressure that the only near term free times in our calendars are breakfast, lunch and dinner. If you use those times, bring refreshments.

3. Arrive early and begin the meeting promptly
Meetings take a lot of energy to organize. Time is the one commodity that people cannot make more of. If the meeting organizer arrives late, it's a sign of disrespect. Begin and end the meeting on time. The attendees will appreciate it.

4. Circulate meeting materials including a formal agenda, ahead of time
I've attended several meetings without a clear understanding of who's attending, the purpose of the meeting, the desired outcome, and any preparation I should do ahead of time. Circulate an agenda a few days before the meeting containing

a. The attendees, location, and call in number (for remote attendees)
b. The overall purpose of the meeting
c. The items to be discussed, identifying the main presenters
d. The background briefing materials to help with decisionmaking

The meeting is likely to be much more productive.

5. Ensure everyone has a chance to speak and interacts professionally
As the master of ceremonies of the meeting, the convener must prevent hegemony of any one speaker and should rapidly quash any emotional outbursts. It's fine to criticize ideas in a meeting but not to criticize or attack a person. I often use humor, real time compromise, and personal stories of similar past controversies to bring out the best behavior in everyone. If I know that a meeting has potental landmines, I'll openly state them at the beginning of the meeting and give permission to everyone to openly discuss them in a non-judgemental forum.

6. Put your Blackberry away
During long meetings that I run, I will very occassionally (once an hour) scan the inbox on my Blackberry for downtime messages, urgent notices, email from the CEO etc., but I will never use the Blackberry to respond to an issue while I am running a meeting.

7. Adhere to the agenda
Running a meeting requires focus. If there are tangential discussions, bring the focus back to the agenda. If there are private sidebar conversations, bring them into the whole meeting. The only way for the meeting to move forward is with everyone listening and progressing through the agenda.

8. Do a mid meeting summary
Halfway through the meeting, I take a checkpoint. Are we on track, what have we decided thus far? I summarize the main points of agreement and the remaining items on the agenda, so we can bring closure to the work we've done thus far and move forward.

9. Own the last two minutes
The last two minutes of the meeting are the most important. The convener should review all the decisions made, the next steps decided upon, the resources committed, the timelines, the deliverables, and most critically, the paths not chosen i.e. we will not do this project, spend this money, pursue that opportunity. By ending the meeting with an overall summary, all will leave the meeting with a common understanding of the consensus, even if some do not agree with it. The last two minutes can make or break the meeting by ensuring there is no ambiguity about the meeting's results. With a clear summary, all participants will understand the value of the meeting and their role in the meeting events.

10. Send a followup email summary with minutes
Even with a great last two minute summary, some stakeholders will have selective memory. It's best to memorialize the last two minutes of the meeting in meeting minutes, circulated to all meeting attendees and stakeholders. Also, take the time to thank those who attended the meeting and praise them for their work.

By following these steps, you maximize your meeting time, optimize communication, and maybe even reduce the number of meetings needed because of your remarkable efficiency running the process!

You're the Doctor- Whats the Diagnosis?

This young woman developed this skin condition on her leg after a trip to the Grand Canyon. She was treated with two different antibiotics but it worsened. The winner gets a years free subscription to EverythingHealth. Oh, yea...it's already free. OK the winner gets bragging rights.Answer will be posted tomorrow.

Ricin - what you should know

The poison Ricin was found in a Las Vegas Hotel room and one man is hospitalized and unconscious after exposure. Because it takes a deliberate act to poison with Ricin, this brings up concerns about National Security and many questions that are not yet answered.Ricin is the "waste" mash left after Castor beans are processed into castor oil. Except for cancer research, there are no medical uses