Downtime in 2002 verses 2013

On November 13, 2002, the network core at Beth Israel Deaconess failed due to a complex series of events and the hospital lost access to all applications.   Clinicians had no email, no lab results, no PACS images, and no order entry.    All centrally stored files were unavailable.   The revenue cycle could not flow.   For 2 days, the hospital of 2002 became the hospital of 1972.  Much has been written about this incident including a CIO Magazine article and a Harvard Business School case.

On July 25, 2013, a storage virtualization appliance at BIDMC failed in a manner which gave us Hobson's choice  - do nothing and risk potential data loss; or intervene and create slowness/downtime.   Since data loss was not an option, we chose slowness.  Here's the email I sent to all staff on the morning of July 25.

"Last evening, the vendor of the storage components that support Home directories (H:) and Shared drives (S:) recommended that we run a re-indexing maintenance task in order avoid potential data corruption. They anticipated this task could be run in the middle of the night and would not impact our users.   They were mistaken.

The indexing continues to run and must run to completion to protect H: and S: drive data.  While it is running, access to H: and S: will be slow, but also selected clinical web applications such as Provider Order Entry, webOMR, Peri-operative Information System, and the ED Dashboard will be slow.  Our engineers are monitoring the clinical web applications minute to minute and making adjustments to ensure they are as functional as possible.   We are also investigating options to separate clinical web applications from the storage systems which are causing the slowness.

All available IS resources are focused on resolving this as soon as possible.  We ask that all staff and clinical services affected by the interruption utilize downtime procedures  until the issue is resolved.  We apologize for the disruption this issue has caused to patients, providers, and staff."

2002 and 2013 were very different experiences.   Here's a brief analysis:

1.  Although 2002 was an enterprise downtime of all applications, there was an expectation and understanding that failure happens.   The early 2000's were still early in the history of the web.   There was no cloud, no app-enabled smartphones, and no universal adoption of social networking. Technology was not massively redundant.  Planned downtime still occurred on nights and weekends.

In 2013, there is a sense that IT is like heat, power, and light - always there and assumed to be high performing.   Any downtime is unacceptable as emphasized by the typical emails I received from clinicians:

"My patients are still coming on time and expect the high quality care they normally receive. They also want it in a timely manner.  Telling them the computer system is down is not an acceptable answer to them.   Having an electronic health care record is vital but when we as physicians rely on it and when it is not available, it leads to gaps in care."

"Any idea how long we will be down? I am at the point where I may cancel my office for the rest of the day as I cannot provide adequate care without access to electronic records."

In 2013, we've become dependent on technology and any downtime procedures seem insufficient.

2. The burden of regulation is much different in 2013.  Meaningful Use, the Affordable Care Act, ICD10, the HIPAA Omnibus rule, and the Physician's Quality Reporting System did not exist in 2002.   There is a sense now that clinicians cannot get through each day unless every tool  and process, especially IT related, is working perfectly.

Add downtime/slowness and the camel's back is broken.

3.   Society, in general, has more anxiety and less optimism.    Competition for scarce resources  translates into less flexibility, impatience, and lack of a long-term perspective.

4.  The failure modes of technology in 2013 are more subtle and are harder to anticipate.

In 2002, networking was simple.  Servers were physical.  Storage was physical.  Today, networks are multi-layered.  Servers are virtual.  Storage is virtual. More moving parts and more complexity lead to more capabilities but when failure occurs, it takes a multi-disciplinary team to diagnose and treat it.

5.  Users are more savvy.   Here's another email:

"Although I was profoundly impacted by today's events as a PCP trying to see 21 patients, I understand how difficult it is to balance all that goes into making a decision with a vendor on hardware/software maintenance. However, I was responsible for this for a large private group on very sophisticated IT, and I would urge you to consider doing future maintenance and upgrade projects starting on Friday nights, so as to have as little impact as possible on ambulatory patient care."

My experience with last week's event will shape the way I think about future communications for any IT related issues.    Expectations are higher, tolerance is lower, and clinician stress is overwhelming.    No data was lost, no patient harm occurred, and the entire event lasted a few hours, not a few days.     However, it will take months of perfection to regain the trust of my stakeholders.

It's been 10 years since we had to use downtime procedures.   We'll continue to reduce single points of failure and remove complexity, reducing the potential for downtime.   As a clinician I know that reliability, security, and usability are critical.   As a CIO I know how hard this is to deliver every day.

Building Unity Farm - Scenes of Summer

Unity Farm is at the peak of Summer.  Everything is in bloom, the forest is bursting with wildlife, and all our outdoor activities are in full swing as we finish creation of our growing areas before we retreat inside for 6 months of winter.   Here are a few photos with the scenes of Summer at Unity Farm.

1.  Afternoon thunderstorms pop up during the hot and humid weather.    They skies are filled with billowing clouds that dwarf the barn and paddocks.

2.  The animals cling to their barn fans, run through the sprinklers and enjoy an afternoon snack of chilled romaine lettuce to keep cool

3.   In the stream, Muskrat Susie and Muskrat Sam whirl and twirl among the reeds.  (I know the song is awful)

4.  Mom and Dad proudly walk our country lane with their new fawn

5.  Guinea fowl build nests of 20-30 eggs in the deepest part of our fern forests

6.  The Great Pyrenees enjoy playing in the shade with their new ball toy

7.  The bees are storing honey for the winter.   Here's a closeup of the queen from one of our 8 hives

8.  The orchard grass has gone to seed and needs mowing .   I maintain the orchard with a push mower and a trimmer for more delicate edge work.   Here's a view of the mowing in progress and the finished result.

9.  Ground hogs (also know as Woodchucks) nibble at the grass in the meadow.

10. Garter snakes sun themselves on the rocks in the garden

The Era of Epic

In the Boston marketplace, Partners Healthcare is is replacing 30 years of self developed software with Epic.   Boston Medical Center is replacing Eclipsys (Allscripts) with Epic.   Lahey Clinic is replacing Meditech/Allscripts with Epic.  Cambridge Health Alliance and Atrius already run Epic.   Rumors abound that others are in Eastern Massachusetts are considering Epic.  Why has Epic gained such momentum over the past few years?   Watching the implementations around me, here are a few observations

1.  Epic sells software, but more importantly it has perfected a methodology to gain clinician buy in to adopt a single configuration of a single product.   Although there are a few clinician CIOs, most IT senior management teams have difficulty motivating clinicians to standardize work.  Epic's project methodology establishes the governance, the processes, and the staffing to accomplish what many administrations cannot do on their own.

2.  Epic eases the burden of demand management.   Every day, clinicians ask me for innovations because they know our self-built, cloud hosted, mobile friendly core clinical systems are limited only by our imagination.   Further, they know that we integrate department specific niche applications very well, so best of breed or best of suite is still a possibility. Demand for automation is infinite but supply is always limited.   My governance committees balance requests with scope, time, and resources.   It takes a great deal of effort and political capital.   With Epic, demand is more easily managed by noting that desired features and functions depend on Epic's release schedule.   It's not under IT control.  

3.  It's a safe bet for Meaningful Use Stage 2.   Epic has a strong track record of providing products and the change management required to help hospital and professionals achieve meaningful use.  There's no meaningful use certification or meaningful use related product functionality risk.

4.  No one got fired by buying Epic.   At the moment, buying Epic is the popular thing to do.   Just as the axiom of purchasing agents made IBM a safe selection,   the brand awareness of Epic has made it a safe choice for hospital senior management.   It does rely on 1990's era client server technology delivered via terminal services that require significant staffing to support, but purchasers overlook this fact because Epic is seen in some markets as a competitive advantage to attract and retain doctors.

5.  Most significantly, the industry pendulum has swung from best of breed/deep clinical functionality to the need for integration.   Certainly Epic has many features and overall is a good product.   It has few competitors, although Meditech and Cerner may provide a lower total cost of ownership which can be a deciding factor for some customers.   There are niche products that provide superior features for a department or specific workflow.   However,  many hospital senior managers see that Accountable Care/global capitated risk depends upon maintaining continuous wellness not  treating episodic illness, so a fully integrated record for all aspects of a patient care at all sites seems desirable.  In my experience, hospitals are now willing to give up functionality so that they can achieve the integration they believe is needed for care management and population health.

Beth Israel Deaconess builds and buys systems. I continue to believe that clinicians building core components of EHRs for clinicians using a cloud-hosted, thin client, mobile friendly, highly interoperable approach offers lower cost, faster innovation, and strategic advantage to BIDMC.  We may be the last shop in healthcare building our own software and it's one of those unique aspects of our culture that makes BIDMC so appealing.

The next few years will be interesting to watch.   Will a competitor to Epic emerge with agile, cloud hosted, thin client features such as Athenahealth?   Will Epic's total cost of ownership become an issue for struggling hospitals?   Will the fact that Epic uses Visual Basic and has been slow to adopt mobile and web-based approaches provide to be a liability?

Or alternatively, will BIDMC and Children's hospital be the last academic medical centers in Eastern Massachusetts that have not replaced their entire application suite with Epic?   There's a famous scene at the end of the classic film Invasion of the Body Snatchers, which depicts the last holdout from the alien invasion becoming a pod person himself.  At times, in the era of Epic, I feel that screams to join the Epic bandwagon are directed at me.

Image Exchange

Last week, the Clinical Operations Workgroup of the HIT Standards Committee held its third hearing on image exchange, seeking testimony from Hamid Tabatabaie, CEO of LifeImage and Michael Baglio, CTO of LifeImage.

He made several important points
1.  We should think of image exchange as having two major categories - local and long distance.    DICOM works well for modality to PACS connectivity within an enterprise (local).   DICOM was never designed for internet-based cross organizational image sharing.   DICOM images tend to be large, including a vast amount of metadata with every image object in a study.    DICOM was also never designed to work well with the kind of firewalls, load balancers, and network security appliances we have today.

2.  Two image exchange architectures have been used in the marketplace to date, which Hamid called "iTunes" and "Napster",  classifications first suggested by Dr. Keith Dreyer.

iTunes refers to the centralization of images into a single repository or what a appears to be single repository - it may actually be a clearinghouse to other image stores, but the user never knows that.   Query/response transactions against this central repository can be straightforward, using standards such as Blue Button Plus/Direct for share, access, download.

Napster refers to a decentralized, federated model in which images are not placed in a single repository -    an index of images and their location is all that is centralized.   Typically, query/response is done with standards such as XDS-i.   XDS itself was never designed for image exchange and is incomplete.  It can be challenging to search for a single exam on a known date of a known modality type.

3. Current standards do not include any privacy metadata and security is not built in.  Future standards should enable applications to restrict image flows based on consent/patient preferences.

4.  We need a web friendly method for visualization that does not require the download of a proprietary viewer, one that is often operating system specific.   Consumers should be able to view thumbnails of images on a smartphone, tablet, or the device of their choosing without special software.   If the full DICOM object is needed (patient mediated provider to provider image exchange), download and transmission should be enabled using standards such as REST, OAUTH2/OpenID, and secure email.

5.  Hamid made a forward looking statement that should be carefully considered as we plan the lifecycle of existing Radiology Information Systems (RIS) and Picture Archiving and Communication Systems (PACS) systems.   He is seeing EHR features expand to cover many aspects of RIS workflow.   If scheduling, image viewing, report construction with templates/front end voice recognition, and easy exchange of reports with clinicians are supported by the EHR, maybe radiologists (some of which want to qualify for meaningful use payments) will start using increasingly capable EHRs instead of RIS.   Vendor neutral archives (VNA) which store images of all "-ologies"  and enable easy search and retrieval may replace PACS.   Imagine 5 to 10 years from now when RIS/PACS no longer exists and is replaced by EHR, HIE,  and VNA.   Interesting possibility.


Great testimony.    In the past when I've suggested DICOM is not ideal for internet-based multi-organizational exchange, I've been criticized.   In the past when I've suggested that DICOM has issues of vendor-specific proprietary metadata extensions, cumbersome viewers, and heavy payloads, I've been challenged.   It's refreshing to hear from someone doing the hard work of high volume image sharing that current standards not ideal.  We need new approaches to move payloads efficiently on the internet, view images via web-browsers, facilitate easy searching, support security, and enable multiple provider/patient/group sharing use cases.

Building Unity Farm - The Barn Swallows of Unity Farm

One of the side effects of creating the orchard at Unity farm was opening about 2 acres of airspace adjacent to our pasture.  This has attracted  many new species of birds which now dance and dive in the clearing between our woodland and marsh.   Since adding the orchard we've seen a significant increase in our barn swallow population.  Throughout the day, at least 5 adults dash at high speed around their own open air playground, eating mosquitos and enjoying their social community of birds.

Our barn has two sliding doors which open to the male and female paddocks.  The swallows retreat to the barn for shelter at night and during the rain.   Recently a mating pair built a nest on top of a porcelain light socket.   We use LED lightbulbs in the barn, so the socket does not get hot.   The nest is a delicate combination of mud, sticks, great pyrenees fur, and feathers from our chickens/guinea fowl, pictured above.

This week, 4 swallow babies are fledging.  Two are pictured below balanced on a barn door rail.   Watching the parents feed them for the past several weeks, I've read a great deal about swallows and can now answer some lifelong questions I've had since 1975, when King Arthur first had a dialog about swallows with the guardian of the bridge.

1.  Are the swallows of Unity Farm considered European or African?

European swallows are migratory and are widespread throughout the Northern Hemisphere.   African swallows are non-migratory and are typically found in in Botswana, Republic of the Congo, Democratic Republic of the Congo, Gabon, Lesotho, Malawi, Namibia, South Africa, Zambia, and Zimbabwe.    Thus, if asked, you can definitively answer that the barn swallows of Unity Farm are European.

2.  Much of the day the adult swallows gather insects to feed to their young.  I've watched them busily carrying food and nesting materials as the cruise through the paddocks and into the barn.     When they're done feeding their babies and reinforcing the nest, they fly above the paddock turning and twisting at high speed before deftly returning to the barn.   Watching them makes me wonder - just how fast do they fly when unencumbered i.e. What is the airspeed velocity of an unladen swallow (European)?

Luckily much research has been done on this topic.


A 54-year survey of 26,285 European Swallows captured and released by the Avian Demography Unit of the University of Capetown finds that the average adult European swallow has a wing length of 12.2 cm and a body mass of 20.3 grams.

European Swallow flies at cruising speed with a frequency of roughly 15 beats per second, and an amplitude of roughly 22 cm.   However, some other researchers have measured a lower frequency of 7-9 beats per second among some swallows.

Because wing beat frequency and wing amplitude both scale with body mass and flight kinematic data is available,  we can  estimate airspeed (U).

Graham K. Taylor et al. show that as a rule of thumb, the speed of a flying animal is roughly 3 times frequency times amplitude (U ≈ 3fA).

Based on wing beats per minute, body mass, and amplitude, the answer for our swallows appears to be 24 miles per hour (11 meters per second).   So if on your quest to visit Unity farm, you are asked the airspeed velocity of an unladen swallow (European), you know the answer!