The Reality of SaaS


Recently I was asked if SaaS/Cloud computing is appropriate for small practice EHR hosting.

I responded

"SaaS in general is good.

However, most SaaS is neither private nor secure.

Current regulatory and compliance mandates require that you find a cloud hosting firm which will indemnify you against privacy breeches caused by security issues in the SaaS hosting facility.

Also, SaaS is only as good as the internet connections of the client sites.   We've had a great deal of experience with 'last mile' issues"

To add further detail, Bill Gillis, the CIO of the Beth Israel Deaconess Care Organization (BIDCO) responded

"We built, manage & maintain our own private cloud in a Co-location facility.  Our EHR cloud is served to the practice via public internet over SSL. One challenge we struggle with is ISP availability and service level/stability.  In Metro Boston one would expect a robust internet infrastructure.  We've found heterogeneous public internet capabilities and quality of service.  We've found that getting a good ping response is not truly an indicator of meeting application performance requirements.  Many cloud hosted applications are sensitive to latency, packet loss, fragmentation & jitter.

In the first year of our project deployment we struggled because the ISP connectivity did not appear to be the culprit.  A practice would have 10+ megabit connections with ping returns under 25ms.  Yet the practice would experience application freezing, crashes or very poor/slow response time.  From the public ISP's perspective 'the lights were green' and they would take no further action.  After engaging third party network sniffing firm, we discovered the real culprit impacting performance - network latency.  We were able to take the data from that engagement back to the ISP to illustrate the problems with the packets in transit.

Implementing network sniffing engagement was time consuming and costly.  Doing this for the 100+ practice locations we were supporting is not sustainable.  Luckily we found a company in Boston called Apparent Networks (now called Appneta).  Appneta makes a small, low cost black box application that provides deep and detailed network data back to a secure cloud.  We place a device in a practice that communicates back to a device we keep in our hosted/central site.  The devices continually communicate with each other and log all of the various degrees of network performance up to the cloud.  The best part is we preconfigure the devices and mail them to the practices reporting issues.

 All the practice staff need to do is provide power and plug it into an open Ethernet port.  This saves us from deploying a technician on-site.  Since we first deployed these devices we've been able to get to the root cause of performance issues and resolve them rapidly.  We've been able to identify everything from an ISP charging for a certain level of bandwidth while only providing 1/2 that speed to staff streaming media during high volume hours saturating the local router.  The performance data is stored in the cloud indefinitely.  This give us a longitudinal view of the network/internet connectivity for a specific practice.  Recently we were able to avoid a potential issue by noticing that a practice's connection stability was slowly degrading over the past year.  We were able to work with the ISP to discover they had an issue with a local Central Office/substation.  The reality is most ISP's are not that willing to work with us until we show them the data.  Once we have the smoking gun, they tend to dig deeper and work with us to resolve the problems.  For all the high-tech equipment we've leveraged for our private cloud, this device was the real swiss army knife of the project."

I've described Cloud Computing as "your mess run by someone else".   It can be done successfully, but SaaS is only as good as the privacy protections you purchase or build yourself.   Performance is only as good as your network connection.

I hope this is helpful.

Building Unity Farm - Managing the Farm in Our Absence

Paul Harvey wonderfully captured the responsibilities of being a farmer.  Just like being a CIO, being a farmer is not a job but a lifestyle.

When I called my wife and daughter in the hours after my father's death last week, we had to figure out how to manage Unity Farm during a situation that required the absence of the entire family.

We created a workplan and contacted a nearby farm to ask for their help.

Here's what we had to document

Alpaca/Llama care
  Hay management in two indoor and two outdoor feeders
  Twice daily grain feedings
  Heated water bucket filling
  Snow clearing in the alpaca paddocks
  Manure management in the barn, paddocks, and composting area

Chickens/Guinea fowl
  Feeder filling (with multi-flock crumbles)
  Providing greens (romaine lettuce)
  Heated waterer filling
  Coop cleaning
  Opening and closing the coop during daylight hours
  Counting all the birds to ensure they return from their free range adventures
  Turning on the inside lights during the day and off at night (guineas will not roost in a dark coop)

Dogs
  Feeding the dogs in the evening
  Heated water bucket filling
  Providing them snacks and toys

Rabbits
  Lettuce feeding
  Hutch cleaning

General
  Ensuring fences are not damaged by falling tree limbs or snow drifts
  Managing the electric fences
  Ensuring protective lighting is working to keep predators away
  Keep barn doors closed
  Keep property safe

We never expected to all be away from the farm at the same time, so creating detailed instructions required significant short term work.

A farm hand from our adjacent farm did a wonderful job and I returned on Sunday night to find Unity Farm in perfect shape, other than one barn light that was knocked down by a falling tree limb.    Kathy stayed with my mother an extra week and returns to the farm on Saturday.     Just as IT professionals plan for redundancy and disaster recovery,  Kathy and I have learned the importance of contingency planning for the farm.    We have been an inseparable team for 33 years.   Keeping all our professional and personal activities in balance is easy when we're together, but nearly impossible when we're apart.


A Unified Software Development Lifecycle


Recently, in response to an audit, I was asked to document our Software Development Lifecycle across all our platforms - clinical, financial, and web.    Here's what I wrote.  I hope you find it useful.

1.  Project Definition

Multi-stakeholder governance bodies of business owners and IS professionals meet on a regular basis to define the scope and requirements of new projects.    The priorities of these new projects are based on business owner strategic alignment, regulatory/compliance requirements, quality/safety imperative, impact factor (employees, clinicians, patients),  and return on investment.    Governance Committees with oversight over the software development life cycle include:

Clinical - webOMR User's group sets ambulatory development priorities.    Inpatient Clinical Applications Steering Committee sets inpatient development priorities.

Financial/Billing - IS project manager and IS fiscal manager work with Patient Financial Services stakeholders to mutually agree on the work tasks of all programmers.

Financial/Supply Chain Manaagement/Research/ERP -  Human Resources/Payroll, General Accounting, Research Finance and Supply Chain Management Steering Committees set Peoplesoft and related application priorities

Web Applications  - the Portal steering committee sets web development priorities

Once a project is approved by a governance committee, it is assigned a tracking number and assigned to programmer

In addition to project definition, these committees also oversee the portfolio of work their specific domain.    This includes monitoring progress and identifying/eliminating barriers to success.

2. User Requirements Definition, Analysis and Design

The development process for approved projects begins with user requirements definition.   This is a collaborative effort involving developers, analysts and business owners.  It is an iterative process in which a prototype is developed based on an initial set of user requirements and then modified in response to user feedback.   Multiple cycles of revising requirements and prototypes typically occur.   We employ an agile development methodology with source code control systems/versioning for every development platform.

3.   System requirements definition

Application projects may have infrastructure implications.   An infrastructure project manager is engaged if additional server capacity, novel desktop configuration, or new client hardware (mobile, specialized printers, bar code scanners) is required as part of the application

4.  Testing

BIDMC maintains dedicated development and testing environments that are separate from live/production.     Developers first unit test their changes.   Application analysts independently test changes.    End users perform acceptance testing.

Test scripts are used to perform integrated testing before go live.

No code is moved into production before end user signoff approval.

5.  Go live and continuous improvement

Go live is planned in collaboration with business owners which includes communication and post go live support planning.     Changes to infrastructure and communication plans for major go lives are presented to the IS Change Control Board to ensure awareness and coordinate timelines among all IS projects.

The push of code from a testing environment to a production environment is done via a source control system, is logged for auditing purposes, and may be rolled back quickly.   For clinical applications, this push is done by the most experienced developer, as experience has shown that this person is most able to ensure a successful deployment and rapidly identify any defects, minimizing risk.   For financial/billing  applications, this push is done by the non-IS Production Control group, since mainframe workflows are more batch oriented and more amenable to segregation of duties in go live processes.   The organization accepts this difference between the clinical and financial/billing go live process as necessary to reduce overall risk.

Once the go live push is completed, success of the process is validated by the most experienced developer, the analysts, and/or the business owner as is appropriate for the application.

Based on feedback during the go live validation process, a rollback may be done if there are unexpected consequences.  This is a very rare event, but it supported by the source code control systems which drive the go live process.

Once application  changes are live, user feedback is provided at the governance committee level and products are continuously improved to meet users needs, always following the above Software Development Lifecycle.

Thank You to the Village


From March 8 to March 17, I was focused entirely on my father - from serving as his healthcare navigator to arranging his funeral/memorial to ensuring my mother had a path forward.

For 10 days, I had to minimize my roles as a CIO/professor and maximize my roles as son/clinician.

I cancelled numerous meetings, speaking engagements, and classes.  I backed out of commitments made months ago.   My response to calls/emails/texts went from minutes to days.  

All of this was necessary and appropriate to support my father.

Now that I'm back in Boston and restarting my usual schedule, I can say that the past 10 days were only possible because of the incredible outpouring of support I received from the village of people around me.

My wife and daughter flew to Los Angeles to support my father, my mother, and me.

My parents' friends brought food to the hospital, helped with funeral arrangements, and provided emotional support.

My  staff at BIDMC covered for me in all my meetings and phone calls.   Nothing bad happened and no urgent issue was overlooked.

My colleagues in the State and Federal government ensured the cadence of all our work continued without me.

The lessons learned
*Family must come first
*There is no work related urgency that trumps a focus on major life events
*The people who surround you in life make all the difference

Thanks again to the people who supported me.   I've now completed all the tasks surrounding my father's death - from comfort care, to cremation, to memorial, to preparing  the house for my mother's needs, to working on all the financial/administrative matters surrounding the death of a father/spouse.

The healing will take time, but with the great people who came together over the past 10 days, I'm confident that all will be well.

Building Unity Farm - "Planting" the Mushroom Farm

As I mentioned last week, we're planting the orchard and developing a mushroom farm this Spring.

What is the scope and scale of the mushroom farm effort?

We've cut 200 feet of poplar trees that were too near our buildings for safety.   I chainsawed the trees into 12 four foot logs that 6-8" in diameter and 60 two foot logs that are 8-12" in diameter.  

Poplar is an ideal substrate for oyster mushrooms.    Our mushroom farm supplier, Field and Forest Products, recommends the "totem" method for inoculating poplar.

Here's what I'm planning for the last week of April.

I've cut 192 feet of pine 2x4s into 16 inch segments.  These boards will serve as the bases for 72 "totem poles".     I'll cut the 4 foot logs into three 16 inch segments.   I'll cut the 2 foot logs into 12 inch segments.   I'll lay down the 2x4 bases every 33 inches to create a 200 foot line in the moist and shaded area of our north wood.   I'll place a large trash bag on top of each base the generously add sawdust inoculated with oyster mushroom mycelium (spawn).   I'll add a log, add more spawn, add a log, and seal the totem pole in the trash bag to establish the perfect environment for growing mushrooms.

Field and Forest Products supplies 6 different subspecies of oyster, so we'll inoculate 12 totems with each type.

I've cut 25 four foot oak 3-6" oak logs from trees damaged during Hurricane Sandy for Shitake growing.   As we clear an acre for the orchard, we'll have enough oak for 220 logs arranged as 11 stacks of 20 logs.   I'll place one stack every 18 feet in the 200 foot mushroom growing area of our north wood.   I've cut 88 feet of pine 4x4s into four foot segments to serve as bases for the stacks.

Field and Forest Products supplies 11 different subspecies of Shitake, so we'll inoculate each stack with a different species.  Inoculating requires drilling 1.5 inch deep holes every 4 inches around the entire circumference and length of each log.    For 220 four foot logs, that means  220 logs * 4 feet/log * 12 inches/foot * 1 hole/4 inches for each row around the circumference * 4 rows  = 10,560 holes.

How do you use a drill to make 10,560 holes?   You don't.   You use an 8000 rpm grinder retrofitted with a drill chuck and high speed bit.   In my case, I'm using the Makita 9557pb with the Field and Forest chuck that attaches to the 5/8" inch coarse threads of the grinder.

I will fill these holes with sawdust spawn using a special inoculator then seal the holes with 25 pounds of melted food grade paraffin.

The entire process for processing the poplar and oak logs will be done in three weekends at the end of April and the beginning of May.

I've completed the survey work, the brush clearing, and layout of the mushroom farm.  I have all the tools and technologies I'll need.  The rest is the muscle power to process a few tons of wood into a production configuration.

We may see some fruiting this Fall, but likely next Spring we'll have our first mushroom harvest.   Like our Orchard, we've chosen subspecies that fruit at different times in different conditions so we'll have yields throughout the season when the temperature is over 40 degrees.   If we're lucky this batch wood of yield for many seasons, so the "heavy lifting" only needs to be done every 5 years.