DESperately Seeking Simulation - Tips and Potential Pitfalls

Author
Affiliation

Sammi Rosser, Dr Dan Chalk

Health Service Modelling Associates Programme Lambda

How to Make a Good Model

A model is doomed to fail if it’s not designed well.

So how can you help ensure that you are enabling your analysts to make good models?

Hopefully from what you’ve seen so far, you’re able to see applications of DES in your own system, and keen to implement it.

But how can you maximise the chance of the project being successful?

Understanding the Process

What is the process - really?

  • You’re setting the model up for failure if you don’t have the experts in the process in the room from the start

  • Even then, it can be complex to work out what to model and what to simplify

    • But the scenarios you want to test can really influence this!

  • And do you actually hold the data you need to parameterise the model?

    • Expert opinions can be used where data is missing, but this needs to be documented!

Measuring Up

  • From the beginning, you need to be thinking about what metrics/KPIs do you want to be able to measure
  • How are you going to measure that the model reflects the current reality?

Scenarios

  • What do you need to be able to change?
    • and what can you actually change in your system?
  • Will the simplified model logic support the scenarios you want to test?
  • How do you want to be able to compare scenarios?
    • looking in-depth at scenarios one at a time?
    • or displaying outputs side-by-side in a web app?

Using it for decisions

  • Is this a one and done output, or an ongoing tool?
  • What do you need to be able to export?
  • Have you asked the modeller how they are assessing it against the real system, and refine it?

Understanding the limits

A model is not a crystal ball!

It should generally be thought of as presenting a ‘direction of travel’ - not exact estimates.

As well, a common mistake amongst those newer to modelling is to assume that

  • models need to capture everything (and in detail)
  • and that a more “realistic” model is a better one

Model Detail and Scope

“All models are wrong, but some are useful.”

  • George E.P. Box

(a British Statistician considered one of the greatest statistical minds of the 20th century)

Tube Map

This model is wrong…

Actual Layout


but very useful!

Model Simplifications

All models are built on assumptions and simplifications.


Assumptions are things we have to include because we don’t or can’t know something

Simplifications are things in a model which we choose to represent in less detail than the real world equivalent because the added detail won’t give us added benefit

As models are essentially collections of assumptions and simplifications, the more of the real world we choose to capture, the more potential inaccuracy we introduce!

Scope and Detail

When designing a model, the modeller needs to consider scope and level of detail.

Scope determines which section of the real world is carved out and represented in the model - where are the boundaries?

Level of detail is how much of the real world detail is represented vs simplified

Imagine you’re modelling an ED to identify strategies to reduce waits.

Does modelling all the tasks a nurse does during a triage give you anything above simplifying to a “triage” process that takes x amount of time?


If I want to use the model to explore changing some of those processes then maybe…

But if it will just change the overall amount of time with the nurse, it’s equally effective to say “What if the average triage time was reduced by 2 minutes?”

The more complexity and assumptions you’re building into that unnecessary task breakdown, the more risk you have of making your model not representative!

A good rule of thumb



Build the simplest model to sufficiently answer the question for which the model is being used


If extra scope or detail is needed, how can its representation be simplified?

How soon do you need it?

How long does a model take?

  • A good model can take time to create
    • it’s not quite like a regular analysis
  • You can help it go faster by carefully considering the things on the preceding slides
  • Set your analysts up for success!
  • A rushed model can do more harm than good
  • But it’s generally still quicker than trialling the real world changes!

No, really, how long?

  • A simple model can go together pretty quickly (hours to days)
    • but your analysts may need time to upskill in R or Python…
    • and upskill in DES-specific concepts…
    • and add in a Streamlit frontend…
    • and clear visuals…
    • and animations…
    • and more complex inputs and scenarios…
    • and the ability to compare things side-by-side…
    • and the ability to download a summary table…
    • and robust tests…

Summary

In reality, a good model may take weeks to months - and may be iterated on for years

But the time and effort can be worth it!


“I can give you an answer by this afternoon that’s probably wrong and you’ll probably have to ask me again next week.

Or I can build a model, which will take a bit longer, but you’ll probably never need to ask me that question again.”

  • One of our HSMAs from 2016 to their senior managers