Technology AI

Running a four-week AI call handling pilot

A week-by-week plan for piloting AI call handling in a UK practice: scope, baseline, measures, stop rules, patient feedback and the decision meeting.

Ava research teamPublished 7 min read

On this page
  1. Why a pilot needs a plan of its own
  2. Before week one: scope, owner and stop rules
  3. Preparing the team and the wording
  4. Taking a baseline
  5. Measures that tell you something
  6. The four weeks
  7. Patient feedback without bias
  8. Weekly call reviews
  9. Example: a pilot summary
  10. The decision meeting
  11. Frequently asked questions
  12. References

Key takeaways

  • Agree scope, measures and stop rules in writing before the first patient call, and take a two-week baseline.
  • Four weeks is enough to see Mondays, a month-end and a normal mix of calls, as long as you avoid bank holidays and Christmas.
  • Measure outcomes, not just answered calls: requests finished, transfers that connected, wrong answers and complaints.
  • Collect patient feedback with two short questions and review a sample of calls every week.
  • End with a decision meeting that has three options: stop, extend with changes, or continue.

Why a pilot needs a plan of its own

Most AI call handling pilots fail for organisational reasons rather than technical ones. Nobody wrote down what success looked like, the baseline was a guess, the pilot ran into a bank holiday, and at the end the partners were left choosing between impressions. A four-week pilot avoids that if every decision is made before the first call.

Four weeks gives you four Mondays, at least one month-end and enough routine calls to see patterns, without committing the practice to a long contract. It is short enough that staff stay engaged and long enough that the novelty wears off and you see the tool's normal performance.

Before week one: scope, owner and stop rules

Write a one-page pilot charter and get it signed by the practice manager and a clinical lead. It should answer five questions.

  1. Which calls are in scope? For example overflow during the morning peak and lunchtime, and all calls after closing, but not the dedicated line for care homes.
  2. Which tasks will the tool handle? List them, such as practice information, explaining how to get an appointment, urgent-help wording and passing patient matters to reception.
  3. Who owns the pilot day to day, and who can switch it off?
  4. What are the measures and targets, and where will the data come from?
  5. What are the stop rules that end the pilot immediately?

Keep the scope to what the product does live today. A pilot of Ava, for example, would cover practice information, explaining routes, urgent-help wording and passing patient matters to reception, because appointment and prescription requests are still labelled Planned. Testing a planned feature in a pilot only tells you about the roadmap.

Stop rules to agree in advance: any urgent caller who did not receive the agreed 999 or NHS 111 wording; any disclosure of patient information to the wrong person; any period when calls could not reach the practice at all.

Preparing the team and the wording

Spend the week before the pilot on the content and the people. The tool will only be as accurate as the practice information behind it, and the team will only trust it if they helped shape it.

  • Review every approved answer with reception: opening hours, closures, how to order repeat medicines, how to register, parking and access, and what happens on the online form.
  • Agree the urgent-help wording with a clinician and test it with ten fictional urgent calls.
  • Check transfer numbers and ring times with your telephony supplier, including what happens when the line is busy.
  • Brief all staff on what the tool does, what it does not do and how to flag a problem in one click or one message.
  • Update the phone greeting, the website and the privacy notice to say an AI tool may answer some calls and how to reach a person.

Plan the timing of in-scope hours around the demand you actually see. Nationally, 45% of GP appointments are booked for the same day[1], which is why so many calls cluster in the first hour after opening. If your baseline shows the same pattern, start the pilot on the morning peak, where any improvement will be easiest to see and matters most to patients.

Taking a baseline

Pull two weeks of data from your telephony system before the pilot starts, covering the same hours and lines that will be in scope. Pick ordinary weeks. NHS England notes that call volumes are affected by the number of days in the month, public holidays and weekend days[2], so a baseline that includes a bank holiday will not compare fairly with a pilot that does not. In England and Wales, Christmas 2026 creates a four-day closure from Friday 25 to Monday 28 December[3]; do not run any part of the pilot across it.

  • Calls offered, answered and abandoned, by hour.
  • Average and longest wait to answer.
  • Call-backs requested and completed.
  • Calls by reason, from a manual tally by reception over three sample days.
  • Complaints and compliments about phone access.

National figures give useful context but are not a target. In the 2026 GP Patient Survey 56.9% of patients found it easy to contact their practice by phone and 72.6% rated their last contact experience as good[4]. NHS 111 answered in an average of 82 seconds in July 2026, with 4.6% of calls abandoned after its call-steering menu[2].

Patients finding it easy to contact their GP practice by phone
Patients finding it easy to contact their GP practice by phoneColumn chart. 2024: 49.7%; 2025: 52.9%; 2026: 56.9%.0%20%40%60%2024: 49.7%49.7%20242025: 52.9%52.9%20252026: 56.9%56.9%2026Patients finding it easy to contact their GP practice by phoneColumn chart. 2024: 49.7%; 2025: 52.9%; 2026: 56.9%.0%20%40%60%2024: 49.7%49.7%20242025: 52.9%52.9%20252026: 56.9%56.9%2026

Source: Ipsos (GP Patient Survey provider for NHS England), 2026[4]

Show the data
ItemValue
202449.7%
202552.9%
202656.9%

Measures that tell you something

Pilot measures, how to collect them and why each matters
MeasureHow to collect itWhy it matters
Calls handled by the toolSupplier reportVolume only; never the main measure
Requests finished without a callbackSample of 30 call records a weekShows whether work actually left reception
Transfers that connectedTelephony report plus call recordsA transfer that rings out is a failure
Wrong or outdated answersStaff flags plus weekly sampleAccuracy and safety
Urgent-help wording given correctlyEvery call with urgent words, reviewedStop rule
Abandoned calls in scoped hoursTelephony report against baselinePatient access
Patient feedbackTwo-question text surveyAcceptability
Staff viewFive-minute weekly huddleWorkload and trust

The four weeks

  1. Week 1: narrow and watched

    Run the tool on overflow only during one peak period. Review every call record each afternoon and correct wording the same day.

  2. Week 2: full scope

    Extend to all in-scope hours. Review a daily sample, check every call that mentioned urgent symptoms and hold the first staff huddle.

  3. Week 3: steady state

    Change nothing unless a stop rule or clear error requires it, so the data reflects normal running. Send the patient feedback text.

  4. Week 4: measure and prepare

    Pull the same reports as the baseline, finish the sample reviews and draft a two-page summary for the decision meeting.

Patient feedback without bias

Send a short text to a random sample of callers whose call the tool handled, within a day of the call. Keep it to two questions so response rates stay reasonable, and do not lead the answer.

Script: "You called [practice name] on [day]. Did you get what you needed from the call? Reply 1 for yes, 2 for partly, 3 for no. If you want to tell us more, reply with a few words."

Expect a range of views. Healthcare is the area where UK adults most often say AI needs careful management to prevent harm, chosen by 29%[5]. Comments are more useful than scores: read every one, sort them into themes and bring the themes to the staff huddle.

Weekly call reviews

Pick 30 call records a week at random, plus every call that triggered urgent-help wording or a transfer failure. Two people review them separately, ideally a receptionist and a clinician, and compare notes. For each call, ask:

  • Did the caller get the right answer or the right route?
  • Was the wording the practice's approved wording?
  • Did the tool ask for anything it did not need?
  • Would a receptionist have handled it differently, and would that have been better?
  • Was anything in the record that should not be there?

Example: a pilot summary

Example (fictional): a two-site practice with 11,000 patients pilots overflow and after-hours call handling for four weeks in February.

In this example abandoned calls in the morning peak fell compared with the baseline, and most information-only calls were finished without a callback. Two answers about the flu clinic were outdated in week one and corrected the same day. One transfer failed because the reception extension was set to ring for too short a time, which the telephony supplier fixed. No stop rule was triggered. Patient replies were mostly positive, with several asking for the option to speak to a person to be offered earlier, which the practice added. Staff reported fewer repeat callers in the mid-morning.

The decision meeting

Hold it within a week of the pilot ending, with the practice manager, a clinical lead, a receptionist and the supplier for the first half only. Present the measures against the baseline, the call review findings and the patient comments, then decide between three options.

  • Stop: a stop rule was triggered, or the tool created more work than it removed.
  • Extend with changes: results are promising but a specific problem needs fixing, with a new end date.
  • Continue: measures were met; agree ongoing monitoring, review dates and contract terms.

Changes in phone systems can move patient experience a long way. In NHS England trials, moving practices to high-quality digital telephony raised patients' ease of getting through by 30 percentage points[6]. A good pilot tells you whether an AI tool adds to that, and on what terms.

Frequently asked questions

How long should an AI call handling pilot run?

Four weeks is a practical minimum: long enough to include several Mondays and a month-end, short enough to keep staff engaged and avoid a long commitment.

What is the most important pilot measure?

Requests finished without a callback, checked by sampling call records. Answered call counts alone can hide work that has just moved elsewhere.

When should a pilot stop early?

When a stop rule agreed in advance is triggered, such as an urgent caller not receiving the agreed 999 or NHS 111 wording.

Should we tell patients they are part of a pilot?

Tell them they are speaking to an AI tool and how to reach a person. You can also mention the trial on your website and in the feedback text.

References

  1. British Medical Association (2026). Pressures in general practice data analysis (opens in a new tab) ↑

  2. NHS England (2026). Statistical Note: Integrated Urgent Care Aggregate Data Collection (IUC ADC) July 2026 (opens in a new tab) ↑

  3. GOV.UK (2026). UK bank holidays (opens in a new tab) ↑

  4. Ipsos (GP Patient Survey provider for NHS England) (2026). 2026 GP Patient Survey results released (opens in a new tab) ↑

  5. GOV.UK (Department for Science, Innovation and Technology) (2024). Public attitudes to data and AI: Tracker survey (Wave 4) report (opens in a new tab) ↑

  6. NHS England and DHSC (2023). Delivery plan for recovering access to primary care (opens in a new tab) ↑

Book a demo

See Ava handle your calls.

Book a short demo. We’ll walk through your calls and workflow, and agree what Ava should take care of.

Book a demo