Key takeaways
- Agree scope, measures and stop rules in writing before the first patient call, and take a two-week baseline.
- Four weeks is enough to see Mondays, a month-end and a normal mix of calls, as long as you avoid bank holidays and Christmas.
- Measure outcomes, not just answered calls: requests finished, transfers that connected, wrong answers and complaints.
- Collect patient feedback with two short questions and review a sample of calls every week.
- End with a decision meeting that has three options: stop, extend with changes, or continue.
Why a pilot needs a plan of its own
Most AI call handling pilots fail for organisational reasons rather than technical ones. Nobody wrote down what success looked like, the baseline was a guess, the pilot ran into a bank holiday, and at the end the partners were left choosing between impressions. A four-week pilot avoids that if every decision is made before the first call.
Four weeks gives you four Mondays, at least one month-end and enough routine calls to see patterns, without committing the practice to a long contract. It is short enough that staff stay engaged and long enough that the novelty wears off and you see the tool's normal performance.
Before week one: scope, owner and stop rules
Write a one-page pilot charter and get it signed by the practice manager and a clinical lead. It should answer five questions.
- Which calls are in scope? For example overflow during the morning peak and lunchtime, and all calls after closing, but not the dedicated line for care homes.
- Which tasks will the tool handle? List them, such as practice information, explaining how to get an appointment, urgent-help wording and passing patient matters to reception.
- Who owns the pilot day to day, and who can switch it off?
- What are the measures and targets, and where will the data come from?
- What are the stop rules that end the pilot immediately?
Keep the scope to what the product does live today. A pilot of Ava, for example, would cover practice information, explaining routes, urgent-help wording and passing patient matters to reception, because appointment and prescription requests are still labelled Planned. Testing a planned feature in a pilot only tells you about the roadmap.
Stop rules to agree in advance: any urgent caller who did not receive the agreed 999 or NHS 111 wording; any disclosure of patient information to the wrong person; any period when calls could not reach the practice at all.
Preparing the team and the wording
Spend the week before the pilot on the content and the people. The tool will only be as accurate as the practice information behind it, and the team will only trust it if they helped shape it.
- Review every approved answer with reception: opening hours, closures, how to order repeat medicines, how to register, parking and access, and what happens on the online form.
- Agree the urgent-help wording with a clinician and test it with ten fictional urgent calls.
- Check transfer numbers and ring times with your telephony supplier, including what happens when the line is busy.
- Brief all staff on what the tool does, what it does not do and how to flag a problem in one click or one message.
- Update the phone greeting, the website and the privacy notice to say an AI tool may answer some calls and how to reach a person.
Plan the timing of in-scope hours around the demand you actually see. Nationally, 45% of GP appointments are booked for the same day[1], which is why so many calls cluster in the first hour after opening. If your baseline shows the same pattern, start the pilot on the morning peak, where any improvement will be easiest to see and matters most to patients.
Taking a baseline
Pull two weeks of data from your telephony system before the pilot starts, covering the same hours and lines that will be in scope. Pick ordinary weeks. NHS England notes that call volumes are affected by the number of days in the month, public holidays and weekend days[2], so a baseline that includes a bank holiday will not compare fairly with a pilot that does not. In England and Wales, Christmas 2026 creates a four-day closure from Friday 25 to Monday 28 December[3]; do not run any part of the pilot across it.
- Calls offered, answered and abandoned, by hour.
- Average and longest wait to answer.
- Call-backs requested and completed.
- Calls by reason, from a manual tally by reception over three sample days.
- Complaints and compliments about phone access.
National figures give useful context but are not a target. In the 2026 GP Patient Survey 56.9% of patients found it easy to contact their practice by phone and 72.6% rated their last contact experience as good[4]. NHS 111 answered in an average of 82 seconds in July 2026, with 4.6% of calls abandoned after its call-steering menu[2].
Source: Ipsos (GP Patient Survey provider for NHS England), 2026[4]
Show the data
| Item | Value |
|---|---|
| 2024 | 49.7% |
| 2025 | 52.9% |
| 2026 | 56.9% |
Measures that tell you something
| Measure | How to collect it | Why it matters |
|---|---|---|
| Calls handled by the tool | Supplier report | Volume only; never the main measure |
| Requests finished without a callback | Sample of 30 call records a week | Shows whether work actually left reception |
| Transfers that connected | Telephony report plus call records | A transfer that rings out is a failure |
| Wrong or outdated answers | Staff flags plus weekly sample | Accuracy and safety |
| Urgent-help wording given correctly | Every call with urgent words, reviewed | Stop rule |
| Abandoned calls in scoped hours | Telephony report against baseline | Patient access |
| Patient feedback | Two-question text survey | Acceptability |
| Staff view | Five-minute weekly huddle | Workload and trust |
The four weeks
Week 1: narrow and watched
Run the tool on overflow only during one peak period. Review every call record each afternoon and correct wording the same day.
Week 2: full scope
Extend to all in-scope hours. Review a daily sample, check every call that mentioned urgent symptoms and hold the first staff huddle.
Week 3: steady state
Change nothing unless a stop rule or clear error requires it, so the data reflects normal running. Send the patient feedback text.
Week 4: measure and prepare
Pull the same reports as the baseline, finish the sample reviews and draft a two-page summary for the decision meeting.
Patient feedback without bias
Send a short text to a random sample of callers whose call the tool handled, within a day of the call. Keep it to two questions so response rates stay reasonable, and do not lead the answer.
Script: "You called [practice name] on [day]. Did you get what you needed from the call? Reply 1 for yes, 2 for partly, 3 for no. If you want to tell us more, reply with a few words."
Expect a range of views. Healthcare is the area where UK adults most often say AI needs careful management to prevent harm, chosen by 29%[5]. Comments are more useful than scores: read every one, sort them into themes and bring the themes to the staff huddle.
Weekly call reviews
Pick 30 call records a week at random, plus every call that triggered urgent-help wording or a transfer failure. Two people review them separately, ideally a receptionist and a clinician, and compare notes. For each call, ask:
- Did the caller get the right answer or the right route?
- Was the wording the practice's approved wording?
- Did the tool ask for anything it did not need?
- Would a receptionist have handled it differently, and would that have been better?
- Was anything in the record that should not be there?
Example: a pilot summary
Example (fictional): a two-site practice with 11,000 patients pilots overflow and after-hours call handling for four weeks in February.
In this example abandoned calls in the morning peak fell compared with the baseline, and most information-only calls were finished without a callback. Two answers about the flu clinic were outdated in week one and corrected the same day. One transfer failed because the reception extension was set to ring for too short a time, which the telephony supplier fixed. No stop rule was triggered. Patient replies were mostly positive, with several asking for the option to speak to a person to be offered earlier, which the practice added. Staff reported fewer repeat callers in the mid-morning.
The decision meeting
Hold it within a week of the pilot ending, with the practice manager, a clinical lead, a receptionist and the supplier for the first half only. Present the measures against the baseline, the call review findings and the patient comments, then decide between three options.
- Stop: a stop rule was triggered, or the tool created more work than it removed.
- Extend with changes: results are promising but a specific problem needs fixing, with a new end date.
- Continue: measures were met; agree ongoing monitoring, review dates and contract terms.
Changes in phone systems can move patient experience a long way. In NHS England trials, moving practices to high-quality digital telephony raised patients' ease of getting through by 30 percentage points[6]. A good pilot tells you whether an AI tool adds to that, and on what terms.
Frequently asked questions
How long should an AI call handling pilot run?
Four weeks is a practical minimum: long enough to include several Mondays and a month-end, short enough to keep staff engaged and avoid a long commitment.
What is the most important pilot measure?
Requests finished without a callback, checked by sampling call records. Answered call counts alone can hide work that has just moved elsewhere.
When should a pilot stop early?
When a stop rule agreed in advance is triggered, such as an urgent caller not receiving the agreed 999 or NHS 111 wording.
Should we tell patients they are part of a pilot?
Tell them they are speaking to an AI tool and how to reach a person. You can also mention the trial on your website and in the feedback text.
References
British Medical Association (2026). Pressures in general practice data analysis (opens in a new tab) ↑
NHS England (2026). Statistical Note: Integrated Urgent Care Aggregate Data Collection (IUC ADC) July 2026 (opens in a new tab) ↑
GOV.UK (2026). UK bank holidays (opens in a new tab) ↑
Ipsos (GP Patient Survey provider for NHS England) (2026). 2026 GP Patient Survey results released (opens in a new tab) ↑
GOV.UK (Department for Science, Innovation and Technology) (2024). Public attitudes to data and AI: Tracker survey (Wave 4) report (opens in a new tab) ↑
NHS England and DHSC (2023). Delivery plan for recovering access to primary care (opens in a new tab) ↑