An AI Proof of Concept Should End with a Decision

The demo works. An assistant reads a document, produces a tidy answer and cites a source. Everyone can see the potential. Then someone asks whether the business should fund the next stage—and the room goes quiet.

That is the question an AI proof of concept should help answer. A useful PoC tests a specific uncertainty against agreed evidence. Its output is a decision: proceed to a bounded pilot, change the approach, or stop spending on this use case.

This guide follows one illustrative AI proof of concept example: helping a supplier’s sales team draft answers to product-specification questions. The company, dataset, thresholds and results below are invented for explanation. They are not Pixelfield client results or a model benchmark. You can use the accompanying worksheet to design your own test.

Define the investment decision before the experiment

An AI PoC is a limited investigation into whether a proposed capability is feasible and useful under stated conditions. It need not include a polished interface or a production integration. It does need an explicit question and a way to decide what the answer means.

For the supplier, “Can AI answer product questions?” is too broad. The proposed question is:

Can a draft-only assistant help sales staff answer supported product-specification questions faster, while identifying the correct product variant and declining questions the approved documents cannot answer?

The proposed next investment is a supervised pilot with a small sales group. Automatic replies, recommendations about product suitability and changes to customer records are outside scope.

AWS makes a useful distinction between experimenting with an interesting capability and using a PoC to reduce the risks around a business objective. Its guide to moving generative AI proofs of concept towards production argues for defining success and the path beyond the experiment. That is the organising principle here; the worked example and its numerical thresholds are our own illustration.

For broader product-validation terminology, see our general proof-of-concept guide. This article deals specifically with the evidence behind an AI investment decision.

Write the test brief in one page

Before choosing a model, the sponsor and technical lead should agree what they will evaluate. In this example, the brief looks like this:

The exclusions matter when the demonstration becomes attractive. A stakeholder may ask for CRM updates or live stock information halfway through. Record that as a separate proposed capability; do not quietly add it to the experiment and then claim that the original test covered it.

The distinction between an answering interface and a system that performs actions is covered in our chatbot and AI agent comparison. For this PoC, draft-only output deliberately limits what the team must investigate.

Build a dataset that can disappoint you

The supplier starts with approved documents and a collection of representative questions. Somebody who understands the catalogue records the acceptable answer, the correct variant and the supporting source for each question. Where the documents do not support an answer, the expected result is an abstention—not an educated guess.

A proposed test set contains 100 questions: 70 supported questions, 20 with insufficient evidence and 10 involving contradictory or outdated documents. Those counts are a planning example, not a recommended sample size for every project. A real evaluation needs enough coverage for its users, document types and consequences of error.

Keep development examples separate from the final decision set. If the team reads a failed case and changes the prompt to handle it, that case has become development material. Repeated success on the same corrected examples is useful debugging evidence, but weak evidence about unseen work. Record that distinction in the report.

Include the cases people tend to avoid in a sales demonstration:

• Two products have almost identical names but different limits.
• The answer appears in a table rather than a paragraph.
• A discontinued document conflicts with the current version.
• The question combines a supported fact with an unsupported request.
• A relevant document is absent or inaccessible to the intended user.

Do not describe an evaluation as representative merely because it contains 100 rows. Record where the cases came from and which kinds of work are missing. Use synthetic material for initial plumbing checks; before an investment decision, establish whether the approach works with appropriately authorised, realistic data.

Agree what passes—and what stops the test

A single “accuracy” percentage can hide the failure that matters. An answer may sound right but cite the wrong variant. Another may be factually correct yet take longer to check than writing the reply manually.

AWS’s guidance on baseline performance targets recommends identifying measurable targets in the context of existing performance and user needs. Apply that idea by separating quality, handling of unknowns and the effort required from people.

For the fictional supplier, the sponsor agrees these example thresholds before the run:

These are illustrative business choices, not universal acceptance criteria. Zero observed critical errors does not prove that the error rate is zero. Nor does a small test justify removing human review. State both limitations alongside the result.

Define the scoring rules too. If two reviewers disagree about whether a response is supported, resolve the disagreement and record why. Otherwise the team can appear to improve simply because the reviewer becomes more forgiving.

Run the baseline and the AI-assisted task fairly

Measure the complete job: locating documents, producing a reply, checking it and correcting it. Keep response quality comparable between the manual and assisted conditions. A fast unchecked draft is not equivalent to a checked customer response.

Avoid timing an adviser on a question they have just answered manually and treating the second attempt as an independent comparison. Prior knowledge can make the assisted process look quicker. Use matched work, counterbalance the order where practical, and document the limitations of the design.

Preserve the configuration behind each run: model identifier, prompt version, document snapshot, retrieval settings and any tools used. Save the answer and retrieved evidence against the case identifier. If a configuration changes, label the new run rather than mixing its results into the previous one.

Also note failures outside the model. If the document retrieval step times out, the response should not imply that a complete search found no answer. That is an integration failure to investigate, not a correct abstention. Our AI integration guide explains the role of API boundaries and failure handling.

Read the result as a decision, not a headline

Suppose the invented results are 65 correct supported answers out of 70, 20 appropriate abstentions out of 20, and eight correctly flagged conflicts out of 10. No critical errors are observed. Median complete handling time falls from eight minutes to six.

The time reduction is (8 − 6) ÷ 8 = 25%. The supported-answer gate passes too. But two contradictory-document cases fail. The agreed decision is therefore revise and retest, not proceed to the pilot. Reporting only the time saving would conceal the reason the system is not ready.

The next experiment should investigate document-version handling. It might change which sources are eligible for retrieval or how conflicts are surfaced. Re-run the affected cases, then test the revised approach on fresh examples. Do not simply remove the two difficult questions from the scorecard.

There are three defensible outcomes:

Proceed to a restricted pilot: the agreed gates pass, remaining risks are documented and someone owns the next stage. The approval applies to that scope and configuration.

Revise and retest: a specific unresolved issue has a plausible remedy. Name the issue, the next test and the limit on further work.

Stop: the approach cannot meet the required quality, data access is unavailable, or the complete economics do not justify continued investment. A clear stop decision is a useful result.

Check the economics without turning minutes into guaranteed savings

Two minutes saved on one measured task does not automatically become two minutes of cash savings. Staff may use the capacity for other work; review demand may vary; adoption may be uneven.

Use the time result as an input to a business case. Estimate eligible task volume, realistic usage, model and retrieval charges, integration costs, document maintenance, evaluation effort and continuing human oversight. Separate one-off build costs from recurring operation. Include the cost of cases that need escalation or a second attempt.

For this example, the business still needs evidence about how often suitable questions occur and whether advisers will use the assistant. The PoC answers part of the investment question. A pilot can investigate everyday usefulness without pretending that every production dependency has already been proven.

Ask for a decision pack you can inspect

At handover, ask for more than a slide with a success rate. The pack should contain the original brief, dataset description, scoring rules, configuration, case-level outputs, measured effort, failed examples and the recommendation. Keep confidential source material in an appropriately controlled location.

Download the AI PoC decision worksheet. It contains the illustrative gates and fictional results from this article, clearly labelled, with space for your own evidence. Replace the numbers before using it for a real decision.

Bring one task, a few suitable examples and the uncertainty you need to resolve to a conversation about Pixelfield’s AI proof-of-concept service. The useful starting point is the decision you need to make—and the evidence that would change it.