In brief
- Start with the task, data and consequences of error, not a generic vendor questionnaire.
- Assess the exact plan, endpoint and feature set; statements about the brand alone are too broad.
- Separate training, retention, data location, subprocessors, support access and internal logging.
- Record the decision, conditions, accountable owners and date of the next review.
Define the use case first
A provider cannot be assessed meaningfully without a bounded use case. A writing assistant for public content creates different requirements from analysing recruitment files or health information.
- Step 1
Task
What specific output should the application produce?
- Step 2
Data
Which inputs, sources and personal data will it process?
- Step 3
Users
Who may use the application and underlying information?
- Step 4
Consequences
What happens if an output is wrong or information is disclosed?
- Step 5
Risk level
Do the technology, purpose, scale or data indicate the need for a deeper impact assessment?
Map the exact product and data flow
Many providers operate several products under different terms. A consumer chat, enterprise workspace and API can differ materially in training, retention and administration.
- Exact product, plan, region, model and API endpoint
- Inputs, outputs, files, conversation history and derived metadata
- RAG sources, web search, plug-ins, connectors and code execution
- Storage locations, retention periods, backups and deletion paths
- Subprocessors, support access and possible international transfers
- Internal logs, monitoring, analytics and error systems
Roles and contract must match the data flow
When a provider processes data on the organisation’s behalf, the organisation remains responsible for careful selection, instruction and appropriate oversight. The agreement needs to cover the actual technical use.
- Roles of all parties and permitted processing purposes
- Processing only on documented instructions
- Confidentiality and technical and organisational safeguards
- Rules for new or changing subprocessors
- Safeguards for transfers to countries without adequate protection
- Support for access, correction, deletion and security incidents
- Return, export and deletion at contract end
- Notice and termination rights for material product changes
Evidence is stronger than product promises
Certifications and audit reports can support an assessment but do not replace review of the use case. The key question is whether required controls are available, configured correctly and verifiable.
| Review area | Concrete question | Expected evidence | Warning sign |
|---|---|---|---|
| Identity and access | Can SSO, MFA, roles, tenant separation and automated revocation be enforced? | Test-tenant configuration, role matrix and an audit log of a revoked account. | Shared accounts, coarse admin roles or no exportable access logs. |
| Retention and deletion | Which content, metadata, backups and features are stored, and for how long? | Contractual periods, product matrix and a successfully tested deletion case. | A generic ‘no training’ statement without retention or downstream-system details. |
| Data flow and subprocessors | Which parties process which data in which countries? | Current subprocessor list, regions, transfer safeguards and change process. | Unknown recipients, a global default region or changes without an objection route. |
| Security and incidents | How are vulnerabilities, administrative changes and data breaches detected and reported? | Audit report, incident process, notification deadlines and sample audit entries. | A certificate without scope, unclear reporting channels or no binding timelines. |
| Quality and product change | How are model, feature and security changes announced and re-evaluated? | Release process, change log, evaluation results and a suspension or termination route. | Silent model changes or material updates without reviewable evidence. |
Approve with evidence and continue reviewing
Provider assessment does not end when the contract is signed. Models, features and subprocessors change. A limited pilot shows whether commitments, configuration and working practices align in reality.
- Step 1
Start safely
Pilot first with synthetic, anonymised or low-sensitivity data.
- Step 2
Verify configuration
Technically check region, retention, training, roles and logs.
- Step 3
Test realistically
Evaluate quality, data leakage, failure cases and human review with representative cases.
- Step 4
Document approval
Record permitted data, features, conditions and accountable owners.
- Step 5
Monitor operation
Review changes, incidents, access and provider performance; rehearse export and exit.
Example: extracting data from incoming invoices
A company assesses a service that classifies invoices and extracts fields. The documents contain contact, banking and order information. Before the pilot, the team clarifies the data flow, product region, subprocessors, deletion periods and training use. The agreement covers instructions, incidents, return and deletion. SSO, roles and content-minimising logs are enabled. The pilot begins with synthetic invoices and then a limited approved dataset. Extraction, failure cases, human correction and deletion are tested before production approval.
What to remember
A good provider assessment ends with a traceable decision: approved use case, permitted data, active safeguards, contractual evidence, residual risks, accountable owners and next review date.
Sources and further reading
These primary sources provide further detail on definitions, technical foundations or responsible use.
Content reviewed
General assessment guidance based on official regulator and standards sources. This is not legal advice; privacy and contractual questions must be assessed for the actual deployment.