Bulk Data Extract Overview

Definitions

Term	Description
Direct Care	"The term ‘direct care’ is defined as a clinical, social or public health activity concerned with the prevention, investigation and treatment of illness and the alleviation of suffering of individuals (all activities that directly contribute to the diagnosis, care and treatment of an individual).  It includes: supporting individuals’ ability to function and improve their participation in life and society; the local audit/assurance of the quality of care provided;  the management of untoward or adverse incidents;  the measurement of outcomes undertaken by one or more registered and regulated health or social care professionals and their team with whom the individual has a legitimate relationship for their care It does not include research, teaching, financial audit, service management activities or risk stratification (see note below on borderline cases).” Source: https://digital.nhs.uk/data-and-information/looking-after-information/data-security-and-information-governance/codes-of-practice-for-handling-information-in-health-and-care/a-guide-to-confidentiality-in-health-and-social-care/hscic-guide-to-confidentiality-references/section-10
Indirect Care	"The term ‘indirect care’ is defined as activities that contribute to the overall provision of services to a population as a whole or a group of patients with a particular condition, but which fall outside the scope of direct care. It covers health services management, preventative medicine, and medical research. Examples of indirect care activities include risk prediction and stratification (see note below on borderline cases), service evaluation, needs assessment, and financial audit. The key reason for distinguishing between purposes in this way is that it is generally possible to imply consent for the use of confidential information for direct care purposes but not for other purposes. There are some exceptions and some tricky borderline cases on which specific guidance is provided."  Source: https://digital.nhs.uk/data-and-information/looking-after-information/data-security-and-information-governance/codes-of-practice-for-handling-information-in-health-and-care/a-guide-to-confidentiality-in-health-and-social-care/hscic-guide-to-confidentiality-references/section-10

Supported Use Cases

The only supported use cases for IM1 Bulk Extracts are:

• Targeted extracts for specific reporting use cases (e.g. CH-IS extracts)

• Broad data extract to transfer to data warehouse (e.g. PHM/extracts)

• Direct care planning where the organisation has a legitimate relationship (e.g. the GP Practice themselves running call/recall processes)

IM1 bulk must not be used for the following use cases:

• Direct care for third party organisations delivering care (e.g. record viewers or clinical decision support) - use IM1 transactional

• Patient facing services (e.g. patients viewing their own records) - use IM1 PFS

• Data migration - Use the NHS data migration standard

Data Extract Definition

We are able to query the reporting database tables to meet your needs and can define multiple different queries to be exported in the same extract, or to join on reporting database tables as required.

The Data Extract Definition is defined as an “SQL” query against the reporting data tables. Inline with the principle of minimisation of data egress, your queries must select the minimum data and fields required for the use case.

Multiple CSV files will be extracted to handle multiple data sources. There will be a single ZIP folder that contains all CSV files (i.e. Medicus will send all files at once).

All data extracts are processed by Medicus in the United Kingdom.

The extract definition must include the following information:

• Which reporting data & fields are required

• Patient registration status [permanently registered | all patients]

• Include deceased patients?

• Full extract or “deltas”

  • For full data extracts, a CSV file containing the data will be generated at the agreed interval

  • For data extracts that require “deltas”, there will be an initial data extraction, followed by subsequent delta extracts that contain the changes to the data since the last extraction

    • If a data extract definition changes, the process will need to be performed from the beginning as a new extract

• Frequency of extract: Daily

  • Timing: overnight preferred to minimise any impact on Medicus servers during core working hours

As per the IM1 contractual standard (requirement im1_bulk_001), bulk schemas and consumer requests will be reviewed every 3 months and new entities or fields required for inclusion must incorporated as agreed & directed by the ‘Minor or patch changes’ section of the Change Management Process. For the avoidance of doubt, this includes updates to existing data extracts.

Data Protection Obligations

All consumers must:

• Provide a definition of the “intended use” of the data

• Outline the legal basis for extraction: Details of any data provision notice or explanation of the purpose

• Provide a DPIA & other documentation explaining the data flows

  • This documentation must detail how data will be access controlled, particularly when the feed is multi-use (e.g. direct and indirect care) and controlled via RBAC within the consumer system

• Maintain ISO 27001 certification from a UKAS-approved accreditation body, with SoA covering all processing activities

• Maintain Cyber Essentials Plus certification, with scope covering all processing activities

• Maintain DSP Toolkit certification to "Standards Exceeded" level

• Legally guarantee that all processing and sub-processing only happens within the United Kingdom, unless an exception has been agreed with the Data Controllers - Medicus Health shares no liability for any breaches of this agreement

• Legally guarantee that there will be no secondary transfers without explicit Data Controller approval - Medicus Health shares no liability for any breaches of this agreement

• Have an agreed Data Retention Policy which includes a mechanism for cascading GDPR obligations from Data Controllers

• Enforce at least NIST AAL2 authentication levels for access to the data

• Detail the procedures they have in place to protect sensitive data from illegal breach

• Provide a statement outlining how data remains encrypted once it has been downloaded from MESH or SFTP

• Provide a statement that confirms all data will remain in the United Kingdom during the end-to-end data flow, including technical details

Patient Consent & Type 1 Opt-out

• Patients with Type 1 opt-out will not be included in the extract unless the use case is direct care OR there is a statutory data provision notice which overrides this

• For patients with a Type 1 opt-out preference, only new data extracts will exclude this patient’s details. If the patient explicitly requests to be removed from historical data extracts then the Data Controller and consumer are jointly responsible for removing historical data from their systems

• If a patient opts-out and then opts-in again, only there will be no back-filling of data unless the extract specifically includes these parameters

• Data will only be shared for permanently registered patients at a GP Practice unless there is an explicit reason which overrides this (for example, a payment related extract that includes all procedures done) - this must be defined in the data extract definition with clear justification

• More information about patient consent for information sharing can be found here

• Data Controllers are responsible for ensuring removal of historical data by the consumer where it is no longer appropriate or legal (e.g. historical data extracted over many years and then retained beyond what is reasonable or lawful)

Anonymisation & Minimisation of Data Egress

• For all data extracts, the consumer must give special consideration to, and provide a statement on the risk of:

  • a) “singling out”

  • b) “linkability” (otherwise known as the mosaic effect)

• See the ICO guidance on singling out and linkability for more information.

• Other IG considerations:

  • GDPR - Minimisation of Data Egress

    • The consumer will only be granted access to data fields that have a clear legal basis for disclosure. As part of the extract definition, consumers must provide clear justification for the data extracted

  • NHS Act 2006 s251 - Sensitive Medical Records

    • Without an explicit justification, no sensitive medical records will be included (e.g. codes on RCGP exclusion lists)

Data Controller Legal Obligations

• Bulk data extracts are always approved by the GP Practice - no bulk data will be extracted without prior approval from the GP Practice, acting in their capacity as Data Controller

  • In the scenario where the GP Practice and ICB act as joint Data Controllers, the GP Practice must approve the extract from their Medicus database

• Unless there is a clear legal basis, the following information is not available via bulk extracts. We will work with any consumers who have a clear legal basis when a use case arises:

  • Gender reassignment

    • Any concept in 999004351000000109 | General practice summary data sharing exclusion for gender related issues simple reference set (foundation metadata concept)

    • Any concept in 1955851000000101 | National Health Service secondary use data exclusions due to Gender Recognition Act 2004 simple reference set (foundation metadata concept)

  • Sexuality

    • >> 118199002 | Finding relating to sexuality and sexual activity (finding)

    • >> 365957000 | Finding related to sexual relationship (finding)

    • >> 118200004 | Finding related to sexual state (finding)

    • >> 171023003 | Psychosexual counseling (procedure)

  • HIV status

    • Any concept in 999004381000000103 | General practice summary data sharing exclusion for sexually transmitted disease simple reference set [the most robust way to exclude HIV status]

  • Legal allegations against others

    • Covered by GP practices marking items as confidential from third parties + not extracting free text or documentation

  • Items marked as confidential from third parties

  • Other legally restricted information be protected from breaching Data Controller legal obligations

    • Any concept in the “RCGP sensitive dataset“:

      • 999004361000000107 | General practice summary data sharing exclusion for termination of pregnancy simple reference set

      • 999004351000000109 | General practice summary data sharing exclusion for gender related issues simple reference set

      • 999004371000000100 | General practice summary data sharing exclusion for assisted fertilisation simple reference set

      • 999004381000000103 | General practice summary data sharing exclusion for sexually transmitted disease simple reference set

• Free-text and documents are not able to be included as there is no mechanism to ensure this type of legal information is not exported, leading to a legal breach

• Whilst we endeavour to protect sensitive data, the Data Controller holds the ultimate responsibility for ensuring that data is captured in a way that allows for automatic redaction from bulk data extracts

Transmission Mechanism

SFTP Server

[Hosted by Consumer]

Pros/cons

• Pros:

  • Consumers do not need to be assured to use MESH

  • Works in all contexts (NHS, Channel Islands)

• Cons:

  • Additional cost for consumers to host

What we require:

• SFTP Server Host

• SFTP Server Port

• SFTP Server Username

• SFTP Directory (where the files will be uploaded)

We will authenticate using public/private SSH keys (Algorithm: ed25519).

Please add the Medicus public key to your SFTP server for the environments you will be receiving extracts from.

Public keys for SFTP uploads:

UAT: 

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHmuZOGYiUQVyOsso8SDhx8GmRhsno2oNmuBM0tE5WPg [e-mail]

Staging: 

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJDCqiL07SBXzIFwxgloWrhkLH5mXLHfr9dIc8/EOF7q [e-mail]

Production:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICgV1uyg1WqW6W/e4j9b5KL3C/2iTCbOhfbFeCn+SZ9t [e-mail]

The SFTP server must be on the public internet (not HSCN).

If you need to add our IP addresses to your firewall to access the SFTP server then please use:

• UAT - 3.11.49.207

• Staging - 3.8.218.189

• Production - 18.135.65.70

Encryption during transit:

Data sent via SFTP is encrypted using the Secure File Transfer Protocol pattern.

MESH

Pros/cons

• Pros:

  • Very high availability (NHS Platinum service)

  • Retry mechanism built into outbox sending pattern

  • Secure by design - established pattern for sharing PII

  • Low cost for consumers

• Cons:

  • This is a secure method if you are transferring data from NHS England organisations only. (If the product needs to work outside of England or for private organisations, then they are not eligible for MESH Mailboxes.)

What we require:

• MESH Mailbox ID

We will send the files from the local organisation's MESH mailbox, using MESH Workflow ID GFTD_INIT

Encryption during transit:

Data is sent via MESH over HTTPS and downloaded by the consumer from MESH over HTTPS.

File Encryption

We can optionally encrypt the .zip files that are being transferred.

We encrypt using "PKCS#7 / S/MIME ZIP Encryption" which is an industry-standard method for encrypting files using a combination of symmetric and public key cryptography.

How it works:

1. The ZIP file is encrypted with a random one-time AES-256 key

2. That key is then encrypted using your public key (RSA/ECC), so only your private key can unlock it

3. Both are bundled into a single encrypted package (.enc)

Security features:

• No passwords to transmit or guess — security depends on possession of your private key

• AES-256 is the same standard used by governments and financial institutions

• Built on globally vetted, open standards

• Without your private key, the file is unreadable

Technical instructions for encrypted ZIP files from Medicus:

1. Generate a private key

For a modern RSA key (2048-bit):

openssl genpkey -algorithm RSA -out private.pem -pkeyopt rsa_keygen_bits:2048

This generates private.pem which is the private key. Keep this secret - it should stay on your server to decrypt the files that we upload.

2. Generate a self-signed certificate

For PKCS#7 / S/MIME, you need a certificate with your public key. Use your private key to generate it:

openssl req -new -x509 -key private.pem -out certificate.pem -days 3650

This generates certificate.pem which is the certificate. It has an expiry date of 10 years. There is no need to fill in any of the certificate fields.

3. Provide the certificate to the practice

Provide certificate.pem to the practice when they enable the extract:

Medicus will encrypt the data extract file.

4. Decrypt the encrypted .zip file that Medicus sends:

openssl smime -decrypt \
  -in medicus-data-extract.zip.enc \
  -recip certificate.pem \
  -inkey private.pem \
  -out medicus-data-extract.zip