Hybrid Patron Pseudonymizer

README.md

HybridPatronPseudonymizer

Protecting the privacy of patrons is of utmost importance for institutions that handle sensitive data, and in some cases, it's mandated by law or other regulations. In this context, the provided Python module serves as a powerful tool to ensure that patron data remains confidential and protected while still being usable for certain types of analytical use.

Pseudonymization is a data protection technique where personally identifiable information (PII) fields within a data record are replaced with artificial identifiers or pseudonyms. This method ensures data subject privacy while still permitting data analysis; given proper keys and credentials, aspects of the data set may be decrypted for closer analysis while still maintaining the integrity of both the data and the privacy of patrons.

The HybridPatronPseudonymizer is a Python class that implements a hybrid encryption system. Hybrid encryption combines the benefits of both symmetric and asymmetric encryption.

Potential Uses:

1. Data Storage: Before saving patron data in external databases, or csv, Excel, etc, use the encrypt method to ensure it's stored securely.
2. Analysis: For data analysis tasks, patron identifiers remain unique, but are unable to be linked to patrons directly. Should it be necessary, decryption is possible given the encrypted RSA key and the password needed to unlock it; data at-rest always remains encrypted, ensuring stored security.
3. Trust: Trust is foundational in the relationship between patrons and institutions. By taking steps to be transparent and safeguard personal data through pseudonymization, institutions bolster this trust, ensuring that patrons feel secure in their engagements.
4. Privacy Regulations With the rise of data privacy regulations globally, such as the GDPR in the European Union, organizations have a responsibility to protect the privacy of their users. Pseudonymizing patron data aids libraries and similar institutions in adhering to these regulations.
5. Maintaining both Data and Privacy Integrity In the event of a data breach, pseudonymized data becomes less valuable and useless to hackers. This is because the essential personal identifiers have been substituted for encrypted values, making the linkage of the data to individual patrons more challenging or impossible without other pieces of information--such as private encrypted keys and application secrets. These other pieces of information are securely kept outside of the scope of the data.

By using this module, institutions can maintain the highest standards of data privacy while still accessing and using the data as needed.

It should be noted that the data can be fully anonymized by simply destroying the private key file. Some analytical usefullness will of course be lost in this action. Identifiers indicating the uniqueness of a patrons remain which are still useful for certain analytical purposes, but the ablility to decrypt and link actions or behaviour to a specific patron are lost.

Overview

The class revolves around the following key functionalities:

1. Initialization: The constructor initializes the pseudonymizer with RSA private and public keys and requires an app_secret for some operations.
2. RSA Key Pair Generation: A method is provided to generate an RSA key pair and save it to disk.
3. Encryption: The class offers a method to encrypt data using AES-SIV (a symmetric encryption algorithm). The AES key is encrypted using the RSA public key.
4. Decryption: The decryption method decrypts the provided data by decrypting the AES key using the RSA private key and then using that AES key to decrypt the actual data.
5. Key Derivation: A private method derives a symmetric encryption key using PBKDF2 with the SHA256 hash function.

Detailed Breakdown

Initialization

The constructor (__init__) initializes the pseudonymizer with:

• RSA private and public keys.
• An app_secret which is mandatory for certain operations.

Key Pair Generation

The method generate_rsa_key_pair generates an RSA key pair and saves it to specified file paths.

Encryption

The encrypt method:

1. Encodes the provided data to bytes if it is a string.
2. Derives the AES key using the _derive_aes_key method, which uses the app_secret and patron_record as inputs.
3. Encrypts the data using AES-SIV mode with the derived AES key.
4. Encrypts the derived AES key using the RSA public key with OAEP padding.
5. Base64-encodes both the encrypted AES key and the ciphertext before returning.

Decryption

The decrypt method decrypts the provided data by:

1. Decrypting the AES key using the RSA private key.
2. Decrypting the actual data using the derived AES key.

Key Derivation

The _derive_key method derives a symmetric encryption key using PBKDF2 with the SHA256 hash function. This derived key is used as the AES key for symmetric encryption.

Collision Considerations

In encryption, collisions refer to different plaintexts producing the same ciphertext. The risk of collisions in this implementation is influenced by:

1. AES-SIV Mode: Being deterministic, AES-SIV will produce the same ciphertext for the same plaintext and key. However, for different plaintexts or keys, the ciphertexts should be different.
2. Key Derivation: The AES key is derived from both the app_secret and the patron_record. As long as this combination is unique for each encryption, the derived AES key should be unique.
3. RSA Encryption: RSA encryption with OAEP padding is considered secure. The encrypted AES key will only collide if two different AES keys produce the same RSA-encrypted output, which is highly improbable.

For the HybridPatronPseudonymizer to function as intended, it's crucial to ensure that the combination of app_secret and patron_record is unique for each encryption.

Pseudonymization and Data Analysis:

Pseudonymization doesn't hinder meaningful data analysis. Here's why:

1. Maintaining Data Relationships:

Even with the pseudonyms replacing actual identifiers, the relationships between data records remain. This intact relationship ensures that analyses, like determining the circulation of items, remain accurate.

2. Aggregated Analysis:

Pseudonymized data sets still hold value for aggregated analyses. For example, one can compute the average number of books borrowed per patron or ascertain the most popular genres.

3. Temporal Analysis:

Over time, the behavior and preferences of patrons can change. Pseudonymized data allows institutions to track these changes over periods without revealing individual identities.

4. Consistency of Pseudonyms:

For each real-world entity, pseudonyms remain consistent. This consistency, for instance, ensures that a patron borrowing multiple books will have the same pseudonym across all records, enabling individualized analysis without exposing the patron's identity.

5. Joining Multiple Data Sets:

If several data sets undergo pseudonymization using the same methodology and pseudonyms, they can be merged for a more extensive analysis. This combination can reveal insights like the correlation between book borrowing patterns and event attendance.

Limitations

1. Key Management: The strength of any encryption system largely depends on the secure management of cryptographic keys and secrets. If the RSA private key or the app_secret were to be compromised, the security of all encrypted data would be at risk. Proper storage and backup strategies should be in place.

2. Uniqueness of Inputs: For the HybridPatronPseudonymizer to function optimally and prevent potential collisions, it's imperative to ensure the combination of app_secret and patron_record is unique for each encryption operation. Failing to maintain this uniqueness might compromise the deterministic nature of the encryption and could lead to potential data ambiguities.

3. Deterministic Nature: While the deterministic nature of AES-SIV encryption ensures the same plaintext produces the same ciphertext, it also means that repeated encryption of the same data can make the system more vulnerable to certain types of analysis over time. Users should be aware of this when using the system for datasets with a lot of repeated entries.

4. Performance Overhead: Hybrid encryption, by its nature, involves both symmetric and asymmetric encryption operations. This can introduce a performance overhead, especially when dealing with large datasets.

Conclusion

Pseudonymization strikes the perfect balance between safeguarding data privacy and retaining its utility. By ensuring the protection of personal identifiers, data remains meaningful for deriving insights and analysis.

The HybridPatronPseudonymizer class provides a robust encryption mechanism that combines the efficiency of symmetric encryption with the key exchange benefits of asymmetric encryption. As long as the combination of app_secret and patron_record is unique for each encryption, the system should offer a low probability of collisions, making it suitable for most practical purposes.

hybrid_patron_pseudonymizer.py

import base64
import os
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric import rsa, padding as rsa_padding
from cryptography.hazmat.primitives.ciphers.aead import AESSIV
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives.kdf.pbkdf2 import PBKDF2HMAC
from cryptography.hazmat.primitives import hashes

class HybridPatronPseudonymizer:
    def __init__(
        self, 
        private_key_path='private_key.pem', 
        public_key_path='public_key.pem', 
        app_secret=None,
        rsa_key_password=None
    ):
        if app_secret is None:
            raise ValueError('Error: need app_secret to continue')

        self.app_secret = app_secret
        
        # If an RSA key password is provided, encode it, otherwise, set it to None
        self.rsa_key_password = rsa_key_password.encode() if rsa_key_password else None

        # Load RSA public key if it exists
        if os.path.exists(public_key_path):
            with open(public_key_path, 'rb') as f:
                self.rsa_public_key = serialization.load_pem_public_key(
                    f.read(), backend=default_backend()
                )
        else:
            raise ValueError("Public key file not found.")
        
        # Load RSA private key if it exists
        if os.path.exists(private_key_path):
            with open(private_key_path, 'rb') as f:
                self.rsa_private_key = serialization.load_pem_private_key(
                    f.read(), password=self.rsa_key_password, backend=default_backend()
                )
        else:
            self.rsa_private_key = None  # Set to None if private key file is missing

    @staticmethod
    def generate_rsa_key_pair(private_key_path='private_key.pem', public_key_path='public_key.pem', rsa_key_password=None):
        # Generate an RSA key pair
        rsa_private_key = rsa.generate_private_key(
            public_exponent=65537,
            key_size=2048,
            backend=default_backend()
        )
        rsa_public_key = rsa_private_key.public_key()

        # Serialize and save the private key
        encryption_algo = \
            serialization.BestAvailableEncryption(rsa_key_password.encode()) \
            if rsa_key_password \
            else serialization.NoEncryption()
        
        pem_private = rsa_private_key.private_bytes(
            encoding=serialization.Encoding.PEM,
            format=serialization.PrivateFormat.PKCS8,
            encryption_algorithm=encryption_algo
        )
        with open(private_key_path, 'wb') as f:
            f.write(pem_private)
        
        # Serialize and save the public key
        pem_public = rsa_public_key.public_bytes(
            encoding=serialization.Encoding.PEM,
            format=serialization.PublicFormat.SubjectPublicKeyInfo
        )
        with open(public_key_path, 'wb') as f:
            f.write(pem_public)
        
        print("RSA key pair generated and saved.")

    def _derive_aes_key(self, app_secret, patron_record):
        # Combine patron record's id and createdDate and app_secret
        combined_data = (
            str(patron_record["id"]) \
            + str(patron_record["createdDate"]) \
            + str(app_secret)
        ).encode()
        
        # Use PBKDF2 to derive the AES key
        # AES-128-SIV requires a key that's double the size of standard AES-128 ...
            # AES-128: Requires a 128-bit (or 16-byte) key.
            # AES-192: Requires a 192-bit (or 24-byte) key.
            # AES-256: Requires a 256-bit (or 32-byte) key.
        kdf = PBKDF2HMAC(
            algorithm=hashes.SHA256(),
            length=32,  # see above
            salt=combined_data,
            iterations=1000,
            backend=default_backend()
        )
        
        return kdf.derive(combined_data)

    def encrypt(self, data, patron_record):
        if isinstance(data, str):
            data = data.encode()
    
        # Derive the AES key using password (app_secret _and_ patron record
        aes_key = self._derive_aes_key(self.app_secret, patron_record)
        
        # Encrypt the data using AES-SIV mode
        ciphertext = AESSIV(aes_key).encrypt(data, [])
        
        # Encrypt the derived AES key using RSA public key
        encrypted_aes_key = self.rsa_public_key.encrypt(
            aes_key,
            rsa_padding.OAEP(
                mgf=rsa_padding.MGF1(algorithm=hashes.SHA256()),
                algorithm=hashes.SHA256(),
                label=None
            )
        )
        
        # Base64-encode both the encrypted AES key and the ciphertext before returning
        return {
            'encrypted_key': base64.b64encode(encrypted_aes_key).decode(),
            'ciphertext': base64.b64encode(ciphertext).decode()
        }

    def decrypt(self, encrypted_data, patron_record):
        # Check if private key is loaded
        if not self.rsa_private_key:
            raise ValueError("Private key is missing. Decryption not possible.")
        
        # Base64-decode the encrypted AES key and ciphertext
        encrypted_aes_key = base64.b64decode(encrypted_data['encrypted_key'].encode())
        ciphertext = base64.b64decode(encrypted_data['ciphertext'].encode())
        
        # Decrypt the AES key using RSA private key
        aes_key = self.rsa_private_key.decrypt(
            encrypted_aes_key,
            rsa_padding.OAEP(
                mgf=rsa_padding.MGF1(algorithm=hashes.SHA256()),
                algorithm=hashes.SHA256(),
                label=None
            )
        )
    
        # Decrypt the data using AES-SIV
        plaintext = AESSIV(aes_key).decrypt(ciphertext, [])
    
        return plaintext

if __name__ == "__main__":
    # %%timeit

    # Below is a general use pattern for this module:

    # 1. Generate the RSA Key Pair (only run once to generate and save the keys)
    HybridPatronPseudonymizer.generate_rsa_key_pair(
        private_key_path='dummy-test-private_key.pem', 
        public_key_path='dummy-test-public_key.pem',
        rsa_key_password="secret (don't really use this for the love of Cthulhu)"        
    )
    
    # Load the public key and (optionally) the encrypted private key
    pseudonymizer = HybridPatronPseudonymizer(
        private_key_path='dummy-test-private_key.pem', 
        public_key_path='dummy-test-public_key.pem',
        rsa_key_password="secret (don't really use this for the love of Cthulhu)",
        app_secret="again, pick something different if you enjoy the love of Cthulhu"
    )
    
    for i in range(2):
    
        patron_record = {"id": 2198439, "createdDate": "2017-03-28T01:10:11Z"}
        encrypted_data_bundle_name = pseudonymizer.encrypt("Chimperson, Chimpy H", patron_record)
        encrypted_data_bundle_barcode = pseudonymizer.encrypt("678999", patron_record)
        
        # 2. Decrypt Data
        # If the private key is encrypted, provide the password when creating the pseudonymizer instance
        try:
            decrypted_data1 = pseudonymizer.decrypt(encrypted_data_bundle_name, patron_record)
            decrypted_data2 = pseudonymizer.decrypt(encrypted_data_bundle_barcode, patron_record)
        except Exception as e:
            print(e)
        
        print(
            encrypted_data_bundle_name,
            encrypted_data_bundle_barcode,
            "---",
            decrypted_data1.decode(),
            decrypted_data2.decode(),
            sep="\n",
            end="\n\n\n"
        )
"""result:

RSA key pair generated and saved.
{
    'encrypted_key': 'CzlozHw4JM1VAJ6X14yN60NgydIAPVUa12cUTzkbwgB4BTi3BhSEJ8agpGvV2x/9Bu76LgyTPMVkPanIvSNO/5nlZBhXunrppm0FEXjW10lSV5PksTHYYR9scF125yvvM13bn9qixddxMTTo9Xhj6MX0959cB2de313Dcx6fo94IW7RmBRVeXfNmJUSZGhZnqNenoLda2LztOmkY3yfqeWBvXpPaT2S5ouHBR2+soHIx93pACAe6AwEM2dHh/zlDQk3JF1fUteYEpLzIU1WPni0dkIPdzAA5e6IxYNGF8nQz26imb8wNIyvDCqm1ypKhkwdoGKPO0yQfA5flER+PRQ==',
    'ciphertext': 'U6EtsvmZcFdy0NoRHThVt3yAafIkqUzqAS1n8nlZbzPSuyYR'
}
{
    'encrypted_key': 'D2JejrxWdkb8jM3nAHx3KB1D3uVtC+fAN7aKnIvQwfKQ4bE1y0lCk7U6R3eFNiXWYQwhsYONRO31en35CTEcb4nuqvAfnL4eBSdYn4oelRw1D+HkXH1ZO+qHn0uYjBbUOswdnTyduzxy89VxAOoD7AC8C8EVqquqr/YyxyDVz6OXtw4AtdpEpP5ZnFYHRVkx2YY6fyl5BEU/VOjcI6Or77pA9sZgHiY8lCZD31CZsCzcHa3m/qfnfS7ZaWrf959bmlYLuCD1nSnDNzq+YTu7Dhwu4qvJpmohjGgunC8pBmTJT0V/tk8JpOCY78XUaMcLk2ooKdf1/rH7XNam/UdmwQ==',
    'ciphertext': 'Aqma5Vj4DG5RnsOlOx72/9aFoS7bAw=='
}
---
Chimperson, Chimpy H
678999


{
    'encrypted_key': 'kVdocyJ13C6wpu1Lx3kLbGs2Mz2KUlucXNsswROmhrfMvaWMzwsOZYGOWerXMBqrqfnlc2Qebtcga/rmbBdoV0/lzOHNapED2Fr0yjuduye/XcepV+ZTu0RCE2MmFcFKyU8RqgX26aRIUdiwpVTAkGa6MgC3Zib30U32zmf31tqGvA6/HO/+/H29Hn4rZTo+7ga8ZydVh9jZRAdWZV0+tBJC2uP9kxQ0iG9Bx0GvDVxbvc5AZmb+Ua7ksijmvXncoOUwc19DOP3E+f27G0PnL0gFbSJVl/RzWUSkk6ByGwO2Ps7poVe6S4D6tH8ESP1e9u0lZWlvB7OkNaMzCMdrVw==',
    'ciphertext': 'U6EtsvmZcFdy0NoRHThVt3yAafIkqUzqAS1n8nlZbzPSuyYR'
}
{
    'encrypted_key': 'LAogNnUXPS3s4OaTRGK7EpBIo1EoZUsOgL9GKH1GCvvRB4+MuADt0VS2UdWUPJiZn+1EyuG4KPgAhpDIn/6s9qV0xv9r7mg5DqB5fmZKMWMxnjFZezwd+UEKMKClNNCxYFmJAwnnXFGvJMGFsHkLElJ36PpesQyzyWyxpIFv+aWiBL2B5FsVW/ikPAoQYp8P+dTYXKq54LAx5SVTftOhLvK+FLQu70bixk6sN7nAgrnjfOfP7mxxYnkdIjCW5I5dwXb4KAaq5z1o6CsLkadmgtOO0R+8JhLCOjiK6JCHJTQwuX5AlR8n6zwBPFS4RbYYAjQmaBiYMFPUPmPISG5oiw==',
    'ciphertext': 'Aqma5Vj4DG5RnsOlOx72/9aFoS7bAw=='
}
---
Chimperson, Chimpy H
678999
"""