Python crypto for USB: SecureBytes and streaming encryption

Secure USB Encryption in Python: Memory Management & Stream Processing

For secure file transfer via USB, simplicity is key: insert the drive, enter a password, access encrypted data. VeraCrypt uses container files, complicating direct file handling across different operating systems. The main challenge? Protecting against data corruption during sudden power loss when a flash drive is pulled out mid-encryption.

A crypto engine built in Python leverages AES-GCM and ChaCha20. Focus is on secure memory storage of keys, streaming encryption of gigabyte-sized files without exhausting RAM, and robust data integrity mechanisms.

The SecureBytes class eliminates a critical vulnerability in Python and Java: garbage collection (GC) copying sensitive data (passwords, keys) to new memory locations, leaving remnants in RAM. Here's how it works:

class SecureBytes:
    def __init__(self, data: Union[bytes, bytearray, int]):
        if isinstance(data, int):
            self._buffer = bytearray(data)
        else:
            self._buffer = bytearray(data)
        self._finalized = False
        # Register a weak finalizer
        self._weak_ref = weakref.ref(self, self._cleanup_callback)

    def wipe(self, passes: int = 3):
        if self._finalized or len(self._buffer) == 0:
            return
        # Pass 1: random bytes
        self._buffer[:] = secrets.token_bytes(len(self._buffer))
        # Pass 2: zeros
        self._buffer[:] = b'\x00' * len(self._buffer)
        self._finalized = True
        gc.collect()

    def __del__(self):
        if not self._finalized:
            self.wipe()

Key features:

• Uses bytearray for in-place overwrites—unlike immutable bytes.
• Multi-pass wiping: random data followed by zeros (per NIST SP 800-88).
• Designed for use with context managers (with secure_key(...)) to ensure cleanup even on exceptions.

This minimizes key exposure in memory and prevents leaks through GC.

Streaming Encryption: MemorySensitiveReader

Encrypting a 10 GB file on a machine with only 4 GB of RAM demands a streaming approach. The MemorySensitiveReader class automatically switches modes based on file size and available memory:

class MemorySensitiveReader:
    def __init__(self, file_path: str, memory_threshold: int = 100 * 1024 * 1024):
        self.file_size = os.path.getsize(file_path)
        # Threshold to switch to streaming mode
        self.use_streaming = self.file_size > memory_threshold 

    def iter_chunks(self, chunk_size: int = 8192):
        # Read and encrypt in chunks
        ...

Solving the nonce problem: Reusing the same nonce with a single key in AES-GCM or ChaCha20 is a catastrophic flaw. To prevent this, each block gets a unique nonce derived from a base nonce and block index:

def _derive_block_nonce_12bit(base_nonce: bytes, block_index: int) -> bytes:
    # First 8 bytes = prefix, last 4 = block counter
    prefix = base_nonce[:8]
    block_counter = block_index.to_bytes(4, byteorder='big')
    return prefix + block_counter

Blocks are read in 8 KB chunks, with nonces generated from the base value and index—ensuring cryptographic strength regardless of file size.

Fault Tolerance in Operations

Standard encryption that deletes the original file upon failure leads to permanent data loss. This system includes:

• Lock file .encryption_lock.json: Tracks status as in_progress and lists processed files.
• Temporary .tmp files: Encrypt into a copy; delete the original only after successful verification.
• Integrity checks: Decrypt a block, compare HMAC and SHA-256 hashes.
• Rollback mechanism: On interruption, automatically restore originals from encrypted files.

This ensures that after a power failure, data is either fully encrypted or completely untouched—no partial states.

Supported Algorithms

• AES-256-GCM: Leverages hardware acceleration via AES-NI.
• ChaCha20-Poly1305: Optimal performance on ARM devices without AES-NI.
• XChaCha20-Poly1305: 24-byte nonce for large-scale encryption.

Parallelization using ThreadPoolExecutor boosts I/O performance for many small files, overcoming the GIL limitation.

What matters most:

• SecureBytes prevents key leaks via GC through multi-pass wiping.
• Streaming encryption with unique nonces for files over 100 MB—zero collision risk.
• Full fault tolerance: lock files and automatic rollback on power loss.
• Support for AES-GCM, ChaCha20, XChaCha20—all without compromising security.
• Metadata stored in .usb_crypt_meta.json, passwords ≥12 characters with validation.

Limitations & Threat Model

The tool protects against lost or stolen drives but does not hide filenames or directory structure. It offers no defense against keyloggers on the host machine. Best suited for mid-to-senior developers needing custom USB encryption with strong focus on memory safety and resilience.

— Editorial Team