FrameworkGuidesModulesRelease NotesAbout

DEP and PyInstaller

Data Execution Prevention (DEP) is a system-level security feature that prevents code from executing in memory regions marked as non-executable. PyInstaller is a tool that packages Python scripts into standalone executables. The intersection of these two technologies is relevant for security testing because it affects how Python-based payloads behave on modern Windows systems — and what defenders can detect.

This page covers the technical relationship between DEP and PyInstaller, the implications for both payload developers and detection engineers, and practical guidance for lab testing.

What DEP Does

DEP marks memory pages as either executable or non-executable. When code attempts to execute from a non-executable page (such as the stack or heap), DEP triggers an access violation and terminates the process. This prevents a class of attacks that rely on writing shellcode to data regions and then executing it.

Modern Windows systems enable DEP by default for system processes and can be configured to enforce it for all processes. Hardware DEP (NX bit) is supported by all modern CPUs.

How PyInstaller Works

PyInstaller bundles a Python script, the Python interpreter, and all dependencies into a single executable (or a directory). At runtime, it extracts the bundled contents to a temporary directory and launches the Python interpreter with the target script.

Key characteristics relevant to security:

  • Temporary file extraction — Files are written to %TEMP%\_MEIxxxxx directories
  • Process spawning — The stub executable launches the Python interpreter as a child process
  • File system artifacts — Extracted .pyc files, DLLs, and the Python runtime are visible on disk
  • Signature — PyInstaller executables have recognizable structures in their PE headers

DEP's Impact on PyInstaller Payloads

When a PyInstaller-packaged payload attempts to execute shellcode from a non-executable memory region, DEP blocks the execution. This affects certain payload types that rely on allocating memory and executing shellcode directly.

The practical implications:

  • For payload testing: Payloads that use VirtualAlloc with execute permissions are not affected by DEP. Payloads that write to the stack or heap without proper memory permissions will fail.
  • For detection: DEP violations generate Windows Error Reporting (WER) entries and can be monitored. A process terminated by DEP is worth investigating.

Detection Opportunities

PyInstaller-packaged payloads create multiple detection opportunities:

  • Temporary file creation — Monitor %TEMP% for _MEI directories
  • PE header analysis — PyInstaller executables have characteristic section names and structures
  • Unusual Python runtime execution — Python.dll loaded by a non-Python application
  • DEP violations — WER events indicating DEP terminations
  • Behavioral patterns — A new executable extracting files and spawning interpreters

Lab Testing

When testing DEP interaction with PyInstaller payloads in your lab:

  1. Configure DEP settings on your target (OptIn vs. AlwaysOn)
  2. Generate a PyInstaller-packaged payload
  3. Execute and observe whether DEP intervenes
  4. Check WER logs for DEP violation records
  5. Verify that your endpoint monitoring captures the extraction and execution artifacts

Related