BaseTools/Plugin: Add DebugMacroCheck

Adds a plugin that finds debug macro formatting issues. These errors
often creep into debug prints in error conditions not frequently
executed and make debug more difficult when they are encountered.

The code can be as a standalone script which is useful to find
problems in a large codebase that has not been checked before or as
a build plugin that notifies a developer of an error right away.

The script was already used to find numerous issues in edk2 in the
past so there's not many code fixes in this change. More details
are available in the readme file:

.pytool\Plugin\DebugMacroCheck\Readme.md

Cc: Sean Brogan <sean.brogan@microsoft.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Rebecca Cran <rebecca@bsdio.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Signed-off-by: Michael Kubacki <michael.kubacki@microsoft.com>
Reviewed-by: Michael D Kinney <michael.d.kinney@intel.com>

1 files changed, 253 insertions, 0 deletions

diff --git a/BaseTools/Plugin/DebugMacroCheck/Readme.md b/BaseTools/Plugin/DebugMacroCheck/Readme.md
new file mode 100644
index 0000000000..33f1ad9790
--- /dev/null
+++ b/BaseTools/Plugin/DebugMacroCheck/Readme.md
@@ -0,0 +1,253 @@
+# Debug Macro Check

+

+This Python application scans all files in a build package for debug macro formatting issues. It is intended to be a

+fundamental build-time check that is part of a normal developer build process to catch errors right away.

+

+As a build plugin, it is capable of finding these errors early in the development process after code is initially

+written to ensure that all code tested is free of debug macro formatting errors. These errors often creep into debug

+prints in error conditions that are not frequently executed making debug even more difficult and confusing when they

+are encountered. In other cases, debug macros with these errors in the main code path can lead to unexpected behavior

+when executed. As a standalone script, it can be easily run manually or integrated into other CI processes.

+

+The plugin is part of a set of debug macro check scripts meant to be relatively portable so they can be applied to

+additional code bases with minimal effort.

+

+## 1. BuildPlugin/DebugMacroCheckBuildPlugin.py

+

+This is the build plugin. It is discovered within the Stuart Self-Describing Environment (SDE) due to the accompanying

+file `DebugMacroCheck_plugin_in.yaml`.

+

+Since macro errors are considered a coding bug that should be found and fixed during the build phase of the developer

+process (before debug and testing), this plugin is run in pre-build. It will run within the scope of the package

+being compiled. For a platform build, this means it will run against the package being built. In a CI build, it will

+run in pre-build for each package as each package is built.

+

+The build plugin has the following attributes:

+

+  1. Registered at `global` scope. This means it will always run.

+

+  2. Called only on compilable build targets (i.e. does nothing on `"NO-TARGET"`).

+

+  3. Runs as a pre-build step. This means it gives results right away to ensure compilation follows on a clean slate.

+     This also means it runs in platform build and CI. It is run in CI as a pre-build step when the `CompilerPlugin`

+     compiles code. This ensures even if the plugin was not run locally, all code submissions have been checked.

+

+  4. Reports any errors in the build log and fails the build upon error making it easy to discover problems.

+

+  5. Supports two methods of configuration via "substitution strings":

+

+     1. By setting a build variable called `DEBUG_MACRO_CHECK_SUB_FILE` with the name of a substitution YAML file to

+        use.

+

+        **Example:**

+

+        ```python

+        shell_environment.GetBuildVars().SetValue(

+                                            "DEBUG_MACRO_CHECK_SUB_FILE",

+                                            os.path.join(self.GetWorkspaceRoot(), "DebugMacroCheckSub.yaml"),

+                                            "Set in CISettings.py")

+        ```

+

+        **Substitution File Content Example:**

+

+        ```yaml

+        ---

+        # OvmfPkg/CpuHotplugSmm/ApicId.h

+        # Reason: Substitute with macro value

+        FMT_APIC_ID: 0x%08x

+

+        # DynamicTablesPkg/Include/ConfigurationManagerObject.h

+        # Reason: Substitute with macro value

+        FMT_CM_OBJECT_ID: 0x%lx

+

+        # OvmfPkg/IntelTdx/TdTcg2Dxe/TdTcg2Dxe.c

+        # Reason: Acknowledging use of two format specifiers in string with one argument

+        #         Replace ternary operator in debug string with single specifier

+        'Index == COLUME_SIZE/2 ? " | %02x" : " %02x"': "%d"

+

+        # DynamicTablesPkg/Library/Common/TableHelperLib/ConfigurationManagerObjectParser.c

+        # ShellPkg/Library/UefiShellAcpiViewCommandLib/AcpiParser.c

+        # Reason: Acknowledge that string *should* expand to one specifier

+        #         Replace variable with expected number of specifiers (1)

+        Parser[Index].Format: "%d"

+        ```

+

+     2. By entering the string substitutions directory into a dictionary called `StringSubstitutions` in a

+        `DebugMacroCheck` section of the package CI YAML file.

+

+        **Example:**

+

+        ```yaml

+        "DebugMacroCheck": {

+          "StringSubstitutions": {

+            "SUB_A": "%Lx"

+          }

+        }

+        ```

+

+### Debug Macro Check Build Plugin: Simple Disable

+

+The build plugin can simply be disabled by setting an environment variable named `"DISABLE_DEBUG_MACRO_CHECK"`. The

+plugin is disabled on existence of the variable. The contents of the variable are not inspected at this time.

+

+## 2. DebugMacroCheck.py

+

+This is the main Python module containing the implementation logic. The build plugin simply wraps around it.

+

+When first running debug macro check against a new, large code base, it is recommended to first run this standalone

+script and address all of the issues and then enable the build plugin.

+

+The module supports a number of configuration parameters to ease debug of errors and to provide flexibility for

+different build environments.

+

+### EDK 2 PyTool Library Dependency

+

+This script has minimal library dependencies. However, it has one dependency you might not be familiar with on the

+Tianocore EDK 2 PyTool Library (edk2toollib):

+

+```py

+from edk2toollib.utility_functions import RunCmd

+```

+

+You simply need to install the following pip module to use this library: `edk2-pytool-library`

+(e.g. `pip install edk2-pytool-library`)

+

+More information is available here:

+

+- PyPI page: [edk2-pytool-library](https://pypi.org/project/edk2-pytool-library/)

+- GitHub repo: [tianocore/edk2-pytool-library](https://github.com/tianocore/edk2-pytool-library)

+

+If you strongly prefer not including this additional dependency, the functionality imported here is relatively

+simple to substitute with the Python [`subprocess`](https://docs.python.org/3/library/subprocess.html) built-in

+module.

+

+### Examples

+

+Simple run against current directory:

+

+`> python DebugMacroCheck.py -w .`

+

+Simple run against a single file:

+

+`> python DebugMacroCheck.py -i filename.c`

+

+Run against a directory with output placed into a file called "debug_macro_check.log":

+

+`> python DebugMacroCheck.py -w . -l`

+

+Run against a directory with output placed into a file called "custom.log" and debug log messages enabled:

+

+`> python DebugMacroCheck.py -w . -l custom.log -v`

+

+Run against a directory with output placed into a file called "custom.log", with debug log messages enabled including

+python script function and line number, use a substitution file called "file_sub.yaml", do not show the progress bar,

+and run against .c and .h files:

+

+`> python DebugMacroCheck.py -w . -l custom.log -vv -s file_sub.yaml -n -e .c .h`

+

+> **Note**: It is normally not recommended to run against .h files as they and many other non-.c files normally do

+  not have full `DEBUG` macro prints.

+

+```plaintext

+usage: Debug Macro Checker [-h] (-w WORKSPACE_DIRECTORY | -i [INPUT_FILE]) [-l [LOG_FILE]] [-s SUBSTITUTION_FILE] [-v] [-n] [-q] [-u]

+                           [-df] [-ds] [-e [EXTENSIONS ...]]

+

+Checks for debug macro formatting errors within files recursively located within a given directory.

+

+options:

+  -h, --help            show this help message and exit

+  -w WORKSPACE_DIRECTORY, --workspace-directory WORKSPACE_DIRECTORY

+                        Directory of source files to check.

+

+  -i [INPUT_FILE], --input-file [INPUT_FILE]

+                        File path for an input file to check.

+

+                        Note that some other options do not apply if a single file is specified such as the

+                        git options and file extensions.

+

+  -e [EXTENSIONS ...], --extensions [EXTENSIONS ...]

+                        List of file extensions to include.

+                        (default: ['.c'])

+

+Optional input and output:

+  -l [LOG_FILE], --log-file [LOG_FILE]

+                        File path for log output.

+                        (default: if the flag is given with no file path then a file called

+                        debug_macro_check.log is created and used in the current directory)

+

+  -s SUBSTITUTION_FILE, --substitution-file SUBSTITUTION_FILE

+                        A substitution YAML file specifies string substitutions to perform within the debug macro.

+

+                        This is intended to be a simple mechanism to expand the rare cases of pre-processor

+                        macros without directly involving the pre-processor. The file consists of one or more

+                        string value pairs where the key is the identifier to replace and the value is the value

+                        to replace it with.

+

+                        This can also be used as a method to ignore results by replacing the problematic string

+                        with a different string.

+

+  -v, --verbose-log-file

+                        Set file logging verbosity level.

+                         - None:    Info & > level messages

+                         - '-v':    + Debug level messages

+                         - '-vv':   + File name and function

+                         - '-vvv':  + Line number

+                         - '-vvvv': + Timestamp

+                        (default: verbose logging is not enabled)

+

+  -n, --no-progress-bar

+                        Disables progress bars.

+                        (default: progress bars are used in some places to show progress)

+

+  -q, --quiet           Disables console output.

+                        (default: console output is enabled)

+

+  -u, --utf8w           Shows warnings for file UTF-8 decode errors.

+                        (default: UTF-8 decode errors are not shown)

+

+

+Optional git control:

+  -df, --do-not-ignore-git-ignore-files

+                        Do not ignore git ignored files.

+                        (default: files in git ignore files are ignored)

+

+  -ds, --do-not-ignore-git_submodules

+                        Do not ignore files in git submodules.

+                        (default: files in git submodules are ignored)

+```

+

+## String Substitutions

+

+`DebugMacroCheck` currently runs separate from the compiler toolchain. This has the advantage that it is very portable

+and can run early in the build process, but it also means pre-processor macro expansion does not happen when it is

+invoked.

+

+In practice, it has been very rare that this is an issue for how most debug macros are written. In case it is, a

+substitution file can be used to inform `DebugMacroCheck` about the string substitution the pre-processor would

+perform.

+

+This pattern should be taken as a warning. It is just as difficult for humans to keep debug macro specifiers and

+arguments balanced as it is for `DebugMacroCheck` pre-processor macro substitution is used. By separating the string

+from the actual arguments provided, it is more likely for developers to make mistakes matching print specifiers in

+the string to the arguments. If usage is reasonable, a string substitution can be used as needed.

+

+### Ignoring Errors

+

+Since substitution files perform a straight textual substitution in macros discovered, it can be used to replace

+problematic text with text that passes allowing errors to be ignored.

+

+## Python Version Required (3.10)

+

+This script is written to take advantage of new Python language features in Python 3.10. If you are not using Python

+3.10 or later, you can:

+

+  1. Upgrade to Python 3.10 or greater

+  2. Run this script in a [virtual environment](https://docs.python.org/3/tutorial/venv.html) with Python 3.10

+     or greater

+  3. Customize the script for compatibility with your Python version

+

+These are listed in order of recommendation. **(1)** is the simplest option and will upgrade your environment to a

+newer, safer, and better Python experience. **(2)** is the simplest approach to isolate dependencies to what is needed

+to run this script without impacting the rest of your system environment. **(3)** creates a one-off fork of the script

+that, by nature, has a limited lifespan and will make accepting future updates difficult but can be done with relatively

+minimal effort back to recent Python 3 releases.