1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
|
# Debug Macro Check
This Python application scans all files in a build package for debug macro formatting issues. It is intended to be a
fundamental build-time check that is part of a normal developer build process to catch errors right away.
As a build plugin, it is capable of finding these errors early in the development process after code is initially
written to ensure that all code tested is free of debug macro formatting errors. These errors often creep into debug
prints in error conditions that are not frequently executed making debug even more difficult and confusing when they
are encountered. In other cases, debug macros with these errors in the main code path can lead to unexpected behavior
when executed. As a standalone script, it can be easily run manually or integrated into other CI processes.
The plugin is part of a set of debug macro check scripts meant to be relatively portable so they can be applied to
additional code bases with minimal effort.
## 1. BuildPlugin/DebugMacroCheckBuildPlugin.py
This is the build plugin. It is discovered within the Stuart Self-Describing Environment (SDE) due to the accompanying
file `DebugMacroCheck_plugin_in.yaml`.
Since macro errors are considered a coding bug that should be found and fixed during the build phase of the developer
process (before debug and testing), this plugin is run in pre-build. It will run within the scope of the package
being compiled. For a platform build, this means it will run against the package being built. In a CI build, it will
run in pre-build for each package as each package is built.
The build plugin has the following attributes:
1. Registered at `global` scope. This means it will always run.
2. Called only on compilable build targets (i.e. does nothing on `"NO-TARGET"`).
3. Runs as a pre-build step. This means it gives results right away to ensure compilation follows on a clean slate.
This also means it runs in platform build and CI. It is run in CI as a pre-build step when the `CompilerPlugin`
compiles code. This ensures even if the plugin was not run locally, all code submissions have been checked.
4. Reports any errors in the build log and fails the build upon error making it easy to discover problems.
5. Supports two methods of configuration via "substitution strings":
1. By setting a build variable called `DEBUG_MACRO_CHECK_SUB_FILE` with the name of a substitution YAML file to
use.
**Example:**
```python
shell_environment.GetBuildVars().SetValue(
"DEBUG_MACRO_CHECK_SUB_FILE",
os.path.join(self.GetWorkspaceRoot(), "DebugMacroCheckSub.yaml"),
"Set in CISettings.py")
```
**Substitution File Content Example:**
```yaml
---
# OvmfPkg/CpuHotplugSmm/ApicId.h
# Reason: Substitute with macro value
FMT_APIC_ID: 0x%08x
# DynamicTablesPkg/Include/ConfigurationManagerObject.h
# Reason: Substitute with macro value
FMT_CM_OBJECT_ID: 0x%lx
# OvmfPkg/IntelTdx/TdTcg2Dxe/TdTcg2Dxe.c
# Reason: Acknowledging use of two format specifiers in string with one argument
# Replace ternary operator in debug string with single specifier
'Index == COLUME_SIZE/2 ? " | %02x" : " %02x"': "%d"
# DynamicTablesPkg/Library/Common/TableHelperLib/ConfigurationManagerObjectParser.c
# ShellPkg/Library/UefiShellAcpiViewCommandLib/AcpiParser.c
# Reason: Acknowledge that string *should* expand to one specifier
# Replace variable with expected number of specifiers (1)
Parser[Index].Format: "%d"
```
2. By entering the string substitutions directory into a dictionary called `StringSubstitutions` in a
`DebugMacroCheck` section of the package CI YAML file.
**Example:**
```yaml
"DebugMacroCheck": {
"StringSubstitutions": {
"SUB_A": "%Lx"
}
}
```
### Debug Macro Check Build Plugin: Simple Disable
The build plugin can simply be disabled by setting an environment variable named `"DISABLE_DEBUG_MACRO_CHECK"`. The
plugin is disabled on existence of the variable. The contents of the variable are not inspected at this time.
## 2. DebugMacroCheck.py
This is the main Python module containing the implementation logic. The build plugin simply wraps around it.
When first running debug macro check against a new, large code base, it is recommended to first run this standalone
script and address all of the issues and then enable the build plugin.
The module supports a number of configuration parameters to ease debug of errors and to provide flexibility for
different build environments.
### EDK 2 PyTool Library Dependency
This script has minimal library dependencies. However, it has one dependency you might not be familiar with on the
Tianocore EDK 2 PyTool Library (edk2toollib):
```py
from edk2toollib.utility_functions import RunCmd
```
You simply need to install the following pip module to use this library: `edk2-pytool-library`
(e.g. `pip install edk2-pytool-library`)
More information is available here:
- PyPI page: [edk2-pytool-library](https://pypi.org/project/edk2-pytool-library/)
- GitHub repo: [tianocore/edk2-pytool-library](https://github.com/tianocore/edk2-pytool-library)
If you strongly prefer not including this additional dependency, the functionality imported here is relatively
simple to substitute with the Python [`subprocess`](https://docs.python.org/3/library/subprocess.html) built-in
module.
### Examples
Simple run against current directory:
`> python DebugMacroCheck.py -w .`
Simple run against a single file:
`> python DebugMacroCheck.py -i filename.c`
Run against a directory with output placed into a file called "debug_macro_check.log":
`> python DebugMacroCheck.py -w . -l`
Run against a directory with output placed into a file called "custom.log" and debug log messages enabled:
`> python DebugMacroCheck.py -w . -l custom.log -v`
Run against a directory with output placed into a file called "custom.log", with debug log messages enabled including
python script function and line number, use a substitution file called "file_sub.yaml", do not show the progress bar,
and run against .c and .h files:
`> python DebugMacroCheck.py -w . -l custom.log -vv -s file_sub.yaml -n -e .c .h`
> **Note**: It is normally not recommended to run against .h files as they and many other non-.c files normally do
not have full `DEBUG` macro prints.
```plaintext
usage: Debug Macro Checker [-h] (-w WORKSPACE_DIRECTORY | -i [INPUT_FILE]) [-l [LOG_FILE]] [-s SUBSTITUTION_FILE] [-v] [-n] [-q] [-u]
[-df] [-ds] [-e [EXTENSIONS ...]]
Checks for debug macro formatting errors within files recursively located within a given directory.
options:
-h, --help show this help message and exit
-w WORKSPACE_DIRECTORY, --workspace-directory WORKSPACE_DIRECTORY
Directory of source files to check.
-i [INPUT_FILE], --input-file [INPUT_FILE]
File path for an input file to check.
Note that some other options do not apply if a single file is specified such as the
git options and file extensions.
-e [EXTENSIONS ...], --extensions [EXTENSIONS ...]
List of file extensions to include.
(default: ['.c'])
Optional input and output:
-l [LOG_FILE], --log-file [LOG_FILE]
File path for log output.
(default: if the flag is given with no file path then a file called
debug_macro_check.log is created and used in the current directory)
-s SUBSTITUTION_FILE, --substitution-file SUBSTITUTION_FILE
A substitution YAML file specifies string substitutions to perform within the debug macro.
This is intended to be a simple mechanism to expand the rare cases of pre-processor
macros without directly involving the pre-processor. The file consists of one or more
string value pairs where the key is the identifier to replace and the value is the value
to replace it with.
This can also be used as a method to ignore results by replacing the problematic string
with a different string.
-v, --verbose-log-file
Set file logging verbosity level.
- None: Info & > level messages
- '-v': + Debug level messages
- '-vv': + File name and function
- '-vvv': + Line number
- '-vvvv': + Timestamp
(default: verbose logging is not enabled)
-n, --no-progress-bar
Disables progress bars.
(default: progress bars are used in some places to show progress)
-q, --quiet Disables console output.
(default: console output is enabled)
-u, --utf8w Shows warnings for file UTF-8 decode errors.
(default: UTF-8 decode errors are not shown)
Optional git control:
-df, --do-not-ignore-git-ignore-files
Do not ignore git ignored files.
(default: files in git ignore files are ignored)
-ds, --do-not-ignore-git_submodules
Do not ignore files in git submodules.
(default: files in git submodules are ignored)
```
## String Substitutions
`DebugMacroCheck` currently runs separate from the compiler toolchain. This has the advantage that it is very portable
and can run early in the build process, but it also means pre-processor macro expansion does not happen when it is
invoked.
In practice, it has been very rare that this is an issue for how most debug macros are written. In case it is, a
substitution file can be used to inform `DebugMacroCheck` about the string substitution the pre-processor would
perform.
This pattern should be taken as a warning. It is just as difficult for humans to keep debug macro specifiers and
arguments balanced as it is for `DebugMacroCheck` pre-processor macro substitution is used. By separating the string
from the actual arguments provided, it is more likely for developers to make mistakes matching print specifiers in
the string to the arguments. If usage is reasonable, a string substitution can be used as needed.
### Ignoring Errors
Since substitution files perform a straight textual substitution in macros discovered, it can be used to replace
problematic text with text that passes allowing errors to be ignored.
## Python Version Required (3.10)
This script is written to take advantage of new Python language features in Python 3.10. If you are not using Python
3.10 or later, you can:
1. Upgrade to Python 3.10 or greater
2. Run this script in a [virtual environment](https://docs.python.org/3/tutorial/venv.html) with Python 3.10
or greater
3. Customize the script for compatibility with your Python version
These are listed in order of recommendation. **(1)** is the simplest option and will upgrade your environment to a
newer, safer, and better Python experience. **(2)** is the simplest approach to isolate dependencies to what is needed
to run this script without impacting the rest of your system environment. **(3)** creates a one-off fork of the script
that, by nature, has a limited lifespan and will make accepting future updates difficult but can be done with relatively
minimal effort back to recent Python 3 releases.
|
