1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
|
.. -*- coding: utf-8; mode: rst -*-
.. _v4l2-meta-fmt-params:
.. _v4l2-meta-fmt-stat-3a:
******************************************************************
V4L2_META_FMT_IPU3_PARAMS ('ip3p'), V4L2_META_FMT_IPU3_3A ('ip3s')
******************************************************************
.. c:type:: ipu3_uapi_stats_3a
3A statistics
=============
For IPU3 ImgU, the 3A statistics accelerators collect different statistics over
an input bayer frame. Those statistics, defined in data struct :c:type:`ipu3_uapi_stats_3a`,
are obtained from "ipu3-imgu 3a stat" metadata capture video node, which are then
passed to user space for statistics analysis using :c:type:`v4l2_meta_format` interface.
The statistics collected are AWB (Auto-white balance) RGBS (Red, Green, Blue and
Saturation measure) cells, AWB filter response, AF (Auto-focus) filter response,
and AE (Auto-exposure) histogram.
struct :c:type:`ipu3_uapi_4a_config` saves configurable parameters for all above.
.. code-block:: c
struct ipu3_uapi_stats_3a {
struct ipu3_uapi_awb_raw_buffer awb_raw_buffer;
struct ipu3_uapi_ae_raw_buffer_aligned ae_raw_buffer[IPU3_UAPI_MAX_STRIPES];
struct ipu3_uapi_af_raw_buffer af_raw_buffer;
struct ipu3_uapi_awb_fr_raw_buffer awb_fr_raw_buffer;
struct ipu3_uapi_4a_config stats_4a_config;
__u32 ae_join_buffers;
__u8 padding[28];
struct ipu3_uapi_stats_3a_bubble_info_per_stripe stats_3a_bubble_per_stripe;
struct ipu3_uapi_ff_status stats_3a_status;
};
.. c:type:: ipu3_uapi_params
Pipeline parameters
===================
IPU3 pipeline has a number of image processing stages, each of which takes a
set of parameters as input. The major stages of pipelines are shown here:
Raw pixels -> Bayer Downscaling -> Optical Black Correction ->
Linearization -> Lens Shading Correction -> White Balance / Exposure /
Focus Apply -> Bayer Noise Reduction -> ANR -> Demosaicing -> Color
Correction Matrix -> Gamma correction -> Color Space Conversion ->
Chroma Down Scaling -> Chromatic Noise Reduction -> Total Color
Correction -> XNR3 -> TNR -> DDR
The table below presents a description of the above algorithms.
======================== =======================================================
Name Description
======================== =======================================================
Optical Black Correction Optical Black Correction block subtracts a pre-defined
value from the respective pixel values to obtain better
image quality.
Defined in :c:type:`ipu3_uapi_obgrid_param`.
Linearization This algo block uses linearization parameters to
address non-linearity sensor effects. The Lookup table
table is defined in
:c:type:`ipu3_uapi_isp_lin_vmem_params`.
SHD Lens shading correction is used to correct spatial
non-uniformity of the pixel response due to optical
lens shading. This is done by applying a different gain
for each pixel. The gain, black level etc are
configured in :c:type:`ipu3_uapi_shd_config_static`.
BNR Bayer noise reduction block removes image noise by
applying a bilateral filter.
See :c:type:`ipu3_uapi_bnr_static_config` for details.
ANR Advanced Noise Reduction is a block based algorithm
that performs noise reduction in the Bayer domain. The
convolution matrix etc can be found in
:c:type:`ipu3_uapi_anr_config`.
Demosaicing Demosaicing converts raw sensor data in Bayer format
into RGB (Red, Green, Blue) presentation. Then add
outputs of estimation of Y channel for following stream
processing by Firmware. The struct is defined as
:c:type:`ipu3_uapi_dm_config`. (TODO)
Color Correction Color Correction algo transforms sensor specific color
space to the standard "sRGB" color space. This is done
by applying 3x3 matrix defined in
:c:type:`ipu3_uapi_ccm_mat_config`.
Gamma correction Gamma correction :c:type:`ipu3_uapi_gamma_config` is a
basic non-linear tone mapping correction that is
applied per pixel for each pixel component.
CSC Color space conversion transforms each pixel from the
RGB primary presentation to YUV (Y: brightness,
UV: Luminance) presentation. This is done by applying
a 3x3 matrix defined in
:c:type:`ipu3_uapi_csc_mat_config`
CDS Chroma down sampling
After the CSC is performed, the Chroma Down Sampling
is applied for a UV plane down sampling by a factor
of 2 in each direction for YUV 4:2:0 using a 4x2
configurable filter :c:type:`ipu3_uapi_cds_params`.
CHNR Chroma noise reduction
This block processes only the chrominance pixels and
performs noise reduction by cleaning the high
frequency noise.
See struct :c:type:`ipu3_uapi_yuvp1_chnr_config`.
TCC Total color correction as defined in struct
:c:type:`ipu3_uapi_yuvp2_tcc_static_config`.
XNR3 eXtreme Noise Reduction V3 is the third revision of
noise reduction algorithm used to improve image
quality. This removes the low frequency noise in the
captured image. Two related structs are being defined,
:c:type:`ipu3_uapi_isp_xnr3_params` for ISP data memory
and :c:type:`ipu3_uapi_isp_xnr3_vmem_params` for vector
memory.
TNR Temporal Noise Reduction block compares successive
frames in time to remove anomalies / noise in pixel
values. :c:type:`ipu3_uapi_isp_tnr3_vmem_params` and
:c:type:`ipu3_uapi_isp_tnr3_params` are defined for ISP
vector and data memory respectively.
======================== =======================================================
A few stages of the pipeline will be executed by firmware running on the ISP
processor, while many others will use a set of fixed hardware blocks also
called accelerator cluster (ACC) to crunch pixel data and produce statistics.
ACC parameters of individual algorithms, as defined by
:c:type:`ipu3_uapi_acc_param`, can be chosen to be applied by the user
space through struct :c:type:`ipu3_uapi_flags` embedded in
:c:type:`ipu3_uapi_params` structure. For parameters that are configured as
not enabled by the user space, the corresponding structs are ignored by the
driver, in which case the existing configuration of the algorithm will be
preserved.
Both 3A statistics and pipeline parameters described here are closely tied to
the underlying camera sub-system (CSS) APIs. They are usually consumed and
produced by dedicated user space libraries that comprise the important tuning
tools, thus freeing the developers from being bothered with the low level
hardware and algorithm details.
It should be noted that IPU3 DMA operations require the addresses of all data
structures (that includes both input and output) to be aligned on 32 byte
boundaries.
The meta data :c:type:`ipu3_uapi_params` will be sent to "ipu3-imgu parameters"
video node in ``V4L2_BUF_TYPE_META_CAPTURE`` format.
.. code-block:: c
struct ipu3_uapi_params {
/* Flags which of the settings below are to be applied */
struct ipu3_uapi_flags use;
/* Accelerator cluster parameters */
struct ipu3_uapi_acc_param acc_param;
/* ISP vector address space parameters */
struct ipu3_uapi_isp_lin_vmem_params lin_vmem_params;
struct ipu3_uapi_isp_tnr3_vmem_params tnr3_vmem_params;
struct ipu3_uapi_isp_xnr3_vmem_params xnr3_vmem_params;
/* ISP data memory (DMEM) parameters */
struct ipu3_uapi_isp_tnr3_params tnr3_dmem_params;
struct ipu3_uapi_isp_xnr3_params xnr3_dmem_params;
/* Optical black level compensation */
struct ipu3_uapi_obgrid_param obgrid_param;
};
Intel IPU3 ImgU uAPI data types
===============================
.. kernel-doc:: include/uapi/linux/intel-ipu3.h
|
