MRPT  2.0.1
COpenNI2Sensor.h
Go to the documentation of this file.
1 /* +------------------------------------------------------------------------+
2  | Mobile Robot Programming Toolkit (MRPT) |
3  | https://www.mrpt.org/ |
4  | |
5  | Copyright (c) 2005-2020, Individual contributors, see AUTHORS file |
6  | See: https://www.mrpt.org/Authors - All rights reserved. |
7  | Released under BSD License. See: https://www.mrpt.org/License |
8  +------------------------------------------------------------------------+ */
9 #pragma once
10 
15 
16 namespace mrpt::hwdrivers
17 {
18 /** A class for grabing "range images", intensity images (either RGB or IR) and
19  *other information from an OpenNI2 sensor.
20  * This class permits to access several sensors simultaneously. The same
21  *options (resolution, fps, etc.) are used for every sensor.
22  *
23  * <h2>Configuration and usage:</h2> <hr>
24  * Data is returned as observations of type mrpt::obs::CObservation3DRangeScan.
25  * See those classes for documentation on their fields.
26  *
27  * As with any other CGenericSensor class, the normal sequence of methods to be
28  *called is:
29  * - CGenericSensor::loadConfig() - Or calls to the individual setXXX() to
30  *configure the sensor parameters.
31  * - COpenNI2Sensor::initialize() - to start the communication with the
32  *sensor.
33  * - call COpenNI2Sensor::getNextObservation() for getting the data.
34  *
35  * <h2>Calibration parameters</h2><hr>
36  * In this class we employ the OpenNI2 method to return depth images refered
37  *to the RGB camera. Otherwise we could specify
38  * an accurate transformation of depth images to 3D points, you'll have to
39  *calibrate your RGBD sensor for that, and supply
40  * the following <b>threee pieces of information</b> (default calibration
41  *data will be used otherwise, but they'll be not optimal for all sensors!):
42  * - Camera parameters for the RGB camera. See
43  *COpenNI2Sensor::setCameraParamsIntensity()
44  * - Camera parameters for the depth camera. See
45  *COpenNI2Sensor::setCameraParamsDepth()
46  * - The 3D relative pose of the two cameras. See
47  *COpenNI2Sensor::setRelativePoseIntensityWrtDepth()
48  *
49  * See https://www.mrpt.org/Kinect_calibration for a procedure to calibrate
50  *RGBD sensors with an interactive GUI program.
51  *
52  * <h2>Coordinates convention</h2><hr>
53  * The origin of coordinates is the focal point of the depth camera, with the
54  *axes oriented as in the
55  * diagram shown in mrpt::obs::CObservation3DRangeScan. Notice in that
56  *picture that the RGB camera is
57  * assumed to have axes as usual in computer vision, which differ from those
58  *for the depth camera.
59  *
60  * The X,Y,Z axes used to report the data from accelerometers coincide with
61  *those of the depth camera
62  * (e.g. the camera standing on a table would have an ACC_Z=-9.8m/s2).
63  *
64  * Notice however that, for consistency with stereo cameras, when loading the
65  *calibration parameters from
66  * a configuration file, the left-to-right pose increment is expected as if
67  *both RGB & IR cameras had
68  * their +Z axes pointing forward, +X to the right, +Y downwards (just like
69  *it's the standard in stereo cameras
70  * and in computer vision literature). In other words: the pose stored in
71  *this class uses a different
72  * axes convention for the depth camera than in a stereo camera, so when a
73  *pose L2R is loaded from a calibration file
74  * it's actually converted like:
75  *
76  * L2R(this class convention) = CPose3D(0,0,0,-90deg,0deg,-90deg) (+)
77  *L2R(in the config file)
78  *
79  *
80  * <h2>Some general comments</h2><hr>
81  * - Depth is grabbed in millimeters
82  * - This sensor can be also used from within rawlog-grabber to grab
83  *datasets
84  *within a robot with more sensors.
85  * - There is no built-in threading support, so if you use this class
86  *manually
87  *(not with-in rawlog-grabber),
88  * the ideal would be to create a thread and continuously request data
89  *from
90  *that thread (see mrpt::system::createThread ).
91  * - The intensity channel default to the RGB images, but it can be changed
92  *with setVideoChannel() to read the IR camera images (useful for calibrating).
93  * - There is a built-in support for an optional preview of the data on a
94  *window, so you don't need to even worry on creating a window to show them.
95  * - This class relies on an embedded version of libfreenect (you do NOT
96  *need
97  *to install it in your system). Thanks guys for the great job!
98  *
99  * <h2>Converting to 3D point cloud </h2><hr>
100  * You can convert the 3D observation into a 3D point cloud with this piece
101  *of code:
102  *
103  * \code
104  * mrpt::obs::CObservation3DRangeScan obs3D;
105  * mrpt::maps::CColouredPointsMap pntsMap;
106  * pntsMap.colorScheme.scheme = CColouredPointsMap::cmFromIntensityImage;
107  * pntsMap.loadFromRangeScan(obs3D);
108  * \endcode
109  *
110  * Then the point cloud mrpt::maps::CColouredPointsMap can be converted into
111  *an OpenGL object for
112  * rendering with mrpt::maps::CMetricMap::getAs3DObject() or alternatively
113  *with:
114  *
115  * \code
116  * mrpt::opengl::CPointCloudColoured::Ptr gl_points =
117  *mrpt::opengl::CPointCloudColoured::Create();
118  * gl_points->loadFromPointsMap(&pntsMap);
119  * \endcode
120  *
121  *
122  * <h2>Platform-specific comments</h2><hr>
123  * For more details, refer to <a href="http://openkinect.org/wiki/Main_Page"
124  *>libfreenect</a> documentation:
125  * - Linux: You'll need root privileges to access Kinect. Or, install
126  *<code>
127  *MRPT/scripts/51-kinect.rules </code> in <code>/etc/udev/rules.d/</code> to
128  *allow access to all users.
129  * - Windows:
130  * - Since MRPT 0.9.4 you'll only need to install <a
131  *href="http://sourceforge.net/projects/libusb-win32/files/libusb-win32-releases/"
132  *>libusb-win32</a>: download and extract the latest
133  *libusb-win32-bin-x.x.x.x.zip
134  * - To install the drivers, read this:
135  *http://openkinect.org/wiki/Getting_Started#Windows
136  * - MacOS: (write me!)
137  *
138  *
139  * <h2>Format of parameters for loading from a .ini file</h2><hr>
140  *
141  * \code
142  * PARAMETERS IN THE ".INI"-LIKE CONFIGURATION STRINGS:
143  * -------------------------------------------------------
144  * [supplied_section_name]
145  * sensorLabel = OPENNI2 // A text description
146  * preview_window = false // Show a window with a preview of the
147  *grabbed data in real-time
148  *
149  * device_number = 0 // Device index to open (0:first Kinect,
150  *1:second Kinect,...)
151  *
152  * grab_image = true // Grab the RGB image channel?
153  *(Default=true)
154  * grab_depth = true // Grab the depth channel? (Default=true)
155  * grab_3D_points = true // Grab the 3D point cloud? (Default=true)
156  *If disabled, points can be generated later on.
157  *
158  * video_channel = VIDEO_CHANNEL_RGB // Optional. Can be:
159  *VIDEO_CHANNEL_RGB (default) or VIDEO_CHANNEL_IR
160  *
161  * pose_x=0 // Camera position in the robot (meters)
162  * pose_y=0
163  * pose_z=0
164  * pose_yaw=0 // Angles in degrees
165  * pose_pitch=0
166  * pose_roll=0
167  *
168  *
169  * // Kinect sensor calibration:
170  * // See https://www.mrpt.org/Kinect_and_MRPT
171  *
172  * // Left/Depth camera
173  * [supplied_section_name_LEFT]
174  * rawlog-grabber-ignore = true // Instructs rawlog-grabber to ignore this
175  *section (it is not a separate device!)
176  *
177  * resolution = [640 488]
178  * cx = 314.649173
179  * cy = 240.160459
180  * fx = 572.882768
181  * fy = 542.739980
182  * dist = [-4.747169e-03 -4.357976e-03 0.000000e+00 0.000000e+00
183  *0.000000e+00] // The order is: [K1 K2 T1 T2 K3]
184  *
185  * // Right/RGB camera
186  * [supplied_section_name_RIGHT]
187  * rawlog-grabber-ignore = true // Instructs rawlog-grabber to ignore this
188  *section (it is not a separate device!)
189  *
190  * resolution = [640 480]
191  * cx = 322.515987
192  * cy = 259.055966
193  * fx = 521.179233
194  * fy = 493.033034
195  * dist = [5.858325e-02 3.856792e-02 0.000000e+00 0.000000e+00
196  *0.000000e+00] // The order is: [K1 K2 T1 T2 K3]
197  *
198  * // Relative pose of the right camera wrt to the left camera:
199  * // This assumes that both camera frames are such that +Z points
200  * // forwards, and +X and +Y to the right and downwards.
201  * // For the actual coordinates employed in 3D observations, see figure in
202  *mrpt::obs::CObservation3DRangeScan
203  * [supplied_section_name_LEFT2RIGHT_POSE]
204  * rawlog-grabber-ignore = true // Instructs rawlog-grabber to ignore this
205  *section (it is not a separate device!)
206  *
207  * pose_quaternion = [0.025575 -0.000609 -0.001462 0.999987 0.002038
208  *0.004335 -0.001693]
209  *
210  * \endcode
211  *
212  * More references to read:IMPEXP mrpt
213  * - http://http://www.openni.org/
214  * \ingroup mrpt_hwdrivers_grp
215  */
218 {
220 
221  public:
222  /** Default ctor
223  */
224  COpenNI2Sensor();
225  /** Default ctor
226  */
227  ~COpenNI2Sensor() override;
228 
229  /** Set the serial number of the device to open.
230  * \exception This method must throw an exception when such serial number
231  * is not found among the connected devices.
232  */
233  inline void setSerialToOpen(const unsigned serial)
234  {
235  m_serial_number = serial;
236  }
237 
238  /** Set the sensor_id of the device to open.
239  * \exception This method must throw an exception when such serial number
240  * is not found among the connected devices.
241  */
242  inline void setSensorIDToOpen(const unsigned sensor_id)
243  {
244  m_user_device_number = sensor_id;
245  }
246 
247  /** Initializes the 3D camera - should be invoked after calling loadConfig()
248  * or setting the different parameters with the set*() methods.
249  * \exception This method must throw an exception with a descriptive
250  * message if some critical error is found.
251  */
252  void initialize() override;
253 
254  /** To be called at a high rate (>XX Hz), this method populates the
255  * internal buffer of received observations.
256  * This method is mainly intended for usage within rawlog-grabber or
257  * similar programs.
258  * For an alternative, see getNextObservation()
259  * \exception This method must throw an exception with a descriptive
260  * message if some critical error is found.
261  * \sa getNextObservation
262  */
263  void doProcess() override;
264 
265  /** The main data retrieving function, to be called after calling
266  * loadConfig() and initialize().
267  * \param out_obs The output retrieved observation (only if
268  * there_is_obs=true).
269  * \param there_is_obs If set to false, there was no new observation.
270  * \param hardware_error True on hardware/comms error.
271  *
272  * \sa doProcess
273  */
274  void getNextObservation(
275  mrpt::obs::CObservation3DRangeScan& out_obs, bool& there_is_obs,
276  bool& hardware_error);
277 
278  /** Set the path where to save off-rawlog image files (this class DOES take
279  * into account this path).
280  * An empty string (the default value at construction) means to save
281  * images embedded in the rawlog, instead of on separate files.
282  * \exception std::exception If the directory doesn't exists and cannot be
283  * created.
284  */
285  void setPathForExternalImages(const std::string& directory) override;
286 
287  /** @name Sensor parameters (alternative to \a loadConfig ) and manual
288  control
289  @{ */
290 
291  /** Get the maximum range (meters) that can be read in the observation field
292  * "rangeImage" */
293  inline double getMaxRange() const { return m_maxRange; }
294  /** Get the row count in the camera images, loaded automatically upon camera
295  * open(). */
296  inline size_t rows() const { return m_cameraParamsRGB.nrows; }
297  /** Get the col count in the camera images, loaded automatically upon camera
298  * open(). */
299  inline size_t cols() const { return m_cameraParamsRGB.ncols; }
300  /** Get a const reference to the depth camera calibration parameters */
302  {
303  return m_cameraParamsRGB;
304  }
306  {
307  m_cameraParamsRGB = p;
308  }
309 
310  /** Get a const reference to the depth camera calibration parameters */
312  {
313  return m_cameraParamsDepth;
314  }
316  {
318  }
319 
320  /** Set the pose of the intensity camera wrt the depth camera \sa See
321  * mrpt::obs::CObservation3DRangeScan for a 3D diagram of this pose */
323  {
325  }
327  {
329  }
330 
331  /** Enable/disable the grabbing of the RGB channel */
332  inline void enableGrabRGB(bool enable = true) { m_grab_image = enable; }
333  inline bool isGrabRGBEnabled() const { return m_grab_image; }
334  /** Enable/disable the grabbing of the depth channel */
335  inline void enableGrabDepth(bool enable = true) { m_grab_depth = enable; }
336  inline bool isGrabDepthEnabled() const { return m_grab_depth; }
337  /** Enable/disable the grabbing of the 3D point clouds */
338  inline void enableGrab3DPoints(bool enable = true)
339  {
340  m_grab_3D_points = enable;
341  }
342  inline bool isGrab3DPointsEnabled() const { return m_grab_3D_points; }
343  /** @} */
344 
345  protected:
347  const mrpt::config::CConfigFileBase& configSource,
348  const std::string& section) override;
349 
351 
352  /** Show preview window while grabbing
353  */
354  bool m_preview_window{false};
355  /** If preview is enabled, only show 1 out of N images.
356  */
360 
361  /** Params for the RGB camera
362  */
364  /** Params for the Depth camera
365  */
367  /** See mrpt::obs::CObservation3DRangeScan for a diagram of this pose
368  */
370 
371  /** Sensor max range (meters)
372  */
373  double m_maxRange = 5.0;
374 
375  /** Number of device to open (0:first,...)
376  */
378  /** Serial number of device to open
379  */
381 
382 }; // End of class
383 } // namespace mrpt::hwdrivers
A generic interface for a wide-variety of sensors designed to be used in the application RawLogGrabbe...
mrpt::img::TCamera m_cameraParamsDepth
Params for the Depth camera.
void setCameraParamsIntensity(const mrpt::img::TCamera &p)
bool m_grab_image
The data that the RGBD sensors can return.
uint32_t nrows
Definition: TCamera.h:40
const mrpt::img::TCamera & getCameraParamsIntensity() const
Get a const reference to the depth camera calibration parameters.
mrpt::gui::CDisplayWindow::Ptr m_win_int
mrpt::gui::CDisplayWindow::Ptr m_win_range
void getNextObservation(mrpt::obs::CObservation3DRangeScan &out_obs, bool &there_is_obs, bool &hardware_error)
The main data retrieving function, to be called after calling loadConfig() and initialize().
A range or depth 3D scan measurement, as from a time-of-flight range camera or a structured-light dep...
Contains classes for various device interfaces.
size_t cols() const
Get the col count in the camera images, loaded automatically upon camera open().
A class for grabing "range images", intensity images (either RGB or IR) and other information from an...
This class allows loading and storing values and vectors of different types from a configuration text...
const mrpt::poses::CPose3D & getRelativePoseIntensityWrtDepth() const
mrpt::img::TCamera m_cameraParamsRGB
Params for the RGB camera.
int m_serial_number
Serial number of device to open.
const mrpt::img::TCamera & getCameraParamsDepth() const
Get a const reference to the depth camera calibration parameters.
void setSerialToOpen(const unsigned serial)
Set the serial number of the device to open.
Parameters for the Brown-Conrady camera lens distortion model.
Definition: TCamera.h:26
An abstract class for accessing OpenNI2 compatible sensors.
void enableGrab3DPoints(bool enable=true)
Enable/disable the grabbing of the 3D point clouds.
mrpt::poses::CPose3D m_sensorPoseOnRobot
double getMaxRange() const
Get the maximum range (meters) that can be read in the observation field "rangeImage".
void enableGrabRGB(bool enable=true)
Enable/disable the grabbing of the RGB channel.
void setSensorIDToOpen(const unsigned sensor_id)
Set the sensor_id of the device to open.
void setPathForExternalImages(const std::string &directory) override
Set the path where to save off-rawlog image files (this class DOES take into account this path)...
#define DEFINE_GENERIC_SENSOR(class_name)
This declaration must be inserted in all CGenericSensor classes definition, within the class declarat...
mrpt::poses::CPose3D m_relativePoseIntensityWRTDepth
See mrpt::obs::CObservation3DRangeScan for a diagram of this pose.
void setCameraParamsDepth(const mrpt::img::TCamera &p)
void doProcess() override
To be called at a high rate (>XX Hz), this method populates the internal buffer of received observati...
double m_maxRange
Sensor max range (meters)
A class used to store a 3D pose (a 3D translation + a rotation in 3D).
Definition: CPose3D.h:85
int m_user_device_number
Number of device to open (0:first,...)
void enableGrabDepth(bool enable=true)
Enable/disable the grabbing of the depth channel.
size_t rows() const
Get the row count in the camera images, loaded automatically upon camera open().
size_t m_preview_window_decimation
If preview is enabled, only show 1 out of N images.
~COpenNI2Sensor() override
Default ctor.
bool m_preview_window
Show preview window while grabbing.
void initialize() override
Initializes the 3D camera - should be invoked after calling loadConfig() or setting the different par...
void loadConfig_sensorSpecific(const mrpt::config::CConfigFileBase &configSource, const std::string &section) override
Loads specific configuration for the device from a given source of configuration parameters, for example, an ".ini" file, loading from the section "[iniSection]" (see config::CConfigFileBase and derived classes)
void setRelativePoseIntensityWrtDepth(const mrpt::poses::CPose3D &p)
Set the pose of the intensity camera wrt the depth camera.
uint32_t ncols
Camera resolution.
Definition: TCamera.h:40



Page generated by Doxygen 1.8.14 for MRPT 2.0.1 Git: 0fef1a6d7 Fri Apr 3 23:00:21 2020 +0200 at vie abr 3 23:20:28 CEST 2020