在进行Robot Learning的研究中，我们常常会有如下问题：如何得到渲染环境中的相机小孔模型中的内参矩阵。如果已经知道一个相机的内参矩阵，我们又如何将其在环境中建模出来。为了解决这个问题我开发了UE4的插件：PinholeCamera - UE4

PinhonleCamera-UE4Plugin

Nowadays many pieces of research are based on synthetic images which can be found in this repository. This repository implements a pinhole camera model in UE4, which could be very useful for researchers working on Reinforcement learning,3D-Vision,SLAM,Robot navigation, so on.

Features

PinhonleCamera-UE4Plugin could help researchers put a camera in the real world (such as iphone) into Unreal world, and keep the Geometric consistency. We expect the work will do some benefits to communities. The following images will help readers to figure out the core contribution of this repository.

Experiment 1: We take a set of photos of a checkboard using the camera of Iphone7 in RealWorld. And then we take another set of photos of a checkboard (same size) using the camera implemented with our code at the same position in UnrealWorld. We then compare the photos taken in RealWord and UnrealWorld using “projection error”.

Experiment 2: We take two photos of a checkboard using the camera of Iphone7 in RealWorld. And then we take another two photos of a checkboard (same size) using the camera implemented with our code at the same position in UnrealWorld. We then using Multiple View Geometry to reconstruct the 3D points of checkboard, and we compare consequences with “reconstruction error”.

Before you start

PinhonleCamera-UE4Plugin is designed as a plugin of Unreal Engine. You should first install Unreal Engine before you start. And you can imstall Unreal Engine as following step:

1. Download the Epic Games Launcher. While the Unreal Engine is open source and free to download, registration is still required.
2. Run the Epic Games Launcher, open the Library tab from left, click on the “Add Versions” which should show the option to download Unreal 4.18 as shown below. If you have multiple versions of Unreal installed then make sure 4.18 is “Current” by clicking down arrow next to the Launch button for the version.

Setup PinhonleCamera-UE4Plugin

1. You will need an Unreal project that hosts the environment for PinhonleCamera-UE4Plugin. You can follow the instruction here.

2. Create new folder named as “Plugins” in project dictionary. And then copy “Pinhole_camera” folder into “Plugins” folder. The last structure looks like following:

   /Projectdir
   /Plugins
   /Pinhole_camera
     /Binaries
     /Intermediate
     /Resources
     /Source
     Pinhole_camera.uplugin
   

3. Right click XXX.uproject in your and then click Generate Visual Studio project files.

4. Double click XXX.sln and rebuild project in VS2015.

Usage of PinhonleCamera-UE4Plugin

1. In “Window->Developer Tools->Class Viewer”, you can find “camera_actor” and this is the camera model of our plugin. Drag it into your own environment.
2. You can change the parameter of camera in the detail tab of “camera_actor”. “savelocation”: The position to save photos. “camerainfo”: parameters of a camera model you want to setup.
   

Note: There are many other parameters you can set in detail tab, and these parameters provide with many powerful functions including changing texture, adding noise etc. They are implemented in USceneCaptureComponent2D calss of UnrealEngine and you can get information here