A Low-Resolution IR-Array as a Doorway Occupancy Counter in a Smart Building

A doorway counter, which detects a person underpass at a room entry/exit, may be the most accurate type of occupancy counters used in buildings. An occupancy counter, which uses a low-resolution IR-imager and Raspberry Pi board has been constructed. The imager provides only 8 x 8 pixels initial resolution, but it has been enhanced using two-dimensional interpolation. Due to the low absolute accuracy in temperature measurements, the imager is set to measure temperature difference between a target and background. Signal-tonoise ratio is also increased using discrete two-dimensional convolution filtering. The blob detection and tracking algorithm deduces the direction of an occupant and finally increments or decrements the counter. A heat signature varies between people and depends on person’s clothing. An on-board server on Raspberry Pi distributes the data via Wi-Fi to any client device in the net. The complete system includes also wireless PIR-sensors. The low-resolution IR occupancy counter has been compared with counters based on different technologies. The benefits of a low-resolution IR-imager are privacy preservation, operation capability in total darkness, energy-efficient passive operation and a low price. Keywords—Occupancy counter, low-resolution IR, smart building, wireless sensor, Kalman filter tracking, PIR-sensor, GridEye, Raspberrry Pi.


Introduction
Around 40% of all the energy used in the USA [1] and in the world [2] is used in buildings. This energy is mainly used in heating, ventilation and air conditioning (HVAC) and lighting purposes. Conventional systems in buildings use this energy despite the buildings level of usage. It is clear, that significant energy savings can be reached by taking consideration the actual need for HVAC functions which depends on the number of people in a building. Taking into consideration the presence or absence of people, 10-15 % energy savings can be reached in HVAC [2]. The highest saving can be reached in air conditioning (40%) and in lighting (35-75%) [2]. Demand based HVAC and lighting control not only means energy savings but also improved comfortable indoor environments, benefits from the building management point of view or improved security in emergency situations in buildings. Occupancy detectors and counters are needed to provide this information for a building automation system. Sensors based on infrared technology are passive, low-power and privacy preserving. [3].
A low-resolution IR-imager is an infrared sensitive imaging device having much less pixels than a typical IR camera array. As a result of lower resolution, its price is also lower. Its operation is based on thermopile technology distinct to pyroelectricity which PIR sensor uses. While PIR-sensor is a motion detector, an IR array detects both stationary and moving targets. The clear benefits of low-resolution arrays are: Privacy preservation, low power consumption (passive device), low price, insensitivity to ambient lighting level and changes, operation in total darkness, fast response time, easy deployment and easy image processing.
GridEye AMG88xx (Panasonic), 8 x 8 pixels infrared array is a widely known component and has been used previous human occupancy detection, counting or tracking studies [1] [4][5][6][7][8][9][10][11][12][13]. IR-arrays with the same amount of pixels as GridEye, but in different dimensions, are available. MLX90620 (Melexis) has 16 x 4 array. Its use in similar applications has been studied by [14]. The next resolution level in imaging arrays is 16 x 16. IR 1011 (AMK Semiconductor) has been used in similar applications by [15] and [16]. At the moment, this paper is written 32 x 24 pixels IR-arrays MLX90640 (Melexis) are already available. Authors of [17] have used this array in their studies. Arrays of 128 x 128 pixels and more can already be considered as an IR-camera array.
Most of the previous studies have focused on detecting stationary or tracking moving people in a room using multiple sensors as a sensor network. The most common application for this is an assisted living of elderly people in their home for detecting possible emergency situations [15]. There are also studies consider a doorway people detection and counting. A doorway counter is an example of perimeter sensors which sense people at an entry/exit points of a room. A doorway counter is the most obvious type of device for counting the number of people in a certain room or part of a building. [18] presents a review of available technologies for such a counter. A doorway counter has 96% accuracy in indicating whether a room is occupied or not, 91 % accuracy in showing the time a room has been occupied and 87% accuracy in showing times a person has entered in or exited from a room. [9] presents a doorway counter which is a single sensor located on the side of a doorway at a height between 125 cm to 140 cm. Its operation has been verified with a door width of 90 cm and 180 cm. This assembly can be considered as a replacement for a traditional break beam sensor. In previous studies GridEye has shown surprisingly good accuracy in occupant counting. According to [9], accuracy in estimating the number of occupants in a room was 93%. As a doorway sensor, mounted on the side at height of 140 cm, the accuracy was 89-92%.

A Doorway Counter
In this research, the idea is to use a single low-resolution GridEye AMG8833 IRarray which locates above a doorway. This location for a sensor makes it less visible, but more efficient for detecting human IR-radiation. Most of the IR-radiation is originated from head and torso area, less from limbs and other parts of a body. The imaging view from ceilings downwards is also the most privacy preserving. Fig. 1. shows the developed counter assembled over a doorway. The sensor locates at the end of a bar which allows to change the assembly height and tilt of the sensor. The sensor's field of view extends some amount outside the room and this extension is affected by the tilt angle. The sensor mount has been manufactured using 3D-printer and it is intended only for this research platform.

Interpolation
The initial image size is 8 x 8 pixels. The raw image is too coarse for occupancy counting. Using interpolation and adding one intermediate pixel between original pixels, the size of 15 x 15 pixels is reached. By adding two more pixels results in size of 22 x 22 pixels. By adding more pixels, the size increases as (n x 7 + 1) x (n x 7 +1) when n = 4,5,6. In this application, the image size was set at 71 x 71 pixels. This resolution has used also by [5].
There are a few choices for an interpolation method: 2D nearest-neighbor, bilinear and bicubic interpolation. Bilinear interpolation uses 4 points of a square to calculate an interpolated value within the square. This requires only two calculations with linear equations. That is simple to program and the code is extremely fast to be executed. Bilinear interpolation produces an image which looks clearly rasterized. Bicubic interpolation uses 3 rd. order polynomial fit. It is numerically heavier because gradients and the cross derivatives are also needed for 4 x 4 grid points. This can be done using Lanczos interpolation which, is the most accurate multi-lobe method. Bicubic interpolation results in a smoother misty image, which visually looks more pleasant for a human observer than a rasterized one. This application uses bicubic interpolation. Interpolation can be presented in a general equation (1). Fig. 3. shows the original and interpolated images using thermal light colors. Rainbow colors are the most often used color coding. In rainbow colors blue indicates cool, red hot temperature.
The thermal light presentation is related to the real color of a hot body. Cooler targets appear dark red and at very high temperature it lights on white. Colors have been calculated using algorithm given by [19]. In the image, a test person is standing still and heading to the right.

3.2
Signal-to-noise enhancement using convolution The first implementation of the counter used a direct conversion method where measured temperature values were converted directly to blob pixels. This implementation suffered from high noise and low sensitivity. The sensitivity had to be made changeable between two levels. First the sensor operates in low sensitivity while waiting a target to appear in the field of view. After a target has been detected the sensitivity was increased in order to make it detectable all the time during the traverse. After leaving the field of view the sensitivity was reduced again to avoid noise triggered events. This was implemented by using a squelch. This implementation results in a sensor of an operational distance only up to 0.5 meters, too short for detecting all people of different heights. This implementation was also rather unstable and was finally rejected. In order to improve the signal-to-noise ratio SNR, 2-dimensional convolution is a standard procedure in all image processing applications. A discrete time convolution in 2dimensions over the matrix of M x N pixels can be calculated using Equ. 2.
Where x contains the values of an IR-matrix and h is a kernel matrix. The kernel matrix h should have the image of the expected thermal heat source.

3.3
Filtering using 2D-convolution Gaussian filtering: [5] stated that both interpolation, adaptive threshold for background reduction and Gaussian filtering is needed for reducing noise and detecting multiple people in the field of view. Gaussian filtering matrix is calculated using Equ. 3.
Gaussian filtering smooths an image by reducing noise. Blob radius after Gaussian filtering is Laplacian of Gaussian: Laplacian method emphasizes changes in the image. Laplacian-Gaussian is widely used in various image processing tasks [20]. It is used to edge detection and shows the outline of a blob. As it also amplifies noise, therefore Gaussian filtering is needed for noise removal. Equ. 5. calculated Laplacian of Gaussian filtering matrix.
In the Fig. 4 a person is standing under the imaging array and heading to the left. IRimage is on the left and the blob image on the right side. 71 x 71 pixels resolution is used. Fig. a) shows a Gaussian filtered image (σ is 1, matrix size n is 7). Fig. 4 b) shows Laplacian of Gaussian filtered image (σ is 1, matrix size n is 7). This shows the borderline of the blob. Due to the low resolution, the outline looks thick.
For the images in Fig. 4 iJOE -Vol. 16, No. 6, 2020 Gaussian filtering seemed to be the best method for this application. To maintain the maximum speed 10 frames in a second, filtering matrix size n was reduced to 5. Just about the same output can be reached also with n = 3.

Blob detection
A blob algorithm is based on grassfire-algorithm. There are different implementations for the algorithm. The most common one starts blob searching in one corner of the IR-array and advances row by row or column by column basis towards the other corner. An alternative algorithm called flood fill takes a specific starting point and extends search until finds the borders of a blob. It might proceed slightly faster than grassfire search. Despite the selected algorithm, the search proceeds as follows: When pixel with higher value than background noise is detected, it will be marked active. Then nearby pixels will be checked. Using the 4-connectivity principle adjacent pixels to the left, right, up and down will be checked. The 8-connectivity principle means that also pixels locating diagonally will be checked. In this application, the 4-connectivity is in use. Active pixels which are connected together forms a blob. While doing so detected blobs will be numbered according to their order of appearance. In case more than one blob is visible, the blob order may have been changed in adjacent frames. This sets difficulties for tracking the particular blob. In the current stage, the algorithm is able to detect and track two blobs.

Blob tracking and counter action
Single blob tracking: The counter logic use defined three zones separated by the decision lines shown as vertical red lines in Fig. 4. The center line has also been added in the Fig. 5. The start zones are the outermost zones on the left and the right. The center zone locates between them. A person can enter to the field of view from two directions. The person enters first to the start zone, proceeds to the center zone and finally crosses the decision line on the opposite side and causes a counter event. Depending in the traversing direction the counter is either incremented or decremented. This procedure using three zones was found the best to eliminate non-intended counter events caused by noise.
The frame rate of the IR array set the limit for the velocity of a target which can be detected. The target must be detected at least in each of three zones once. According to [21] human walking speed is 2.5-3 m/s, according to [9] average walking speed is 1.38 m/s. 10 frame/s makes it possible to detect a walking person using sensor assembly shown in Fig. 2.
Multiple blob tracking: Blob tracking can either take place by tracking the center or gravity point of a blob. The logic handles cases up to two blobs. The counter is successfully updated in cases whether two people are traversing parallel, opposite directions or another person is standing still or not reaching the decision line. However, the blobs should be seen separate ones all the time in order to handle them correctly.
In case more blobs are detected they are ignored. An error follows in a case where the minimum distance between people does not exist. In that case blobs are merged together and they may not be separated by any means. The minimum distance depends on target distances, sizes, speeds etc. Typically, the minimum distance is some tens of centimeters. [9] has studied this with their doorway counter.
If two people are merged into one blob, there is not much to be done to separate them. However, [9] presents an idea to separate the blob. If a blob is large enough to be merged two blobs, then the threshold is increased by steps, say 0.25 °C (equivalent to resolution of GridEye) until either two blobs become visible or the size of the single blob becomes small enough, say 10% of the original size. If only one blob remains visible, the conclusion is that only one person is seen in the image. Kalman filter (KF) is widely used in many applications. In image processing target tracking in 2D is often based on the trajectory calculated from real measurements processed using linear Kalman filter (KF). KF is considered to the best method in case process and measurement noise is zero averaged and Gaussian distributed. KF principle uses state equations where the target position, the velocity, the acceleration, the affecting forces etc. can be included. In this application only position and velocities are included. Matrix state equations can be written for prediction and correction.
Prediction equations Correction equations A is the state transition matrix. Pk is the process covariance matrix. Q is the covariance matrix of process noise. Q is often hard to be found. According to [8], Q in this application is as shown below. xk is a vector of the variables including positions and velocities in both directions. B is the control force matrix and in this application B = 0. wk-1 is the process noise and it can also be set at zero. t is the time step between measurements. http://www.i-joe.org In the correction equations, H is the observation matrix. zk is the vector of the measurements. vk is measurement noise and in this case it can be set at zero. R is a matrix of measurements errors. As in the case with matrix Q, matrix R can be hard to be defined. In this case, we need accuracy of position (ep) and velocity measurements (ev).
Using KF in tracking requires the history of measured target position and velocities. The velocities can also be calculated values. First prediction values for xk will be calculated using Equ. 1-2. Then the calculated xk is used in Equ. 3-4 and the calculated Pk is used in Equ 5-6. The result is a path which in the most confident way in the presence of Gaussian noise shows the traverse of the measured target. Using KF method helps to avoid mixing blobs in case of multiple blobs in the field of view at the same time.

Thermal Signatures
A test person having an average body and height of 183 cm used for thermal signature tests. The sensor locates at height of 245 cm. In case 1, the test person is wearing a light T-shirt, in case 2 a long-sleeved shirt and in case 3 winter sport jacket with an insulating layer. In all cases, no headwear was used. In the tests, the person was standing still under the sensor for 3 minutes resulting in 1800 images. Statistical analysis was executed and a normal distribution curve was fitted on the measurements. Using 71 x 71 pixels array, the total amount of pixels is 5041. In the Fig. 7 density functions shows the probability of incidence for the number of active pixels. It shows that the estimated number of active pixels in case 1: 907, in case 2: 761 and in case 3: 607. Variation in active pixels results from the noise and slight movements of the test person. Variance in case 3 seems to be twofold compared in cases 1 and 2. In Fig. 7, the analysis has been done for the product of active pixel and its value. The statistical presentation shows similar results than in Fig. 7.

Sensor Network
Several GridEye counters can be connected for a network. Each sensor device has its own server which keeps tracking the current and history occupancy information. Additionally, each GridEye counter stores 2 x 3 minutes data records into files including the frame number, the frame time, the blob number and center/gravity point of blobs. Each sensor device has Apache 2 web server with PHP installed.
GridEye counters have been used in a meeting and an office room of a university building. Typically, people arrive at a meeting room with large time separations, but after the end of a meeting they exit the room almost at the same time. In the exit situation multiple people may be detected as a single blob by the counter. This counter type in a meeting room is prone to an error showing people in the room although the room is already empty. By adding PIR sensors for indicating the empty room condition improves accuracy. The system utilizes several wired and wireless PIR sensors.

User Interface
The user interface uses OpenGL graphics in Raspberry Pi. The graphic window is shown below. Both the processed IR-image and blob's image are available. Maximum two blobs are tracked, the coordinates for the center of gravities and the blob sizes in pixels are displayed. Calibration takes once in 60 seconds if there is no target in the field of view. In case there is a target, the calibration is postponed. Value of the occupancy counter and timestamp for the last counter action are displayed. All data is continuously recorded in files all the time a target is visible. Each counter has one wired PIR and one wireless PIR sensor connected. Status of PIR sensors is displayed on the graphics screen. Each counter unit has a web server and PHP written scripts. The counter value and its history records as well as recorded data can be read via Wi-Fi by any client device in the net.

Conclusion
A doorway occupancy counter based on GridEye 8 x 8 IR-array and Raspberry Pi 3B board has been demonstrated. All software has been created from a scratch making it the best platform for a research. Despite the high interpolation, 8 x 8 pixels resolution with only 10 frames/s speed shows not entirely adequate performance in occupancy counting in cases of two or more people present in the field of view. The counter performs fine in cases of a single person moving in the field of view with a normal walking speed up to 2.5 m/s. The software handles a situation of two people at the same time in the field of view. Kalman filter tracking has been used. The counter has both a graphics user and a web server interface. The system uses added wired and wireless PIR sensors for offset error elimination. A heat signature of a person has been studied. A heat signature changes much between people and between the same person at different times. Clothing has prominent effect on the heat signature. By using covering or thermally well insulating clothing, the heat signature may totally disappear.