How to connect Flir Lepton to NVIDIA Jetson TK1

5 years ago · 9 min read

Introduction

In this post, I will discuss how to connect FLIR Lepton module with NVIDIA Jetson TK1.If you are new to Jetson TK1 and struggling on how to kick off the development with that , you can refer to my previous post to get to know about how to get started with NVIDIA Jetson TK1.In this post I assume, you have already installed Operating system(Ubuntu 14.04) using Jetpack or any other way. There is an excellent guide on how to enable SPI with Jetson TK1 and connect to Arduino Due, It helped me a lot understanding  the hardware capabilities of Jetson TK1 to enable SPI communication , and how kernel device drivers and device tree helps to allow communication with SPI hardware in TK1 with programmers in Operating system level.It is better to first to go through that article up to part 3.My Jetson TK1 setup is Ubuntu 14.04 installed with Jetpack R21.5 and update the kernel with Grinch 21.3.4 for Jetson TK1.If you have already installed jetpack and running the OS following topic will cover how to install Grinch 21.3.4 kernel, and configure to use SPI.

Update Kernel

I actually wanted to connect Lepton with Jetson without doing any Kernal modification , so I tried building official kernel enabling the SPI module but It didn't work and ended up with hanging the OS after few seconds of booting. So re-flash the Jetson and tried with latest Grinch kernel.As the steps mention here updating the stock kernel with Grinch kernel is easy.If you have fresh Jetpack installation(In my case Jetpack 21.5). Then run the following commands to update the kernel.

  1. Download required firmware,kernel modules and boot zimage files

    wget http://www.jarzebski.pl/files/jetsontk1/grinch-21.3.4/zImage
wget http://www.jarzebski.pl/files/jetsontk1/grinch-21.3.4/jetson-tk1-grinch-21.3.4-modules.tar.bz2
wget http://www.jarzebski.pl/files/jetsontk1/grinch-21.3.4/jetson-tk1-grinch-21.3.4-firmware.tar.bz2

  2. Make sure you check the MD5 hash with following (It sometimes download 135K junk file, nothing in it)

    a4a4ea10f2fe74fbb6b10eb2a3ad5409  zImage
3f84d425a13930af681cc463ad4cf3e6  jetson-tk1-grinch-21.3.4-modules.tar.bz2
f80d37ca6ae31d03e86707ce0943eb7f  jetson-tk1-grinch-21.3.4-firmware.tar.bz2

  3. Then extract and copy the files to system

    sudo tar -C /lib/modules -vxjf jetson-tk1-grinch-21.3.4-modules.tar.bz2
sudo tar -C /lib -vxjf jetson-tk1-grinch-21.3.4-firmware.tar.bz2
sudo cp zImage /boot/zImage

  4. If you are following original NVIDIA dev post sofa , stop here and the coping DTB file into /boot directory, Because we have to make some changes before copying the device tree blob file into /boot

    • The reason is , we need to change the frequency of the SPI clock or reduce the default Jetson touch SPI CLK frequency to comfortable range to work with Lepton, For that we will decompile the .dtb file and generate the .dts(Device tree source file) and do the modification to the .dts file and the compile the source code using dtc(Device Tree Compiler) and copy the modified device tree blob(.dtb) file to /boot directory
    So for doing the above task first we need the dtc tool, The source code for the dtc comes with the kernel it self in the kernel/scripts/dtc/ you can compile it and use.Else I have uploaded here the pre-compiled dtc tool for Jetson TK1 , you can use that if you don't want to do any kernel builds.
    1. One you have the dtc tool , go to the directory where you have downloaded the tegra124-jetson_tk1-pm375-000-c00-00.dtb file and run the following command ```shell ./dtc -I dtb -O dts -o tegra124-jetson_tk1-pm375-000-c00-00.dts tegra124-jetson_tk1-pm375-000-c00-00.dtb ```
    2. Then open the *.dts file from your favourite text editor and locate the section spi@7000d400 and change the below highlighted lines, ``` spi@7000d400 { compatible = "nvidia,tegra114-spi"; reg = <0x0 0x7000d400 0x0 0x200>; interrupts = <0x0 0x3b 0x4>; nvidia,dma-request-selector = <0x7 0xf>; nvidia,memory-clients = <0xe>; #address-cells = <0x1>; #size-cells = <0x0>; clocks = <0xc 0x29>; status = "okay"; dmas = <0x7 0xf 0x7 0xf>; dma-names = "rx", "tx"; spi-max-frequency = <0xF42400>; 
              spi0_0 {
                #address-cells = &lt;0x1&gt;;
                #size-cells = &lt;0x0&gt;;
                compatible = "spidev";
                reg = &lt;0x0&gt;;
                spi-max-frequency = &lt;0xF42400&gt;;
                spi-cpha;
                nvidia,enable-hw-based-cs;
                nvidia,cs-setup-clk-count = &lt;0x1e&gt;;
                nvidia,cs-hold-clk-count = &lt;0x1e&gt;;
                nvidia,rx-clk-tap-delay = &lt;0x1f&gt;;
                nvidia,tx-clk-tap-delay = &lt;0x0&gt;;
        };
};
    I have only change the original <strong>0x17D7840 </strong>value to <strong>0xF42400</strong>, setting the max frequency value from 25MHz to 16MHz</li>

<li>Then compile the device tree source file again and copy it to <strong>/boot</strong> directory, to do that use the following command

```shell
./dtc -I dts -O dtb -o /boot/tegra124-jetson_tk1-pm375-000-c00-00.dtb tegra124-jetson_tk1-pm375-000-c00-00.dts
As for the instruction in the Nvidia forum , You should get the /dev/spidev0.0 after a restart , But it seems it's not working after Jetpack ~R21.3 onwards.So I google for a solution for this and finally found a workaround to get the SPI work

Fixing conflicts with Jetson TK1 SPI touch drivers

After following exact steps as mentioned in neurorobotictech article and grinch kernel update post , I still couldn't get the /dev/spidev0.0 appear in the device directory, So to check what is going on with the kernel , I check for spi logs in dmesg and tried to find a clue dmesg | grep "spi"  it shows that 
spi-tegra114 spi-tegra114.0: chipselect 0 already in use
spi_master spi0: spi_device register error /spi@7000d400/spi0_0

But it was quite confident that , I haven't using that port for any reason,So I further look for what could use this port other than spidev, and finally was able to find the answer from this post

This is because the touch controller fails to start up, becuase MCP251x has taken it already. This can be ignored, or you can disable the touch controller by editing /boot/extlinux/extlinux.conf and setting "touch_id=3@3"
So simply open the /boot/extlinux/extlinux.conf file and change the touch_id value from 0@0 to 3@3, And that's it.Now restart the jetson and check whether the /dev/spidev0.0 is there , and most probably you should be able to locate it now.

So let's discuss how to connect the FLIR wire connections physically into Jetson TK1 GPIO adapter.

Connecting Hardware

We have bought an FLIR Lepton module with the pureengineering lepton breakout board attach from the groupgets

IMG 20161120 112012

When I tried to connect the Lepton breakout board to Jetson , the harder part was to find the correct pins for SPI0 in Jetson J3A1 pad, Luckily descriptive explanation about all the GPIO port in Jetson was here. Then it was just a matter of picking up the correct pin from Jetson and connect to Lepton via a jumper cable, To make the process easy I have marked the correct pin number and it's named from above elinux wiki and the pin name appear in breakout board along with the pin number from right to left starting from SCL. 

Before start connecting the lines , Make sure you have power off the jetson and unplug it from the power supply.

Jetson lepton pin config low

Now connect the cables and start the jetson TK1 to proceed with the next step, Capturing the data.

BTW Is I2C connections missing ? , No you can simply connect the SCL and SDA lines from Lepton breakout board to 18 GEN2I2CSCL3.3V  and 20 GEN2I2CSDA3.3V and get the I2C working, But I haven't tested this yet.

Software setup for capturing data from Lepton

After connecting the device , I still didn't know that whether I have connected the device to right pins or even all the above steps have enabled the SPI in jetson, The next challenge was to make the connection between hardware and software. I tried with the pylepton library from groupgets GitHub repo, and need to do some slight changes to get it work.

One thing is we need to change the SPEED constant value in Lepton.py to SPEED = 16000000 to make it comply with the maxfrequency value we specified in device tree source, And also we need to change the MODE constant value to MODE = SPIMODE0  Because according to Jetson TK1 technical reference manual(TRM) ,SPI0 in jetson works in SPI mode0. After making those changes run the <a href="https://github.com/groupgets/pylepton/blob/master/pyleptoncapture">pylepton_capture script with sudo permission and you wold be able to take a thermal grayscale image from Lepton.

image sample 33

image sample 44

image sample 74

image sample 127

image sample 144

Here you can find some raw sample jpg images along with the value received from the Lepton before converting the values to 0-255 range.

What next. . .

Next, I will write a post on how to use the thermal data(Not specifically image just raw data) and train a neural network model using Caffe deep learning library to detect human presence.Well, there is a lot of work there in the previous phrase.Will try to put them under one post.Since then good luck connecting the lepton to Jetson :)

References :

  • https://goo.gl/6sqUBH
  • http://www.electricstuff.co.uk/lepton.html
  • https://github.com/groupgets/pylepton
  • https://devtalk.nvidia.com/default/topic/740390/?comment=4421685
  • http://elinux.org/Jetson/Installing_OpenCV
  • http://neurorobotictech.com/Community/Blog/tabid/184/ID/13/Using-the-Jetson-TK1-SPI--Part-3-Configuring-SPI-in-the-device-tree.aspx
  • https://github.com/chneukirchen/linux-jetson-tk1/blob/master/Documentation/devicetree/bindings/spi/spi-bus.txt
  • https://learn.sparkfun.com/tutorials/serial-peripheral-interface-spi