home/mr_halfword/Documents/MPSOC2019.1_DVD_V1.1/01-Document/UserManual/English/MYD-CZU3EG-4EV Tutorial.pdf was created using Vivado 2019.1. The "4.1 Create a Block Design" section selects the Presets option on the "Re-customize IP" dialog to import the configuration for the MYC-CZU4EV module.
However, in Vivado 2020.1 the Presets options is not present on the dialog.
https://www.xilinx.com/support/answers/75467.html is 'Zynq UltraScale+ MPSoC, Vivado 2020.1 - Processing System "Presets" option temporarily removed'
That answer mentions how to run "write_bd_tcl" to save the configuration, but doesn't mention how to apply an existing configuratuion.
https://forums.xilinx.com/t5/Processor-System-Design-and-AXI/Apply-Zynq-processor-preset-from-TCL/td-p/861912 has some TCL commands which can be run to apply a saved configuration.
From the TCL console:
source /home/mr_halfword/Documents/MPSOC2019.1_DVD_V1.1/05-ProgrammableLogic_Source/CZU4EV/tutorial/tutorial.tcl
set presets [apply_preset 0]
foreach {k v} $presets {
if {![info exists preset_list]} {
set preset_list [dict create $k $v]
} else {
dict set preset_list $k $v
}
}
set_property -dict $preset_list [get_bd_cells zynq_ultra_ps_e_0]
Which results in the following being enabled in the Block Design:
- QSPI
- GEM 3
- USB 0
- SD 0/eMMC
- SD 1/eMMC
- IC2 1
- UART 0
Output from Z:\home\mr_halfword\myir_workspace\lwip_ide\bootimage\BOOT.bin booted from SD card:
Xilinx Zynq MP First Stage Boot Loader
Release 2020.1 Aug 30 2020 - 20:23:25
Programming SI5338 for 125MHZ ref clock
Progarmming SI5338 with 125MHZ ref clock successfull
CSU_IDCODE=0x04721093 EFUSE_IPDISABLE=0x00000000
Silicon name=4
ProcName=EV
-----lwIP TCP echo server ------
TCP packets sent to port 6001 will be echoed back
Start PHY autonegotiation
Waiting for PHY to complete autonegotiation.
autonegotiation complete
link speed for phy address 5: 100
Board IP: 192.168.0.3
Netmask : 255.255.255.0
Gateway : 192.168.0.1
TCP echo server started @ port 7
This was built from Vivado 2020.1.
Initially attempted to create the bitstream for a XCZU5EV based apon the info from the programs in the on-board flash. However, the bitstream for the XCZU5EV failed to load, whereas when changed the device in the Vivado project to a XCZU4EV then the bistream was loaded.
Output from u-boot in on-board flash as delivered (using an older 2017.01 version of the Xilinx FSBL and U-Boot):
Xilinx Zynq MP First Stage Boot Loader
Release 2017.4 Mar 16 2020 - 15:41:32
Programming SI5338 for 125MHZ ref clock
Progarmming SI5338 with 125MHZ ref clock successfull
NOTICE: ATF running on XCZU5EV/silicon v4/RTL5.1 at 0xfffea000, with PMU firmware
NOTICE: BL31: Secure code at 0x0
NOTICE: BL31: Non secure code at 0x8000000
NOTICE: BL31: v1.3(release):0d9d51a
NOTICE: BL31: Built : 07:43:08, Jan 17 2018
PMUFW: v0.3
U-Boot 2017.01 (Mar 12 2020 - 16:07:20 +0800) Xilinx ZynqMP
I2C: ready
DRAM: 4 GiB
chip id is : 0x0020
Chip ID: xczu5ev
MMC: sdhci@ff170000: 0
SF: Detected n25q512a with page size 512 Bytes, erase size 128 KiB, total 128 MiB
*** Warning - bad CRC, using default environment
In: serial
Out: serial
Err: serial
Net: ZYNQ GEM: ff0e0000, phyaddr c, interface rgmii-id
eth0: ethernet@ff0e0000
Hit any key to stop autoboot: 0
ZynqMP> fpga info 0
Xilinx Device
Descriptor @ 0x000000007ffaad98
Family: ZynqMP PL
Interface type: csu_dma configuration interface (ZynqMP)
Device Size: 1 bytes
Cookie: 0x0 (0)
Device name: xczu5ev
Device Function Table @ 0x000000007ff903b0
ZynqMP>
https://github.com/Xilinx/arm-trusted-firmware/commit/e27d3b59b5cff88c01bbfa799c5c5b9211727484#diff-726e091045a4645a44e4a7775213024a corrected the definition of ZYNQMP_CSU_IDCODE_SVD_MASK from:
#define ZYNQMP_CSU_IDCODE_SVD_MASK (0xE << ZYNQMP_CSU_IDCODE_SVD_SHIFT)
To:
#define ZYNQMP_CSU_IDCODE_SVD_MASK (0x7 << \
ZYNQMP_CSU_IDCODE_SVD_SHIFT)
Where the commit was made on 17 May 2018
The effect of the incorrect definition of ZYNQMP_CSU_IDCODE_SVD_MASK was to clear the least significant bit of the CSU IDCODE which changes the value of 0x21 for a 4EV into a 0x20 for a 5EV.
Copying the contents of /home/mr_halfword/Documents/MPSOC2019.1_DVD_V1.1/02-Images/Petalinux-Images/CZU4EV onto a sdcard and booting that resulted in:
Xilinx Zynq MP First Stage Boot Loader
Release 2019.1 Apr 16 2020 - 01:49:38
Programming SI5338 for 125MHZ ref clock
Progarmming SI5338 with 125MHZ ref clock successfull
NOTICE: ATF running on XCZU4EV/silicon v4/RTL5.1 at 0xfffea000
NOTICE: BL31: Secure code at 0x0
NOTICE: BL31: Non secure code at 0x8000000
NOTICE: BL31: v2.0(release):xilinx-v2018.3-720-g80d1c790
NOTICE: BL31: Built : 08:15:05, Apr 16 2020
PMUFW: v1.1
U-Boot 2019.01-07037-gd895ac5e94-dirty (Apr 16 2020 - 09:03:45 +0000)
Board: Xilinx ZynqMP
I2C: ready
DRAM: 4 GiB
usb dr_mode not found
EL Level: EL2
Chip ID: zu4ev
MMC: mmc@ff160000: 0, mmc@ff170000: 1
Loading Environment from SPI Flash... SF: Detected n25q512a with page size 512 Bytes, erase size 128 KiB, total 128 MiB
*** Warning - bad CRC, using default environment
In: serial@ff000000
Out: serial@ff000000
Err: serial@ff000000
Board: Xilinx ZynqMP
Bootmode: SD_MODE1
Reset reason: EXTERNAL
Net: ZYNQ GEM: ff0e0000, phyaddr 5, interface rgmii-id
eth0: ethernet@ff0e0000
U-BOOT for myd_czu4ev
ethernet@ff0e0000 Waiting for PHY auto negotiation to complete.... done
BOOTP broadcast 1
BOOTP broadcast 2
BOOTP broadcast 3
BOOTP broadcast 4
DHCP client bound to address 192.168.0.3 (1756 ms)
Hit any key to stop autoboot: 0
ZynqMP> fpga info 0
Xilinx Device
Descriptor @ 0x000000007ffbaf60
Family: ZynqMP PL
Interface type: csu_dma configuration interface (ZynqMP)
Device Size: 1 bytes
Cookie: 0x0 (0)
Device name: zu4ev
Device Function Table @ 0x000000007ff9b5d8
PCAP status 0xa0002fde
ZynqMP>
I.e. the later 2019.01 ATF and U-Boot have the fixes to correctly identify the XCZU4EV device correctly.
Copied the contents of /home/mr_halfword/Documents/MPSOC2019.1_DVD_V1.1/02-Images/Linux-Images/CZU4EV/sfp to a sdcard
Gets as far as "Starting kernel..." reported but then reboots and re-starts the FSBL:
Xilinx Zynq MP First Stage Boot Loader
Release 2019.1 Mar 26 2020 - 14:39:17
Programming SI5338 for 125MHZ ref clock
Progarmming SI5338 with 125MHZ ref clock successfull
NOTICE: ATF running on XCZU4EV/silicon v4/RTL5.1 at 0xfffea000
NOTICE: BL31: Secure code at 0x0
NOTICE: BL31: Non secure code at 0x10080000
NOTICE: BL31: v2.0(release):xilinx-v2018.3-720-g80d1c790
NOTICE: BL31: Built : 08:15:05, Apr 16 2020
PMUFW: v1.1
U-Boot 2019.01 (Apr 16 2020 - 17:45:07 +0800)
Model: ZynqMP ZCU102 Rev1.0
Board: Xilinx ZynqMP
I2C: ready
DRAM: 4 GiB
EL Level: EL2
Chip ID: zu4ev
MMC: mmc@ff170000: 0
Loading Environment from SPI Flash... SF: Detected n25q512a with page size 512 Bytes, erase size 128 KiB, total 128 MiB
*** Warning - bad CRC, using default environment
In: serial@ff000000
Out: serial@ff000000
Err: serial@ff000000
Model: ZynqMP ZCU102 Rev1.0
Board: Xilinx ZynqMP
Bootmode: SD_MODE1
Reset reason: EXTERNAL
Net: ZYNQ GEM: ff0e0000, phyaddr c, interface rgmii-id
eth0: ethernet@ff0e0000
Hit any key to stop autoboot: 0
Device: mmc@ff170000
Manufacturer ID: 3
OEM: 5344
Name: SU04G
Bus Speed: 50000000
Mode : SD High Speed (50MHz)
Rd Block Len: 512
SD version 3.0
High Capacity: Yes
Capacity: 3.7 GiB
Bus Width: 4-bit
Erase Group Size: 512 Bytes
16079360 bytes read in 1081 ms (14.2 MiB/s)
5931402 bytes read in 405 ms (14 MiB/s)
34092 bytes read in 16 ms (2 MiB/s)
## Loading init Ramdisk from Legacy Image at 13000000 ...
Image Name: Ramdisk
Image Type: AArch64 Linux RAMDisk Image (gzip compressed)
Data Size: 5931338 Bytes = 5.7 MiB
Load Address: 00000000
Entry Point: 00000000
Verifying Checksum ... OK
## Flattened Device Tree blob at 16000000
Booting using the fdt blob at 0x16000000
Loading Ramdisk to 07a57000, end 07fff14a ... OK
Loading Device Tree to 0000000007a4b000, end 0000000007a5652b ... OK
Starting kernel ...
Xilinx Zynq MP First Stage Boot Loader
The above was with 4 SFP transeivers plugged in, but no fibres connected.
The Linux-Images/CZU4EV/hdmi_lcd and Linux-Images/CZU4EV/tutorial boot Linux successfully, it is only Linux-Images/CZU4EV/sfp which fails to boot Linux.
The Image, rootfs.tar and uramdisk.image.gz files are identical between the variants, it is only the BOOT.bin and *.dtb files which have differences.
For these tests all 4 SFP tranceivers were fitted, but with no cables connected. Booted inux-Images/CZU4EV/hdmi_lcd from a sdcard.
Detecting the IC2 buses, which detects the muxed buses behind the TCA9548A on the baseboard.
[root@myir ~]# i2cdetect -l
i2c-3 i2c i2c-0-mux (chan_id 0) I2C adapter
i2c-10 i2c i2c-0-mux (chan_id 7) I2C adapter
i2c-1 i2c Cadence I2C at ff030000 I2C adapter
i2c-8 i2c i2c-0-mux (chan_id 5) I2C adapter
i2c-6 i2c i2c-0-mux (chan_id 3) I2C adapter
i2c-4 i2c i2c-0-mux (chan_id 1) I2C adapter
i2c-11 i2c ZynqMP DP AUX I2C adapter
i2c-2 i2c xiic-i2c I2C adapter
i2c-0 i2c Cadence I2C at ff020000 I2C adapter
i2c-9 i2c i2c-0-mux (chan_id 6) I2C adapter
i2c-7 i2c i2c-0-mux (chan_id 4) I2C adapter
i2c-5 i2c i2c-0-mux (chan_id 2) I2C adapter
The muxed chan_id's 3 to 6 are the SFP IC2 buses:
- i2c-0-mux (chan_id 3) SFP+_RB
- i2c-0-mux (chan_id 4) SFP+_RT
- i2c-0-mux (chan_id 5) SFP+_LB
- i2c-0-mux (chan_id 6) SFP+_LT
Probing these I2C buses shows:
[root@myir ~]# i2cdetect -y -r 6
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: 50 51 -- -- -- -- -- -- -- -- -- -- -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
70: UU -- -- -- -- -- -- --
[root@myir ~]# i2cdetect -y -r 7
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: 50 51 -- -- -- -- -- -- -- -- -- -- -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
70: UU -- -- -- -- -- -- --
[root@myir ~]# i2cdetect -y -r 8
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: 50 51 -- -- -- -- -- -- -- -- -- -- -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
70: UU -- -- -- -- -- -- --
[root@myir ~]# i2cdetect -y -r 9
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: 50 51 -- -- -- -- -- -- -- -- -- -- -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
70: UU -- -- -- -- -- -- --
Where the addreses 51 and 52 are the SFP Digital Diagnostic Memory Map
And dumping the Serial ID of the transeivers shows FINISAR FTLF8524P2BNV modules with as expected different serial numbers:
[root@myir ~]# i2cdump 6 0x50
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-6, address 0x50, mode byte
Continue? [Y/n] y
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 03 04 07 00 00 00 01 20 40 0c 15 01 2a 00 00 00 ???...? @???*...
10: 0f 07 00 00 46 49 4e 49 53 41 52 20 43 4f 52 50 ??..FINISAR CORP
20: 2e 20 20 20 00 00 90 65 46 54 4c 46 38 35 32 34 . ..?eFTLF8524
30: 50 32 42 4e 56 20 20 20 41 20 20 20 03 52 00 71 P2BNV A ?R.q
40: 00 32 00 00 50 42 32 30 42 54 53 20 20 20 20 20 .2..PB20BTS
50: 20 20 20 20 30 37 30 31 30 38 20 20 68 d8 01 e0 070108 h???
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[root@myir ~]# i2cdump 7 0x50
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-7, address 0x50, mode byte
Continue? [Y/n] y
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 03 04 07 00 00 00 01 20 40 0c 15 01 2a 00 00 00 ???...? @???*...
10: 0f 07 00 00 46 49 4e 49 53 41 52 20 43 4f 52 50 ??..FINISAR CORP
20: 2e 20 20 20 00 00 90 65 46 54 4c 46 38 35 32 34 . ..?eFTLF8524
30: 50 32 42 4e 56 20 20 20 41 20 20 20 03 52 00 71 P2BNV A ?R.q
40: 00 3a 00 00 50 48 4c 37 48 4a 30 20 20 20 20 20 .:..PHL7HJ0
50: 20 20 20 20 31 30 30 35 31 35 20 20 68 f8 01 04 100515 h???
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[root@myir ~]# i2cdump 8 0x50
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-8, address 0x50, mode byte
Continue? [Y/n] y
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 03 04 07 00 00 00 01 20 40 0c 15 01 2a 00 00 00 ???...? @???*...
10: 0f 07 00 00 46 49 4e 49 53 41 52 20 43 4f 52 50 ??..FINISAR CORP
20: 2e 20 20 20 00 00 90 65 46 54 4c 46 38 35 32 34 . ..?eFTLF8524
30: 50 32 42 4e 56 20 20 20 41 20 20 20 03 52 00 71 P2BNV A ?R.q
40: 00 32 00 00 50 46 42 32 34 50 54 20 20 20 20 20 .2..PFB24PT
50: 20 20 20 20 30 39 30 33 31 30 20 20 68 d8 01 e2 090310 h???
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[root@myir ~]# i2cdump 9 0x50
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-9, address 0x50, mode byte
Continue? [Y/n] y
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 03 04 07 00 00 00 01 20 40 0c 15 01 2a 00 00 00 ???...? @???*...
10: 0f 07 00 00 46 49 4e 49 53 41 52 20 43 4f 52 50 ??..FINISAR CORP
20: 2e 20 20 20 00 00 90 65 46 54 4c 46 38 35 32 34 . ..?eFTLF8524
30: 50 32 42 4e 56 20 20 20 41 20 20 20 03 52 00 71 P2BNV A ?R.q
40: 00 3a 00 00 50 50 36 35 52 59 39 20 20 20 20 20 .:..PP65RY9
50: 20 20 20 20 31 33 30 32 31 33 20 20 68 f8 01 14 130213 h???
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
The Enhanced Options at byte 0x5d differs:
- The two with the oldest datecodes have 0xd8
- The two with the newest datecodes have 0xf8
Reading back the "Optional Status/Control Bits" shows the Tx Disable Input Pin and RS_LOS State are both one:
[root@myir ~]# i2cdump -r 110-110 6 0x51
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-6, address 0x51, mode byte
Probe range limited to 0x6e-0x6e.
Continue? [Y/n] y
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
60: 82 ?
[root@myir ~]# i2cdump -r 110-110 7 0x51
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-7, address 0x51, mode byte
Probe range limited to 0x6e-0x6e.
Continue? [Y/n] y
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
60: 82 ?
[root@myir ~]# i2cdump -r 110-110 8 0x51
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-8, address 0x51, mode byte
Probe range limited to 0x6e-0x6e.
Continue? [Y/n] y
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
60: 82 ?
[root@myir ~]# i2cdump -r 110-110 9 0x51
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-9, address 0x51, mode byte
Probe range limited to 0x6e-0x6e.
Continue? [Y/n] y
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
60: 82 ?
With the contents of the sdcard as that from Linux-Images/CZU4EV/hdmi_lcd, replaced just the BOOT.bin from Linux-Images/CZU4EV/sfp, in order to use the FPGA bitstream which has presumably set to use the SFP interfaces. This time the transceiver "Optional Status/Control Bits" shows the Tx Disable Input Pin is zero and RS_LOS State are still one:
i2cdump -r 110-110 6 0x51[root@myir ~]# i2cdump -r 110-110 6 0x51
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-6, address 0x51, mode byte
Probe range limited to 0x6e-0x6e.
Continue? [Y/n] y
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
60: 02 ?
[root@myir ~]# i2cdump -r 110-110 7 0x51
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-7, address 0x51, mode byte
Probe range limited to 0x6e-0x6e.
Continue? [Y/n] y
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
60: 02 ?
[root@myir ~]# i2cdump -r 110-110 8 0x51
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-8, address 0x51, mode byte
Probe range limited to 0x6e-0x6e.
Continue? [Y/n] y
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
60: 02 ?
[root@myir ~]# i2cdump -r 110-110 9 0x51
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-9, address 0x51, mode byte
Probe range limited to 0x6e-0x6e.
Continue? [Y/n] y
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
60: 02 ?
This implies changing to use the FPGA bitstream for the Linux-Images/CZU4EV/sfp is now driving the TX_disable signals as zero (i.e. tx enabled), rather than before when the TX_disable signals were being pull-uped.
Connected fibres between the two bott0m transceiver SFP cages and RX_LOS clears on the SFP addresses for the lower transceivers:
[root@myir ~]# [ 208.671987] random: crng init done
i2cdump -r 110-110 6 0x51
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-6, address 0x51, mode byte
Probe range limited to 0x6e-0x6e.
Continue? [Y/n] y
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
60: 00 .
[root@myir ~]# i2cdump -r 110-110 7 0x51
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-7, address 0x51, mode byte
Probe range limited to 0x6e-0x6e.
Continue? [Y/n] y
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
60: 02 ?
[root@myir ~]# i2cdump -r 110-110 8 0x51
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-8, address 0x51, mode byte
Probe range limited to 0x6e-0x6e.
Continue? [Y/n] y
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
60: 00 .
[root@myir ~]# i2cdump -r 110-110 9 0x51
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-9, address 0x51, mode byte
Probe range limited to 0x6e-0x6e.
Continue? [Y/n] y
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
60: 02 ?
Move the loopback fibres from the bottom to top SFP cages, and the RX_LOS signals follow:
[root@myir ~]# i2cdump -r 110-110 6 0x51
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-6, address 0x51, mode byte
Probe range limited to 0x6e-0x6e.
Continue? [Y/n] y
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
60: 02 ?
[root@myir ~]# i2cdump -r 110-110 7 0x51
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-7, address 0x51, mode byte
Probe range limited to 0x6e-0x6e.
Continue? [Y/n] y
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
60: 00 .
[root@myir ~]# i2cdump -r 110-110 8 0x51
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-8, address 0x51, mode byte
Probe range limited to 0x6e-0x6e.
Continue? [Y/n] y
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
60: 02 ?
[root@myir ~]# i2cdump -r 110-110 9 0x51
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-9, address 0x51, mode byte
Probe range limited to 0x6e-0x6e.
Continue? [Y/n] y
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
60: 00 .
Insert loopback fibres for all transceivers and now all RX_LOS signals are low:
[root@myir ~]# i2cdump -r 110-110 6 0x51
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-6, address 0x51, mode byte
Probe range limited to 0x6e-0x6e.
Continue? [Y/n] y
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
60: 00 .
[root@myir ~]# i2cdump -r 110-110 7 0x51
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-7, address 0x51, mode byte
Probe range limited to 0x6e-0x6e.
Continue? [Y/n] y
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
60: 00 .
[root@myir ~]# i2cdump -r 110-110 8 0x51
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-8, address 0x51, mode byte
Probe range limited to 0x6e-0x6e.
Continue? [Y/n] y
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
60: 00 .
[root@myir ~]# i2cdump -r 110-110 9 0x51
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-9, address 0x51, mode byte
Probe range limited to 0x6e-0x6e.
Continue? [Y/n] y
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
60: 00 .
The pre-built Linux using 2019.1 didn't detect the PCIe root port.
Attmepting to follow https://www.xilinx.com/Attachment/Xilinx_Answer_71493_ZCU102_ZC706.pdf which was written for 2018.2
Started with a new project and applied MPSOC2019.1_DVD_V1.1/05-ProgrammableLogic_Source/CZU4EV/tutorial/tutorial.tcl
Re-customize IP steps were:
- In I/O Configuration enable PCIe
- In PCIe Configuration select:
- Device Port Type as "Root Port".
- Enable "CRS Software Visibility"
- Change "Device ID" from 0xD011 to 0xD021
- Change "Sub Class" from 0x80 to 0x4 The Link Speed of "5.0 GB/s" is OK.
- In I/O Configuration set Rootport Mode Reset to "MIO 31" to match the MYG-CZU4EV schamatic. The Lane Selection of x1 and Reset Polarity of "Active Low" are OK.
- In PS-PL Interfaces unticked "Fabric Reset Enable"
Some of the The PCIe Configuration -> Device IDs are different from "Figure 16 - Customizing Zynq IP (PCIe Configuration)" in the Xilinx_Answer_71493_ZCU102_ZC706:
Values in 2020.1 Values in Xilinx_Answer_71493_ZCU102_ZC706
Device ID 0xD011 0xD021
Sub Class 0x80 0x04
Create PetaLinux project:
$ source /opt/pkg/petalinux/2020.1/settings.sh
PetaLinux environment set to '/opt/pkg/petalinux/2020.1'
WARNING: This is not a supported OS
INFO: Checking free disk space
INFO: Checking installed tools
INFO: Checking installed development libraries
INFO: Checking network and other services
The "This is not a supported OS" is due to running Ubuntu 18.04.5 LTS, where the latest supported by PetaLinux 2020.1 is 18.04.4.
$ cd $PETALINUX
$ mkdir hdf_MYD_CZU4EV_RC
$ cd hdf_MYD_CZU4EV_RC/
$ petalinux-create -t project -n MYD_CZU4EV_RC --template zynqMP
INFO: Create project: MYD_CZU4EV_RC
INFO: New project successfully created in /opt/pkg/petalinux/2020.1/MYD_CZU4EV_RC
The get-hw-description optiion in 2020.1 is different to the Xilinx_Answer_71493_ZCU102_ZC706.pdf in that now takes the diretory containing the .XSA file rather than a hdf file. Followed the menu selections in Xilinx_Answer_71493_ZCU102_ZC706.pdf:
$ petalinux-config --get-hw-description=~/myir_workspace/pcie_root_complex
INFO: sourcing build tools
INFO: Getting hardware description...
INFO: Rename MYD_CZU4EV_RC_wrapper.xsa to system.xsa
[INFO] generating Kconfig for project
[INFO] menuconfig project
configuration written to /opt/pkg/petalinux/2020.1/MYD_CZU4EV_RC/project-spec/configs/config
*** End of the configuration.
*** Execute 'make' to start the build or try 'make help'.
[INFO] extracting yocto SDK to components/yocto
[INFO] sourcing build environment
[INFO] generating kconfig for Rootfs
[INFO] silentconfig rootfs
[INFO] generating plnxtool conf
[INFO] generating user layers
[INFO] generating workspace directory
Configured the rootfs, following the menu selections in Xilinx_Answer_71493_ZCU102_ZC706.pdf
$ petalinux-config -c rootfs
INFO: sourcing build tools
[INFO] silentconfig project
Configured the Kernel:
$ petalinux-config -c kernel
[INFO] generating kconfig for Rootfs
[INFO] menuconfig rootfs
configuration written to /opt/pkg/petalinux/2020.1/MYD_CZU4EV_RC/project-spec/configs/rootfs_config
*** End of the configuration.
*** Execute 'make' to start the build or try 'make help'.
[INFO] generating plnxtool conf
[INFO] successfully configured rootfs
When the build got to "do_menuconfig" a "Linux/arm64 5.4.0. Kernel Configuration" menu openeded in a new terminal window. Looking under Device Drivers -> PCI Support -> PCI Controller drivers "NWL PCIe Core" is selected. Device Drivers -> PCI Support -> PCI Endpoint Support is not selected
As per Xilinx_Answer_71493_ZCU102_ZC706.pdf used Exit and then saved the configuration (after making no changes).
Started the build:
$ petalinux-build
INFO: sourcing build tools
[INFO] building project
[INFO] sourcing build environment
[INFO] generating user layers
[INFO] generating workspace directory
INFO: bitbake petalinux-image-minimal
Loading cache: 100% |############################################| Time: 0:00:00
Loaded 4229 entries from dependency cache.
Parsing recipes: 100% |##########################################| Time: 0:00:05
Parsing of 2961 .bb files complete (2959 cached, 2 parsed). 4230 targets, 167 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies
WARNING: /opt/pkg/petalinux/2020.1/MYD_CZU4EV_RC/components/yocto/layers/meta-xilinx/meta-xilinx-bsp/recipes-kernel/linux/linux-xlnx_2020.1.bb:do_compile is tainted from a forced run
Initialising tasks: 100% |#######################################| Time: 0:00:04
Checking sstate mirror object availability: 100% |###############| Time: 0:00:10
Sstate summary: Wanted 996 Found 791 Missed 205 Current 42 (79% match, 80% complete)
NOTE: Executing Tasks
Currently 12 running tasks (1165 of 3690) 31% |######### |
Currently 12 running tasks (1165 of 3690) 31% |######### |
Currently 12 running tasks (1165 of 3690) 31% |######### |
Currently 12 running tasks (1165 of 3690) 31% |######### |
Currently 12 running tasks (1165 of 3690) 31% |######### |
Currently 12 running tasks (1165 of 3690) 31% |######### |
Currently 12 running tasks (1165 of 3690) 31% |######### |
Currently 12 running tasks (1165 of 3690) 31% |######### |
NOTE: linux-xlnx: compiling from external source tree /opt/pkg/petalinux/2020.1/MYD_CZU4EV_RC/components/yocto/workspace/sources/linux-xlnx
NOTE: Setscene tasks completed
NOTE: Tasks Summary: Attempted 3690 tasks of which 2680 didn't need to be rerun and all succeeded.
Summary: There was 1 WARNING message shown.
INFO: Successfully copied built images to tftp dir: /tftpboot
[INFO] successfully built project
Generate the boot image:
$ petalinux-package --boot --fsbl images/linux/zynqmp_fsbl.elf --atf images/linux/bl31.elf --u-boot images/linux/u-boot.elf --fpga images/linux/system.bit
INFO: sourcing build tools
INFO: File in BOOT BIN: "/opt/pkg/petalinux/2020.1/MYD_CZU4EV_RC/images/linux/zynqmp_fsbl.elf"
INFO: File in BOOT BIN: "/opt/pkg/petalinux/2020.1/MYD_CZU4EV_RC/images/linux/pmufw.elf"
INFO: File in BOOT BIN: "/opt/pkg/petalinux/2020.1/MYD_CZU4EV_RC/images/linux/system.bit"
INFO: File in BOOT BIN: "/opt/pkg/petalinux/2020.1/MYD_CZU4EV_RC/images/linux/bl31.elf"
INFO: File in BOOT BIN: "/opt/pkg/petalinux/2020.1/MYD_CZU4EV_RC/images/linux/system.dtb"
INFO: File in BOOT BIN: "/opt/pkg/petalinux/2020.1/MYD_CZU4EV_RC/images/linux/u-boot.elf"
INFO: Generating zynqmp binary package BOOT.BIN...
****** Xilinx Bootgen v2020.1
**** Build date : May 26 2020-14:07:15
** Copyright 1986-2020 Xilinx, Inc. All Rights Reserved.
[INFO] : Bootimage generated successfully
INFO: Binary is ready.
Copied the following files from /opt/pkg/petalinux/2020.1/MYD_CZU4EV_RC/images/linux to the SD card BOOT partition:
- image.ub
- BOOT.BIN
- boot.scr
And copied the root file system to the SD card root partition with:
mr_halfword@Haswell-Ubuntu:/media/mr_halfword/root$ sudo rm -rf *
mr_halfword@Haswell-Ubuntu:/media/mr_halfword/root$ sudo tar -xzf /opt/pkg/petalinux/2020.1/MYD_CZU4EV_RC/images/linux/rootfs.tar.gz .
The serial console output was:
Xilinx Zynq MP First Stage Boot Loader
Release 2020.1 Sep 28 2020 - 11:26:18
NOTICE: ATF running on XCZU4EV/silicon v4/RTL5.1 at 0xfffea000
NOTICE: BL31: v2.2(release):v1.1-5588-g5918e656e
NOTICE: BL31: Built : 11:19:33, Sep 28 2020
U-Boot 2020.01 (Sep 28 2020 - 11:24:21 +0000)
Board: Xilinx ZynqMP
DRAM: 4 GiB
usb dr_mode not found
PMUFW: v1.1
EL Level: EL2
Chip ID: zu4ev
NAND: 0 MiB
MMC: mmc@ff160000: 0, mmc@ff170000: 1
In: serial@ff000000
Out: serial@ff000000
Err: serial@ff000000
Bootmode: SD_MODE1
Reset reason: EXTERNAL
Net:
ZYNQ GEM: ff0e0000, mdio bus ff0e0000, phyaddr -1, interface rgmii-id
No ethernet found.
Hit any key to stop autoboot: 0
switch to partitions #0, OK
mmc1 is current device
Scanning mmc 1:1...
Found U-Boot script /boot.scr
2007 bytes read in 13 ms (150.4 KiB/s)
## Executing script at 20000000
22197128 bytes read in 1465 ms (14.4 MiB/s)
## Loading kernel from FIT Image at 10000000 ...
Using 'conf@system-top.dtb' configuration
Trying 'kernel@1' kernel subimage
Description: Linux kernel
Type: Kernel Image
Compression: gzip compressed
Data Start: 0x100000e8
Data Size: 8135353 Bytes = 7.8 MiB
Architecture: AArch64
OS: Linux
Load Address: 0x00080000
Entry Point: 0x00080000
Hash algo: sha256
Hash value: c97cc5591a3baab3d1e0eb4d6d9009e8c3c7764c54c2be55e4b385d37be627f8
Verifying Hash Integrity ... sha256+ OK
## Loading ramdisk from FIT Image at 10000000 ...
Using 'conf@system-top.dtb' configuration
Trying 'ramdisk@1' ramdisk subimage
Description: petalinux-image-minimal
Type: RAMDisk Image
Compression: uncompressed
Data Start: 0x107cb840
Data Size: 14022094 Bytes = 13.4 MiB
Architecture: AArch64
OS: Linux
Load Address: unavailable
Entry Point: unavailable
Hash algo: sha256
Hash value: 959d99b6a520a37691425b0f1d6baf8be70e9009b3ff448350fbe43d8121c112
Verifying Hash Integrity ... sha256+ OK
## Loading fdt from FIT Image at 10000000 ...
Using 'conf@system-top.dtb' configuration
Trying 'fdt@system-top.dtb' fdt subimage
Description: Flattened Device Tree blob
Type: Flat Device Tree
Compression: uncompressed
Data Start: 0x107c24b4
Data Size: 37562 Bytes = 36.7 KiB
Architecture: AArch64
Hash algo: sha256
Hash value: 517b4ca4e4785b75259ca69f320d777a0755c6d3a323bc0f42848a5f03f50c92
Verifying Hash Integrity ... sha256+ OK
Booting using the fdt blob at 0x107c24b4
Uncompressing Kernel Image
Loading Ramdisk to 782a0000, end 78fff5ce ... OK
Loading Device Tree to 000000000fff3000, end 000000000ffff2b9 ... OK
Starting kernel ...
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
[ 0.000000] Linux version 5.4.0-xilinx-v2020.1 (oe-user@oe-host) (gcc version 9.2.0 (GCC)) #1 SMP Mon Sep 28 11:18:59 UTC 2020
[ 0.000000] Machine model: xlnx,zynqmp
[ 0.000000] earlycon: cdns0 at MMIO 0x00000000ff000000 (options '115200n8')
[ 0.000000] printk: bootconsole [cdns0] enabled
[ 0.000000] efi: Getting EFI parameters from FDT:
[ 0.000000] efi: UEFI not found.
[ 0.000000] cma: Reserved 256 MiB at 0x0000000068000000
[ 0.000000] psci: probing for conduit method from DT.
[ 0.000000] psci: PSCIv1.1 detected in firmware.
[ 0.000000] psci: Using standard PSCI v0.2 function IDs
[ 0.000000] psci: MIGRATE_INFO_TYPE not supported.
[ 0.000000] psci: SMC Calling Convention v1.1
[ 0.000000] percpu: Embedded 22 pages/cpu s49944 r8192 d31976 u90112
[ 0.000000] Detected VIPT I-cache on CPU0
[ 0.000000] CPU features: detected: ARM erratum 845719
[ 0.000000] Speculative Store Bypass Disable mitigation not required
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1032192
[ 0.000000] Kernel command line: earlycon console=ttyPS0,115200 clk_ignore_unused root=/dev/ram0 rw
[ 0.000000] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[ 0.000000] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[ 0.000000] software IO TLB: mapped [mem 0x7c000000-0x80000000] (64MB)
[ 0.000000] Memory: 3755036K/4194304K available (11644K kernel code, 672K rwdata, 3624K rodata, 704K init, 323K bss, 177124K reserved, 262144K cma-reserved)
[ 0.000000] rcu: Hierarchical RCU implementation.
[ 0.000000] rcu: RCU event tracing is enabled.
[ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=4.
[ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[ 0.000000] GIC: Adjusting CPU interface base to 0x00000000f902f000
[ 0.000000] GIC: Using split EOI/Deactivate mode
[ 0.000000] random: get_random_bytes called from start_kernel+0x2a8/0x42c with crng_init=0
[ 0.000000] arch_timer: cp15 timer(s) running at 33.33MHz (phys).
[ 0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x7b00c3848, max_idle_ns: 440795203375 ns
[ 0.000003] sched_clock: 56 bits at 33MHz, resolution 30ns, wraps every 2199023255545ns
[ 0.008254] Console: colour dummy device 80x25
[ 0.012391] Calibrating delay loop (skipped), value calculated using timer frequency.. 66.66 BogoMIPS (lpj=133333)
[ 0.022667] pid_max: default: 32768 minimum: 301
[ 0.027390] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[ 0.034615] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[ 0.043306] ASID allocator initialised with 32768 entries
[ 0.047810] rcu: Hierarchical SRCU implementation.
[ 0.052688] EFI services will not be available.
[ 0.057138] smp: Bringing up secondary CPUs ...
[ 0.061838] Detected VIPT I-cache on CPU1
[ 0.061869] CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
[ 0.062213] Detected VIPT I-cache on CPU2
[ 0.062232] CPU2: Booted secondary processor 0x0000000002 [0x410fd034]
[ 0.062555] Detected VIPT I-cache on CPU3
[ 0.062574] CPU3: Booted secondary processor 0x0000000003 [0x410fd034]
[ 0.062621] smp: Brought up 1 node, 4 CPUs
[ 0.096970] SMP: Total of 4 processors activated.
[ 0.101642] CPU features: detected: 32-bit EL0 Support
[ 0.106746] CPU features: detected: CRC32 instructions
[ 0.111878] CPU: All CPU(s) started at EL2
[ 0.115926] alternatives: patching kernel code
[ 0.121379] devtmpfs: initialized
[ 0.127539] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[ 0.133314] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
[ 0.146158] xor: measuring software checksum speed
[ 0.184211] 8regs : 2375.000 MB/sec
[ 0.224239] 32regs : 2725.000 MB/sec
[ 0.264268] arm64_neon: 2365.000 MB/sec
[ 0.264305] xor: using function: 32regs (2725.000 MB/sec)
[ 0.268173] pinctrl core: initialized pinctrl subsystem
[ 0.273967] NET: Registered protocol family 16
[ 0.278707] DMA: preallocated 256 KiB pool for atomic allocations
[ 0.283851] audit: initializing netlink subsys (disabled)
[ 0.289305] audit: type=2000 audit(0.228:1): state=initialized audit_enabled=0 res=1
[ 0.296909] cpuidle: using governor menu
[ 0.300904] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[ 0.318990] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[ 0.320050] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages
[ 0.326707] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[ 0.333366] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages
[ 1.413725] DRBG: Continuing without Jitter RNG
[ 1.490175] raid6: neonx8 gen() 1543 MB/s
[ 1.558199] raid6: neonx8 xor() 1461 MB/s
[ 1.626251] raid6: neonx4 gen() 1478 MB/s
[ 1.694287] raid6: neonx4 xor() 1419 MB/s
[ 1.762336] raid6: neonx2 gen() 1124 MB/s
[ 1.830369] raid6: neonx2 xor() 1172 MB/s
[ 1.898443] raid6: neonx1 gen() 729 MB/s
[ 1.966460] raid6: neonx1 xor() 880 MB/s
[ 2.034514] raid6: int64x8 gen() 1162 MB/s
[ 2.102565] raid6: int64x8 xor() 760 MB/s
[ 2.170625] raid6: int64x4 gen() 977 MB/s
[ 2.238662] raid6: int64x4 xor() 733 MB/s
[ 2.306733] raid6: int64x2 gen() 677 MB/s
[ 2.374761] raid6: int64x2 xor() 591 MB/s
[ 2.442854] raid6: int64x1 gen() 449 MB/s
[ 2.510874] raid6: int64x1 xor() 450 MB/s
[ 2.510910] raid6: using algorithm neonx8 gen() 1543 MB/s
[ 2.514870] raid6: .... xor() 1461 MB/s, rmw enabled
[ 2.519800] raid6: using neon recovery algorithm
[ 2.524683] iommu: Default domain type: Translated
[ 2.529434] SCSI subsystem initialized
[ 2.533081] usbcore: registered new interface driver usbfs
[ 2.538425] usbcore: registered new interface driver hub
[ 2.543698] usbcore: registered new device driver usb
[ 2.548733] mc: Linux media interface: v0.10
[ 2.552946] videodev: Linux video capture interface: v2.00
[ 2.558399] pps_core: LinuxPPS API ver. 1 registered
[ 2.563310] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[ 2.572404] PTP clock support registered
[ 2.576295] EDAC MC: Ver: 3.0.0
[ 2.579766] zynqmp-ipi-mbox mailbox@ff990400: Registered ZynqMP IPI mbox with TX/RX channels.
[ 2.588036] FPGA manager framework
[ 2.591359] Advanced Linux Sound Architecture Driver Initialized.
[ 2.597556] Bluetooth: Core ver 2.22
[ 2.600866] NET: Registered protocol family 31
[ 2.605264] Bluetooth: HCI device and connection manager initialized
[ 2.611580] Bluetooth: HCI socket layer initialized
[ 2.616423] Bluetooth: L2CAP socket layer initialized
[ 2.621444] Bluetooth: SCO socket layer initialized
[ 2.626571] clocksource: Switched to clocksource arch_sys_counter
[ 2.632436] VFS: Disk quotas dquot_6.6.0
[ 2.636279] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[ 2.647095] NET: Registered protocol family 2
[ 2.647708] tcp_listen_portaddr_hash hash table entries: 2048 (order: 3, 32768 bytes, linear)
[ 2.655913] TCP established hash table entries: 32768 (order: 6, 262144 bytes, linear)
[ 2.663939] TCP bind hash table entries: 32768 (order: 7, 524288 bytes, linear)
[ 2.671381] TCP: Hash tables configured (established 32768 bind 32768)
[ 2.677554] UDP hash table entries: 2048 (order: 4, 65536 bytes, linear)
[ 2.684225] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes, linear)
[ 2.691401] NET: Registered protocol family 1
[ 2.695815] RPC: Registered named UNIX socket transport module.
[ 2.701460] RPC: Registered udp transport module.
[ 2.706127] RPC: Registered tcp transport module.
[ 2.710795] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 2.717430] PCI: CLS 0 bytes, default 64
[ 2.721186] Trying to unpack rootfs image as initramfs...
[ 3.357997] Freeing initrd memory: 13692K
[ 3.358467] hw perfevents: no interrupt-affinity property for /pmu, guessing.
[ 3.363649] hw perfevents: enabled with armv8_pmuv3 PMU driver, 7 counters available
[ 3.371944] Initialise system trusted keyrings
[ 3.375669] workingset: timestamp_bits=46 max_order=20 bucket_order=0
[ 3.382782] NFS: Registering the id_resolver key type
[ 3.387001] Key type id_resolver registered
[ 3.391144] Key type id_legacy registered
[ 3.395126] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[ 3.401794] jffs2: version 2.2. (NAND) © 2001-2006 Red Hat, Inc.
[ 3.422824] NET: Registered protocol family 38
[ 3.422866] Key type asymmetric registered
[ 3.425686] Asymmetric key parser 'x509' registered
[ 3.430558] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247)
[ 3.437889] io scheduler mq-deadline registered
[ 3.442385] io scheduler kyber registered
[ 3.471215] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[ 3.474518] cacheinfo: Unable to detect cache hierarchy for CPU 0
[ 3.482103] brd: module loaded
[ 3.486395] loop: module loaded
[ 3.487024] mtdoops: mtd device (mtddev=name/number) must be supplied
[ 3.491537] libphy: Fixed MDIO Bus: probed
[ 3.495529] tun: Universal TUN/TAP device driver, 1.6
[ 3.499684] CAN device driver interface
[ 3.504046] usbcore: registered new interface driver asix
[ 3.508813] usbcore: registered new interface driver ax88179_178a
[ 3.514852] usbcore: registered new interface driver cdc_ether
[ 3.520644] usbcore: registered new interface driver net1080
[ 3.526266] usbcore: registered new interface driver cdc_subset
[ 3.532148] usbcore: registered new interface driver zaurus
[ 3.537694] usbcore: registered new interface driver cdc_ncm
[ 3.543914] usbcore: registered new interface driver uas
[ 3.548594] usbcore: registered new interface driver usb-storage
[ 3.555055] rtc_zynqmp ffa60000.rtc: registered as rtc0
[ 3.559767] i2c /dev entries driver
[ 3.564445] usbcore: registered new interface driver uvcvideo
[ 3.568892] USB Video Class driver (1.1.1)
[ 3.573369] Bluetooth: HCI UART driver ver 2.3
[ 3.577373] Bluetooth: HCI UART protocol H4 registered
[ 3.582473] Bluetooth: HCI UART protocol BCSP registered
[ 3.587763] Bluetooth: HCI UART protocol LL registered
[ 3.592853] Bluetooth: HCI UART protocol ATH3K registered
[ 3.598230] Bluetooth: HCI UART protocol Three-wire (H5) registered
[ 3.604475] Bluetooth: HCI UART protocol Intel registered
[ 3.609819] Bluetooth: HCI UART protocol QCA registered
[ 3.615019] usbcore: registered new interface driver bcm203x
[ 3.620645] usbcore: registered new interface driver bpa10x
[ 3.626179] usbcore: registered new interface driver bfusb
[ 3.631628] usbcore: registered new interface driver btusb
[ 3.637088] usbcore: registered new interface driver ath3k
[ 3.642618] EDAC MC: ECC not enabled
[ 3.646171] EDAC DEVICE0: Giving out device to module zynqmp-ocm-edac controller zynqmp_ocm: DEV ff960000.memory-controller (INTERRUPT)
[ 3.658502] sdhci: Secure Digital Host Controller Interface driver
[ 3.664303] sdhci: Copyright(c) Pierre Ossman
[ 3.668626] sdhci-pltfm: SDHCI platform and OF driver helper
[ 3.674539] ledtrig-cpu: registered to indicate activity on CPUs
[ 3.680258] zynqmp_firmware_probe Platform Management API v1.1
[ 3.686015] zynqmp_firmware_probe Trustzone version v1.0
[ 3.715033] alg: No test for xilinx-zynqmp-aes (zynqmp-aes)
[ 3.716869] zynqmp_aes zynqmp_aes: AES Successfully Registered
[ 3.716869]
[ 3.722495] alg: No test for xilinx-keccak-384 (zynqmp-keccak-384)
[ 3.730197] alg: No test for xilinx-zynqmp-rsa (zynqmp-rsa)
[ 3.735745] usbcore: registered new interface driver usbhid
[ 3.739535] usbhid: USB HID core driver
[ 3.745613] fpga_manager fpga0: Xilinx ZynqMP FPGA Manager registered
[ 3.750010] usbcore: registered new interface driver snd-usb-audio
[ 3.756599] pktgen: Packet Generator for packet performance testing. Version: 2.75
[ 3.763922] Initializing XFRM netlink socket
[ 3.767725] NET: Registered protocol family 10
[ 3.772443] Segment Routing with IPv6
[ 3.775794] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[ 3.781898] NET: Registered protocol family 17
[ 3.785993] NET: Registered protocol family 15
[ 3.790411] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[ 3.803283] can: controller area network core (rev 20170425 abi 9)
[ 3.809445] NET: Registered protocol family 29
[ 3.813837] can: raw protocol (rev 20170425)
[ 3.818072] can: broadcast manager protocol (rev 20170425 t)
[ 3.823699] can: netlink gateway (rev 20190810) max_hops=1
[ 3.829216] Bluetooth: RFCOMM TTY layer initialized
[ 3.833997] Bluetooth: RFCOMM socket layer initialized
[ 3.839106] Bluetooth: RFCOMM ver 1.11
[ 3.842815] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[ 3.848087] Bluetooth: BNEP filters: protocol multicast
[ 3.853279] Bluetooth: BNEP socket layer initialized
[ 3.858208] Bluetooth: HIDP (Human Interface Emulation) ver 1.2
[ 3.864091] Bluetooth: HIDP socket layer initialized
[ 3.869124] 9pnet: Installing 9P2000 support
[ 3.873277] Key type dns_resolver registered
[ 3.877737] registered taskstats version 1
[ 3.881565] Loading compiled-in X.509 certificates
[ 3.886708] Btrfs loaded, crc32c=crc32c-generic
[ 3.899080] ff000000.serial: ttyPS0 at MMIO 0xff000000 (irq = 45, base_baud = 6249999) is a xuartps
[ 3.908094] printk: console [ttyPS0] enabled
[ 3.908094] printk: console [ttyPS0] enabled
[ 3.912389] printk: bootconsole [cdns0] disabled
[ 3.912389] printk: bootconsole [cdns0] disabled
[ 3.921607] of-fpga-region fpga-full: FPGA Region probed
[ 3.931949] nwl-pcie fd0e0000.pcie: Link is DOWN
[ 3.936597] nwl-pcie fd0e0000.pcie: host bridge /amba/pcie@fd0e0000 ranges:
[ 3.943574] nwl-pcie fd0e0000.pcie: MEM 0xe0000000..0xefffffff -> 0xe0000000
[ 3.950793] nwl-pcie fd0e0000.pcie: MEM 0x600000000..0x7ffffffff -> 0x600000000
[ 3.958380] nwl-pcie fd0e0000.pcie: PCI host bridge to bus 0000:00
[ 3.964556] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 3.970039] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xefffffff]
[ 3.976904] pci_bus 0000:00: root bus resource [mem 0x600000000-0x7ffffffff pref]
[ 3.984399] pci 0000:00:00.0: [10ee:d021] type 01 class 0x060400
[ 3.990463] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot
[ 3.998828] pci 0000:00:00.0: PCI bridge to [bus 01-0c]
[ 4.004333] xilinx-zynqmp-dma fd500000.dma: ZynqMP DMA driver Probe success
[ 4.011426] xilinx-zynqmp-dma fd510000.dma: ZynqMP DMA driver Probe success
[ 4.018522] xilinx-zynqmp-dma fd520000.dma: ZynqMP DMA driver Probe success
[ 4.025619] xilinx-zynqmp-dma fd530000.dma: ZynqMP DMA driver Probe success
[ 4.032720] xilinx-zynqmp-dma fd540000.dma: ZynqMP DMA driver Probe success
[ 4.039819] xilinx-zynqmp-dma fd550000.dma: ZynqMP DMA driver Probe success
[ 4.046916] xilinx-zynqmp-dma fd560000.dma: ZynqMP DMA driver Probe success
[ 4.054022] xilinx-zynqmp-dma fd570000.dma: ZynqMP DMA driver Probe success
[ 4.061189] xilinx-zynqmp-dma ffa80000.dma: ZynqMP DMA driver Probe success
[ 4.068296] xilinx-zynqmp-dma ffa90000.dma: ZynqMP DMA driver Probe success
[ 4.075388] xilinx-zynqmp-dma ffaa0000.dma: ZynqMP DMA driver Probe success
[ 4.082485] xilinx-zynqmp-dma ffab0000.dma: ZynqMP DMA driver Probe success
[ 4.089581] xilinx-zynqmp-dma ffac0000.dma: ZynqMP DMA driver Probe success
[ 4.096676] xilinx-zynqmp-dma ffad0000.dma: ZynqMP DMA driver Probe success
[ 4.103780] xilinx-zynqmp-dma ffae0000.dma: ZynqMP DMA driver Probe success
[ 4.110875] xilinx-zynqmp-dma ffaf0000.dma: ZynqMP DMA driver Probe success
[ 4.118280] spi_master spi0: cannot find modalias for /amba/spi@ff0f0000/flash@0
[ 4.125675] spi_master spi0: Failed to create SPI device for /amba/spi@ff0f0000/flash@0
[ 4.133967] macb ff0e0000.ethernet: Not enabling partial store and forward
[ 4.141328] libphy: MACB_mii_bus: probed
[ 4.149761] macb ff0e0000.ethernet eth0: Could not attach to PHY
[ 4.175308] xilinx-axipmon ffa00000.perf-monitor: Probed Xilinx APM
[ 4.181813] zynqmp_pll_disable() clock disable failed for apll_int, ret = -13
[ 4.189032] xilinx-axipmon fd0b0000.perf-monitor: Probed Xilinx APM
[ 4.195505] xilinx-axipmon fd490000.perf-monitor: Probed Xilinx APM
[ 4.201987] xilinx-axipmon ffa10000.perf-monitor: Probed Xilinx APM
[ 4.208473] dwc3-of-simple ff9d0000.usb0: dwc3_simple_set_phydata: Can't find usb3-phy
[ 4.216785] dwc3 fe200000.dwc3: Failed to get clk 'ref': -2
[ 4.223131] cdns-i2c ff030000.i2c: 400 kHz mmio ff030000 irq 31
[ 4.229371] cdns-wdt fd4d0000.watchdog: Xilinx Watchdog Timer with timeout 60s
[ 4.236835] cdns-wdt ff150000.watchdog: Xilinx Watchdog Timer with timeout 10s
[ 4.276071] mmc0: SDHCI controller on ff160000.mmc [ff160000.mmc] using ADMA 64-bit
[ 4.315333] mmc1: SDHCI controller on ff170000.mmc [ff170000.mmc] using ADMA 64-bit
[ 4.325370] rtc_zynqmp ffa60000.rtc: setting system clock to 1970-01-01T00:00:06 UTC (6)
[ 4.333459] of_cfs_init
[ 4.335919] of_cfs_init: OK
[ 4.338829] cfg80211: Loading compiled-in X.509 certificates for regulatory database
[ 4.373495] mmc0: new HS200 MMC card at address 0001
[ 4.378869] mmcblk0: mmc0:0001 Q2J54A 3.64 GiB
[ 4.383577] mmcblk0boot0: mmc0:0001 Q2J54A partition 1 2.00 MiB
[ 4.389669] mmcblk0boot1: mmc0:0001 Q2J54A partition 2 2.00 MiB
[ 4.395674] mmcblk0rpmb: mmc0:0001 Q2J54A partition 3 512 KiB, chardev (245:0)
[ 4.403952] mmcblk0: p1
[ 4.421791] mmc1: new high speed SDHC card at address aaaa
[ 4.427607] mmcblk1: mmc1:aaaa SU04G 3.69 GiB
[ 4.435109] mmcblk1: p1 p2
[ 4.480192] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
[ 4.486723] clk: Not disabling unused clocks
[ 4.490986] ALSA device list:
[ 4.493937] No soundcards found.
[ 4.497622] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
[ 4.506234] cfg80211: failed to load regulatory.db
[ 4.511205] Freeing unused kernel memory: 704K
[ 4.530603] Run /init as init process
INIT: version 2.88 booting
Starting udev
[ 4.645149] udevd[164]: starting version 3.2.8
[ 4.649893] random: udevd: uninitialized urandom read (16 bytes read)
[ 4.656404] random: udevd: uninitialized urandom read (16 bytes read)
[ 4.662904] random: udevd: uninitialized urandom read (16 bytes read)
[ 4.673854] udevd[165]: starting eudev-3.2.8
[ 4.921031] random: fast init done
[ 5.079858] EXT4-fs (mmcblk0p1): mounted filesystem with ordered data mode. Opts: (null)
[ 5.216133] EXT4-fs (mmcblk1p2): mounted filesystem with ordered data mode. Opts: (null)
[ 5.230453] FAT-fs (mmcblk1p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
[ 5.297964] dmaproxy: loading out-of-tree module taints kernel.
Mon Sep 28 11:25:10 UTC 2020
[ 6.705129] urandom_read: 4 callbacks suppressed
[ 6.705134] random: dd: uninitialized urandom read (512 bytes read)
Configuring packages on first boot....
(This may take several minutes. Please do not power off the machine.)
Running postinst /etc/rpm-postinsts/100-sysvinit-inittab...
update-rc.d: /etc/init.d/run-postinsts exists during rc.d purge (continuing)
INIT: Entering runlevel: 5
Configuring network interfaces... Cannot find device "eth0"
Starting haveged: haveged: listening socket at 3
haveged: haveged starting up
Starting Dropbear SSH server: [ 6.865679] random: dropbearkey: uninitialized urandom read (32 bytes read)
Generating 2048 bit rsa key, this may take a while...
[ 6.875315] random: dropbearkey: uninitialized urandom read (32 bytes read)
haveged: haveged: ver: 1.9.5; arch: generic; vend: ; build: (gcc 9.2.0 CTV); collect: 128K
haveged: haveged: cpu: (VC); data: 16K (D); inst: 16K (D); idx: 11/40; sz: 15456/64452
haveged: haveged: tot tests(BA8): A:1/1 B:1/1 continuous tests(B): last entropy estimate 7.99902
haveged: haveged: fills: 0, generated: 0
[ 7.624975] random: crng init done
Public key portion is:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFqak1GqU29wNQCUG+oXDo1eRfB3L+mkF0VVs9syDRHdiacv862cMBKog7Kh126y0s6jmuHeNgeiYO6lyrIdAOqlvIkXTzTCco1ciF+At1ztBDvGO6i81tQ0FDBvDOH0yanrrEQzRPYXhZUIrbAtAcS/TVq97qtAuHsejzTTmhHi+Y2YMWeuYso9keGlxShcefqlbijIVrCOIc/VGdi+BeSrXaaQ6kZlaYxSw3Wg8io2B7IH3vn520REKImyX4xAYe/Aup9W188NgeGsqJpGn91axvHYFv5LNrAoZd73wbHKyUkSvTsgYANUb6Nu/JN/CmcURdNtFO5u92IHUs92xz root@MYD_CZU4EV_RC
Fingerprint: sha1!! 8a:40:17:b0:03:12:e0:3c:d3:c4:fb:e1:e7:fc:0f:1d:10:11:71:8e
dropbear.
Starting internet superserver: inetd.
Starting syslogd/klogd: done
Starting tcf-agent: OK
PetaLinux 2020.1 MYD_CZU4EV_RC ttyPS0
MYD_CZU4EV_RC login:
After linux has booted the PCIe root bridge is visible:
root@MYD_CZU4EV_RC:~# lspci -v
00:00.0 PCI bridge: Xilinx Corporation Device d021 (prog-if 00 [Normal decode])
Flags: fast devsel, IRQ 255
Bus: primary=00, secondary=01, subordinate=0c, sec-latency=0
I/O behind bridge: 00000000-00000fff [size=4K]
Memory behind bridge: None
Prefetchable memory behind bridge: None
Capabilities: [40] Power Management version 3
Capabilities: [60] Express Root Port (Slot-), MSI 00
Capabilities: [100] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [10c] Virtual Channel
Capabilities: [128] Vendor Specific Information: ID=1234 Rev=1 Len=018 <?>
Neither u-boot nor Linux can access the eth0.
U-Boot reports:
ZYNQ GEM: ff0e0000, mdio bus ff0e0000, phyaddr -1, interface rgmii-id
No ethernet found.
Linux reports:
[ 4.149761] macb ff0e0000.ethernet eth0: Could not attach to PHY
dtc --in-format dtb /opt/pkg/petalinux/2020.1/MYD_CZU4EV_RC/images/linux/system.dtb shows no phy entry for ethernet@ff0e0000
Whereas dtc --in-format dtb ~/Documents/MPSOC2019.1_DVD_V1.1/02-Images/Linux-Images/CZU4EV/tutorial/devicetree.dtb has the following in ethernet@ff0e0000 :
phy@21 {
reg = <0x5>;
ti,rx-internal-delay = <0x8>;
ti,tx-internal-delay = <0xa>;
ti,fifo-depth = <0x1>;
ti,dp83867-rxctrl-strap-quirk;
phandle = <0xe>;
};
https://www.xilinx.com/support/answers/61117.html explains that
The device tree DTS and DTSI files created by PetaLinux do not include PHY or MDIO information.
U-Boot can report the correct phy status when manually access phy address 0x5 which the myir documentation gives as the gem3 phy address.
With no cable plugged in:
ZynqMP> mii info 0x5
PHY 0x05: OUI = 0x80028, Model = 0x23, Rev = 0x01, 10baseT, HDX
With cable connected to 100M switch port:
ZynqMP> mii info 0x5
PHY 0x05: OUI = 0x80028, Model = 0x23, Rev = 0x01, 100baseT, FDX
With cable connected to 1G switch port:
ZynqMP> mii info 0x5
PHY 0x05: OUI = 0x80028, Model = 0x23, Rev = 0x01, 1000baseT, FDX
https://www.xilinx.com/support/documentation/sw_manuals/xilinx2020_1/ug1144-petalinux-tools-reference-guide.pdf notes that board and user specific dtsi entries should be added to project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi.
Added the following to MYD_CZU4EV_RC/project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi :
&gem3 {
phy-handle = <&phy3>;
phy3: phy@5 {
reg = <0x5>;
ti,rx-internal-delay = <0x8>;
ti,tx-internal-delay = <0xa>;
ti,fifo-depth = <0x1>;
ti,dp83867-rxctrl-strap-quirk;
};
};
And re-built with:
$ petalinux-build
$ petalinux-package --boot --fsbl images/linux/zynqmp_fsbl.elf --atf images/linux/bl31.elf --u-boot images/linux/u-boot.elf --fpga images/linux/system.bit --force
The --force on the petalinux-package was to avoid the following error: ERROR: Output file "/opt/pkg/petalinux/2020.1/MYD_CZU4EV_RC/images/linux/BOOT.BIN" already exists. Please use --force to overwrite it.
Having copied the re-built BOOT.BIN and image.ub into the SD card BOOT partition, the Ethernet is now working in U-Boot and Linux.
Did notice that the fingerprint for the RSA key changes on every re-boot, which causes ssh from a PC to report
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Switched from dropbear to sshd but the keys were still re-generated on a reboot. According to https://forums.xilinx.com/t5/Embedded-Linux/Generating-a-persistent-ssh-key/td-p/834672 regenerating the keys on a reboot is by design.
In fact, the petalinux image.ub file contains the root file system which is unpacked to RAM at startup. I.e. erase the root partition and still boots.
Also noticed that the FPGA bitstream created by tutorial.tcl only had ic2 1 enabled which is for the clock generator on the SOM.
Other projects have enabled ic2 0 using EMIO. After enabling ic2 and regnerating the MYD_CZU4EV_RC_wrapper.xsa ran the following:
$ petalinux-config --get-hw-description=~/myir_workspace/pcie_root_complex
And selected "Subsystem AUTO Hardware Settings"
$ petalinux-build
$ petalinux-package --boot --fsbl images/linux/zynqmp_fsbl.elf --atf images/linux/bl31.elf --u-boot images/linux/u-boot.elf --fpga images/linux/system.bit --force
Copying the i2c0 i2c-mux@70 dts entries from ~/Documents/MPSOC2019.1_DVD_V1.1/02-Images/Linux-Images/CZU4EV/hdmi_lcd/devicetree.dtb into project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi didn't help.
Looking at the Diagram in the myd-czu4ev-hdmi_lcd shows the IIC_0 from the zynq_ultra_ps_e_0 connected to an IIC_0 external interface. Whereas isn't connected in the pci_root_complex project where ic20 doesn't work. I.e. EMIO requires FPGA interconnects.
Added constraints for IIC_0_sda_io and IIC_0_scl_io and re-created the bitstream. In the MYD_CZU4EV_RC_wrapper_io_places.rpt checked that the IIC_0 I2C signals mapped to the "Pin Name" given on the MYB-CZU3EG-C_V13.pdf schematic.
Exported the hardware and then re-build the software:
$ petalinux-config --get-hw-description=~/myir_workspace/pcie_root_complex
$ petalinux-build
$ petalinux-package --boot --fsbl images/linux/zynqmp_fsbl.elf --atf images/linux/bl31.elf --u-boot images/linux/u-boot.elf --fpga images/linux/system.bit --force
Not sure what happened, but the first attempt didn't pick up the modified FPGA bitstream, and images/linux/system.bit was unchanged. Worked on a second attempt (in which all .bit files first removed from the /opt/pkg/petalinux/2020.1/MYD_CZU4EV_RC directory.
With the above Kernel configuration with PCIe root port enabled plugged in a 2 port PCIe serial card. But the Kernel paniced during boot up:
[ 5.918747] nwl-pcie fd0e0000.pcie: Link is UP
[ 5.923222] nwl-pcie fd0e0000.pcie: host bridge /amba/pcie@fd0e0000 ranges:
[ 5.930193] nwl-pcie fd0e0000.pcie: MEM 0xe0000000..0xefffffff -> 0xe0000000
[ 5.937411] nwl-pcie fd0e0000.pcie: MEM 0x600000000..0x7ffffffff -> 0x600000000
[ 5.944995] nwl-pcie fd0e0000.pcie: PCI host bridge to bus 0000:00
[ 5.951176] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 5.956656] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xefffffff]
[ 5.963523] pci_bus 0000:00: root bus resource [mem 0x600000000-0x7ffffffff pref]
[ 5.971019] pci 0000:00:00.0: [10ee:d021] type 01 class 0x060400
[ 5.977086] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot
[ 5.984464] pci 0000:01:00.0: [1c00:3253] type 00 class 0x070005
[ 5.990507] pci 0000:01:00.0: reg 0x10: [io 0x0000-0x00ff]
[ 5.996086] pci 0000:01:00.0: reg 0x14: [mem 0x00000000-0x00007fff pref]
[ 6.002795] pci 0000:01:00.0: reg 0x18: [io 0x0000-0x0003]
[ 6.008411] pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x00007fff pref]
[ 6.015205] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[ 6.022550] pci 0000:00:00.0: BAR 8: assigned [mem 0xe0000000-0xe00fffff]
[ 6.029340] pci 0000:01:00.0: BAR 1: assigned [mem 0xe0000000-0xe0007fff pref]
[ 6.036564] pci 0000:01:00.0: BAR 6: assigned [mem 0xe0008000-0xe000ffff pref]
[ 6.043779] pci 0000:01:00.0: BAR 0: no space for [io size 0x0100]
[ 6.050036] pci 0000:01:00.0: BAR 0: failed to assign [io size 0x0100]
[ 6.056641] pci 0000:01:00.0: BAR 2: no space for [io size 0x0004]
[ 6.062899] pci 0000:01:00.0: BAR 2: failed to assign [io size 0x0004]
[ 6.069506] pci 0000:00:00.0: PCI bridge to [bus 01-0c]
[ 6.074725] pci 0000:00:00.0: bridge window [mem 0xe0000000-0xe00fffff]
[ 6.081627] pci 0000:00:00.0: enabling device (0000 -> 0002)
[ 6.087290] serial 0000:01:00.0: enabling device (0000 -> 0002)
[ 6.093638] 0000:01:00.0: ttyS0 at I/O 0xc0 (irq = 50, base_baud = 115200) is a XR16850
[ 6.101636] Unable to handle kernel paging request at virtual address fffffffefec000c3
[ 6.109544] Mem abort info:
[ 6.112328] ESR = 0x96000006
[ 6.115377] EC = 0x25: DABT (current EL), IL = 32 bits
[ 6.120678] SET = 0, FnV = 0
[ 6.123726] EA = 0, S1PTW = 0
[ 6.126858] Data abort info:
[ 6.129727] ISV = 0, ISS = 0x00000006
[ 6.133552] CM = 0, WnR = 0
[ 6.136512] swapper pgtable: 4k pages, 39-bit VAs, pgdp=0000000000f71000
[ 6.143203] [fffffffefec000c3] pgd=00000000010d9003, pud=00000000010d9003, pmd=0000000000000000
[ 6.151894] Internal error: Oops: 96000006 [#1] SMP
[ 6.156757] Modules linked in:
[ 6.159799] CPU: 0 PID: 38 Comm: kworker/0:1 Not tainted 5.4.0-xilinx-v2020.1 #1
[ 6.167181] Hardware name: xlnx,zynqmp (DT)
[ 6.171357] Workqueue: events deferred_probe_work_func
[ 6.176480] pstate: 60000005 (nZCv daif -PAN -UAO)
[ 6.181255] pc : io_serial_in+0x1c/0x38
[ 6.185073] lr : serial8250_set_sleep+0xc0/0x138
[ 6.189671] sp : ffffffc011293440
[ 6.192970] x29: ffffffc011293440 x28: ffffffc011071490
[ 6.198264] x27: 0000000000000032 x26: ffffff886e69fd00
[ 6.203559] x25: ffffffc0112934f8 x24: ffffffc010ec64e0
[ 6.208854] x23: ffffff886e8c3900 x22: ffffffc011070f90
[ 6.214149] x21: ffffff887aba9240 x20: 0000000000000000
[ 6.219443] x19: ffffffc011116f48 x18: 0000000000000010
[ 6.224738] x17: 0000000000000000 x16: 0000000000000001
[ 6.230033] x15: ffffff887aba9668 x14: 3631525820612073
[ 6.235328] x13: 6920293030323531 x12: 31203d2064756162
[ 6.240623] x11: 5f65736162202c30 x10: 35203d2071726928
[ 6.245918] x9 : ffffffc0110e1000 x8 : 000000000000010a
[ 6.251212] x7 : 0000000000000006 x6 : ffffffc0110e10ba
[ 6.256507] x5 : 000000000000000f x4 : 0000000000000000
[ 6.261802] x3 : ffffffc0105d4630 x2 : 0000000000000000
[ 6.267097] x1 : fffffffefec000c3 x0 : fffffffefec00000
[ 6.272392] Call trace:
[ 6.274824] io_serial_in+0x1c/0x38
[ 6.278295] serial8250_pm+0x28/0x30
[ 6.281856] uart_add_one_port+0x35c/0x500
[ 6.285935] serial8250_register_8250_port+0x288/0x418
[ 6.291057] pciserial_init_ports+0x124/0x218
[ 6.295395] pciserial_init_one+0xd4/0x1c8
[ 6.299476] pci_device_probe+0xb4/0x168
[ 6.303382] really_probe+0xd8/0x2f8
[ 6.306940] driver_probe_device+0x54/0xe8
[ 6.311020] __device_attach_driver+0x80/0xb8
[ 6.315360] bus_for_each_drv+0x74/0xc0
[ 6.319179] __device_attach+0xdc/0x138
[ 6.322998] device_attach+0x10/0x18
[ 6.326558] pci_bus_add_device+0x4c/0xa8
[ 6.330550] pci_bus_add_devices+0x38/0x80
[ 6.334629] pci_bus_add_devices+0x64/0x80
[ 6.338710] nwl_pcie_probe+0xa10/0xa18
[ 6.342529] platform_drv_probe+0x50/0xa0
[ 6.346521] really_probe+0xd8/0x2f8
[ 6.350080] driver_probe_device+0x54/0xe8
[ 6.354160] __device_attach_driver+0x80/0xb8
[ 6.358499] bus_for_each_drv+0x74/0xc0
[ 6.362319] __device_attach+0xdc/0x138
[ 6.366138] device_initial_probe+0x10/0x18
[ 6.370304] bus_probe_device+0x90/0x98
[ 6.374124] deferred_probe_work_func+0x6c/0xa0
[ 6.378638] process_one_work+0x1c4/0x338
[ 6.382629] worker_thread+0x260/0x488
[ 6.386363] kthread+0x120/0x128
[ 6.389574] ret_from_fork+0x10/0x18
[ 6.393135] Code: 8b21c001 b25f7be0 f2bfd800 8b000021 (39400021)
[ 6.399217] ---[ end trace 7b4ea446dd8a1f5a ]---
The 8250 PCI serial drivers were built into the Kernel. Change them to be loadable modules. The loaded module generates the same "Unable to handle kernel paging request at virtual address fffffffefec000c3", but now a loadable module doesn't cause the Kernel boot to stop.
The PCI information after boot:
# lspci -v
00:00.0 PCI bridge: Xilinx Corporation Device d021 (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 255
Bus: primary=00, secondary=01, subordinate=0c, sec-latency=0
I/O behind bridge: 00000000-00000fff [size=4K]
Memory behind bridge: e0000000-e00fffff [size=1M]
Prefetchable memory behind bridge: None
Capabilities: [40] Power Management version 3
Capabilities: [60] Express Root Port (Slot-), MSI 00
Capabilities: [100] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [10c] Virtual Channel
Capabilities: [128] Vendor Specific Information: ID=1234 Rev=1 Len=018 <?>
01:00.0 Serial controller: Device 1c00:3253 (rev 10) (prog-if 05 [16850])
Subsystem: Device 1c00:3253
Flags: fast devsel, IRQ 53
I/O ports at <unassigned> [disabled]
Memory at e0000000 (32-bit, prefetchable) [size=32K]
I/O ports at <unassigned> [disabled]
[virtual] Expansion ROM at e0008000 [disabled] [size=32K]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/32 Maskable+ 64bit+
Capabilities: [80] Express Legacy Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Kernel driver in use: serial
Kernel modules: 8250_pci
The components/yocto/workspace/sources/linux-xlnx/Documentation/devicetree/bindings/pci/xilinx-nwl-pcie.txt file describes the device tree entries read by the RC driver, including the ranges. The documentation notes that the hardware doesn't support the I/O space region.
The lspci -vvv output shows the root port is PCIe2x1, but is running at at PCIe1 speed:
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM not supported
ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s (downgraded), Width x1 (ok)
TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
And the serial card is PCIe1x1:
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 unlimited
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s (ok), Width x1 (ok)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
Enabled Infiniband support in the Kernel configuration, including Mellanox ConnectX drivers.
This created a mlx4_core.ko which supports the same PCI devices as that in the Ubuntu systems.
To get user mode support added the following to project-spec/meta-user/conf/user-rootfsconfig, and enabled it:
CONFIG_rdma-core
The ConnectX-2 enumerates on the PCIe bus and the drivers are loaded:
root@MYD_CZU4EV_RC:~# lspci -v
00:00.0 PCI bridge: Xilinx Corporation Device d021 (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 255
Bus: primary=00, secondary=01, subordinate=0c, sec-latency=0
I/O behind bridge: 00000000-00000fff [size=4K]
Memory behind bridge: e0000000-e00fffff [size=1M]
Prefetchable memory behind bridge: 0000000600000000-00000006007fffff [size=8M]
Capabilities: [40] Power Management version 3
Capabilities: [60] Express Root Port (Slot-), MSI 00
Capabilities: [100] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [10c] Virtual Channel
Capabilities: [128] Vendor Specific Information: ID=1234 Rev=1 Len=018 <?>
01:00.0 InfiniBand: Mellanox Technologies MT25408A0-FCC-QI ConnectX, Dual Port 40Gb/s InfiniBand / 10GigE Adapter IC with PCIe 2.0 x8 5.0GT/s In... (rev b0)
Subsystem: Mellanox Technologies Device 0036
Flags: bus master, fast devsel, latency 0, IRQ 53
Memory at e0000000 (64-bit, non-prefetchable) [size=1M]
[virtual] Memory at 600000000 (64-bit, prefetchable) [size=8M]
Capabilities: [40] Power Management version 3
Capabilities: [48] Vital Product Data
Capabilities: [9c] MSI-X: Enable- Count=128 Masked-
Capabilities: [60] Express Endpoint, MSI 00
Capabilities: [100] Alternative Routing-ID Interpretation (ARI)
Capabilities: [148] Device Serial Number 00-02-c9-03-00-4b-fe-ee
Kernel driver in use: mlx4_core
Kernel modules: mlx4_core
root@MYD_CZU4EV_RC:~# lsmod
Tainted: G
dmaproxy 16384 0 - Live 0xffffffc008c34000 (O)
mlx4_ib 167936 0 - Live 0xffffffc008cc6000
ib_uverbs 118784 1 mlx4_ib, Live 0xffffffc008c9d000
ib_core 270336 2 mlx4_ib,ib_uverbs, Live 0xffffffc008c46000
mlx4_core 307200 1 mlx4_ib, Live 0xffffffc008be8000
uio_pdrv_genirq 16384 0 - Live 0xffffffc008be0000
The Infiniband ports become active.
Attempting to use the rdma-core programs found that the Queue-Pairs would connect, but the transfers would not hang.
Trying a single short transfer and using events on the Zync allows can allow the test to complete on the Ubuntu end:
$ ibv_rc_pingpong --ib-port=1 192.168.0.100 --size=28 -c -n 1
local address: LID 0x0002, QPN 0x00022f, PSN 0xec3315, GID ::
remote address: LID 0x0007, QPN 0x00022c, PSN 0xbea5b9, GID ::
56 bytes in 0.00 seconds = 5.97 Mbit/sec
1 iters in 0.00 seconds = 75.00 usec/iter
But doesn't complete on the Zync end:
root@MYD_CZU4EV_RC:~# ibv_rc_pingpong --ib-port=2 --size=28 -c -n 1 -e
local address: LID 0x0007, QPN 0x00022c, PSN 0xbea5b9, GID ::
remote address: LID 0x0002, QPN 0x00022f, PSN 0xec3315, GID ::
^C
Suspect an issue with cache-coherency. https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842098/Zynq+UltraScale+MPSoC+Cache+Coherency says:
Zynq UltraScale+ MPSOC defaults to software managed coherency by default at this time (2017.4) and the following page describes the details of turning on hardware managed coherency.
In Vivado under Re-Customise IP -> Advanced Configuration -> CCI Enablement ticked "FPD Masters(PCIe,SATA)"
In the MYD_CZU4EV_RC_wrapper.xsa the only functional change seems to be that value of PSU__FPDMASTERS_COHERENCY changed from 0 to 1.
Re-built software with:
$ petalinux-config --get-hw-description=~/myir_workspace/pcie_root_complex
$ petalinux-build
$ petalinux-package --boot --fsbl images/linux/zynqmp_fsbl.elf --atf images/linux/bl31.elf --u-boot images/linux/u-boot.elf --fpga images/linux/system.bit --force
Following the change, the driver fails to load reporting:
root@MYD_CZU4EV_RC:~# dmesg|grep mlx4
[ 6.994770] mlx4_core: Mellanox ConnectX core driver v4.0-0
[ 7.000407] mlx4_core: Initializing 0000:01:00.0
[ 7.010752] mlx4_core 0000:01:00.0: enabling device (0000 -> 0002)
[ 8.030943] mlx4_core 0000:01:00.0: Installed FW has unsupported command interface revision 0
[ 8.039468] mlx4_core 0000:01:00.0: (Installed FW version is 0.0.000)
[ 8.045904] mlx4_core 0000:01:00.0: This driver version supports only revisions 2 to 3
[ 8.053812] mlx4_core 0000:01:00.0: QUERY_FW command failed, aborting
[ 8.060241] mlx4_core 0000:01:00.0: Failed to init fw, aborting.
And the Inifniband ports don't become active.
Reverted the CCI Enablement in the Vivado project, ahd the Infiniband ports then become active again:
root@MYD_CZU4EV_RC:~# lsmod
Tainted: G
dmaproxy 16384 0 - Live 0xffffffc008c34000 (O)
mlx4_ib 167936 0 - Live 0xffffffc008cc6000
ib_uverbs 118784 1 mlx4_ib, Live 0xffffffc008c9d000
ib_core 270336 2 mlx4_ib,ib_uverbs, Live 0xffffffc008c46000
mlx4_core 307200 1 mlx4_ib, Live 0xffffffc008be8000
uio_pdrv_genirq 16384 0 - Live 0xffffffc008be0000
root@MYD_CZU4EV_RC:~# dmesg|grep mlx4
[ 6.995567] mlx4_core: Mellanox ConnectX core driver v4.0-0
[ 7.001212] mlx4_core: Initializing 0000:01:00.0
[ 7.011555] mlx4_core 0000:01:00.0: enabling device (0000 -> 0002)
[ 9.758531] mlx4_core 0000:01:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5 GT/s x1 link at 0000:00:00.0 (capable of 32.000 Gb/s with 5 GT/s x8 link)
[ 9.798022] <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v4.0-0
[ 9.806574] <mlx4_ib> mlx4_ib_add: counter index 0 for port 1 allocated 0
[ 9.813390] <mlx4_ib> mlx4_ib_add: counter index 1 for port 2 allocated 0
Trying with the CCI Enablement and adding a regs.init file with:
.set. 0xFF41A040 = 0x3;
Which is the lpd_apu (LPD_SLCR) Register - https://www.xilinx.com/html_docs/registers/ug1087/lpd_slcr___lpd_apu.html
And the regs.init is specified during the boot generation:
$ petalinux-package --boot --fsbl images/linux/zynqmp_fsbl.elf --atf images/linux/bl31.elf --u-boot images/linux/u-boot.elf --fpga images/linux/system.bit --bif-attribute init --bif-attribute-value regs.init --force
The mlx4 driver initialisation still failed:
root@MYD_CZU4EV_RC:~# dmesg|grep mlx4
[ 6.995764] mlx4_core: Mellanox ConnectX core driver v4.0-0
[ 7.001420] mlx4_core: Initializing 0000:01:00.0
[ 7.011758] mlx4_core 0000:01:00.0: enabling device (0000 -> 0002)
[ 8.030948] mlx4_core 0000:01:00.0: Installed FW has unsupported command interface revision 0
[ 8.039477] mlx4_core 0000:01:00.0: (Installed FW version is 0.0.000)
[ 8.045912] mlx4_core 0000:01:00.0: This driver version supports only revisions 2 to 3
[ 8.053819] mlx4_core 0000:01:00.0: QUERY_FW command failed, aborting
[ 8.060249] mlx4_core 0000:01:00.0: Failed to init fw, aborting.
As above, when CCI was enabled for "FPD Masters(PCIe,SATA)" in Vivado the only function in the MYD_CZU4EV_RC_wrapper.xsa appeared to be that the value of PSU__FPDMASTERS_COHERENCY changed from 0 to 1.
Looking for differences that caused in the output from the petalinux build:
-
rootfs.tar.gz has only non-functional changes:
- /etc/shadow has different user IDs
- /etc/timestamp and /etc/version have time of build
- /var/cache/ldconfig/aux-cache has the same size but appears to have been re-ordered. https://patchwork.archlinux.org/patch/1221/ for says "Making archiso deterministic" this file appears to be generated non-deterministically, and it is not neceesary.
-
Enbling CCI added dma-coherent to the system.dtb entry for the pcie@fd0e0000 PCIe root complex "xlnx,nwl-pcie-2.11"
-
The difference in the size of u-boot.bin and loadable section section size of u-boot.elf is 25 bytes, which is the difference in size of system.dtb The only difference in u-boot.bin seems to be the build timestamps and the embedded system.dtb.
-
The size of the XPfw_Main() function in pmufw.elf increased by 20 bytes, based upon changes in symbol addresses, when CCI enabled. pmufw.elf has no debug information and looks like has been automatically inlined by the optimiser which from comparing the diassembled MicroBlaze assembler made it difficult to see the change.
The inserted instructions were:
ffdc9f20: b000ff41 imm -191 ffdc9f24: 3080a040 addik r4, r0, -24512 ffdc9f28: e8640000 lwi r3, r4, 0 ffdc9f2c: a0630002 ori r3, r3, 2 ffdc9f30: f8640000 swi r3, r4, 0Which looks to set bit 1 at address FF41A040 which is the brdc_inner field in the lpd_apu (LPD_SLCR) Register, which is the same change as made in the regs.init file above.
To look at the difference in the generated pmu-firmware source files added the following to project-spec/meta-user/conf/petalinuxbsp.conf to stop the source code being deleted after the build:
RM_WORK_EXCLUDE += "pmu-firmware"
The "CCI enablement in FPD Masters(PCIe,SATA)" had the effect of causing build/tmp/work/zynqmp_generic-xilinx-linux/pmu-firmware/2020.1+gitAUTOINC+6cbb920f4d-r0/git/lib/bsp/standalone/data/standalone.tcl to add the following to build/tmp/work/zynqmp_generic-xilinx-linux/pmu-firmware/2020.1+gitAUTOINC+6cbb920f4d-r0/build/pmu-firmware/zynqmp_pmufw_bsp/psu_pmu_0/include/xparameters.h :
#define XPAR_FPD_IS_CACHE_COHERENT
Where XPAR_FPD_IS_CACHE_COHERENT causes build/tmp/work/zynqmp_generic-xilinx-linux/pmu-firmware/2020.1+gitAUTOINC+6cbb920f4d-r0/build/pmu-firmware/xpfw_user_startup.c to call the following:
/*****************************************************************************
*
* Enable the broadcasting of Inner Shareable transactions for APU.
*
* @param None.
*
* @return None.
*
******************************************************************************/
static void XPfw_Enable_Inner_Shareable_Broadcast(void)
{
u32 val = XPfw_Read32(LPD_SLCR_LPD_APU);
val |= (1U << LPD_SLCR_LPD_APU_BRDC_INNER_SHIFT);
XPfw_Write32(LPD_SLCR_LPD_APU , val);
}
- CCI enabablement in Vivado, regs.init to write to set brdc_inner, pmufw loaded by CSU
$ petalinux-package --boot --fsbl images/linux/zynqmp_fsbl.elf --atf images/linux/bl31.elf --u-boot images/linux/u-boot.elf --fpga images/linux/system.bit --bif-attribute init --bif-attribute-value regs.init --force
$ cat build/bootgen.bif
the_ROM_image:
{
[bootloader, destination_cpu=a53-0] /tmp/tmp.6V5iMUZ64H/zynqmp_fsbl.elf
[pmufw_image] /tmp/tmp.6V5iMUZ64H/pmufw.elf
[destination_device=pl] /tmp/tmp.6V5iMUZ64H/system.bit
[destination_cpu=a53-0, exception_level=el-3, trustzone] /tmp/tmp.6V5iMUZ64H/bl31.elf
[destination_cpu=a53-0, load=0x00100000] /tmp/tmp.6V5iMUZ64H/system.dtb
[destination_cpu=a53-0, exception_level=el-2] /tmp/tmp.6V5iMUZ64H/u-boot.elf
[init] regs.init
}
No initial UART messages from pmufw:
Xilinx Zynq MP First Stage Boot Loader
Release 2020.1 Oct 25 2020 - 19:13:41
NOTICE: ATF running on XCZU4EV/silicon v4/RTL5.1 at 0xfffea000
NOTICE: BL31: v2.2(release):v1.1-5588-g5918e656e
NOTICE: BL31: Built : 19:11:13, Oct 25 2020
And results in driver failing to load:
[ 8.030976] mlx4_core 0000:01:00.0: Installed FW has unsupported command interface revision 0
- CCI disabled in Vivado, no regs.init, pmufw loaded by CSU
$ petalinux-package --boot --fsbl images/linux/zynqmp_fsbl.elf --atf images/linux/bl31.elf --u-boot images/linux/u-boot.elf --fpga images/linux/system.bit --force
$ cat build/bootgen.bif
the_ROM_image:
{
[bootloader, destination_cpu=a53-0] /tmp/tmp.Ow94jodOQ3/zynqmp_fsbl.elf
[pmufw_image] /tmp/tmp.Ow94jodOQ3/pmufw.elf
[destination_device=pl] /tmp/tmp.Ow94jodOQ3/system.bit
[destination_cpu=a53-0, exception_level=el-3, trustzone] /tmp/tmp.Ow94jodOQ3/bl31.elf
[destination_cpu=a53-0, load=0x00100000] /tmp/tmp.Ow94jodOQ3/system.dtb
[destination_cpu=a53-0, exception_level=el-2] /tmp/tmp.Ow94jodOQ3/u-boot.elf
}
Results in driver loading, but ibv_rc_pingpong hanging.
From Linux the brdc_inner reads as clear:
root@MYD_CZU4EV_RC:~# devmem 0xFF41A040 32
0x00000001
- CCI disabled in Vivado, regs.init to write to set brdc_inner, pmufw loaded by CSU
$ petalinux-package --boot --fsbl images/linux/zynqmp_fsbl.elf --atf images/linux/bl31.elf --u-boot images/linux/u-boot.elf --fpga images/linux/system.bit --bif-attribute init --bif-attribute-value regs.init --force
$ cat build/bootgen.bif
Results in driver loading, but ibv_rc_pingpong hanging.
From Linux the brdc_inner reads as set:
root@MYD_CZU4EV_RC:~# devmem 0xFF41A040 32
0x00000003
- CCI disabled in Vivado, no regs.init, pmufw loaded by CPU, manual dma-coherent added to device tree
The following was added to project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi to replicate what happened when CCI was enabled in Vivado:
&pcie {
dma-coherent;
};
$ petalinux-package --boot --fsbl images/linux/zynqmp_fsbl.elf --atf images/linux/bl31.elf --u-boot images/linux/u-boot.elf --fpga images/linux/system.bit --force
$ cat build/bootgen.bif
the_ROM_image:
{
[bootloader, destination_cpu=a53-0] /tmp/tmp.3SVzCD0fNy/zynqmp_fsbl.elf
[pmufw_image] /tmp/tmp.3SVzCD0fNy/pmufw.elf
[destination_device=pl] /tmp/tmp.3SVzCD0fNy/system.bit
[destination_cpu=a53-0, exception_level=el-3, trustzone] /tmp/tmp.3SVzCD0fNy/bl31.elf
[destination_cpu=a53-0, load=0x00100000] /tmp/tmp.3SVzCD0fNy/system.dtb
[destination_cpu=a53-0, exception_level=el-2] /tmp/tmp.3SVzCD0fNy/u-boot.elf
}
This had the same effect of the driver not loading as when CCI was enabled in Vivado:
root@MYD_CZU4EV_RC:~# devmem 0xFF41A040 32
0x00000001
root@MYD_CZU4EV_RC:~# lsmod
Tainted: G
dmaproxy 16384 0 - Live 0xffffffc008c34000 (O)
mlx4_core 307200 0 - Live 0xffffffc008be8000
uio_pdrv_genirq 16384 0 - Live 0xffffffc008be0000
root@MYD_CZU4EV_RC:~# dmesg|grep mlx4
[ 6.986230] mlx4_core: Mellanox ConnectX core driver v4.0-0
[ 6.991884] mlx4_core: Initializing 0000:01:00.0
[ 7.002256] mlx4_core 0000:01:00.0: enabling device (0000 -> 0002)
[ 8.030948] mlx4_core 0000:01:00.0: Installed FW has unsupported command interface revision 0
[ 8.039475] mlx4_core 0000:01:00.0: (Installed FW version is 0.0.000)
[ 8.045908] mlx4_core 0000:01:00.0: This driver version supports only revisions 2 to 3
[ 8.053816] mlx4_core 0000:01:00.0: QUERY_FW command failed, aborting
[ 8.060246] mlx4_core 0000:01:00.0: Failed to init fw, aborting.
I.e. appears to indicate it is the addition of dma-coherent to the device tree which prevents the mlx4 driver from loading.
The mlx4_QUERY_FW() function in components/yocto/workspace/sources/linux-xlnx/drivers/net/ethernet/mellanox/mlx4/fw.c which reports the above error calls mlx4_alloc_cmd_mailbox() to allocate a buffer to receive the firmware version information, where the buffer looks to get eventually allocated by dma_alloc_coherent(). I.e. could explain why adding dma-coherent to the device tree for the the PCIe root complex causes the driver to fail to read the firmware version - if Linux doesn't perform software coherency thinking that hardware coherency is enabled.
Using devmem to readback PCIe controller registers in the case the mlx4 driver loads (i.e. software coherency):
- All TRAN_EGRESS_CONTROL (AXIPCIE_EGRESS) Registers are zero, meaning no egress translation
- All TRAN_INGRESS_CONTROL (AXIPCIE_INGRESS) Registers are zero, meaning no ingress translation
- BRIDGE_CORE_CFG_PCIE_RX1 (AXIPCIE_MAIN) Register is zero The cfg_pcie_rx_arcache and cfg_pcie_rx_awcache fields are zero
According to https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842098/Zynq+UltraScale+MPSoC+Cache+Coherency#ZynqUltraScaleMPSoCCacheCoherency-3.1ARCACHE[3:0]andAWCACHE[3:0] non-zero values are required for the upper two bits of AxCACHE for cache coherency. Where that Wiki page suggests setting the AxCACHE[3:0] to 0xb.
Initially tried adding the following to regs.init used with bootgen to write to the BRIDGE_CORE_CFG_PCIE_RX1 register:
.set. 0xFD0E0004 = 0xbb;
However, no sign of any output from the serial console during boot. Probably the PCIe root port wasn't initialised at power on reset to allow the register to be written.
However, a manual register write from Linux prior to starting ibv_rc_pingpong then allowed the test to work, initially a single exchange using events:
root@MYD_CZU4EV_RC:~# devmem 0xFD0E0004
0x00000000
root@MYD_CZU4EV_RC:~# devmem 0xFD0E0004 32 0xbb
root@MYD_CZU4EV_RC:~# devmem 0xFD0E0004
0x000000BB
root@MYD_CZU4EV_RC:~# ibv_rc_pingpong --ib-port=2 --size=28 -c -n 1 -e
local address: LID 0x0007, QPN 0x000208, PSN 0xc334df, GID ::
remote address: LID 0x0002, QPN 0x000210, PSN 0x6cd1fe, GID ::
56 bytes in 0.00 seconds = 1.89 Mbit/sec
1 iters in 0.00 seconds = 237.00 usec/iter
And also a longer test using polling:
root@MYD_CZU4EV_RC:~# ibv_rc_pingpong --ib-port=2 -c -n 1000000
local address: LID 0x0007, QPN 0x00020c, PSN 0x7f37c4, GID ::
remote address: LID 0x0002, QPN 0x000214, PSN 0xea9b74, GID ::
8192000000 bytes in 27.69 seconds = 2366.55 Mbit/sec
1000000 iters in 27.69 seconds = 27.69 usec/iter
Start with: a. CCI Enablement in Vivado, since adds dma-coherent to the device tree for the PCIe root bridge. b. Use a regs.init with the bootgen to set brdc_inner While the pmufw initialisation will also be setting brdc_inner, it should be safe since:
- ORs in the brdc_inner bit, and since the CSU has already set the bit by the time the pmufw starts there won't be a change in the lpd_apu register value.
- While the description of the lpd_apu register contains "This register may be written to only when APU is in reset", it appears to set the value of the Cortex-A53 BROADCASTINNER pin which https://developer.arm.com/documentation/ddi0500/j/Signal-Descriptions/ACE-and-CHI-interface-signals says: "This pin is sampled only during reset of the Cortex-A53 processor."
While originally thought that would have to modify the PCIe root bridge driver found that could use udev to execute a devmem command to writeto the BRIDGE_CORE_CFG_PCIE_RX1 register to enable cache-coherency prior to loading the mlx4_core module:
root@MYD_CZU4EV_RC:~# cat /etc/udev/rules.d/79-coherent.rules
SUBSYSTEM=="pci", ACTION=="add", ATTRS{vendor}=="0x15b3", ATTRS{device}=="0x673c", RUN+="/sbin/devmem 0xFD0E0004 32 0xbb"
The udev rules file had to be named 79-coherent.rules to have its RUN command called prior to /lib/udev/rules.d/80-drivers.rules which loads the module.
A package was created for openslp, based upon autoconf:
mr_halfword@Haswell-Ubuntu:/opt/pkg/petalinux/2020.1/MYD_CZU4EV_RC$ petalinux-create -t apps --template autoconf --name openslp --enable
/opt/pkg/petalinux/2020.1/MYD_CZU4EV_RC/images/linux/image.ub was created which downloaded the source and was unable to build it.
This leaves the following files, which look to be suitable to cross-compile against openslp:
- build/tmp/sysroots-components/aarch64/openslp/usr/include/slp.h
- build/tmp/sysroots-components/aarch64/openslp/usr/lib/libslp.so
And the slptool can find the
root@MYD_CZU4EV_RC:~# slptool findsrvs service:message_bw
service:message_bw://Haswell-Ubuntu/name=tx_0_0_0,65535
service:message_bw://Haswell-Ubuntu/name=rx_0_0_0,65535
root@MYD_CZU4EV_RC:~# slptool findattrs service:message_bw://Haswell-Ubuntu/name=tx_0_0_0
(size=16779264,rkey=4352,addr=0x7fcb210fd000,lid=2,psn=11496758,qp_num=522,qp_rtr=1)
root@MYD_CZU4EV_RC:~# slptool findattrs service:message_bw://Haswell-Ubuntu/name=rx_0_0_0
(size=16779264,rkey=4608,addr=0x7f292d997000,lid=3,psn=9746292,qp_num=523,qp_rtr=1)
For the above slpd hadn't been started at boot, but slptool still worked.
For ibv_message_bw had to start slpd to prevent slp_reg() from failing.
numactl --show was originally reporting "No NUMA support available on this system", meaning numa_available() was returning -1 which would require a code change to run ibv_message_bw.
Modified the Kernel configuration to add NUMA support, and now appears a single NUMA node.
To link ibmad needs infiniband-diags, which has a dependency on opensm.
Compilation of opensm fails, on ./../opensm-3.3.23/libvendor/osm_mad_pool.c see /opt/pkg/petalinux/2020.1/MYD_CZU4EV_RC/build/tmp/work/aarch64-xilinx-linux/opensm/23.2.23-r0/temp/log.do_compile.16212
Looks like the system /usr/include/inttypes.h gets include.
When building libvendor what causes -I/usr/include to be on the command line?
Result of running ibv_message_bw at the same time for transmit and receive
root@MYD_CZU4EV_RC:~# /media/sd-mmcblk1p2/root/ibv_message_bw --ib-dev=mlx4_0 --ib-port=1 --thread=rx:0 --verify-data
Rx_0 connected
Rx_0 received 3601858560 data bytes in 3435 messages over last 10.0 seconds
Rx_0 received 4172283904 data bytes in 3979 messages over last 10.0 seconds
Rx_0 received 4172283904 data bytes in 3979 messages over last 10.0 seconds
Rx_0 received 4161798144 data bytes in 3969 messages over last 10.0 seconds
Rx_0 received 3973054464 data bytes in 3789 messages over last 10.0 seconds
Rx_0 received 3974103040 data bytes in 3790 messages over last 10.0 seconds
Rx_0 received 3973054464 data bytes in 3789 messages over last 10.0 seconds
Rx_0 received 3974103040 data bytes in 3790 messages over last 10.0 seconds
Rx_0 received 3973054464 data bytes in 3789 messages over last 10.0 seconds
Rx_0 received 3974103040 data bytes in 3790 messages over last 10.0 seconds
Rx_0 received 3973054464 data bytes in 3789 messages over last 10.0 seconds
Rx_0 received 3974103040 data bytes in 3790 messages over last 10.0 seconds
Rx_0 received 3973054464 data bytes in 3789 messages over last 10.0 seconds
Rx_0 received 3974103040 data bytes in 3790 messages over last 10.0 seconds
Rx_0 received 3973054464 data bytes in 3789 messages over last 10.0 seconds
Rx_0 received 3974103040 data bytes in 3790 messages over last 10.0 seconds
Rx_0 received 3973054464 data bytes in 3789 messages over last 10.0 seconds
Rx_0 received 3974103040 data bytes in 3790 messages over last 10.0 seconds
Rx_0 received 3973054464 data bytes in 3789 messages over last 10.0 seconds
Rx_0 received 3974103040 data bytes in 3790 messages over last 10.0 seconds
Rx_0 received 3973054464 data bytes in 3789 messages over last 10.0 seconds
Rx_0 received 3974103040 data bytes in 3790 messages over last 10.0 seconds
Rx_0 received 3973054464 data bytes in 3789 messages over last 10.0 seconds
Rx_0 received 3974103040 data bytes in 3790 messages over last 10.0 seconds
Rx_0 received 3973054464 data bytes in 3789 messages over last 10.0 seconds
Rx_0 received 3974103040 data bytes in 3790 messages over last 10.0 seconds
Rx_0 received 3973054464 data bytes in 3789 messages over last 10.0 seconds
Rx_0 received 3973054464 data bytes in 3789 messages over last 10.0 seconds
Rx_0 received 3974103040 data bytes in 3790 messages over last 10.0 seconds
Rx_0 received 3973054464 data bytes in 3789 messages over last 10.0 seconds
Rx_0 received 3974103040 data bytes in 3790 messages over last 10.0 seconds
Rx_0 received 3973054464 data bytes in 3789 messages over last 10.0 seconds
Rx_0 received 3974103040 data bytes in 3790 messages over last 10.0 seconds
Rx_0 received 3973054464 data bytes in 3789 messages over last 10.0 seconds
Rx_0 received 3974103040 data bytes in 3790 messages over last 10.0 seconds
Rx_0 received 3973054464 data bytes in 3789 messages over last 10.0 seconds
Rx_0 received 3974103040 data bytes in 3790 messages over last 10.0 seconds
Rx_0 received 3973054464 data bytes in 3789 messages over last 10.0 seconds
Rx_0 received 3974103040 data bytes in 3790 messages over last 10.0 seconds
Rx_0 received 3973054464 data bytes in 3789 messages over last 10.0 seconds
Rx_0 received 3974103040 data bytes in 3790 messages over last 10.0 seconds
Rx_0 received 3973054464 data bytes in 3789 messages over last 10.0 seconds
Rx_0 received 3974103040 data bytes in 3790 messages over last 10.0 seconds
Rx_0 received 3973054464 data bytes in 3789 messages over last 10.0 seconds
Rx_0 received 3974103040 data bytes in 3790 messages over last 10.0 seconds
Rx_0 received 3973054464 data bytes in 3789 messages over last 10.0 seconds
Rx_0 received 3974103040 data bytes in 3790 messages over last 10.0 seconds
Rx_0 received 3973054464 data bytes in 3789 messages over last 10.0 seconds
Rx_0 received 3974103040 data bytes in 3790 messages over last 10.0 seconds
Rx_0 received 3973054464 data bytes in 3789 messages over last 10.0 seconds
Rx_0 received 3974103040 data bytes in 3790 messages over last 10.0 seconds
Rx_0 received 3984588800 data bytes in 3800 messages over last 10.0 seconds
Rx_0 Total data bytes 206968979456 over 518.915876 seconds; 398.8 Mbytes/second
Rx_0 Total messages 197381 over 518.915876 seconds; 380 messages/second
Rx_0 Min message size=1048576 max message size=1048576 data verification=yes
Rx_0 minor page faults=1 (4196 -> 4197)
Rx_0 major page faults=0 (0 -> 0)
Rx_0 voluntary context switches=0 (27 -> 27)
Rx_0 involuntary context switches=8170 (0 -> 8170)
Rx_0 user time=518.820967 system time=0.002649
root@MYD_CZU4EV_RC:~# /media/sd-mmcblk1p2/root/ibv_message_bw --ib-dev=mlx4_0 --ib-port=2 --thread=tx:1 --verify-data
Press Ctrl-C to tell the 1 transmit thread(s) to stop the test
Tx_1 connected
Tx_1 transmitted 2497708032 data bytes in 2382 messages over last 10.0 seconds
Tx_1 transmitted 3176136704 data bytes in 3029 messages over last 10.0 seconds
Tx_1 transmitted 3177185280 data bytes in 3030 messages over last 10.0 seconds
Tx_1 transmitted 3176136704 data bytes in 3029 messages over last 10.0 seconds
Tx_1 transmitted 3176136704 data bytes in 3029 messages over last 10.0 seconds
Tx_1 transmitted 3177185280 data bytes in 3030 messages over last 10.0 seconds
Tx_1 transmitted 3176136704 data bytes in 3029 messages over last 10.0 seconds
Tx_1 transmitted 3176136704 data bytes in 3029 messages over last 10.0 seconds
Tx_1 transmitted 3176136704 data bytes in 3029 messages over last 10.0 seconds
Tx_1 transmitted 3177185280 data bytes in 3030 messages over last 10.0 seconds
Tx_1 transmitted 3176136704 data bytes in 3029 messages over last 10.0 seconds
Tx_1 transmitted 3176136704 data bytes in 3030 messages over last 10.0 seconds
Tx_1 transmitted 3177185280 data bytes in 3029 messages over last 10.0 seconds
Tx_1 transmitted 3176136704 data bytes in 3029 messages over last 10.0 seconds
Tx_1 transmitted 3176136704 data bytes in 3029 messages over last 10.0 seconds
Tx_1 transmitted 3176136704 data bytes in 3029 messages over last 10.0 seconds
Tx_1 transmitted 3177185280 data bytes in 3030 messages over last 10.0 seconds
Tx_1 transmitted 3176136704 data bytes in 3029 messages over last 10.0 seconds
Tx_1 transmitted 3177185280 data bytes in 3030 messages over last 10.0 seconds
Tx_1 transmitted 3176136704 data bytes in 3029 messages over last 10.0 seconds
Tx_1 transmitted 3176136704 data bytes in 3029 messages over last 10.0 seconds
Tx_1 transmitted 3176136704 data bytes in 3029 messages over last 10.0 seconds
Tx_1 transmitted 3176136704 data bytes in 3029 messages over last 10.0 seconds
Tx_1 transmitted 3177185280 data bytes in 3030 messages over last 10.0 seconds
Tx_1 transmitted 3176136704 data bytes in 3029 messages over last 10.0 seconds
Tx_1 transmitted 3176136704 data bytes in 3030 messages over last 10.0 seconds
Tx_1 transmitted 3177185280 data bytes in 3029 messages over last 10.0 seconds
Tx_1 transmitted 3176136704 data bytes in 3029 messages over last 10.0 seconds
Tx_1 transmitted 3176136704 data bytes in 3029 messages over last 10.0 seconds
Tx_1 transmitted 3177185280 data bytes in 3030 messages over last 10.0 seconds
Tx_1 transmitted 3176136704 data bytes in 3029 messages over last 10.0 seconds
Tx_1 transmitted 3176136704 data bytes in 3029 messages over last 10.0 seconds
Tx_1 transmitted 3177185280 data bytes in 3030 messages over last 10.0 seconds
Tx_1 transmitted 3176136704 data bytes in 3029 messages over last 10.0 seconds
Tx_1 transmitted 3176136704 data bytes in 3029 messages over last 10.0 seconds
Tx_1 transmitted 3177185280 data bytes in 3030 messages over last 10.0 seconds
Tx_1 transmitted 3176136704 data bytes in 3029 messages over last 10.0 seconds
Tx_1 transmitted 3176136704 data bytes in 3030 messages over last 10.0 seconds
Tx_1 transmitted 3177185280 data bytes in 3029 messages over last 10.0 seconds
Tx_1 transmitted 3176136704 data bytes in 3029 messages over last 10.0 seconds
Tx_1 transmitted 3176136704 data bytes in 3029 messages over last 10.0 seconds
Tx_1 transmitted 3177185280 data bytes in 3030 messages over last 10.0 seconds
Tx_1 transmitted 3176136704 data bytes in 3029 messages over last 10.0 seconds
Tx_1 transmitted 3177185280 data bytes in 3030 messages over last 10.0 seconds
Tx_1 transmitted 3176136704 data bytes in 3029 messages over last 10.0 seconds
Tx_1 transmitted 3176136704 data bytes in 3029 messages over last 10.0 seconds
Tx_1 transmitted 3177185280 data bytes in 3030 messages over last 10.0 seconds
Tx_1 transmitted 3176136704 data bytes in 3029 messages over last 10.0 seconds
^C
Tx_1 Total data bytes 152460853248 over 479.971425 seconds; 317.6 Mbytes/second
Tx_1 Total messages 145398 over 479.971425 seconds; 303 messages/second
Tx_1 Min message size=1048576 max message size=1048576 data verification=yes
Tx_1 minor page faults=2 (4197 -> 4199)
Tx_1 major page faults=0 (0 -> 0)
Tx_1 voluntary context switches=0 (44 -> 44)
Tx_1 involuntary context switches=10782 (1 -> 10783)
Tx_1 user time=479.717108 system time=0.027566