Jay Taylor's notes
back to listing indexRamtin Amin Web Blog
[web search]-
FemtoCell
-
Nand Dumping
-
SIM Card
-
Misc
-
Telco protocol
Demystifying the i-Device NVMe NAND
(New storage used by Apple)
Introduction
NVMexpress is the next generation storage technology used in embedded devices. The main idea was to have a memory controller directly on the PCIexpress level, avoiding a lot of overhead. Therefor, the technology is based on PCIe. We could already see on recent Mac Book Air, the details about it in the control pannel:
More recently, Macrumers was showing the characteristic on this screen shot:
So this was in fact very interesting to go look at it, and figure it out.
Removing the Nand
There are several videos out there, that shows how the nand is removed or replaced. I couldn't make a better quality video as the one we find on youtube, so I share one here
Electrical interface
First, after digging a bit here and there, I manage to find the pinout of their chip.
We can see that it requires 3 main voltages: 0.9v, 1.8v, and 3.0v. We have a 2x PCIe Interface with the TX[0..1]+/- and RX[0..1]+/- as well as the CLK+/- pins. Note that those are differential pairs. I assumed that they were routed with a 90R impedance. On top of that, we could find some PCIe control pins such as the CLKREQ, NRST. And some other config pins such as a low power mode, or a firmware strap and SWD jtag. Another interesting part is that the Clkc frequency is directly controlled by the SoC PLL itself. In order to get the chip working elsewhere, one would need to know the frequency. Things easily doable with a scope:
So 24Mhz 1.8v is needed.
My Apple Iphone NVMe Reader
In order to read the NVMe, I therefor developped a PCIe card with a Zero Insertion Force reader. I brought the JTAG part to 20pin header.
The hard pard in here is the signal integrity of the differential pairs. In order to do so, I had to use multi layer PCB, and have the impedence match by knowing the stackup, materials used for prepeg and so on..
As I am not planning to sell those device myself, I want to thank Itead Studio that agreed to do a small set of production. The bards are currently in pre-order. Notice that the sockets that are made for this project are not generic but custom made for those LGA70, which explains this price. https://www.itead.cc/nvme-reader.html
NVMe Enumeration
# lspci -d 106b:2002 -vvv
09:00.0 Mass storage controller: Apple Inc. Device 2002 (rev 11) (prog-if 02)
Subsystem: Apple Inc. Device 2002
Physical Slot: 1-2
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 19
Region 0: Memory at a0800000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <4us, L1 <32us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x2, ASPM L0s L1, Exit Latency L0s <1us, L1 <8us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x2, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
Capabilities: [148 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [168 v1] #19
Capabilities: [178 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [180 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=25us PortTPowerOnTime=10us
So as we can see, the device is using Apple's VID: 0x106b and the PID is 0x2002 . Remember, the version in mac book air are 0x2001 ! When using this card in a Mac book that has the NVMe driver, I couldnt get it to work tho. It looks like the driver does not work on this.
Driver Reverse Engineering
In order to get it to work, I had to manage to get the driver working. NVMe is implemented in Linux and Matthew Wilcox (which gave me great support) has done an amaizing job writing the NVMe driver for linux. Some other have done the quirks in order to get the driver work for Mac book Air. (reducing the amount of queues to 2). So After having added the PID to the driver and recompiled, I saw that it was still not wokring. Had to go the hard way then. It looks like to reduce the size needed, the NVMe core uses the host DDR in order to work. Therefor, apple is not strictly following the specification regarding the initialisation. A scratch region is reserved from the beginning, and it's address and lenght are in the device tree. In order to set those value, the NVMe has some special register starting from address BAR0 + 0x1800. For example, registers at offset 0x1910 0x1914 0x1918 0x191c seams to contain informations about Error codes when the NVMe is setting up it's error bit in its STATUS (CSTS) register. Some other interesting registers when in BFH mode are located around 0x1b00. We will get back on those later.
00000000 6a 90 88 cf 8a fd 63 0a e3 51 e2 48 87 e0 b9 8b |j.....c..Q.H....| 00000010 36 fe 07 7b d4 16 44 bb b8 27 9c 89 98 3b aa fc |6..{..D..'...;..| 00000020 00 a0 ac b9 03 00 00 00 03 00 00 00 de 23 55 c1 |.............#U.| 00000030 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00000200 48 46 53 00 00 00 11 aa aa 11 00 30 65 43 ec ac |HFS........0eC..| 00000210 32 fc 46 79 50 c3 47 42 9b 60 df 45 1d fc 70 5f |2.FyP.GB.`.E..p_| 00000220 00 60 00 00 00 00 00 00 00 60 24 be 00 00 00 00 |.`.......`$.....| 00000230 00 00 00 00 00 00 00 00 53 00 79 00 73 00 74 00 |........S.y.s.t.| 00000240 65 00 6d 00 00 00 00 00 00 00 00 00 00 00 00 00 |e.m.............| 00000250 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00000280 48 46 53 00 00 00 11 aa aa 11 00 30 65 43 ec ac |HFS........0eC..| 00000290 0f 66 b8 0b 6b e7 43 c4 ab d7 dc 45 18 35 1b 4c |.f..k.C....E.5.L| 000002a0 00 60 24 be 00 00 00 00 00 10 61 b0 03 00 00 00 |.`$.......a.....| 000002b0 00 00 00 00 00 00 03 00 44 00 61 00 74 00 61 00 |........D.a.t.a.| 000002c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00000300 48 46 53 00 00 00 11 aa aa 11 00 30 65 43 ec ac |HFS........0eC..| 00000310 8d ba 7b fb ad 12 48 20 8e c2 80 1a 36 cb 45 d5 |..{...H ....6.E.| 00000320 00 10 61 b0 03 00 00 00 00 60 00 b1 03 00 00 00 |..a......`......| 00000330 00 00 00 00 00 00 00 00 42 00 61 00 73 00 65 00 |........B.a.s.e.| 00000340 62 00 61 00 6e 00 64 00 20 00 44 00 61 00 74 00 |b.a.n.d. .D.a.t.| 00000350 61 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |a...............| 00000360 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00000800 00 f0 00 00 01 00 02 00 03 00 04 00 05 00 06 00 |................| 00000810 07 00 08 00 09 00 0a 00 0b 00 0c 00 0d 00 0e 00 |................| 00000820 0f 00 10 00 11 00 12 00 13 00 14 00 15 00 16 00 |................| --More--
Powered by asciinema
# nvme list
Node SN Model Version NS Usage Format FW Rev
---------------- -------------------- -------------------- -------- -- -------------------------- ---------------- ---------
/dev/nvme0n1 01061019E5C211EB0 APPLE SSD AP0016K 1.0 1 16,00 GB / 16,00 GB 4 KiB + 0 B 11.90.01
/dev/nvme0n2 01061019E5C211EB0 APPLE SSD AP0016K 1.0 2 8,39 MB / 8,39 MB 4 KiB + 0 B 11.90.01
/dev/nvme0n3 01061019E5C211EB0 APPLE SSD AP0016K 1.0 3 131,07 kB / 131,07 kB 4 KiB + 0 B 11.90.01
/dev/nvme0n4 01061019E5C211EB0 APPLE SSD AP0016K 1.0 4 8,19 kB / 8,19 kB 4 KiB + 0 B 11.90.01
/dev/nvme0n5 01061019E5C211EB0 APPLE SSD AP0016K 1.0 5 8,19 kB / 8,19 kB 4 KiB + 0 B 11.90.01
/dev/nvme0n6 01061019E5C211EB0 APPLE SSD AP0016K 1.0 6 4,10 kB / 4,10 kB 4 KiB + 0 B 11.90.01
/dev/nvme0n7 01061019E5C211EB0 APPLE SSD AP0016K 1.0 7 1,05 MB / 1,05 MB 4 KiB + 0 B 11.90.01
AS one could see, the nand contains 7 partitions. They are usually the first one is a HFS with the system on it, the second one is the llb + iboot (of course GID crypted). the 3rd one is the syscfg... and so on.
Anatomy of the Names Spaces.
NS1
NS1 is what used to by ‘fsys’, and contains an LwVM formatted partition, which contains three volumes - System (/), Data (/private/var) and the baseband data. These are all visible to iOS as /dev/disk0s1s1..s3 respectively
00000000 6a 90 88 cf 8a fd 63 0a e3 51 e2 48 87 e0 b9 8b |j.....c..Q.H....|
00000010 a3 a2 64 6b a1 b1 41 9b 89 49 95 75 61 74 53 31 |..dk..A..I.uatS1|
00000020 00 80 b2 e6 0e 00 00 00 03 00 00 00 1d 4f 5c b9 |.............O\.|
00000030 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000200 48 46 53 00 00 00 11 aa aa 11 00 30 65 43 ec ac |HFS........0eC..|
00000210 a9 49 c9 cf 91 41 43 bc bc c3 2a 80 c9 71 e1 79 |.I...AC...*..q.y|
00000220 00 60 00 00 00 00 00 00 00 f0 16 da 00 00 00 00 |.`..............|
00000230 00 00 00 00 00 00 00 00 53 00 79 00 73 00 74 00 |........S.y.s.t.|
00000240 65 00 6d 00 00 00 00 00 00 00 00 00 00 00 00 00 |e.m.............|
00000250 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000280 48 46 53 00 00 00 11 aa aa 11 00 30 65 43 ec ac |HFS........0eC..|
00000290 6b a9 57 fb 7e 21 46 78 96 16 b5 15 4f df 4d 3e |k.W.~!Fx....O.M>|
000002a0 00 f0 16 da 00 00 00 00 00 10 61 c3 0e 00 00 00 |..........a.....|
000002b0 00 00 00 00 00 00 03 00 44 00 61 00 74 00 61 00 |........D.a.t.a.|
000002c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000300 48 46 53 00 00 00 11 aa aa 11 00 30 65 43 ec ac |HFS........0eC..|
00000310 7a 69 fe f8 7f 82 45 23 b0 75 e8 1a 43 36 90 b2 |zi....E#.u..C6..|
00000320 00 10 61 c3 0e 00 00 00 00 60 00 c4 0e 00 00 00 |..a......`......|
00000330 00 00 00 00 00 00 00 00 42 00 61 00 73 00 65 00 |........B.a.s.e.|
00000340 62 00 61 00 6e 00 64 00 20 00 44 00 61 00 74 00 |b.a.n.d. .D.a.t.|
00000350 61 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |a...............|
00000360 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
NS2
The second NameSpace contains the illb and iBoot. From what we can see in this hex dump. It is encrypted with the GID key
00000000 30 83 03 96 1e 16 04 49 4d 47 34 30 83 03 80 99 |0......IMG40....|
00000010 16 04 49 4d 34 50 16 04 69 6c 6c 62 16 10 69 42 |..IM4P..illb..iB|
00000020 6f 6f 74 2d 32 38 31 37 2e 32 30 2e 32 36 04 83 |oot-2817.20.26..|
00000030 03 80 00 9f 1c 04 9b 76 e3 d6 d3 70 15 99 89 b7 |.......v...p....|
00000040 8a 73 4e dd f8 00 d8 e7 5c 5a b4 55 85 2a 03 e1 |.sN.....\Z.U.*..|
00000050 6b 4a 38 0a 59 6e 28 9d f1 be c5 41 1b 17 bd dd |kJ8.Yn(....A....|
00000060 89 0c 69 63 54 ed bd 72 a1 3d 4f 97 04 de 28 fc |..icT..r.=O...(.|
00000070 94 58 29 b4 45 67 23 ae cc d7 be 59 bc 07 3f fc |.X).Eg#....Y..?.|
00000080 ba 55 03 71 a4 a2 86 dd 23 8f 5a ce 67 18 6e 1e |.U.q....#.Z.g.n.|
00000090 8a 6a ec 83 f8 d6 8c 32 ca 24 36 50 77 76 1e bd |.j.....2.$6Pwv..|
000000a0 4f a9 ac d3 5e 15 70 4e 32 58 a8 34 16 22 0c d3 |O...^.pN2X.4."..|
...
NS3
The third NS is the SysCFG partition. The SysCFG is visible in the IORegistry through the AppleDiagnosticDataSysCfg property of the AppleDiagnosticDataAccessReadOnly IOregistry node. (Provided by com.apple.driver.AppleDiagnosticDataAccessReadOnly kext). (as discussed in Jonathan Levin’s *OS Internals Volumes III and II)
00000000 67 66 43 53 04 05 00 00 00 00 02 00 02 00 02 00 |gfCS............|
00000010 00 00 00 00 3f 00 00 00 42 54 4e 43 23 42 4c 4d |....?...BTNC#BLM|
00000020 11 00 00 00 ec ff 01 00 ff ff ff ff 23 47 46 43 |............#GFC|
00000030 2f 2f 2f 2f 2a 2f 2a 00 00 00 00 00 00 00 00 00 |////*/*.........|
00000040 63 61 4d 57 38 71 de 30 82 ff 00 00 00 00 00 00 |caMW8q.0........|
00000050 00 00 00 00 63 61 4d 42 38 71 de 30 83 00 00 00 |....caMB8q.0....|
00000060 00 00 00 00 00 00 00 00 42 54 4e 43 56 43 41 50 |........BTNCVCAP|
00000070 29 00 00 00 c0 ff 01 00 ff ff ff ff 42 54 4e 43 |)...........BTNC|
00000080 54 41 42 43 31 00 00 00 8c ff 01 00 ff ff ff ff |TABC1...........|
00000090 54 53 42 56 09 00 00 00 00 00 00 00 00 00 00 00 |TSBV............|
000000a0 00 00 00 00 52 42 50 56 0b 00 00 00 00 00 00 00 |....RBPV........|
000000b0 00 00 00 00 00 00 00 00 42 54 4e 43 41 43 42 56 |........BTNCACBV|
000000c0 14 00 00 00 78 ff 01 00 ff ff ff ff 42 54 4e 43 |....x.......BTNC|
000000d0 4c 41 43 57 ac 02 00 00 cc fc 01 00 ff ff ff ff |LACW............|
NS4
The forth NameSpace contains the wifi infos. In fact it allowes the phone, even while not unlocked, to get access to the password to activate the networks
00000000 69 50 43 42 00 01 00 00 01 00 00 00 80 00 00 00 |iPCB............|
00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000002e0 fc ff ef ff af ec 22 08 57 69 50 41 53 58 4e 65 |......".WiPASXNe|
000002f0 78 74 20 31 38 41 41 64 4d 32 00 00 00 00 00 00 |xt 18AAdM2......|
00000300 fc ff ef ff 97 76 22 08 57 69 50 41 53 20 31 43 |.....v".WiPAS 1C|
00000310 4d 50 32 37 00 00 00 00 00 00 00 00 00 00 00 00 |MP27............|
00000320 fc ff ef ff 79 79 22 08 57 69 50 41 53 20 31 43 |....yy".WiPAS 1C|
00000330 4d 50 32 37 00 00 00 00 00 00 00 00 00 00 00 00 |MP27............|
NS5
The fifth partition is the NVRAM which is accessible as logical keys/values through a corresponding Kext
00000000 5a 82 02 00 6e 76 72 61 6d 00 00 00 00 00 00 00 |Z...nvram.......|
00000010 fd 20 e6 d3 58 01 00 00 00 00 00 00 00 00 00 00 |. ..X...........|
00000020 70 7c 80 00 63 6f 6d 6d 6f 6e 00 00 00 00 00 00 |p|..common......|
00000030 61 75 74 6f 2d 62 6f 6f 74 3d 74 72 75 65 00 62 |auto-boot=true.b|
00000040 61 63 6b 6c 69 67 68 74 2d 6c 65 76 65 6c 3d 31 |acklight-level=1|
00000050 34 39 37 00 62 6f 6f 74 2d 61 72 67 73 3d 00 00 |497.boot-args=..|
00000060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000820 a1 15 81 00 41 50 4c 2c 4f 53 58 50 61 6e 69 63 |....APL,OSXPanic|
00000830 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00001030 7f 17 fd 00 77 77 77 77 77 77 77 77 77 77 77 77 |....wwwwwwwwwwww|
00001040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00002000
NS6
The sixth NameSpace is the PLOG. Where the kbag info are stored. That is used when you unlock the phone, with the crypto engine. Also, 10 wrong passcode and this get erased.
00000000 4e 0b 2a 02 c5 30 09 f8 ae 0e ca 8d 8f c1 88 8c |N.*..0..........|
00000010 2b 68 4b 44 c4 30 09 f8 af 0e ca 8d 8f c1 88 8c |+hKD.0..........|
00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000030 00 00 00 00 00 00 00 00 1b 00 00 00 0c 97 b4 a8 |................|
00000040 6b 4c 50 00 4d 56 77 4c 2b a0 ab f1 3e 62 df a6 |kLP.MVwL+...>b..|
00000050 66 93 52 4e 2a da 12 f9 17 ba 37 ea 08 f2 cd 62 |f.RN*.....7....b|
00000060 d8 02 71 b3 a7 a9 38 b8 a1 db 0e 47 29 50 e9 41 |..q...8....G)P.A|
00000070 eb 57 88 e9 76 7a f7 c3 8f 77 bd c8 08 0f 83 3c |.W..vz...w.....<|
00000080 b7 96 20 46 71 a7 ad 59 e1 e3 b4 cc 1b cf 83 83 |.. Fq..Y........|
00000090 5e 60 c8 be f3 20 8c f5 6b 4c 34 00 31 47 41 42 |^`... ..kL4.1GAB|
000000a0 31 47 41 42 18 1b 7c 83 11 63 64 95 96 0d 0e fc |1GAB..|..cd.....|
000000b0 09 30 c0 3e c5 f6 48 70 a8 f4 86 f8 52 77 49 20 |.0.>..Hp....RwI |
000000c0 57 ad 7e 21 23 38 b0 dd 4a aa 21 05 6e 20 0a 8e |W.~!#8..J.!.n ..|
000000d0 76 24 55 fc 6b 4c 28 00 79 65 6b c4 6a dc ee 85 |v$U.kL(.yek.j...|
000000e0 f3 1e d2 0f 66 f8 25 e1 97 bd ef 61 7e 9a 7b 18 |....f.%....a~.{.|
000000f0 d3 60 89 d8 57 43 3c d8 44 30 e3 1a e6 eb 0c 3e |.`..WC<.D0.....>|
00000100 9b 65 f4 69 6b 4c 00 00 45 4e 4f 44 00 00 00 00 |.e.ikL..ENOD....|
00000110 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
....
NS7
The seventh partition looks empty so far
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00100000
Navigating into the nand
Jonathan Levin’s HFSleuth can read raw device files and inspect HFS+ filesystems in them. http://NewOSXBook.com/tools/hfsleuth.html (thank to @Morpheus_____)
Reusing the NAND
The nand could be re-used in an iphone if you have a proper JIG which uses pogo pins in order to contact the pads. It's quite handy when wanting for example to modify the NAND from outside and then get it back to work.
JTAG / SWD
As mentioned before, the NVMe chip has a JTAG/SWD interface. It is interesting to stick an OpenOCD on it to see a bit what's going on.
Using a KT-Link interface with OpenOCD, we could have a TAP enumeration.
Open On-Chip Debugger 0.10.0-dev-00212-g50d4f76 (2016-02-15-18:11)
Licensed under GNU GPL v2
For bug reports, read
http://openocd.org/doc/doxygen/bugs.html
debug_level: 2
Info : FTDI SWD mode enabled
adapter speed: 10000 kHz
Info : clock speed 10000 kHz
Info : SWD IDCODE 0x5ba02477
Info : SWD IDCODE 0x5ba02477
Info : SWD IDCODE 0x5ba02477
Info : SWD IDCODE 0x5ba02477
Info : mynvme.cpu: hardware has 3 breakpoints, 2 watchpoints
Info : mynvme.cpu rev 1, partnum c05, arch f, variant 0, implementor 41
Info : mynvme.cpu cluster 0 core 0 multi core
After looking a bit a round, it looked first that 0x5ba02477 was a Cortex M, but in fact it is a cortex A.
> dap info
AP ID register 0x44770002
Type is MEM-AP APB
MEM-AP BASE 0x80000000
ROM table in legacy format
MEMTYPE system memory not present: dedicated debug bus
ROMTABLE[0x0] = 0x40000002
Component not present
ROMTABLE[0x4] = 0x74300003
Component base address 0xf4300000, start address 0xf4300000
Component class is 0x1, ROM table
Peripheral ID[4..0] = hex 00 00 0c 85 33
Unrecognized (Part 0x533, designer 0xc8)
[L01] ROM table in legacy format
[L01] MEMTYPE system memory not present: dedicated debug bus
[L01] ROMTABLE[0x0] = 0x1003
Component base address 0xf4301000, start address 0xf4301000
Component class is 0xf, PrimeCell or System component
Peripheral ID[4..0] = hex 00 00 4c 80 00
Unrecognized (Part 0x0, designer 0xc8)
[L01] ROMTABLE[0x4] = 0x10003
Component base address 0xf4310000, start address 0xf4310000
Component class is 0x9, CoreSight component
Type is 0x15, Debug Logic, Processor
Peripheral ID[4..0] = hex 04 00 1b bc 05
Part is Cortex-A5 Debug (Debug Unit)
[L01] ROMTABLE[0x8] = 0x11002
Component not present
[L01] ROMTABLE[0xc] = 0x12003
Component base address 0xf4312000, start address 0xf4312000
Component class is 0x9, CoreSight component
Type is 0x13, Trace Source, Processor
Peripheral ID[4..0] = hex 04 00 2b b9 55
Part is Cortex-A5 ETM (Embedded Trace)
[L01] ROMTABLE[0x10] = 0x13003
Component base address 0xf4313000, start address 0xf4313000
Component class is 0x9, CoreSight component
Type is 0x21, Trace Sink, Buffer
Peripheral ID[4..0] = hex 04 00 3b b9 07
Part is CoreSight ETB (Trace Buffer)
[L01] ROMTABLE[0x14] = 0x14003
Component base address 0xf4314000, start address 0xf4314000
Component class is 0x9, CoreSight component
Type is 0x14, Debug Control, Trigger Matrix
Peripheral ID[4..0] = hex 04 00 4b b9 06
Part is CoreSight CTI (Cross Trigger)
[L01] ROMTABLE[0x18] = 0x15003
Component base address 0xf4315000, start address 0xf4315000
Component class is 0x9, CoreSight component
Type is 0x14, Debug Control, Trigger Matrix
Peripheral ID[4..0] = hex 04 00 4b b9 06
Part is CoreSight CTI (Cross Trigger)
[L01] ROMTABLE[0x1c] = 0x20003
Component base address 0xf4320000, start address 0xf4320000
Component class is 0x9, CoreSight component
Type is 0x15, Debug Logic, Processor
Peripheral ID[4..0] = hex 04 00 4b bc 15
Part is Cortex-R5 Debug (Debug Unit)
[L01] ROMTABLE[0x20] = 0x21002
Component not present
[L01] ROMTABLE[0x24] = 0x22003
Component base address 0xf4322000, start address 0xf4322000
Component class is 0x9, CoreSight component
Type is 0x13, Trace Source, Processor
Peripheral ID[4..0] = hex 04 00 0b b9 31
Part is Cortex-R5 ETM (Embedded Trace)
[L01] ROMTABLE[0x28] = 0x23003
Component base address 0xf4323000, start address 0xf4323000
Component class is 0x9, CoreSight component
Type is 0x21, Trace Sink, Buffer
Peripheral ID[4..0] = hex 04 00 3b b9 07
Part is CoreSight ETB (Trace Buffer)
[L01] ROMTABLE[0x2c] = 0x24003
Component base address 0xf4324000, start address 0xf4324000
Component class is 0x9, CoreSight component
Type is 0x14, Debug Control, Trigger Matrix
Peripheral ID[4..0] = hex 04 00 4b b9 06
Part is CoreSight CTI (Cross Trigger)
[L01] ROMTABLE[0x30] = 0x25003
Component base address 0xf4325000, start address 0xf4325000
Component class is 0x9, CoreSight component
Type is 0x14, Debug Control, Trigger Matrix
Peripheral ID[4..0] = hex 04 00 4b b9 06
Part is CoreSight CTI (Cross Trigger)
[L01] ROMTABLE[0x34] = 0x26003
Component base address 0xf4326000, start address 0xf4326000
Component class is 0x9, CoreSight component
Type is 0x12, Trace Link, Funnel, router
Peripheral ID[4..0] = hex 04 00 2b b9 08
Part is CoreSight CSTF (Trace Funnel)
[L01] ROMTABLE[0x38] = 0xffe00002
Component not present
[L01] ROMTABLE[0x3c] = 0x0
[L01] End of ROM table
ROMTABLE[0x8] = 0x0
End of ROM table
It is interesting at this stage to dump their bootrom in order to find the way the chip works.
(15) pc (/32): 0x74002788 (16) r8_fiq (/32) (17) r9_fiq (/32) (18) r10_fiq (/32) (19) r11_fiq (/32) (20) r12_fiq (/32) (21) sp_fiq (/32) (22) lr_fiq (/32) (23) sp_irq (/32) (24) lr_irq (/32) (25) sp_svc (/32): 0x00008FE0 (26) lr_svc (/32): 0x76A82074 (27) sp_abt (/32) (28) lr_abt (/32) (29) sp_und (/32) (30) lr_und (/32) (31) cpsr (/32): 0x60000133 (32) spsr_fiq (/32) (33) spsr_irq (/32) (34) spsr_svc (/32) (35) spsr_abt (/32) (36) spsr_und (/32) (37) sp (/32) (38) lr (/32) (39) sp_mon (/32) (40) lr_mon (/32) (41) spsr_mon (/32) ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────(16) r8_fiq (/32) (17) r9_fiq (/32) (18) r10_fiq (/32) (19) r11_fiq (/32) (20) r12_fiq (/32) (21) sp_fiq (/32) (22) lr_fiq (/32) (23) sp_irq (/32) (24) lr_irq (/32) (25) sp_svc (/32): 0x00008FE0 (26) lr_svc (/32): 0x76A82074 (27) sp_abt (/32) (28) lr_abt (/32) (29) sp_und (/32) (30) lr_und (/32) (31) cpsr (/32): 0x60000133 (32) spsr_fiq (/32) (33) spsr_irq (/32) (34) spsr_svc (/32) (35) spsr_abt (/32) (36) spsr_und (/32) (37) sp (/32) (38) lr (/32) (39) sp_mon (/32) (40) lr_mon (/32) (41) spsr_mon (/32) > reg [0] 0:telnet* "lenovo-Lenovo-Product" 11:57 18-mai-16
Powered by asciinema
From now, it is possible to attach to it using IDA for example, as if it was GDB on port 3333.
Firmware analysis
It is interesting to see that Apple has the firmware upgrade tools in the ramrod OTA updates. To get it:
# wget http://appldnld.apple.com/iOS9.3/031-28353-20160321-9904D9BA-EA47-11E5-9069-7170BD379832/com_apple_MobileAsset_SoftwareUpdate/352b7321dda0925f84c7beb6128a3dbdb71476fe.zip
# unzip 352b7321dda0925f84c7beb6128a3dbdb71476fe.zip AssetData/payload/replace/usr/standalone/update/ramdisk/H8SURamDisk.dmg
# dd if=AssetData/payload/replace/usr/standalone/update/ramdisk/H8SURamDisk.dmg of=update.dmg bs=27 skip=1
# file update.dmg
update.dmg: Macintosh HFS Extended version 4 data last mounted by: '10.0', created: Fri Mar 15 00:27:05 2148, last modified: Fri Mar 15 09:31:59 2148, block size: 4096, number of blocks: 17228, free blocks: 1
# mkdir d
# mount -o loop,ro update.dmg d
. ├── ./BFH │ ├── ./BFH/s3e-bfh-loader.bin │ └── ./BFH/version.plist ├── ./Hynix-mlc-1y-bdie-2p │ ├── ./Hynix-mlc-1y-bdie-2p/FWUpdate-Hynix-mlc-1y-bdie-2p-1dpc-2ch-NonCES.pak │ ├── ./Hynix-mlc-1y-bdie-2p/FWUpdate-Hynix-mlc-1y-bdie-2p-2dpc-2ch-NonCES.pak -> FWUpdate-Hynix-mlc-1y-bdie-2p-1dpc-2ch-NonCES.pak │ ├── ./Hynix-mlc-1y-bdie-2p/FWUpdate-Hynix-mlc-1y-bdie-2p-4dpc-2ch-NonCES.pak -> FWUpdate-Hynix-mlc-1y-bdie-2p-1dpc-2ch-NonCES.pak │ ├── ./Hynix-mlc-1y-bdie-2p/FWUpdate-Hynix-mlc-1y-bdie-2p.pak -> FWUpdate-Hynix-mlc-1y-bdie-2p-1dpc-2ch-NonCES.pak │ └── ./Hynix-mlc-1y-bdie-2p/version.plist ├── ./Hynix-mlc-1y-cdie-2p │ ├── ./Hynix-mlc-1y-cdie-2p/FWUpdate-Hynix-mlc-1y-cdie-2p-1dpc-2ch-NonCES.pak │ ├── ./Hynix-mlc-1y-cdie-2p/FWUpdate-Hynix-mlc-1y-cdie-2p-2dpc-2ch-NonCES.pak -> FWUpdate-Hynix-mlc-1y-cdie-2p-1dpc-2ch-NonCES.pak │ ├── ./Hynix-mlc-1y-cdie-2p/FWUpdate-Hynix-mlc-1y-cdie-2p-4dpc-2ch-NonCES.pak -> FWUpdate-Hynix-mlc-1y-cdie-2p-1dpc-2ch-NonCES.pak │ ├── ./Hynix-mlc-1y-cdie-2p/FWUpdate-Hynix-mlc-1y-cdie-2p.pak -> FWUpdate-Hynix-mlc-1y-cdie-2p-1dpc-2ch-NonCES.pak │ └── ./Hynix-mlc-1y-cdie-2p/version.plist ├── ./Hynix-tlc-1y-2p │ ├── ./Hynix-tlc-1y-2p/FWUpdate-Hynix-tlc-1y-2p-1dpc-2ch-NonCES.pak │ ├── ./Hynix-tlc-1y-2p/FWUpdate-Hynix-tlc-1y-2p-2dpc-2ch-NonCES.pak -> FWUpdate-Hynix-tlc-1y-2p-1dpc-2ch-NonCES.pak │ ├── ./Hynix-tlc-1y-2p/FWUpdate-Hynix-tlc-1y-2p-4dpc-2ch-NonCES.pak -> FWUpdate-Hynix-tlc-1y-2p-1dpc-2ch-NonCES.pak │ ├── ./Hynix-tlc-1y-2p/FWUpdate-Hynix-tlc-1y-2p.pak -> FWUpdate-Hynix-tlc-1y-2p-1dpc-2ch-NonCES.pak │ └── ./Hynix-tlc-1y-2p/version.plist ├── ./Sandisk-mlc-1y-2p │ ├── ./Sandisk-mlc-1y-2p/FWUpdate-Sandisk-mlc-1y-2p-1dpc-2ch-NonCES.pak │ ├── ./Sandisk-mlc-1y-2p/FWUpdate-Sandisk-mlc-1y-2p-2dpc-2ch-NonCES.pak -> FWUpdate-Sandisk-mlc-1y-2p-1dpc-2ch-NonCES.pak │ ├── ./Sandisk-mlc-1y-2p/FWUpdate-Sandisk-mlc-1y-2p-4dpc-2ch-NonCES.pak -> FWUpdate-Sandisk-mlc-1y-2p-1dpc-2ch-NonCES.pak │ ├── ./Sandisk-mlc-1y-2p/FWUpdate-Sandisk-mlc-1y-2p.pak -> FWUpdate-Sandisk-mlc-1y-2p-1dpc-2ch-NonCES.pak │ └── ./Sandisk-mlc-1y-2p/version.plist ├── ./Sandisk-tlc-1y-2p │ ├── ./Sandisk-tlc-1y-2p/FWUpdate-Sandisk-tlc-1y-2p-1dpc-2ch-NonCES.pak │ ├── ./Sandisk-tlc-1y-2p/FWUpdate-Sandisk-tlc-1y-2p-2dpc-2ch-NonCES.pak -> FWUpdate-Sandisk-tlc-1y-2p-1dpc-2ch-NonCES.pak │ ├── ./Sandisk-tlc-1y-2p/FWUpdate-Sandisk-tlc-1y-2p-4dpc-2ch-NonCES.pak -> FWUpdate-Sandisk-tlc-1y-2p-1dpc-2ch-NonCES.pak │ ├── ./Sandisk-tlc-1y-2p/FWUpdate-Sandisk-tlc-1y-2p.pak -> FWUpdate-Sandisk-tlc-1y-2p-1dpc-2ch-NonCES.pak │ └── ./Sandisk-tlc-1y-2p/version.plist ├── ./Sandisk-tlc-1z-2p │ ├── ./Sandisk-tlc-1z-2p/FWUpdate-Sandisk-tlc-1z-2p-1dpc-2ch-NonCES.pak │ ├── ./Sandisk-tlc-1z-2p/FWUpdate-Sandisk-tlc-1z-2p-2dpc-2ch-NonCES.pak -> FWUpdate-Sandisk-tlc-1z-2p-1dpc-2ch-NonCES.pak │ ├── ./Sandisk-tlc-1z-2p/FWUpdate-Sandisk-tlc-1z-2p-4dpc-2ch-NonCES.pak -> FWUpdate-Sandisk-tlc-1z-2p-1dpc-2ch-NonCES.pak │ ├── ./Sandisk-tlc-1z-2p/FWUpdate-Sandisk-tlc-1z-2p.pak -> FWUpdate-Sandisk-tlc-1z-2p-1dpc-2ch-NonCES.pak │ └── ./Sandisk-tlc-1z-2p/version.plist ├── ./Sandisk-tlc-1z-2p-8dpc │ ├── ./Sandisk-tlc-1z-2p-8dpc/FWUpdate-Sandisk-tlc-1z-2p-8dpc-2ch-CES.pak │ └── ./Sandisk-tlc-1z-2p-8dpc/version.plist ├├── ./Toshiba-mlc-1y-2p │ ├── ./Toshiba-mlc-1y-2p/FWUpdate-Toshiba-mlc-1y-2p-1dpc-2ch-NonCES.pak │ ├── ./Toshiba-mlc-1y-2p/FWUpdate-Toshiba-mlc-1y-2p-2dpc-2ch-NonCES.pak -> FWUpdate-Toshiba-mlc-1y-2p-1dpc-2ch-NonCES.pak │ ├── ./Toshiba-mlc-1y-2p/FWUpdate-Toshiba-mlc-1y-2p-4dpc-2ch-NonCES.pak -> FWUpdate-Toshiba-mlc-1y-2p-1dpc-2ch-NonCES.pak │ ├── ./Toshiba-mlc-1y-2p/FWUpdate-Toshiba-mlc-1y-2p.pak -> FWUpdate-Toshiba-mlc-1y-2p-1dpc-2ch-NonCES.pak │ └── ./Toshiba-mlc-1y-2p/version.plist ├── ./Toshiba-mlc-1z-2p │ ├── ./Toshiba-mlc-1z-2p/FWUpdate-Toshiba-mlc-1z-2p-1dpc-2ch-NonCES.pak │ ├── ./Toshiba-mlc-1z-2p/FWUpdate-Toshiba-mlc-1z-2p-2dpc-2ch-NonCES.pak -> FWUpdate-Toshiba-mlc-1z-2p-1dpc-2ch-NonCES.pak │ ├── ./Toshiba-mlc-1z-2p/FWUpdate-Toshiba-mlc-1z-2p-4dpc-2ch-NonCES.pak -> FWUpdate-Toshiba-mlc-1z-2p-1dpc-2ch-NonCES.pak │ ├── ./Toshiba-mlc-1z-2p/FWUpdate-Toshiba-mlc-1z-2p.pak -> FWUpdate-Toshiba-mlc-1z-2p-1dpc-2ch-NonCES.pak │ └── ./Toshiba-mlc-1z-2p/version.plist ├── ./Toshiba-tlc-1z-2p │ ├── ./Toshiba-tlc-1z-2p/FWUpdate-Toshiba-tlc-1z-2p-1dpc-2ch-NonCES.pak │ ├── ./Toshiba-tlc-1z-2p/FWUpdate-Toshiba-tlc-1z-2p-2dpc-2ch-NonCES.pak -> FWUpdate-Toshiba-tlc-1z-2p-1dpc-2ch-NonCES.pak │ ├── ./Toshiba-tlc-1z-2p/FWUpdate-Toshiba-tlc-1z-2p-4dpc-2ch-NonCES.pak -> FWUpdate-Toshiba-tlc-1z-2p-1dpc-2ch-NonCES.pak │ ├── ./Toshiba-tlc-1z-2p/FWUpdate-Toshiba-tlc-1z-2p.pak -> FWUpdate-Toshiba-tlc-1z-2p-1dpc-2ch-NonCES.pak │ └── ./Toshiba-tlc-1z-2p/version.plist ├── ./Toshiba-tlc-1z-2p-8dpc │ ├── ./Toshiba-tlc-1z-2p-8dpc/FWUpdate-Toshiba-tlc-1z-2p-8dpc-2ch-CES.pak └── ./Toshiba-tlc-1z-2p-8dpc/version.plist
Let's have a look at the head of a hexdump of any of the .pak
00000000 46 57 55 42 00 0e 40 01 00 00 00 00 00 00 00 00 |FWUB..@.........|
00000010 10 be 05 00 11 63 6f e7 c0 1f 04 00 1e 56 ea f7 |.....co......V..|
00000020 00 00 00 00 00 00 00 00 20 0d 00 00 ac c9 02 a3 |........ .......|
00000030 10 1d 00 00 8e 17 3c 92 5e 98 b3 ff 46 57 55 45 |......<.^...FWUE|
00000040 02 00 00 00 00 10 0b 51 01 17 01 64 02 08 87 04 |.......Q...d....|
00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000440 00 00 00 00 0f 09 84 54 46 57 55 65 00 00 00 00 |.......TFWUe....|
00000450 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00001000 41 54 4f 43 36 03 00 00 00 00 00 0e 40 01 00 00 |ATOC6.......@...|
00001010 40 00 00 00 00 80 80 ce 01 00 00 08 b0 21 00 f0 |@............!..|
00001020 05 00 e0 de 03 00 00 1a d8 10 00 20 14 00 10 00 |........... ....|
00001030 00 00 45 54 4f 43 00 00 00 00 00 00 00 00 00 00 |..ETOC..........|
00001040 8e 0a 00 fa 13 00 00 ea 04 00 00 ea 4d 00 00 ea |............M...|
00001050 31 00 00 ea fe ff ff ea 80 5e 00 ea 64 00 00 ea |1........^..d...|
00001060 00 30 4f e1 20 00 13 e3 04 30 1e 05 02 30 5e 15 |.0O. ....0...0^.|
It looks like there are some keywords [FWUB, FWUE, FWUe, ATOC, ETOC...] in the file So far, what is interesting is to see that there is a directory called "BFH". with a file "s3e-bfh-loader.bin". "S3E" was previously in the iphone screenshot and looks like it's the code name for the NVMe cheap. When looking at it:
00000000 0e 00 00 ea fe ff ff ea fe ff ff ea fe ff ff ea |................|
00000010 fe ff ff ea fe ff ff ea 00 00 00 ea fe ff ff ea |................|
That suggest that the file is in fact a raw ARM firmware with the interruption vectors. When looking from closer with IDA, we could see some interesting CMP made:
As one can see, those keyword appears in little endian. So it appears that the bfh file is a bootloader that is loaded into the NVMe in order to install the .pak, as it cas clearly parse it. When paying more attention to the driver, it appears that the chip can either be in normal mode, or what is called the "BFH" mode. In fact, when in this mode, at the address BAR0 + 0x1b18, the value is 0xBFBFBFBF. The s3e binary loaded let the firmware updater tool to get the nand information and find out what file has to be loaded.
int nvme_identify(struct nvme_dev * dev) {
int p;
int brand;
int tech;
int non;
int plane;
int flag;
int dpc;
int dch;
char directory[200];
int dsize;
int die;
const char * brand_list[] = {
"Invalid",
"Hynix",
"Invalid",
"Sandisk",
"Samsung",
"Toshiba"
}; //350
const char * tech_list[] = {
"slc",
"mlc",
"tlc"
}; // 380
const char * plane_list[] = {
"1x",
"1y",
"1z"
}; // 398
const char * die_list[] = {
"a",
"b"
}; //3b0
const int size_list[] = {
64,
128,
256,
512,
1024,
2048
};
const char * non_list[] = {
"Non",
""
};
u32 tst32;
u64 tst64;
u64 tmp;
u64 val;
tst32 = readl(dev - > bar + 0x1b20);
printk("REG1B20 (nand info) %08x\n", tst32);
tst64 = tst32;
if (!(tst32 >> 25)) {
printk("val >> 25 NOK !!!\n");
return 0;
}
tst32 = readl(dev - > bar + 0x1b24);
printk("REG1B24 (nand info )%08x\n", tst32);
tmp = tst32;
tmp = tmp << 32;
tst64 |= tmp;
val = tst64;
p = 1 << ((val >> 12) & 3);
brand = val & 7;
tech = (val >> 7) & 3;
flag = (val >> 25) & 0x7F;
plane = (val >> 3) & 0xF;
die = (val >> 20) & 0xF;
if (!flag) {
dpc = 1;
dch = 1;
non = 0;
} else {
dpc = 1 << ((val >> 32) & 0xf);
dch = 1 << ((val >> 36) & 0x3);
non = ((val >> 38) & 1);
}
if (brand == 1 && tech == 1)
sprintf(directory, "%s-%s-%s-%sdie-%dp", brand_list[brand], tech_list[tech], plane_list[plane], die_list[die], p);
else
sprintf(directory, "%s-%s-%s-%dp", brand_list[brand], tech_list[tech], plane_list[plane], p);
if (!flag) {
printk("%s/FWUpdate-%s.pak\n", directory, directory);
} else {
dsize = (val >> 14) & 0x3;
if (non)
non = 1;
if (((size_list[dsize] >> 3) * dpc * dch) < 0x100)
printk("%s/FWUpdate-%s-%ddpc-%dch-%sCES.pak\n", directory, directory, dpc, dch, non_list[non]);
else
printk("%s-%ddpc/FWUpdate-%s-%ddpc-%dch-%sCES.pak\n", directory, dpc, directory, dpc, dch, non_list[non]);
}
return 0;
}
At that stage, this sample function I added to my driver is capable of telling me what file is needed to be uploaded. Regarding the upload process, the address and lenght are set in 2 special registers and a 1 is sent to register 0x1b10. Once the file is uploaded, the register 0x1b14 becomes 0x10, which means that the chip is ready to continue. By writing 2 to the 0x1b10, the next step happens. This process first happen while in BFH mode, once the s3e binary is uploaded, the NVMe is identified. Once that done, the .pak is uploaded again then the upgrade can happen.
Direct Memmory Access (DMA)
First, on the Bootrom side, when the NVMe is up and running in BFH mode and waiting for the host to give it the order to initiate the DMA, we are in the loop we see in this picture. When arriving on the purpple case, we branch all the way up again if the register that tells the DMA can occur is not set.
Once that being done, and if everything goes well, we end up waiting for the host to tell us that it is ready. Notice the "MOVS R0, #0x10" , "STR R0, [R5,#0x2C]" Which is where our register that tells us the DMA occured get sets to 0x10 as said previously.
Finally, the DMA copies the buffer we want at the address 0x76828000 as seen in the previous picture. Then, when the host says it's ready, it jumps to this address: "MOV R0, #0x76828000", "BLX R0" By the way, on the s3e-bfh.bin loader, we can clearly see that the code relocates immediately itself to 0x00101000 (+1 for the thumb mode)
The idea here would be to see if it was possible to control the NVMe over jtag in order to ask it to perform a DMA read over the PCIe Bus. In order to do so, the PCI_COMMAND_BUS_MASTER has to be set to 1. We can assume that since the chip is using remote RAM, it is allowed to act as a master over PCIe. Here is a snippet of the probing function of the kernel driver.
sstatic int bfh_probe(struct pci_dev *pdev, const struct pci_device_id *id)
{
u64 dma_addr = 0;
void *cpu_addr = NULL;
pci_enable_device(pdev);
pci_set_master(pdev);
cpu_addr = dma_alloc_coherent(NULL, 0x10000, &dma_addr, GFP_DMA32);
memset(cpu_addr, 0x41, 0x10000);
printk("ADDR mem %016llx\n", dma_addr);
return 0;
}
Our goal here is to force the DMA to happen just by controlling the ARM of the NVMe over JTAG, in order to ask it to dump the region we alloc'd in kernel and see if we get the data out of it.
08:03.0 PCI bridge: Intel Corporation Device 1513 08:04.0 PCI bridge: Intel Corporation Device 1513 08:05.0 PCI bridge: Intel Corporation Device 1513 09:00.0 Mass storage controller: Apple Inc. Device 2002 (rev 11) root@x-MacBookPro:/home/x/mydma# lspci -d 106b:2002 -v -xx 09:00.0 Mass storage controller: Apple Inc. Device 2002 (rev 11) (prog-if 02) Subsystem: Apple Inc. Device 2002 Physical Slot: 1-2 Flags: fast devsel, IRQ 255 Memory at a0800000 (64-bit, non-prefetchable) [disabled] [size=16K] Capabilities: [40] Power Management version 3 Capabilities: [50] MSI: Enable- Count=1/8 Maskable- 64bit+ Capabilities: [70] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Capabilities: [148] Virtual Channel Capabilities: [168] #19 Capabilities: [178] Latency Tolerance Reporting Capabilities: [180] L1 PM Substates 00: 6b 10 02 20 00 00 10 00 11 02 80 01 00 00 00 00 10: 04 00 80 a0 00 00 00 00 00 00 00 00 00 00 00 00 20: 00 00 00 00 00 00 00 00 00 00 00 00 6b 10 02 20 30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 00 00 root@x-MacBookPro:/home/x/mydma# emacs mydma.c root@x-MacBookPro:/home/x/mydma# make make -C /lib/modules/4.6.0-rc7+/build M=/home/x/mydma modules make[1]: Entering directory '/usr/src/linux-headers-4.6.0-rc7+' CC [M] /home/x/mydma/mydma.o Building modules, stage 2. MODPOST 1 modules LD [M] /home/x/mydma/mydma.ko make[1]: Leaving directory '/usr/src/linux-headers-4.6.0-rc7+' root@x-MacBookPro:/home/x/mydma# insmod ./mydma.ko root@x-MacBookPro:/home/x/mydma# lspci -d 106b:2002 -v | grep driver Kernel driver in use: dmatest root@x-MacBookPro:/home/x/mydma# root@x-MacBookPro:/home/x/mydma# dmesg
Powered by asciinema