back to listing index

Ramtin Amin Web Blog

[web search]
Original source (ramtin-amin.fr)
Tags: hardware hacking
Clipped on: 2016-11-17




Demystifying the i-Device NVMe NAND

(New storage used by Apple)







Introduction

Image (Asset 1/19) alt= Image (Asset 2/19) alt=

NVMexpress is the next generation storage technology used in embedded devices. The main idea was to have a memory controller directly on the PCIexpress level, avoiding a lot of overhead. Therefor, the technology is based on PCIe. We could already see on recent Mac Book Air, the details about it in the control pannel:

Image (Asset 3/19) alt=

More recently, Macrumers was showing the characteristic on this screen shot:

Image (Asset 4/19) alt=

So this was in fact very interesting to go look at it, and figure it out.







Removing the Nand

There are several videos out there, that shows how the nand is removed or replaced. I couldn't make a better quality video as the one we find on youtube, so I share one here

Image (Asset 5/19) alt=
3:53 / 4:05


Electrical interface

First, after digging a bit here and there, I manage to find the pinout of their chip.

Image (Asset 6/19) alt=


We can see that it requires 3 main voltages: 0.9v, 1.8v, and 3.0v. We have a 2x PCIe Interface with the TX[0..1]+/- and RX[0..1]+/- as well as the CLK+/- pins. Note that those are differential pairs. I assumed that they were routed with a 90R impedance. On top of that, we could find some PCIe control pins such as the CLKREQ, NRST. And some other config pins such as a low power mode, or a firmware strap and SWD jtag. Another interesting part is that the Clkc frequency is directly controlled by the SoC PLL itself. In order to get the chip working elsewhere, one would need to know the frequency. Things easily doable with a scope:

Image (Asset 7/19) alt=

So 24Mhz 1.8v is needed.







My Apple Iphone NVMe Reader

In order to read the NVMe, I therefor developped a PCIe card with a Zero Insertion Force reader. I brought the JTAG part to 20pin header.

Image (Asset 8/19) alt= Image (Asset 9/19) alt= Image (Asset 10/19) alt=

The hard pard in here is the signal integrity of the differential pairs. In order to do so, I had to use multi layer PCB, and have the impedence match by knowing the stackup, materials used for prepeg and so on..



As I am not planning to sell those device myself, I want to thank Itead Studio that agreed to do a small set of production. The bards are currently in pre-order. Notice that the sockets that are made for this project are not generic but custom made for those LGA70, which explains this price. https://www.itead.cc/nvme-reader.html





NVMe Enumeration


	# lspci -d 106b:2002 -vvv
	09:00.0 Mass storage controller: Apple Inc. Device 2002 (rev 11) (prog-if 02)
        Subsystem: Apple Inc. Device 2002
        Physical Slot: 1-2
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 19
        Region 0: Memory at a0800000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable- Count=1/8 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [70] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <4us, L1 <32us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x2, ASPM L0s L1, Exit Latency L0s <1us, L1 <8us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x2, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR+, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
        Capabilities: [148 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
                        Status: NegoPending- InProgress-
        Capabilities: [168 v1] #19
        Capabilities: [178 v1] Latency Tolerance Reporting
                Max snoop latency: 0ns
                Max no snoop latency: 0ns
        Capabilities: [180 v1] L1 PM Substates
                L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
                          PortCommonModeRestoreTime=25us PortTPowerOnTime=10us


So as we can see, the device is using Apple's VID: 0x106b and the PID is 0x2002 . Remember, the version in mac book air are 0x2001 ! When using this card in a Mac book that has the NVMe driver, I couldnt get it to work tho. It looks like the driver does not work on this.







Driver Reverse Engineering

In order to get it to work, I had to manage to get the driver working. NVMe is implemented in Linux and Matthew Wilcox (which gave me great support) has done an amaizing job writing the NVMe driver for linux. Some other have done the quirks in order to get the driver work for Mac book Air. (reducing the amount of queues to 2). So After having added the PID to the driver and recompiled, I saw that it was still not wokring. Had to go the hard way then. It looks like to reduce the size needed, the NVMe core uses the host DDR in order to work. Therefor, apple is not strictly following the specification regarding the initialisation. A scratch region is reserved from the beginning, and it's address and lenght are in the device tree. In order to set those value, the NVMe has some special register starting from address BAR0 + 0x1800. For example, registers at offset 0x1910 0x1914 0x1918 0x191c seams to contain informations about Error codes when the NVMe is setting up it's error bit in its STATUS (CSTS) register. Some other interesting registers when in BFH mode are located around 0x1b00. We will get back on those later.

Image (Asset 11/19) alt=
00000000  6a 90 88 cf 8a fd 63 0a  e3 51 e2 48 87 e0 b9 8b  |j.....c..Q.H....|                                                                        00000010  36 fe 07 7b d4 16 44 bb  b8 27 9c 89 98 3b aa fc  |6..{..D..'...;..|                                                                        00000020  00 a0 ac b9 03 00 00 00  03 00 00 00 de 23 55 c1  |.............#U.|                                                                        00000030  01 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|                                                                        00000040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|                                                                        *                                                                                                                                                     00000200  48 46 53 00 00 00 11 aa  aa 11 00 30 65 43 ec ac  |HFS........0eC..|                                                                        00000210  32 fc 46 79 50 c3 47 42  9b 60 df 45 1d fc 70 5f  |2.FyP.GB.`.E..p_|                                                                        00000220  00 60 00 00 00 00 00 00  00 60 24 be 00 00 00 00  |.`.......`$.....|                                                                        00000230  00 00 00 00 00 00 00 00  53 00 79 00 73 00 74 00  |........S.y.s.t.|                                                                        00000240  65 00 6d 00 00 00 00 00  00 00 00 00 00 00 00 00  |e.m.............|                                                                        00000250  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|                                                                        *                                                                                                                                                     00000280  48 46 53 00 00 00 11 aa  aa 11 00 30 65 43 ec ac  |HFS........0eC..|                                                                        00000290  0f 66 b8 0b 6b e7 43 c4  ab d7 dc 45 18 35 1b 4c  |.f..k.C....E.5.L|                                                                        000002a0  00 60 24 be 00 00 00 00  00 10 61 b0 03 00 00 00  |.`$.......a.....|                                                                        000002b0  00 00 00 00 00 00 03 00  44 00 61 00 74 00 61 00  |........D.a.t.a.|                                                                        000002c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|                                                                        *                                                                                                                                                     00000300  48 46 53 00 00 00 11 aa  aa 11 00 30 65 43 ec ac  |HFS........0eC..|                                                                        00000310  8d ba 7b fb ad 12 48 20  8e c2 80 1a 36 cb 45 d5  |..{...H ....6.E.|                                                                        00000320  00 10 61 b0 03 00 00 00  00 60 00 b1 03 00 00 00  |..a......`......|                                                                        00000330  00 00 00 00 00 00 00 00  42 00 61 00 73 00 65 00  |........B.a.s.e.|                                                                        00000340  62 00 61 00 6e 00 64 00  20 00 44 00 61 00 74 00  |b.a.n.d. .D.a.t.|                                                                        00000350  61 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |a...............|                                                                        00000360  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|                                                                        *                                                                                                                                                     00000800  00 f0 00 00 01 00 02 00  03 00 04 00 05 00 06 00  |................|                                                                        00000810  07 00 08 00 09 00 0a 00  0b 00 0c 00 0d 00 0e 00  |................|                                                                        00000820  0f 00 10 00 11 00 12 00  13 00 14 00 15 00 16 00  |................|                                                                        --More--                                                                                                                                              
00:00

Powered by asciinema


# nvme list
Node             SN                   Model                Version  NS  Usage                      Format           FW Rev
---------------- -------------------- -------------------- -------- --  -------------------------- ---------------- ---------
/dev/nvme0n1     01061019E5C211EB0    APPLE SSD AP0016K    1.0      1      16,00  GB /  16,00  GB      4 KiB +  0 B  11.90.01
/dev/nvme0n2     01061019E5C211EB0    APPLE SSD AP0016K    1.0      2       8,39  MB /   8,39  MB      4 KiB +  0 B  11.90.01
/dev/nvme0n3     01061019E5C211EB0    APPLE SSD AP0016K    1.0      3     131,07  kB / 131,07  kB      4 KiB +  0 B  11.90.01
/dev/nvme0n4     01061019E5C211EB0    APPLE SSD AP0016K    1.0      4       8,19  kB /   8,19  kB      4 KiB +  0 B  11.90.01
/dev/nvme0n5     01061019E5C211EB0    APPLE SSD AP0016K    1.0      5       8,19  kB /   8,19  kB      4 KiB +  0 B  11.90.01
/dev/nvme0n6     01061019E5C211EB0    APPLE SSD AP0016K    1.0      6       4,10  kB /   4,10  kB      4 KiB +  0 B  11.90.01
/dev/nvme0n7     01061019E5C211EB0    APPLE SSD AP0016K    1.0      7       1,05  MB /   1,05  MB      4 KiB +  0 B  11.90.01

AS one could see, the nand contains 7 partitions. They are usually the first one is a HFS with the system on it, the second one is the llb + iboot (of course GID crypted). the 3rd one is the syscfg... and so on.




Anatomy of the Names Spaces.

NS1

NS1 is what used to by ‘fsys’, and contains an LwVM formatted partition, which contains three volumes - System (/), Data (/private/var) and the baseband data. These are all visible to iOS as /dev/disk0s1s1..s3 respectively


00000000  6a 90 88 cf 8a fd 63 0a  e3 51 e2 48 87 e0 b9 8b  |j.....c..Q.H....|
00000010  a3 a2 64 6b a1 b1 41 9b  89 49 95 75 61 74 53 31  |..dk..A..I.uatS1|
00000020  00 80 b2 e6 0e 00 00 00  03 00 00 00 1d 4f 5c b9  |.............O\.|
00000030  01 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200  48 46 53 00 00 00 11 aa  aa 11 00 30 65 43 ec ac  |HFS........0eC..|
00000210  a9 49 c9 cf 91 41 43 bc  bc c3 2a 80 c9 71 e1 79  |.I...AC...*..q.y|
00000220  00 60 00 00 00 00 00 00  00 f0 16 da 00 00 00 00  |.`..............|
00000230  00 00 00 00 00 00 00 00  53 00 79 00 73 00 74 00  |........S.y.s.t.|
00000240  65 00 6d 00 00 00 00 00  00 00 00 00 00 00 00 00  |e.m.............|
00000250  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000280  48 46 53 00 00 00 11 aa  aa 11 00 30 65 43 ec ac  |HFS........0eC..|
00000290  6b a9 57 fb 7e 21 46 78  96 16 b5 15 4f df 4d 3e  |k.W.~!Fx....O.M>|
000002a0  00 f0 16 da 00 00 00 00  00 10 61 c3 0e 00 00 00  |..........a.....|
000002b0  00 00 00 00 00 00 03 00  44 00 61 00 74 00 61 00  |........D.a.t.a.|
000002c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000300  48 46 53 00 00 00 11 aa  aa 11 00 30 65 43 ec ac  |HFS........0eC..|
00000310  7a 69 fe f8 7f 82 45 23  b0 75 e8 1a 43 36 90 b2  |zi....E#.u..C6..|
00000320  00 10 61 c3 0e 00 00 00  00 60 00 c4 0e 00 00 00  |..a......`......|
00000330  00 00 00 00 00 00 00 00  42 00 61 00 73 00 65 00  |........B.a.s.e.|
00000340  62 00 61 00 6e 00 64 00  20 00 44 00 61 00 74 00  |b.a.n.d. .D.a.t.|
00000350  61 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |a...............|
00000360  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

NS2

The second NameSpace contains the illb and iBoot. From what we can see in this hex dump. It is encrypted with the GID key


00000000  30 83 03 96 1e 16 04 49  4d 47 34 30 83 03 80 99  |0......IMG40....|
00000010  16 04 49 4d 34 50 16 04  69 6c 6c 62 16 10 69 42  |..IM4P..illb..iB|
00000020  6f 6f 74 2d 32 38 31 37  2e 32 30 2e 32 36 04 83  |oot-2817.20.26..|
00000030  03 80 00 9f 1c 04 9b 76  e3 d6 d3 70 15 99 89 b7  |.......v...p....|
00000040  8a 73 4e dd f8 00 d8 e7  5c 5a b4 55 85 2a 03 e1  |.sN.....\Z.U.*..|
00000050  6b 4a 38 0a 59 6e 28 9d  f1 be c5 41 1b 17 bd dd  |kJ8.Yn(....A....|
00000060  89 0c 69 63 54 ed bd 72  a1 3d 4f 97 04 de 28 fc  |..icT..r.=O...(.|
00000070  94 58 29 b4 45 67 23 ae  cc d7 be 59 bc 07 3f fc  |.X).Eg#....Y..?.|
00000080  ba 55 03 71 a4 a2 86 dd  23 8f 5a ce 67 18 6e 1e  |.U.q....#.Z.g.n.|
00000090  8a 6a ec 83 f8 d6 8c 32  ca 24 36 50 77 76 1e bd  |.j.....2.$6Pwv..|
000000a0  4f a9 ac d3 5e 15 70 4e  32 58 a8 34 16 22 0c d3  |O...^.pN2X.4."..|
...

NS3

The third NS is the SysCFG partition. The SysCFG is visible in the IORegistry through the AppleDiagnosticDataSysCfg property of the AppleDiagnosticDataAccessReadOnly IOregistry node. (Provided by com.apple.driver.AppleDiagnosticDataAccessReadOnly kext). (as discussed in Jonathan Levin’s *OS Internals Volumes III and II)


00000000  67 66 43 53 04 05 00 00  00 00 02 00 02 00 02 00  |gfCS............|
00000010  00 00 00 00 3f 00 00 00  42 54 4e 43 23 42 4c 4d  |....?...BTNC#BLM|
00000020  11 00 00 00 ec ff 01 00  ff ff ff ff 23 47 46 43  |............#GFC|
00000030  2f 2f 2f 2f 2a 2f 2a 00  00 00 00 00 00 00 00 00  |////*/*.........|
00000040  63 61 4d 57 38 71 de 30  82 ff 00 00 00 00 00 00  |caMW8q.0........|
00000050  00 00 00 00 63 61 4d 42  38 71 de 30 83 00 00 00  |....caMB8q.0....|
00000060  00 00 00 00 00 00 00 00  42 54 4e 43 56 43 41 50  |........BTNCVCAP|
00000070  29 00 00 00 c0 ff 01 00  ff ff ff ff 42 54 4e 43  |)...........BTNC|
00000080  54 41 42 43 31 00 00 00  8c ff 01 00 ff ff ff ff  |TABC1...........|
00000090  54 53 42 56 09 00 00 00  00 00 00 00 00 00 00 00  |TSBV............|
000000a0  00 00 00 00 52 42 50 56  0b 00 00 00 00 00 00 00  |....RBPV........|
000000b0  00 00 00 00 00 00 00 00  42 54 4e 43 41 43 42 56  |........BTNCACBV|
000000c0  14 00 00 00 78 ff 01 00  ff ff ff ff 42 54 4e 43  |....x.......BTNC|
000000d0  4c 41 43 57 ac 02 00 00  cc fc 01 00 ff ff ff ff  |LACW............|

NS4

The forth NameSpace contains the wifi infos. In fact it allowes the phone, even while not unlocked, to get access to the password to activate the networks


00000000  69 50 43 42 00 01 00 00  01 00 00 00 80 00 00 00  |iPCB............|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000002e0  fc ff ef ff af ec 22 08  57 69 50 41 53 58 4e 65  |......".WiPASXNe|
000002f0  78 74 20 31 38 41 41 64  4d 32 00 00 00 00 00 00  |xt 18AAdM2......|
00000300  fc ff ef ff 97 76 22 08  57 69 50 41 53 20 31 43  |.....v".WiPAS 1C|
00000310  4d 50 32 37 00 00 00 00  00 00 00 00 00 00 00 00  |MP27............|
00000320  fc ff ef ff 79 79 22 08  57 69 50 41 53 20 31 43  |....yy".WiPAS 1C|
00000330  4d 50 32 37 00 00 00 00  00 00 00 00 00 00 00 00  |MP27............|

NS5

The fifth partition is the NVRAM which is accessible as logical keys/values through a corresponding Kext


00000000  5a 82 02 00 6e 76 72 61  6d 00 00 00 00 00 00 00  |Z...nvram.......|
00000010  fd 20 e6 d3 58 01 00 00  00 00 00 00 00 00 00 00  |. ..X...........|
00000020  70 7c 80 00 63 6f 6d 6d  6f 6e 00 00 00 00 00 00  |p|..common......|
00000030  61 75 74 6f 2d 62 6f 6f  74 3d 74 72 75 65 00 62  |auto-boot=true.b|
00000040  61 63 6b 6c 69 67 68 74  2d 6c 65 76 65 6c 3d 31  |acklight-level=1|
00000050  34 39 37 00 62 6f 6f 74  2d 61 72 67 73 3d 00 00  |497.boot-args=..|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000820  a1 15 81 00 41 50 4c 2c  4f 53 58 50 61 6e 69 63  |....APL,OSXPanic|
00000830  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00001030  7f 17 fd 00 77 77 77 77  77 77 77 77 77 77 77 77  |....wwwwwwwwwwww|
00001040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00002000

NS6

The sixth NameSpace is the PLOG. Where the kbag info are stored. That is used when you unlock the phone, with the crypto engine. Also, 10 wrong passcode and this get erased.


00000000  4e 0b 2a 02 c5 30 09 f8  ae 0e ca 8d 8f c1 88 8c  |N.*..0..........|
00000010  2b 68 4b 44 c4 30 09 f8  af 0e ca 8d 8f c1 88 8c  |+hKD.0..........|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000030  00 00 00 00 00 00 00 00  1b 00 00 00 0c 97 b4 a8  |................|
00000040  6b 4c 50 00 4d 56 77 4c  2b a0 ab f1 3e 62 df a6  |kLP.MVwL+...>b..|
00000050  66 93 52 4e 2a da 12 f9  17 ba 37 ea 08 f2 cd 62  |f.RN*.....7....b|
00000060  d8 02 71 b3 a7 a9 38 b8  a1 db 0e 47 29 50 e9 41  |..q...8....G)P.A|
00000070  eb 57 88 e9 76 7a f7 c3  8f 77 bd c8 08 0f 83 3c  |.W..vz...w.....<|
00000080  b7 96 20 46 71 a7 ad 59  e1 e3 b4 cc 1b cf 83 83  |.. Fq..Y........|
00000090  5e 60 c8 be f3 20 8c f5  6b 4c 34 00 31 47 41 42  |^`... ..kL4.1GAB|
000000a0  31 47 41 42 18 1b 7c 83  11 63 64 95 96 0d 0e fc  |1GAB..|..cd.....|
000000b0  09 30 c0 3e c5 f6 48 70  a8 f4 86 f8 52 77 49 20  |.0.>..Hp....RwI |
000000c0  57 ad 7e 21 23 38 b0 dd  4a aa 21 05 6e 20 0a 8e  |W.~!#8..J.!.n ..|
000000d0  76 24 55 fc 6b 4c 28 00  79 65 6b c4 6a dc ee 85  |v$U.kL(.yek.j...|
000000e0  f3 1e d2 0f 66 f8 25 e1  97 bd ef 61 7e 9a 7b 18  |....f.%....a~.{.|
000000f0  d3 60 89 d8 57 43 3c d8  44 30 e3 1a e6 eb 0c 3e  |.`..WC<.D0.....>|
00000100  9b 65 f4 69 6b 4c 00 00  45 4e 4f 44 00 00 00 00  |.e.ikL..ENOD....|
00000110  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
....

NS7

The seventh partition looks empty so far


00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00100000



Navigating into the nand

Jonathan Levin’s HFSleuth can read raw device files and inspect HFS+ filesystems in them. http://NewOSXBook.com/tools/hfsleuth.html (thank to @Morpheus_____)

Image (Asset 12/19) alt=


Reusing the NAND

The nand could be re-used in an iphone if you have a proper JIG which uses pogo pins in order to contact the pads. It's quite handy when wanting for example to modify the NAND from outside and then get it back to work.

Image (Asset 13/19) alt= Image (Asset 14/19) alt=


JTAG / SWD

As mentioned before, the NVMe chip has a JTAG/SWD interface. It is interesting to stick an OpenOCD on it to see a bit what's going on.

Image (Asset 15/19) alt=

Using a KT-Link interface with OpenOCD, we could have a TAP enumeration.



Open On-Chip Debugger 0.10.0-dev-00212-g50d4f76 (2016-02-15-18:11)
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.org/doc/doxygen/bugs.html
debug_level: 2
Info : FTDI SWD mode enabled
adapter speed: 10000 kHz
Info : clock speed 10000 kHz
Info : SWD IDCODE 0x5ba02477
Info : SWD IDCODE 0x5ba02477
Info : SWD IDCODE 0x5ba02477
Info : SWD IDCODE 0x5ba02477
Info : mynvme.cpu: hardware has 3 breakpoints, 2 watchpoints
Info : mynvme.cpu rev 1, partnum c05, arch f, variant 0, implementor 41
Info : mynvme.cpu cluster 0 core 0 multi core


After looking a bit a round, it looked first that 0x5ba02477 was a Cortex M, but in fact it is a cortex A.



> dap info
AP ID register 0x44770002
        Type is MEM-AP APB
MEM-AP BASE 0x80000000
        ROM table in legacy format
        MEMTYPE system memory not present: dedicated debug bus
        ROMTABLE[0x0] = 0x40000002
                Component not present
        ROMTABLE[0x4] = 0x74300003
                Component base address 0xf4300000, start address 0xf4300000
                Component class is 0x1, ROM table
                Peripheral ID[4..0] = hex 00 00 0c 85 33
                Unrecognized (Part 0x533, designer 0xc8)
        [L01] ROM table in legacy format
        [L01] MEMTYPE system memory not present: dedicated debug bus
        [L01] ROMTABLE[0x0] = 0x1003
                Component base address 0xf4301000, start address 0xf4301000
                Component class is 0xf, PrimeCell or System component
                Peripheral ID[4..0] = hex 00 00 4c 80 00
                Unrecognized (Part 0x0, designer 0xc8)
        [L01] ROMTABLE[0x4] = 0x10003
                Component base address 0xf4310000, start address 0xf4310000
                Component class is 0x9, CoreSight component
                Type is 0x15, Debug Logic, Processor
                Peripheral ID[4..0] = hex 04 00 1b bc 05
                Part is Cortex-A5 Debug (Debug Unit)
        [L01] ROMTABLE[0x8] = 0x11002
                Component not present
        [L01] ROMTABLE[0xc] = 0x12003
                Component base address 0xf4312000, start address 0xf4312000
                Component class is 0x9, CoreSight component
                Type is 0x13, Trace Source, Processor
                Peripheral ID[4..0] = hex 04 00 2b b9 55
                Part is Cortex-A5 ETM (Embedded Trace)
        [L01] ROMTABLE[0x10] = 0x13003
                Component base address 0xf4313000, start address 0xf4313000
                Component class is 0x9, CoreSight component
                Type is 0x21, Trace Sink, Buffer
                Peripheral ID[4..0] = hex 04 00 3b b9 07
                Part is CoreSight ETB (Trace Buffer)
        [L01] ROMTABLE[0x14] = 0x14003
                Component base address 0xf4314000, start address 0xf4314000
                Component class is 0x9, CoreSight component
                Type is 0x14, Debug Control, Trigger Matrix
                Peripheral ID[4..0] = hex 04 00 4b b9 06
                Part is CoreSight CTI (Cross Trigger)
        [L01] ROMTABLE[0x18] = 0x15003
                Component base address 0xf4315000, start address 0xf4315000
                Component class is 0x9, CoreSight component
                Type is 0x14, Debug Control, Trigger Matrix
                Peripheral ID[4..0] = hex 04 00 4b b9 06
                Part is CoreSight CTI (Cross Trigger)
        [L01] ROMTABLE[0x1c] = 0x20003
                Component base address 0xf4320000, start address 0xf4320000
                Component class is 0x9, CoreSight component
                Type is 0x15, Debug Logic, Processor
                Peripheral ID[4..0] = hex 04 00 4b bc 15
                Part is Cortex-R5 Debug (Debug Unit)
        [L01] ROMTABLE[0x20] = 0x21002
                Component not present
        [L01] ROMTABLE[0x24] = 0x22003
                Component base address 0xf4322000, start address 0xf4322000
                Component class is 0x9, CoreSight component
                Type is 0x13, Trace Source, Processor
                Peripheral ID[4..0] = hex 04 00 0b b9 31
                Part is Cortex-R5 ETM (Embedded Trace)
        [L01] ROMTABLE[0x28] = 0x23003
                Component base address 0xf4323000, start address 0xf4323000
                Component class is 0x9, CoreSight component
                Type is 0x21, Trace Sink, Buffer
                Peripheral ID[4..0] = hex 04 00 3b b9 07
                Part is CoreSight ETB (Trace Buffer)
        [L01] ROMTABLE[0x2c] = 0x24003
                Component base address 0xf4324000, start address 0xf4324000
                Component class is 0x9, CoreSight component
                Type is 0x14, Debug Control, Trigger Matrix
                Peripheral ID[4..0] = hex 04 00 4b b9 06
                Part is CoreSight CTI (Cross Trigger)
        [L01] ROMTABLE[0x30] = 0x25003
                Component base address 0xf4325000, start address 0xf4325000
                Component class is 0x9, CoreSight component
                Type is 0x14, Debug Control, Trigger Matrix
                Peripheral ID[4..0] = hex 04 00 4b b9 06
                Part is CoreSight CTI (Cross Trigger)
        [L01] ROMTABLE[0x34] = 0x26003
                Component base address 0xf4326000, start address 0xf4326000
                Component class is 0x9, CoreSight component
                Type is 0x12, Trace Link, Funnel, router
                Peripheral ID[4..0] = hex 04 00 2b b9 08
                Part is CoreSight CSTF (Trace Funnel)
        [L01] ROMTABLE[0x38] = 0xffe00002
                Component not present
        [L01] ROMTABLE[0x3c] = 0x0
        [L01]   End of ROM table
        ROMTABLE[0x8] = 0x0
                End of ROM table



It is interesting at this stage to dump their bootrom in order to find the way the chip works.

(15) pc (/32): 0x74002788                                                                                                            (16) r8_fiq (/32)                                                                                                                    (17) r9_fiq (/32)                                                                                                                    (18) r10_fiq (/32)                                                                                                                   (19) r11_fiq (/32)                                                                                                                   (20) r12_fiq (/32)                                                                                                                   (21) sp_fiq (/32)                                                                                                                    (22) lr_fiq (/32)                                                                                                                    (23) sp_irq (/32)                                                                                                                    (24) lr_irq (/32)                                                                                                                    (25) sp_svc (/32): 0x00008FE0                                                                                                        (26) lr_svc (/32): 0x76A82074                                                                                                        (27) sp_abt (/32)                                                                                                                    (28) lr_abt (/32)                                                                                                                    (29) sp_und (/32)                                                                                                                    (30) lr_und (/32)                                                                                                                    (31) cpsr (/32): 0x60000133                                                                                                          (32) spsr_fiq (/32)                                                                                                                  (33) spsr_irq (/32)                                                                                                                  (34) spsr_svc (/32)                                                                                                                  (35) spsr_abt (/32)                                                                                                                  (36) spsr_und (/32)                                                                                                                  (37) sp (/32)                                                                                                                        (38) lr (/32)                                                                                                                        (39) sp_mon (/32)                                                                                                                    (40) lr_mon (/32)                                                                                                                    (41) spsr_mon (/32)                                                                                                                                                                                                                                                       ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────(16) r8_fiq (/32)                                                                                                                    (17) r9_fiq (/32)                                                                                                                    (18) r10_fiq (/32)                                                                                                                   (19) r11_fiq (/32)                                                                                                                   (20) r12_fiq (/32)                                                                                                                   (21) sp_fiq (/32)                                                                                                                    (22) lr_fiq (/32)                                                                                                                    (23) sp_irq (/32)                                                                                                                    (24) lr_irq (/32)                                                                                                                    (25) sp_svc (/32): 0x00008FE0                                                                                                        (26) lr_svc (/32): 0x76A82074                                                                                                        (27) sp_abt (/32)                                                                                                                    (28) lr_abt (/32)                                                                                                                    (29) sp_und (/32)                                                                                                                    (30) lr_und (/32)                                                                                                                    (31) cpsr (/32): 0x60000133                                                                                                          (32) spsr_fiq (/32)                                                                                                                  (33) spsr_irq (/32)                                                                                                                  (34) spsr_svc (/32)                                                                                                                  (35) spsr_abt (/32)                                                                                                                  (36) spsr_und (/32)                                                                                                                  (37) sp (/32)                                                                                                                        (38) lr (/32)                                                                                                                        (39) sp_mon (/32)                                                                                                                    (40) lr_mon (/32)                                                                                                                    (41) spsr_mon (/32)                                                                                                                  > reg                                                                                                                                [0] 0:telnet*                                                                                 "lenovo-Lenovo-Product" 11:57 18-mai-16
00:00

Powered by asciinema

From now, it is possible to attach to it using IDA for example, as if it was GDB on port 3333.







Firmware analysis

It is interesting to see that Apple has the firmware upgrade tools in the ramrod OTA updates. To get it:


# wget http://appldnld.apple.com/iOS9.3/031-28353-20160321-9904D9BA-EA47-11E5-9069-7170BD379832/com_apple_MobileAsset_SoftwareUpdate/352b7321dda0925f84c7beb6128a3dbdb71476fe.zip
# unzip 352b7321dda0925f84c7beb6128a3dbdb71476fe.zip AssetData/payload/replace/usr/standalone/update/ramdisk/H8SURamDisk.dmg
# dd if=AssetData/payload/replace/usr/standalone/update/ramdisk/H8SURamDisk.dmg of=update.dmg bs=27 skip=1
# file update.dmg
update.dmg: Macintosh HFS Extended version 4 data last mounted by: '10.0', created: Fri Mar 15 00:27:05 2148, last modified: Fri Mar 15 09:31:59 2148, block size: 4096, number of blocks: 17228, free blocks: 1
# mkdir d
# mount -o loop,ro update.dmg d

.
├── ./BFH
│   ├── ./BFH/s3e-bfh-loader.bin
│   └── ./BFH/version.plist
├── ./Hynix-mlc-1y-bdie-2p
│   ├── ./Hynix-mlc-1y-bdie-2p/FWUpdate-Hynix-mlc-1y-bdie-2p-1dpc-2ch-NonCES.pak
│   ├── ./Hynix-mlc-1y-bdie-2p/FWUpdate-Hynix-mlc-1y-bdie-2p-2dpc-2ch-NonCES.pak -> FWUpdate-Hynix-mlc-1y-bdie-2p-1dpc-2ch-NonCES.pak
│   ├── ./Hynix-mlc-1y-bdie-2p/FWUpdate-Hynix-mlc-1y-bdie-2p-4dpc-2ch-NonCES.pak -> FWUpdate-Hynix-mlc-1y-bdie-2p-1dpc-2ch-NonCES.pak
│   ├── ./Hynix-mlc-1y-bdie-2p/FWUpdate-Hynix-mlc-1y-bdie-2p.pak -> FWUpdate-Hynix-mlc-1y-bdie-2p-1dpc-2ch-NonCES.pak
│   └── ./Hynix-mlc-1y-bdie-2p/version.plist
├── ./Hynix-mlc-1y-cdie-2p
│   ├── ./Hynix-mlc-1y-cdie-2p/FWUpdate-Hynix-mlc-1y-cdie-2p-1dpc-2ch-NonCES.pak
│   ├── ./Hynix-mlc-1y-cdie-2p/FWUpdate-Hynix-mlc-1y-cdie-2p-2dpc-2ch-NonCES.pak -> FWUpdate-Hynix-mlc-1y-cdie-2p-1dpc-2ch-NonCES.pak
│   ├── ./Hynix-mlc-1y-cdie-2p/FWUpdate-Hynix-mlc-1y-cdie-2p-4dpc-2ch-NonCES.pak -> FWUpdate-Hynix-mlc-1y-cdie-2p-1dpc-2ch-NonCES.pak
│   ├── ./Hynix-mlc-1y-cdie-2p/FWUpdate-Hynix-mlc-1y-cdie-2p.pak -> FWUpdate-Hynix-mlc-1y-cdie-2p-1dpc-2ch-NonCES.pak
│   └── ./Hynix-mlc-1y-cdie-2p/version.plist
├── ./Hynix-tlc-1y-2p
│   ├── ./Hynix-tlc-1y-2p/FWUpdate-Hynix-tlc-1y-2p-1dpc-2ch-NonCES.pak
│   ├── ./Hynix-tlc-1y-2p/FWUpdate-Hynix-tlc-1y-2p-2dpc-2ch-NonCES.pak -> FWUpdate-Hynix-tlc-1y-2p-1dpc-2ch-NonCES.pak
│   ├── ./Hynix-tlc-1y-2p/FWUpdate-Hynix-tlc-1y-2p-4dpc-2ch-NonCES.pak -> FWUpdate-Hynix-tlc-1y-2p-1dpc-2ch-NonCES.pak
│   ├── ./Hynix-tlc-1y-2p/FWUpdate-Hynix-tlc-1y-2p.pak -> FWUpdate-Hynix-tlc-1y-2p-1dpc-2ch-NonCES.pak
│   └── ./Hynix-tlc-1y-2p/version.plist
├── ./Sandisk-mlc-1y-2p
│   ├── ./Sandisk-mlc-1y-2p/FWUpdate-Sandisk-mlc-1y-2p-1dpc-2ch-NonCES.pak
│   ├── ./Sandisk-mlc-1y-2p/FWUpdate-Sandisk-mlc-1y-2p-2dpc-2ch-NonCES.pak -> FWUpdate-Sandisk-mlc-1y-2p-1dpc-2ch-NonCES.pak
│   ├── ./Sandisk-mlc-1y-2p/FWUpdate-Sandisk-mlc-1y-2p-4dpc-2ch-NonCES.pak -> FWUpdate-Sandisk-mlc-1y-2p-1dpc-2ch-NonCES.pak
│   ├── ./Sandisk-mlc-1y-2p/FWUpdate-Sandisk-mlc-1y-2p.pak -> FWUpdate-Sandisk-mlc-1y-2p-1dpc-2ch-NonCES.pak
│   └── ./Sandisk-mlc-1y-2p/version.plist
├── ./Sandisk-tlc-1y-2p
│   ├── ./Sandisk-tlc-1y-2p/FWUpdate-Sandisk-tlc-1y-2p-1dpc-2ch-NonCES.pak
│   ├── ./Sandisk-tlc-1y-2p/FWUpdate-Sandisk-tlc-1y-2p-2dpc-2ch-NonCES.pak -> FWUpdate-Sandisk-tlc-1y-2p-1dpc-2ch-NonCES.pak
│   ├── ./Sandisk-tlc-1y-2p/FWUpdate-Sandisk-tlc-1y-2p-4dpc-2ch-NonCES.pak -> FWUpdate-Sandisk-tlc-1y-2p-1dpc-2ch-NonCES.pak
│   ├── ./Sandisk-tlc-1y-2p/FWUpdate-Sandisk-tlc-1y-2p.pak -> FWUpdate-Sandisk-tlc-1y-2p-1dpc-2ch-NonCES.pak
│   └── ./Sandisk-tlc-1y-2p/version.plist
├── ./Sandisk-tlc-1z-2p
│   ├── ./Sandisk-tlc-1z-2p/FWUpdate-Sandisk-tlc-1z-2p-1dpc-2ch-NonCES.pak
│   ├── ./Sandisk-tlc-1z-2p/FWUpdate-Sandisk-tlc-1z-2p-2dpc-2ch-NonCES.pak -> FWUpdate-Sandisk-tlc-1z-2p-1dpc-2ch-NonCES.pak
│   ├── ./Sandisk-tlc-1z-2p/FWUpdate-Sandisk-tlc-1z-2p-4dpc-2ch-NonCES.pak -> FWUpdate-Sandisk-tlc-1z-2p-1dpc-2ch-NonCES.pak
│   ├── ./Sandisk-tlc-1z-2p/FWUpdate-Sandisk-tlc-1z-2p.pak -> FWUpdate-Sandisk-tlc-1z-2p-1dpc-2ch-NonCES.pak
│   └── ./Sandisk-tlc-1z-2p/version.plist
├── ./Sandisk-tlc-1z-2p-8dpc
│   ├── ./Sandisk-tlc-1z-2p-8dpc/FWUpdate-Sandisk-tlc-1z-2p-8dpc-2ch-CES.pak
│   └── ./Sandisk-tlc-1z-2p-8dpc/version.plist
├├── ./Toshiba-mlc-1y-2p
│   ├── ./Toshiba-mlc-1y-2p/FWUpdate-Toshiba-mlc-1y-2p-1dpc-2ch-NonCES.pak
│   ├── ./Toshiba-mlc-1y-2p/FWUpdate-Toshiba-mlc-1y-2p-2dpc-2ch-NonCES.pak -> FWUpdate-Toshiba-mlc-1y-2p-1dpc-2ch-NonCES.pak
│   ├── ./Toshiba-mlc-1y-2p/FWUpdate-Toshiba-mlc-1y-2p-4dpc-2ch-NonCES.pak -> FWUpdate-Toshiba-mlc-1y-2p-1dpc-2ch-NonCES.pak
│   ├── ./Toshiba-mlc-1y-2p/FWUpdate-Toshiba-mlc-1y-2p.pak -> FWUpdate-Toshiba-mlc-1y-2p-1dpc-2ch-NonCES.pak
│   └── ./Toshiba-mlc-1y-2p/version.plist
├── ./Toshiba-mlc-1z-2p
│   ├── ./Toshiba-mlc-1z-2p/FWUpdate-Toshiba-mlc-1z-2p-1dpc-2ch-NonCES.pak
│   ├── ./Toshiba-mlc-1z-2p/FWUpdate-Toshiba-mlc-1z-2p-2dpc-2ch-NonCES.pak -> FWUpdate-Toshiba-mlc-1z-2p-1dpc-2ch-NonCES.pak
│   ├── ./Toshiba-mlc-1z-2p/FWUpdate-Toshiba-mlc-1z-2p-4dpc-2ch-NonCES.pak -> FWUpdate-Toshiba-mlc-1z-2p-1dpc-2ch-NonCES.pak
│   ├── ./Toshiba-mlc-1z-2p/FWUpdate-Toshiba-mlc-1z-2p.pak -> FWUpdate-Toshiba-mlc-1z-2p-1dpc-2ch-NonCES.pak
│   └── ./Toshiba-mlc-1z-2p/version.plist
├── ./Toshiba-tlc-1z-2p
│   ├── ./Toshiba-tlc-1z-2p/FWUpdate-Toshiba-tlc-1z-2p-1dpc-2ch-NonCES.pak
│   ├── ./Toshiba-tlc-1z-2p/FWUpdate-Toshiba-tlc-1z-2p-2dpc-2ch-NonCES.pak -> FWUpdate-Toshiba-tlc-1z-2p-1dpc-2ch-NonCES.pak
│   ├── ./Toshiba-tlc-1z-2p/FWUpdate-Toshiba-tlc-1z-2p-4dpc-2ch-NonCES.pak -> FWUpdate-Toshiba-tlc-1z-2p-1dpc-2ch-NonCES.pak
│   ├── ./Toshiba-tlc-1z-2p/FWUpdate-Toshiba-tlc-1z-2p.pak -> FWUpdate-Toshiba-tlc-1z-2p-1dpc-2ch-NonCES.pak
│   └── ./Toshiba-tlc-1z-2p/version.plist
├── ./Toshiba-tlc-1z-2p-8dpc
│   ├── ./Toshiba-tlc-1z-2p-8dpc/FWUpdate-Toshiba-tlc-1z-2p-8dpc-2ch-CES.pak
    └── ./Toshiba-tlc-1z-2p-8dpc/version.plist

Let's have a look at the head of a hexdump of any of the .pak


00000000  46 57 55 42 00 0e 40 01  00 00 00 00 00 00 00 00  |FWUB..@.........|
00000010  10 be 05 00 11 63 6f e7  c0 1f 04 00 1e 56 ea f7  |.....co......V..|
00000020  00 00 00 00 00 00 00 00  20 0d 00 00 ac c9 02 a3  |........ .......|
00000030  10 1d 00 00 8e 17 3c 92  5e 98 b3 ff 46 57 55 45  |......<.^...FWUE|
00000040  02 00 00 00 00 10 0b 51  01 17 01 64 02 08 87 04  |.......Q...d....|
00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000440  00 00 00 00 0f 09 84 54  46 57 55 65 00 00 00 00  |.......TFWUe....|
00000450  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00001000  41 54 4f 43 36 03 00 00  00 00 00 0e 40 01 00 00  |ATOC6.......@...|
00001010  40 00 00 00 00 80 80 ce  01 00 00 08 b0 21 00 f0  |@............!..|
00001020  05 00 e0 de 03 00 00 1a  d8 10 00 20 14 00 10 00  |........... ....|
00001030  00 00 45 54 4f 43 00 00  00 00 00 00 00 00 00 00  |..ETOC..........|
00001040  8e 0a 00 fa 13 00 00 ea  04 00 00 ea 4d 00 00 ea  |............M...|
00001050  31 00 00 ea fe ff ff ea  80 5e 00 ea 64 00 00 ea  |1........^..d...|
00001060  00 30 4f e1 20 00 13 e3  04 30 1e 05 02 30 5e 15  |.0O. ....0...0^.|

It looks like there are some keywords [FWUB, FWUE, FWUe, ATOC, ETOC...] in the file So far, what is interesting is to see that there is a directory called "BFH". with a file "s3e-bfh-loader.bin". "S3E" was previously in the iphone screenshot and looks like it's the code name for the NVMe cheap. When looking at it:


00000000  0e 00 00 ea fe ff ff ea  fe ff ff ea fe ff ff ea  |................|
00000010  fe ff ff ea fe ff ff ea  00 00 00 ea fe ff ff ea  |................|

That suggest that the file is in fact a raw ARM firmware with the interruption vectors. When looking from closer with IDA, we could see some interesting CMP made:

Image (Asset 16/19) alt=

As one can see, those keyword appears in little endian. So it appears that the bfh file is a bootloader that is loaded into the NVMe in order to install the .pak, as it cas clearly parse it. When paying more attention to the driver, it appears that the chip can either be in normal mode, or what is called the "BFH" mode. In fact, when in this mode, at the address BAR0 + 0x1b18, the value is 0xBFBFBFBF. The s3e binary loaded let the firmware updater tool to get the nand information and find out what file has to be loaded.


int nvme_identify(struct nvme_dev * dev) {
    int p;
    int brand;
    int tech;
    int non;
    int plane;
    int flag;
    int dpc;
    int dch;
    char directory[200];
    int dsize;
    int die;
    const char * brand_list[] = {
        "Invalid",
        "Hynix",
        "Invalid",
        "Sandisk",
        "Samsung",
        "Toshiba"
    }; //350
    const char * tech_list[] = {
        "slc",
        "mlc",
        "tlc"
    }; // 380
    const char * plane_list[] = {
        "1x",
        "1y",
        "1z"
    }; // 398
    const char * die_list[] = {
        "a",
        "b"
    }; //3b0
    const int size_list[] = {
        64,
        128,
        256,
        512,
        1024,
        2048
    };
    const char * non_list[] = {
        "Non",
        ""
    };

    u32 tst32;
    u64 tst64;
    u64 tmp;
    u64 val;

    tst32 = readl(dev - > bar + 0x1b20);
    printk("REG1B20 (nand info)  %08x\n", tst32);
    tst64 = tst32;

    if (!(tst32 >> 25)) {
        printk("val >> 25 NOK !!!\n");
        return 0;
    }

    tst32 = readl(dev - > bar + 0x1b24);
    printk("REG1B24 (nand info )%08x\n", tst32);

    tmp = tst32;
    tmp = tmp << 32;
    tst64 |= tmp;
    val = tst64;

    p = 1 << ((val >> 12) & 3);
    brand = val & 7;
    tech = (val >> 7) & 3;
    flag = (val >> 25) & 0x7F;
    plane = (val >> 3) & 0xF;
    die = (val >> 20) & 0xF;

    if (!flag) {
        dpc = 1;
        dch = 1;
        non = 0;
    } else {
        dpc = 1 << ((val >> 32) & 0xf);
        dch = 1 << ((val >> 36) & 0x3);
        non = ((val >> 38) & 1);
    }

    if (brand == 1 && tech == 1)
        sprintf(directory, "%s-%s-%s-%sdie-%dp", brand_list[brand], tech_list[tech], plane_list[plane], die_list[die], p);
    else
        sprintf(directory, "%s-%s-%s-%dp", brand_list[brand], tech_list[tech], plane_list[plane], p);

    if (!flag) {
        printk("%s/FWUpdate-%s.pak\n", directory, directory);
    } else {
        dsize = (val >> 14) & 0x3;
        if (non)
            non = 1;
        if (((size_list[dsize] >> 3) * dpc * dch) < 0x100)
            printk("%s/FWUpdate-%s-%ddpc-%dch-%sCES.pak\n", directory, directory, dpc, dch, non_list[non]);
        else
            printk("%s-%ddpc/FWUpdate-%s-%ddpc-%dch-%sCES.pak\n", directory, dpc, directory, dpc, dch, non_list[non]);
    }

    return 0;
}

At that stage, this sample function I added to my driver is capable of telling me what file is needed to be uploaded. Regarding the upload process, the address and lenght are set in 2 special registers and a 1 is sent to register 0x1b10. Once the file is uploaded, the register 0x1b14 becomes 0x10, which means that the chip is ready to continue. By writing 2 to the 0x1b10, the next step happens. This process first happen while in BFH mode, once the s3e binary is uploaded, the NVMe is identified. Once that done, the .pak is uploaded again then the upgrade can happen.







Direct Memmory Access (DMA)

First, on the Bootrom side, when the NVMe is up and running in BFH mode and waiting for the host to give it the order to initiate the DMA, we are in the loop we see in this picture. When arriving on the purpple case, we branch all the way up again if the register that tells the DMA can occur is not set.



Image (Asset 17/19) alt=

Once that being done, and if everything goes well, we end up waiting for the host to tell us that it is ready. Notice the "MOVS R0, #0x10" , "STR R0, [R5,#0x2C]" Which is where our register that tells us the DMA occured get sets to 0x10 as said previously.

Image (Asset 18/19) alt=

Finally, the DMA copies the buffer we want at the address 0x76828000 as seen in the previous picture. Then, when the host says it's ready, it jumps to this address: "MOV R0, #0x76828000", "BLX R0" By the way, on the s3e-bfh.bin loader, we can clearly see that the code relocates immediately itself to 0x00101000 (+1 for the thumb mode)

Image (Asset 19/19) alt=

The idea here would be to see if it was possible to control the NVMe over jtag in order to ask it to perform a DMA read over the PCIe Bus. In order to do so, the PCI_COMMAND_BUS_MASTER has to be set to 1. We can assume that since the chip is using remote RAM, it is allowed to act as a master over PCIe. Here is a snippet of the probing function of the kernel driver.




sstatic int bfh_probe(struct pci_dev *pdev, const struct pci_device_id *id)
{

  u64 dma_addr = 0;
  void *cpu_addr = NULL;


  pci_enable_device(pdev);
  pci_set_master(pdev);

  cpu_addr = dma_alloc_coherent(NULL, 0x10000, &dma_addr,   GFP_DMA32);
  memset(cpu_addr, 0x41, 0x10000);

  printk("ADDR mem %016llx\n", dma_addr);
  return 0;
}



Our goal here is to force the DMA to happen just by controlling the ARM of the NVMe over JTAG, in order to ask it to dump the region we alloc'd in kernel and see if we get the data out of it.

08:03.0 PCI bridge: Intel Corporation Device 1513                                                           08:04.0 PCI bridge: Intel Corporation Device 1513                                                           08:05.0 PCI bridge: Intel Corporation Device 1513                                                           09:00.0 Mass storage controller: Apple Inc. Device 2002 (rev 11)                                            root@x-MacBookPro:/home/x/mydma# lspci  -d 106b:2002 -v -xx                                                 09:00.0 Mass storage controller: Apple Inc. Device 2002 (rev 11) (prog-if 02)                                       Subsystem: Apple Inc. Device 2002                                                                           Physical Slot: 1-2                                                                                          Flags: fast devsel, IRQ 255                                                                                 Memory at a0800000 (64-bit, non-prefetchable) [disabled] [size=16K]                                         Capabilities: [40] Power Management version 3                                                               Capabilities: [50] MSI: Enable- Count=1/8 Maskable- 64bit+                                                  Capabilities: [70] Express Endpoint, MSI 00                                                                 Capabilities: [100] Advanced Error Reporting                                                                Capabilities: [148] Virtual Channel                                                                         Capabilities: [168] #19                                                                                     Capabilities: [178] Latency Tolerance Reporting                                                             Capabilities: [180] L1 PM Substates                                                                 00: 6b 10 02 20 00 00 10 00 11 02 80 01 00 00 00 00                                                         10: 04 00 80 a0 00 00 00 00 00 00 00 00 00 00 00 00                                                         20: 00 00 00 00 00 00 00 00 00 00 00 00 6b 10 02 20                                                         30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 00 00                                                                                                                                                                     root@x-MacBookPro:/home/x/mydma# emacs mydma.c                                                              root@x-MacBookPro:/home/x/mydma# make                                                                       make -C /lib/modules/4.6.0-rc7+/build M=/home/x/mydma modules                                               make[1]: Entering directory '/usr/src/linux-headers-4.6.0-rc7+'                                               CC [M]  /home/x/mydma/mydma.o                                                                               Building modules, stage 2.                                                                                  MODPOST 1 modules                                                                                           LD [M]  /home/x/mydma/mydma.ko                                                                            make[1]: Leaving directory '/usr/src/linux-headers-4.6.0-rc7+'                                              root@x-MacBookPro:/home/x/mydma# insmod ./mydma.ko                                                          root@x-MacBookPro:/home/x/mydma# lspci -d 106b:2002 -v | grep driver                                                Kernel driver in use: dmatest                                                                       root@x-MacBookPro:/home/x/mydma#                                                                            root@x-MacBookPro:/home/x/mydma# dmesg                                                                      
00:00

Powered by asciinema