1. Preface: On the shoulders of giants ...

2016.09.11

Revised 2016.09.18

Background and Context

Over the past two months, I've worked on getting a 64-bit OS working on the Raspberry Pi-3:

• I wanted to learn the ARMv8 architecture and do some assembly-language programming on the Pi-3 in the ARMv8 environment as a learning exercise.
• I would have been happy to have continued to use Raspbian as my learning environment.  But Raspbian (as of summer, 2016) runs in ARMv7 mode on the ARMv8-based Broadcom 2837 SOC processor, so you don't see the 64-bit registers and can't use the 64-bit instruction set.
• I found that 64-bit versions of Fedora 23+24 and Centos have been ported to ARMv8 (also called aarch64) [ https://www.kraxel.org/repos/rpi2/images/ ].  Look at the https://www.kraxel.org/repos/ page for other packagings.
• I had used Fedora a decade or more ago, so I chose to attempt to install 64-bit Fedora on the Raspberry Pi-3 under the mistaken assumption that it might be a familiar environment.
• I found that Will Foster had created a very helpful guide to installing 32-bit Fedora on RPis [ https://hobo.house/2016/03/13/installing-fedora-linux-on-the-raspberry-pi-3/ ].  I used that guide as the basis for installing the 64-bit version from Kraxel's repository.
• I did some customizing of the procedures because I wanted to increase /boot and /swap volumes, and I installed on 8MB and 32MB µSDs, 64MB USB drive, and 1.5TB USB-attached hard drive (rebooting from µSD's in the latter two cases).
• I installed 64-bit FC23 and FC24 many times with varying degrees of success.  The distributions evolved over the past few months, so each installation generally started from or upgraded to a new base.  Some installations resulted in non-functioning WiFi; some in non-functioning X windows (I'm using xfce4); some in odd memory-related crashes of key service apps such as polkit or Xwindows.  
• Once I have an installed system running, I generally do a "dnf update" pretty routinely, since I know that there is ongoing development and bug fixes for these new systems.  Most of the time the resulting system improved; on some occasions, the upgraded system failed in some new and mysterious way.
• Kraxel includes the Raspberry firmware updates in his updates to Fedora, so it shouldn't be necessary to run "rpi-update" (though he also says it shouldn't be a problem to do so).
• The most crippling failure was a recent update to FC24 (kernel 4.7.1) that caused polkit to crash routinely with a memory addressing problem.  Happily, SirSpudd was following Will Foster's posting and was investigating another type of failure on FC24/RPi that turned out to be related.  Read through the thread at https://hobo.house/2016/03/13/installing-fedora-linux-on-the-raspberry-pi-3/  to see our exchange about that problem, which turned out to have been caused by setting VA_BITS to 48 instead of 39 in the configuration of the kernel compilation.
• SirSpudd documented how to recompile the kernel to correct the problem.  He's an experienced Linux developer; I'm a retired chemist/CIO.  He cross-compiled for the RPi-3; I compiled on the RPi-3.  It took me many attempts at building/installing the kernel to get one that would actually boot up -- I needed more explicit instructions, and I needed instructions tailored for building on the Pi itself.
• Kraxel fixed the VA_BITS=48->39 issue as of the update to kernel 4.7.3.  That eliminated the polkit memory addressing crashes, but other components (notably xfce) continue to fail, even after that update.  The original install of 4.6.2 continues to be the most stable. 

Anyone who tried to follow our discussion and replicate for themselves might become very frustrated and give up (as I did several times).  So I'm posting this blog to document in more excruciating detail how to install and (if necessary) rebuild the Linux kernel for Fedora 24 on the Raspberry Pi-3.

Caveats

• These instructions are for Raspberry Pi-3B, not Pi-1, Pi-2, or Pi Zero.  The Pi-3B uses the ARM Cortex-A53 design, which implements the 64-bit ARMv8 architecture.  Earlier versions of the Pi use processors that implement 32-bit ARMv6 or ARMv7.  These instructions are for installing the 64-bit version of FC23 or 24 and won't work on earlier processors.
• Any mistakes in here are my fault -- either misinterpreting or erroneously revising the procedures worked out by Will Foster, SirSpudd, and others.  Or just not being clear or sufficiently explicit in my documentation.  Please let me know if you find errors or ambiguities -- I will try to correct them promptly.
• This was a learning experience for me.  I had -- still have! -- many misconceptions about how the OS boots up, how things differ (or don't) between how Raspbian and Fedora work, what config files are needed for startup, how the disks might be partitioned, etc.  I still don't understand how "dnf update" and "rpi-update" locate their repositories or how they might interfere with each other.  Those misunderstandings and misconceptions are likely to show up as incorrect statements in the documents that follow -- read and execute with a critical eye.  If it doesn't make sense based on what you know, then think it through and revise the process I suggest.  And then let me know so I can address it here.
• None of this work is my own!  Any progress I've made is based on the work I've "borrowed" from others: Kraxel, Will Foster, and SirSpudd most notably, but many other sources I've read along the way as well.  I'm simply standing on the shoulders of giants (and Linus Torvalds is at the bottom of that pile!).

Current Status (2016.09.18)

Lest you start the installs and builds assuming that all will be rosy at the end, let me burst that bubble right now!

• I have a 64-bit FC23 install with both the original 4.6.2 kernel and with updates to 4.7.4.  The 4.6.2 kernel works well:  boots up into xfce4 and WiFi and most other services work just fine. 
• I have a 64-bit FC24 install with both the original 4.6.2 kernel and with updates to 4.7.4 that performs similarly to FC23.
• However, in both cases:
• The WiFi driver times out occasionally, dropping ssh connections.
• Lengthy data transmissions (dnf, sftp) report sequence warnings for received packets -- the driver resequences correctly, but the incidents are reported.
• xfce will not start up in 4.7.4 -- reports that it can't access the server; may be a lingering polkit issue, but I haven't resolved this one yet despite having recompiled and installed polkit in the native FC24/RPi environment.  I keep both original 4.6.2 and updated (currently 4.7.4) kernels available for bootup and I use the older kernel when I want a GUI desktop.
• Firefox and Thunderbird crash before putting up windows in 4.6.2.
• I have found that the 64-bit system is slower than the 32-bit systems for the things I really am working on (some old FORTRAN code for theoretical chemistry -- but others have reported the same as a general observation).  For my work, I'll experiment with some other FORTRAN compilers and settings for floating-point arithmetic to see if I can't get the improvement I'd expect.  But at this point, don't undertake a conversion to 64-bit OS believing it'll be much faster: it probably won't be.
• However, both the FC23 and FC24 environments I have running offer the full 64-bit ARMv8 architecture, and the assembler recognizes the ARMv8 instruction set and register namings.  So the system fulfills my initial goals in implementing 64-bit Fedora on the Raspberry Pi.

What Follows

The subsequent pages in this blog document what I did to:

1. Install a 64-bit Fedora OS on the RPi-3 and 
2. Build the 64-bit kernel to eliminate the VA_BITS=48 issue that killed (at least) polkit after one of the upgrades of FC24 in the initial 4.7.1 and 4.7.2 updates (fixed starting with 4.7.3).

Comments, questions, suggestions would be welcome.