Tuesday, September 13, 2016

3. Building and installing 64-bit Fedora 24 kernel on Raspberry Pi-3

First version 2016.09.13
Rev 2016.09.18

Background

I started this page as a guide for others who wanted to build the 64-bit kernel for Fedora 24.  However, the page had a short lifetime of usefulness -- some problems that prompted me to recompile were corrected by Kraxel in bringing the update to kernel 4.7.3-2-main of 64-bit Fedora 24.   But other problems continue even with the corrected updates and in recompiled kernels.   So I'll leave the page around (and try to keep it updated) for those who might need to build the kernel on the RPi-3 in the future.

I installed 64-bit Fedora 24 on a Raspberry Pi-3B using Will Foster's guide for the 32-bit version as a procedural outline [ https://hobo.house/2016/03/13/installing-fedora-linux-on-the-raspberry-pi-3/ ].  The FC24 distribution repository version at kraxel.org, dated 20160623, installs 4.6.2 of the kernel.  After installing FC24 on a Raspberry Pi-3B, with a resulting system that was reasonably functional, I did a "dnf update" that brought the system to kernel 4.7.1 (initially -- now 4.7.4) and resulted in at least one key component (polkit) not functioning.  SirSpudd identified the problem as a change in a kernel configuration parameter (VA_BITS set to 48 rather than 39) and reported that by recompiling the kernel with the original  value (39), the polkit (and other) errors were resolved.  He also noted that it might be some time before that correction migrated into the mainstream distribution.

And he provided instructions for compiling (cross-compiling in his case) and installing the corrected kernel.  It looked fairly straightforward, so I thought I'd try it to see if the corrected kernel would eliminate some of my problems.

It has been a while since I'd compiled a Fedora kernel, and compiling for the Raspberry Pi booting environment (in which the graphical coprocessors actually load the ARM OS code) makes this installation a bit more complicated.

The result was a painful (re-)learning experience, but I did eventually compile and install a kernel that would actually boot and run Fedora 24.  And it eliminated the polkit memory addressing crashes that had made some of the Fedora services inaccessible.  It did not eliminate all my problems, but they may be caused by other issues: the obvious problems were resolved.

So, in case others get to the point where they need to compile and install the 64-bit Fedora kernel, here is an expanded version of SirSpudd's instructions, modified slightly since I compiled on the RPi3 itself rather than cross-compiled.

My Pi-3 Configuration

In case my documentation below is confusing, it might help to know the environment in which my Pi-3B operates:
  • RPi-3B running 64-bit Fedora 24 (see part 2 of this blog to set that up)
  • Boot to a Samsung 32GB µSD with FC24 and three partitions: /boot, /swap, and /
  • Logitech K400r wireless single-unit keyboard+trackpad -- uses a USB port of the Pi-3 (the only device on the USB system)
  • Samsung HDMI TV as display -- also supplies the power to the Pi-3 through its 1A USB port
  • Wired Ethernet connection to Cisco Powerline adapter (when needed)
  • Mac Mini as remote computer when needed (used to access RPi-3 via ssh/sftp) 
  • WiFi connection to Cisco Powerline adapter (once it is configured)

Context

I installed 64-bit Fedora 24 on the Pi-3 (part 2 of this blog).  I worked in multi-user mode (terminal) rather than graphical mode.

I followed SirSpudd's guide (reference hobo.house link above) except that I did a native compile on the RPi-3 rather than a cross-compile.  The RPi-3 is fast enough that with a quad-core system the compilation of the kernel requires about 50 minutes (wall clock).  I decided it would be more efficient (in my time) to build it on the RPi-3 than to install, configure, and then learn about cross-compiling on my Mac.

The process is mostly automated and quite straightforward, except at the final installation, where the make file misses some steps that are likely unique to the Raspberry Pi booting environment.  I haven't tried to amend the make file -- I just handle the steps manually (since I don't expect to do this often!).

Preparation

To start this process, you'll download a .rpm file with the source code for the kernel.  You'll need to then unpack and apply any patches before configuring and compiling the kernel.  To unpack, you can use "bsdtar" (which is not the same as "tar"; "tar" doesn't know how to read .rpm files).  And you'll need "patch".  You'll also need to have installed compilers and development headers, if you haven't done so.  So here are the steps you'll need to do in preparation for the real work:
  1. Connect your Pi-3B to the Internet (Ethernet or WiFi).
  2. Log into your own account.
  3. cd (to connect to your home directory)
  4. sudo dnf install bsdtar patch ncurses-devel
  5. sudo dnf groupinstall "system tools" "C development tools and libraries"
These are quick and straightforward.

Procedure

You'll need root access for a few of the steps, but most of this process is done in non-privileged mode from your own accountDon't put your system at risk by doing this as root.
  1. Log into your own account.
    Don't su.  You may want to do this from a remote computer via ssh into your RPi-3.  The kernel source download is about 140MB and may take a while to download.  The kernel compile itself takes about 50 minutes.  So you may want to be doing something else while waiting for those tasks to complete.
  2. Make a directory for the build and connect into it.
    cd; mkdir FC24; cd FC24
  3. Get a copy of the source repository.
    dnf download --source kernel-main
    After this step, you'll have a .rpm file in your FC24 directory with a name looking something like kernel-main-4.7.4-2.src.rpm.  Next you'll unpack that .rpm distribution file, then unzip and unpack the .tar file that contains the source code.
  4. Unpack the .rpm file and extract its contents:
    bsdtar -xvf kernel-main-x.x.x-x.src.rpm
    gunzip linux-x.x.x.tar.gz
    tar -xf linux-x.x.x.tar
  5. Next apply the patches to that source code (in the linux-x.x.x directory):
    cd linux-x.x.x
    cat ../*.patch | patch -p 1
  6. Make sure to start with a cleaned-up source tree:
    make mrproper
  7. Get a  copy of the .config file that generated your running OS:
    zcat /proc/config.gz > .config
  8. Make/remake the .config file
    I believe that make wants to validate the configuration and write its own copy before you can actually start building the kernel.   If you just do a make, it'll run the configuration program and then start the build.  But you might want to double-check the .config file before starting the hour-long build, so I've divided this into steps.   The simplest approach is to simply validate the .config file, verify it visually, and then do the make.  If you want to use make menuconfig here instead so you get an item-by-item check-box-style menu for selecting configuration parameters, you'll need ncurses-devel to have been installed, and we already did that under Preparation.  So this step is simply:
    make oldconfig
    unless you want to do the complete configuration setup yourself, in which case you'd use make menuconfig.  At the end of this step, the configuration program will tell you that it has written the configuration to .config.
  9. Label your kernel and verify the .config file.
    You'll want to give your kernel a local version number that identifies it as yours rather than from the repositories, and you may want to verify some of the parameter settings. I went through this exercise because VA_BITS was set to 48 rather than 39, so I check the .config file to verify that the parameter is set correctly before proceeding with the build.  At the very least, edit the file, locate the string CONFIG_LOCALVERSION=, and change the distribution value (something like "-2-main") to an identifying string in place of "main" that will be unique to you (something like "-2-yih3" -- using Your Initials Here and a sequence number).  While you're in the file, you might check around to confirm that other parameters are set as you intended, but don't arbitrarily change parameters:  Go through make menuconfig if you want to make other changes as there may be inter-dependencies of which you're unaware.
  10. Build the kernel
    I like to track the time it takes to build the kernel, so I time the build.  Omit it if you don't care. Since the RPi-3 has 4 cores and I don't do any other work during the build, I tell make to keep 4 streams running (the -j 4 part).
    time make -j 4
    It'll take about 50 minutes to complete.
  11. Make the modules
    make modules
  12. Install the kernel and modules (most of it)
    This part has to be done as root, hence the sudo's.
    sudo make modules_install
    sudo make install
    These steps install most of what's needed in /boot and in /lib/modules/.   If you ls /boot, you'll see some files there identified with your unique identifier (e.g., "-2-yih3"). Ditto if you check /lib/modules.

    But unfortunately the makefile misses a couple of steps for the RPi, and we need to finish up manually. The key parts are:
    • Copy the device tree directory from the build tree to /boot
    • Copy the bcm2837 device tree file to a place bootcode will find it
    • Edit /boot/extlinux/extlinux.conf to have bootcode let you choose your new kernel when doing the boot

    Kraxel pointed out that there are a couple of lines we can add to /boot/extlinux/extlinux.conf that will allow you to choose at boot time the kernel you want to run. We'll install two lines at the beginning of that file, just once, and then we can choose whether to boot our new kernel or revert back to the older, distribution version -- always a helpful feature in case your new version doesn't boot correctly.  With your LOCALVERSION identifier that you edited into the .config file, the build process created a boot image and related files with a corresponding unique image identifier, something like 4.7.3-2-yih3, that includes the kernel version from the distribution appended with your unique identifier.   Key files and directories will use that identifier in their names.  In the steps below, we'll use the string "<imageid>".  Use your unique image identifier, 4.7.3-2-yih3 or whatever, wherever you see "<imageid>" below.
  13. Copy the device tree and bcm2837 .dtb file to expected locations
    cd /boot/dtbs
    sudo mkdir <imageid>
    cd <imageid>
    sudo cp -rL <your home>/FC24/linux-x.x.x/arch/arm64/boot/dts/* .
    sudo cp broadcom/bcm2837-rpi-3-b.dtb .
  14. Tell bootcode about the new kernel
    sudo edit /boot/extlinux/extlinux.conf
    Just by looking at that file, you'll see what the general layout of a boot kernel description entry is:
    label kernel-4.7.3-2-main
      kernel /vmlinux-4.7.3-2-main
      initrd /initramfs-4.7.3-2-main.img
      fdtdir /dtbs/4.7.3-2-main/
      append dwc_otg.lpm_enable=0 console=ttyAMA0,115200 \ 

        console=tty1 elevator=deadline root=/dev/mmcblk0p3 \
        rootfstype=ext4 rootwait
    You're just going to replicate that at the bottom of the file with the <imageid> of your new kernel image as the kernel name.  So open the file with your favorite editor, and before the first line, insert the following just the first time you edit the file:
    menu title pick kernel 
    timeout 100

    Now go to the bottom of the file and insert the description of your new kernel with your unique identifier (something like 4.7.3-2-yih3) in place of <imageid>:
    label kernel-<imageid>
      kernel /vmlinuz-<imageid>
      initrd /initramfs-<imageid>.img
      fdtdir /dtbs/<imageid>/
      append dwc_otg.lpm_enable=0 console=ttyAMA0,115200 \
        console=tty1 elevator=deadline root=/dev/mmcblk0p3 \
        rootfstype=ext4 rootwait

    NOTE that the last line ("append ...") should all be on one line of the file (no continuation "\").

    ALSO NOTE that the kernel file is named vmlinuz rather than vmlinux: the make install gets the compressed boot file and names it as a compressed image when it copies it to /boot.
  15. Save the file and exit the editor.
  16. Double check
    cat /boot/extlinux/extlinux.conf and make sure that each of the files or directories mentioned in the section for your kernel is actually in place in /boot, and make sure that the kernel module directory is in place as /lib/modules/<imageid>.
  17. Reboot.  Issue a "sudo reboot" command to initiate the reboot.  You'll now get a menu allowing you to pick the kernel you want to use: type in the number of the kernel with your unique identifier.  After you've booted up and logged in, verify that your kernel is running by typing uname -a and make sure the name string has your <imageid> in it.  
    • If it booted up but didn't boot into your kernel, repeat the double-check in 16: typographical errors are easy to make but fatal to the boot process.  
    • If it appears that everything is in place but your kernel isn't coming up, watch the beginning of the boot process carefully to see if you can determine which step is failing, then correct it. 
    • If the boot process is hanging completely when you specify your kernel, reboot by power cycling and select a distribution kernel (rather than the one you just built).  Again, do the double-check in 16.  If there are no typos, review your configuration (make menuconfig) to make sure you have a valid configuration.  Compare your .config with /proc/config.gz to identify inadvertent changes.  And, finally, go back to "make mrproper" and start again.

1. Preface: On the shoulders of giants ...

2016.09.11
Revised 2016.09.18

Background and Context

Over the past two months, I've worked on getting a 64-bit OS working on the Raspberry Pi-3:
  • I wanted to learn the ARMv8 architecture and do some assembly-language programming on the Pi-3 in the ARMv8 environment as a learning exercise.
  • I would have been happy to have continued to use Raspbian as my learning environment.  But Raspbian (as of summer, 2016) runs in ARMv7 mode on the ARMv8-based Broadcom 2837 SOC processor, so you don't see the 64-bit registers and can't use the 64-bit instruction set.
  • I found that 64-bit versions of Fedora 23+24 and Centos have been ported to ARMv8 (also called aarch64) [ https://www.kraxel.org/repos/rpi2/images/ ].  Look at the https://www.kraxel.org/repos/ page for other packagings.
  • I had used Fedora a decade or more ago, so I chose to attempt to install 64-bit Fedora on the Raspberry Pi-3 under the mistaken assumption that it might be a familiar environment.
  • I found that Will Foster had created a very helpful guide to installing 32-bit Fedora on RPis [ https://hobo.house/2016/03/13/installing-fedora-linux-on-the-raspberry-pi-3/ ].  I used that guide as the basis for installing the 64-bit version from Kraxel's repository.
  • I did some customizing of the procedures because I wanted to increase /boot and /swap volumes, and I installed on 8MB and 32MB µSDs, 64MB USB drive, and 1.5TB USB-attached hard drive (rebooting from µSD's in the latter two cases).
  • I installed 64-bit FC23 and FC24 many times with varying degrees of success.  The distributions evolved over the past few months, so each installation generally started from or upgraded to a new base.  Some installations resulted in non-functioning WiFi; some in non-functioning X windows (I'm using xfce4); some in odd memory-related crashes of key service apps such as polkit or Xwindows.  
  • Once I have an installed system running, I generally do a "dnf update" pretty routinely, since I know that there is ongoing development and bug fixes for these new systems.  Most of the time the resulting system improved; on some occasions, the upgraded system failed in some new and mysterious way.
  • Kraxel includes the Raspberry firmware updates in his updates to Fedora, so it shouldn't be necessary to run "rpi-update" (though he also says it shouldn't be a problem to do so).
  • The most crippling failure was a recent update to FC24 (kernel 4.7.1) that caused polkit to crash routinely with a memory addressing problem.  Happily, SirSpudd was following Will Foster's posting and was investigating another type of failure on FC24/RPi that turned out to be related.  Read through the thread at https://hobo.house/2016/03/13/installing-fedora-linux-on-the-raspberry-pi-3/  to see our exchange about that problem, which turned out to have been caused by setting VA_BITS to 48 instead of 39 in the configuration of the kernel compilation.
  • SirSpudd documented how to recompile the kernel to correct the problem.  He's an experienced Linux developer; I'm a retired chemist/CIO.  He cross-compiled for the RPi-3; I compiled on the RPi-3.  It took me many attempts at building/installing the kernel to get one that would actually boot up -- I needed more explicit instructions, and I needed instructions tailored for building on the Pi itself.
  • Kraxel fixed the VA_BITS=48->39 issue as of the update to kernel 4.7.3.  That eliminated the polkit memory addressing crashes, but other components (notably xfce) continue to fail, even after that update.  The original install of 4.6.2 continues to be the most stable. 

Anyone who tried to follow our discussion and replicate for themselves might become very frustrated and give up (as I did several times).  So I'm posting this blog to document in more excruciating detail how to install and (if necessary) rebuild the Linux kernel for Fedora 24 on the Raspberry Pi-3.

Caveats

  • These instructions are for Raspberry Pi-3B, not Pi-1, Pi-2, or Pi Zero.  The Pi-3B uses the ARM Cortex-A53 design, which implements the 64-bit ARMv8 architecture.  Earlier versions of the Pi use processors that implement 32-bit ARMv6 or ARMv7.  These instructions are for installing the 64-bit version of FC23 or 24 and won't work on earlier processors.
  • Any mistakes in here are my fault -- either misinterpreting or erroneously revising the procedures worked out by Will Foster, SirSpudd, and others.  Or just not being clear or sufficiently explicit in my documentation.  Please let me know if you find errors or ambiguities -- I will try to correct them promptly.
  • This was a learning experience for me.  I had -- still have! -- many misconceptions about how the OS boots up, how things differ (or don't) between how Raspbian and Fedora work, what config files are needed for startup, how the disks might be partitioned, etc.  I still don't understand how "dnf update" and "rpi-update" locate their repositories or how they might interfere with each other.  Those misunderstandings and misconceptions are likely to show up as incorrect statements in the documents that follow -- read and execute with a critical eye.  If it doesn't make sense based on what you know, then think it through and revise the process I suggest.  And then let me know so I can address it here.
  • None of this work is my own!  Any progress I've made is based on the work I've "borrowed" from others: Kraxel, Will Foster, and SirSpudd most notably, but many other sources I've read along the way as well.  I'm simply standing on the shoulders of giants (and Linus Torvalds is at the bottom of that pile!).

Current Status (2016.09.18)

Lest you start the installs and builds assuming that all will be rosy at the end, let me burst that bubble right now!
  • I have a 64-bit FC23 install with both the original 4.6.2 kernel and with updates to 4.7.4.  The 4.6.2 kernel works well:  boots up into xfce4 and WiFi and most other services work just fine. 
  • I have a 64-bit FC24 install with both the original 4.6.2 kernel and with updates to 4.7.4 that performs similarly to FC23.
  • However, in both cases:
    • The WiFi driver times out occasionally, dropping ssh connections.
    • Lengthy data transmissions (dnf, sftp) report sequence warnings for received packets -- the driver resequences correctly, but the incidents are reported.
    • xfce will not start up in 4.7.4 -- reports that it can't access the server; may be a lingering polkit issue, but I haven't resolved this one yet despite having recompiled and installed polkit in the native FC24/RPi environment.  I keep both original 4.6.2 and updated (currently 4.7.4) kernels available for bootup and I use the older kernel when I want a GUI desktop.
    • Firefox and Thunderbird crash before putting up windows in 4.6.2.
  • I have found that the 64-bit system is slower than the 32-bit systems for the things I really am working on (some old FORTRAN code for theoretical chemistry -- but others have reported the same as a general observation).  For my work, I'll experiment with some other FORTRAN compilers and settings for floating-point arithmetic to see if I can't get the improvement I'd expect.  But at this point, don't undertake a conversion to 64-bit OS believing it'll be much faster: it probably won't be.
  • However, both the FC23 and FC24 environments I have running offer the full 64-bit ARMv8 architecture, and the assembler recognizes the ARMv8 instruction set and register namings.  So the system fulfills my initial goals in implementing 64-bit Fedora on the Raspberry Pi.

What Follows

The subsequent pages in this blog document what I did to:
  1. Install a 64-bit Fedora OS on the RPi-3 and 
  2. Build the 64-bit kernel to eliminate the VA_BITS=48 issue that killed (at least) polkit after one of the upgrades of FC24 in the initial 4.7.1 and 4.7.2 updates (fixed starting with 4.7.3).
Comments, questions, suggestions would be welcome.

2. Installing 64-bit Fedora 24 on a Raspberry Pi-3

2016.09.13
Revised 2016.09.18


This post is a guide for installing 64-bit Fedora 23 or 24 on a µSD card for the Raspberry Pi-3B.  These instructions are modifications for 64-bit Fedora of the instructions created by Will Foster for 32-bit Fedora [ https://hobo.house/2016/03/13/installing-fedora-linux-on-the-raspberry-pi-3/ ]: all credit for intellectual content goes to him.  This page simply provides the modifications (and cautions) for the 64-bit Fedora install.  Please refer to that site for more information and for prior work on debugging the 64-bit install process.

Please let me know if there are errors or ambiguities in the instructions below and I will correct them as quickly as I can -- even quicker if you provide suggested language!  I think comments work on this blog site, but if not, email me at hdtodd@gmail.com.

Assumed working environment:

  1. A standard OSX Mac or Linux PC with SD slot and a SD adapter card for a µSD card, and a WiFi or Ethernet connection to the Internet.  You can use a Raspbian-based Raspberry Pi as the workstation for this if you have the USB port available and the µSD-USB adapter described under tools.  (I would think these instructions could be tailored for Windows, but I don't have a Windows systems and so couldn't document the process.)
  2. Comfort using command-line tools on Linux and/or Mac OSX. 
  3. A Raspberry Pi-3B (no earlier version will work) with power supply, keyboard, display, µSD disk.  Mouse optional (but required if you want to use a graphical display system).
  4. Ethernet connection and cable  to connect the Pi-3B to the Internet
  5. WiFi for the Pi is optional

Helpful tools:

  1. µSD (micro SD card) USB adapter that will allow you to insert your µSD card into the adapter, then insert the adapter into a USB port, and have the computer then be able to access that µSD card.

Warnings

As of 2016.09.12, there are problems with systems that result from updates to the Fedora 24 Raspberry Pi distribution after the initial install.  Those problems limit the functionality of the updated system (4.7.1 to 4.7.4, at last check).
  1. The 20160623 distribution (step 1 below) installs kernel 4.6.2-4-main.  That system functions reasonably well: WiFi works, xfce GUI desktop functions, etc.  Compilers, assemblers, linkers, etc all function.  Firefox and Thunderbird crash before they can bring up a window.  There may be problems in other applications.
  2. I understand from Kraxel that the RPi firmware is updated as part of the "dnf update" process. So doing "rpi-update" is not necessary.  I was concerned that it may, in fact, cause problems with some of the firmware as it supports Fedora.  So I've omitted the step of performing an "rpi-update".
  3. The 4.7.3-2-main  and subsequent updates fix the VA_BITS=48-->39 problem that caused polkit memory addressing crashes.  
  4. If you do a "dnf update", as of 2016.09.12, you'll update to kernel 4.7.3-1-main (or subsequent update).  That update breaks a number of things (as do previous updates I had tried earlier).  The VA_BITS=48-->39 .config parameter has been corrected, so "polkit" no longer crashes with memory addressing problems.  But other components continue to fail.  "systemctl default" fails; xfce4 simply hangs the system (and trying to boot with the graphical interface set as the default hangs without giving a prompt).  If you do the "dnf update" and get to the point where you can't boot into 4.7.x, boot back to 4.6.2-4-main. The instructions below will give you a system that puts up a boot menu so you can select the kernel you want to run.
  5. If you simply want to bring up 64-bit Fedora 24 with a GUI desktop that functions as a stable Fedora system, I would recommend not doing the "dnf update" in step 11.  Rather, perform the initial installation, then
    dnf groupinstall "xfce desktop"
    You can then use the GUI tools to configure your WiFi connection, and most of the Fedora functionality will be operating.  Watch the blogs to wait for others to signal that the updates no longer break key Fedora services.

Installation Procedure

  1. Download FC23 or FC24 64-bit OS distribution images from here: https://www.kraxel.org/repos/rpi2/images/ .  I used the file https://www.kraxel.org/repos/rpi2/images/arm64-rpi3-f24-mainline-20160623.raw.xz, but you should use a more recent version if there is one.  What you get is a .xz zip file.  You can unzip it with gunzip on the Mac to get a .raw file.  If you rename it to .img on the Mac and double click on it, OSX will open the vfat partition and mount it.  It can’t do that with the ext4 partition since it doesn’t know about them.  Or if you copy the image (or download it to) a Linux system, you can double-click the .img file to  mount both partitions in order to poke around and see what’s in them.  Dismount the partitions before the next step.  On the Mac, if you've mounted it and it's listed as disk2, use the command diskutil list to verify that!!), you can umount with the command:
    sudo diskutil unmountDisk disk2
  2. Copy the unzipped image file to the µSD drive.  You'll need to be administrator or root to do this.  On the Mac or Linux PC, install a blank µSD at least 8GB in capacity; note the device name (something like /dev/disk2 on the Mac or /dev/sdb on RPi — but be sure you get the drive identity correct before you do this step or you may blow away your Mac OS system!!), and then issue the command:
    sudo dd bs=1m if=arm64-rpi3-f24-mainline-20160623.raw \
       of=/dev/rdisk2
    Note the output is to "rdisk2" rather than "disk2".  Again, if you're on a Mac, it will auto-mount the first partition when the copy has been completed; you'll need to issue the following command to unmount the µSD:
    sudo diskutil unmountDisk disk2
  3. Three partitions on the µSD.  The result of this is that you will have 3 partitions on the µSD.  If you insert it into a Linux (RPi) system on which it is the second attached USB drive (hence is "sdb" rather than "sda"), you'll see:
    1. /dev/sdb1: a vfat partition that will be “/boot” when you start up the RPi
    2. /dev/sdb2: “/swap”
    3. /dev/sdb3: an ext4 partition that will be “/” when you start up the RPi
  4. Check and expand the µSD file system.  The third partition is only about a GB in size, corresponding to the size in the distribution version.   We need to expand it.   You will need to be on a RPi or other Linux system – OSX does not have the tools.  Insert the µSD into a Linux system (e.g., Raspbian, but it doesn’t need to be an RPi-3); don’t mount the partitions and umount them if they've been auto-mounted.  Use the following steps to check the file system and then expand “/” on the µSD to fill the remaining drive (I’m assuming it’s /dev/sdb, but check its identity on your system or you may blow away your operating system!):
  5. sudo e2fsck -f /dev/sdb3
    sudo resize2fs /dev/sdb3
  6. Boot your new Fedora system on the RPi-3.  Insert the µSD as the boot drive on a RPi-3 and boot it up.   It will boot up once, do some configuring, resize "/", and then reboot itself.   Don’t panic!  Just be patient … and it’s fairly fast.   It’ll quickly come back with a login prompt.
  7. Log in as “root”, password “pi”.
  8. Streamline the security environment.  As shipped, FC23 installs SELinux in “permissive” mode.  Selinux may cause problems with operation of some apps — at least it looked to be implicated in some things not running because they didn’t have permissions to access or modify files.  And Will Foster (hobo) reported that SELinux is known to interfere with operations in Fedora.  So on the first boot up, edit the configuration file to disable it:
    vi /etc/selinux/config and change “permissive” to “disabled”.  Exit vi (“:wq”) and reboot.  If you don’t know how to use vi, wait ’til you’ve installed a text editor you do know (eg, nano) and do it then.  And once your tools are all in place and the system is stabilized, you may want to go back and revert back to "permissive" .
  9. Set the host name of your system:
    echo MyHostName > /etc/hostname
    (use your preferred host name in place of “MyHostName”)
  10. Familiar tools aren't available yet.  A number of tools you’re likely to want to use aren’t installed in the distribution version.  But that’s OK: you need to update software to stabilize the system and then you can download apps and get the system you want.
  11. Manually configure your WiFi if necessary.  You will likely find it easier to simply plug into an Ethernet port to get started, but if that's difficult, you might try configuring WiFi manually.  This procedure is adapted from [ https://www.blackmoreops.com/2014/09/18/connect-to-wifi-network-from-command-line-in-linux/ ] and has worked for me, though I had to delete the manual configuration once I had the GUI interface working and wanted to manage via the network widget.
    1. Verify that you have a wlan device with the command:
      iw dev
    2. Check the wlan device status:
      ip link show wlan0
      Should be in the DOWN state
    3. Bring up the wlan0 interface:
      ip link set wlan0 up
      and then verify that it is up:
      ip link show wlan0
      should now show wlan0 in the UP state.
    4. Check the connection status -- shouldn't be connected:
      iw wlan0 link
    5. Scan to find the WiFi network you want.  Skip this step if you know the SSID of the network you want to join:
      iw wlan0 scan
    6. Assuming it's a WPA password protected network,
      wpa_passphrase YourNetworkSSID >> /etc/wpa_supplicant.conf
      (now type the password for that network -- no prompt for this)
      wpa_passphrase will create the network entries for your network, appending to any prior entries (not replacing). Verify by
      cat /etc/wpa_supplicant.conf
    7. Connect to the network:
      wpa_supplicant -B -D wext -i wlan0 -c /etc/wpa_supplicant.conf
    8. Verify that you've connected:
      iw wlan0 link
    9. Get an IP address through DHCP:
      dhclient wlan0
    10. And verify that you've gotten an IP and are connected to your wireless network and have your routing set up:
      ip addr show wlan0
      ifconfig wlan0
      ip route show
    11. Check it by pinging a large Internet host:
      ping 8.8.8.8
  12. Connect your RPi3 to the Internet via the Ethernet port.  If you’re skilled at getting WiFi configured via command line editing, go ahead and do that and see if you can get it to connect to the network, but the easiest way will likely be to plug the RPi3 into an Ethernet connection. If you do that, Fedora will tell you it found a link and made a connection.   Some times it even told me the IP address, but generally it doesn’t.  And “ifconfig” isn’t there yet.  But if you have a way of identifying the IP assigned to it (e.g., by looking at the DHCP assignments made by your router or by ssh'ing to another system and typing "who" to see your source IP), then you can ssh into it from another system (e.g., from another PC type ssh root@192.168.1.101, password is pi ). You may find it convenient to do much of the following work from another system, since you’ll spend a good deal of time waiting for downloads and installs to complete and you can do other things if you’re working from another system.  You'll probably want to sftp to some other host on your net and copy over the /etc/hosts file and install it in your RPi /etc directory so that you can connect to your other network hosts by name.
  13.  Install some essential tools.  Issue the commands:
    dnf install binutils nano net-tools wget
    dnf groupinstall "administration tools" "system tools"
    If you’re an emacs user, you can
    dnf install emacs-nox
    at this point, too, or other tools that you use frequently.
  14. Reboot.  Now many of the tools you might want are available. Your favorite editor should be there (or install it with dnf if it isn’t).  “ifconfig” should be there and will show that you have an Ethernet connection and that wlan0 is there and functioning (but not configured and so not connected to an access point).
  15. Secure your system and set up a user account.  Your preferences may vary, but at least log in as root, password=pi, and type:
    passwd
    And then enter a new password for root.  I prefer to set up a group and account for myself using the same uid/gid I use on other systems, to make nfs connections less troublesome:
    groupadd -g 502 YourUserGroup (your group id and group name as you prefer) useradd -b /home -g YourUserGroup -G root -m -s /bin/bash \
        -u 502 YourUserName
    (again, use your preferences for gid, shell, uid, and username).
    Type
    passwd YourUserName (your name here)
    and enter a password for your personal account.  And finally, I like to work in my own account but have easy access to su when I need it, so if you prefer to work that way, enter the command:
    visudo
    then arrow down to
    ## Allow root to run any commands anywhere
    root ALL=(ALL) ALL
    go to the next line (blank), type “i” to begin inserting, and type in
    YourUserName ALL=(ALL) ALL
    press the ESC key to end the insert, then type “:wq” to save the file and exit.  If you do this, you can now log in as yourself to do most of your work but issue a “sudo" to do a specific administrative task or “sudo -s” whenever you need to jump into root for longer periods.
  16. Install a GUI desktop.  There are now tools installed that would let you configure your WiFi connection from the command line, but you’re likely to want a GUI desktop interface anyway, so it may be easier to install that first.   Choose the one you want.  I found that xfce4 works pretty well — lightweight but familiar look.  You can see what’s available with the command:
    dnf group list
    which will show a number of different application suites as well as desktop systems.   Let’s say you would like to try xfce.  Issue the command:
    sudo dnf groupinstall "xfce desktop"
    After the download and install have finished, reboot.   It will come up with a graphical login screen.  Login as yourself.   For some reason, presumably distribution configuration, the system doesn’t start NetworkManager automatically initially, and as a result the WiFi configuration manager toolbar widget won’t start up.  So open a terminal window and issue the commands:
    sudo systemctl enable NetworkManager
    sudo systemctl start NetworkManager
    Now double-click the network widget icon in the upper right-hand toolbar.  It’ll bring up your connection options.  Ethernet should already be connected; Wifi won’t be.  Go to the left side of that toolbar, click “Applications” to bring down the menu, click “system”, and select “network”.  Now tell it you want to add a WiFi interface, wlan0 as the device, and fill in your SSID and authentication information and tell it to connect.  At this point, you should be able to manage your network connections through the icon on the right side of the top toolbar.

    If you prefer not to have a graphical login screen but want to stick with command line for most of your work, type: 
    sudo systemctl set-default multi-user.target
  17. Add your favorite programs.  Now you can load programs to tailor the system to your own needs. Use
    dnf list | grep MyFavoriteApp
    to identify a specific program that you’d like to install, then
    dnf install MyFavoriteApp
    to actually install it.   For larger packages that involve numerous programs, you can use
    dnf group list
    to identify other packages you’d like to install.   If you want C and associated development tools, for example, issue the command:
    dnf groupinstall "C Development Tools and Libraries"

Detritus

These are steps that we defer for now:
  1. Defer this step until the update doesn't break xfce.  Update the OS distribution.  You’ll use “dnf” — a YUM extension, equivalent to "apt" in Raspbian — to do the updates and installs.   Log in as root, password pi, either on the console or over the network. Issue the command:
    dnf update
    and respond "y" to let it install the updates.
  2. This step not recommended for now but left for possible future use.  Install the firmware updater and update the RPi firmware.  The RPi3 uses firmware in the /boot drive to actually load the OS, and that gets updated occasionally, too.   The tool to do that is called “rpi-update”.   We have to download and install it, first, since it’s not part of the Fedora distribution: 
    wget -O /usr/bin/rpi-update \
        https://raw.githubusercontent.com/Hexxeh/rpi-update/master/rpi-update
    chmod +x /usr/bin/rpi-update
    rpi-update
    Reboot

Rebooting to an attached USB drive

I have not been able to get the new booting process to boot Fedora directly from an attached USB drive (process as described in the MSB site (https://www.raspberrypi.org/blog/pi-3-booting-part-i-usb-mass-storage-boot/).  So if you want to install 64-bit Fedora on an external USB-attached hard drive or thumb drive, you'll need to have the boot process transfer from the µSD to the USB device.  The setup process isn’t terribly complicated, but there are too many variations and possible complications to give a single list of instructions, so I'll just sketch the ideas out here.  The description in this section assumes that you have some experience doing systems work in Linux — you know how to use fdisk, dd, and friends, for example, and you know that the results can be catastrophic if you screw up.   The general idea is to tell the booting firmware, which is located on the µSD (/dev/mmcblk0) to redirect its booting process to a partition on the USB thumb drive or USB-attached hard drive (/dev/sda, for example).   In Raspbian, you do that in /boot/cmdline.txt by changing the reference to “/dev/mmcblk0p3” to point to “/dev/sda3” (for example).  In Fedora (or maybe later versions of Raspbian … not sure where the change came from), you change that reference from mmcblk0p3 to sda3 in /boot/extlinux/extlinux.conf .  But before you do that, you need:
  1. fdisk /dev/sda (assuming that’s your new boot device) to create the partitions you want. I made a copy of /dev/mmcblk0p1 (the µSD /boot) to /dev/sda1 ; I made /dev/sda2 be “/swap”; I made /dev/sda3 be “/”.   There’s no real reason to have /boot on the hard drive (RPi boots its firmware from the µSD, and having /boot point anywhere other than /dev/mmcblk0p1 causes other problems when you update firmware … the real boot partition, on mmcblk0, doesn’t get updated by rpi-update, for example!).  But I use the hard-drive first partition /boot to keep an archival copy of a working /boot from mmcblk0p1.
  2. However you choose to set up your partitions, identify which one is going to be “/swap” (you probably really do want that to be on your hard drive!) and issue a mkswap /dev/sda2 (for example)
  3.  Identify which partition is going to be “/”, and create the file systems on it with "mkfs -t ext4 /dev/sda3" (for example)
  4.  Mount the hard drive “/” partition.  For example, mkdir /mnt/hdslash and then mount the partition with "mount /dev/sda3 /mnt/hdslash".   Copy the system from “/” on /dev/mmcblk0p3 to what will become the new “/” on /dev/sda3.  But if you try to copy the live version of “/” (as the system is in operation), you’ll get lots of error messages and likely (since I’ve never let it complete) to be unsuccessful.   This copying is best done by attaching the hard drive and your new FC23 µSD to a functioning Linux system and copying.  This is the tricky part.  If you get this wrong, you blow away your Linux OS or your new FC23 µSD drive.  (How might I know that, you ask?   Experience.  Typing “/dev/sda” instead of “/dev/sdb” can be a fatal mistake).   You need to be careful to identify your source and destination drives (they’re likely to be /dev/sda and /dev/sdb, but which is which? Better be sure!).
  5. The last change you’ll need to make is in the /etc/fstab on the hard drive to identify /boot, /swap, and /.   And confirm that the file /boot/extlinux/extlinux.conf (make sure it’s the one on /dev/mmcblk0p1) points to /dev/sda3 (for example) rather than /dev/mmcblk0p3.
  6. Put your µSD card in your RPi3, attach and turn on your hard drive, and power up your RPi3.  With any luck (or about the 5th time you try it), you’ll see the initial firmware boot off your µSD card, then see it load the OS from the hard drive.
  7. Remember: if you identify in /etc/fstab that /boot is on your hard drive, any software updates (dnf or rpi-update) will only update that version, not the one you actually boot from on /dev/mmcblk0p1.  After an update, you’ll need to copy the updated boot files back to that µSD partition.  And you can get version skew between /boot and the OS files in / on that µSD, so you have to be careful to keep things in sync.   While having /boot point to /dev/sda1 (for example) is manageable, I've found it's easiest to just leave /boot pointing to /dev/mmcblk0p1 and not mount the /dev/sda1 as /boot any more -- just use it to back up /boot.
  8. There are numerous variations in this process that you might choose, depending on how you intend to use your system, the capacity of your hard drive, etc.  One of the simplest would be to connect your hard drive to a Linux system and power it up; assuming it’s /dev/sda (CHECK THAT TWICE!), use the dd command and then the e2fsck and resize2fs commands from the beginning of these instructions to copy the repository distribution image to that hard drive.  Edit /dev/mmcblk0p1/boot/extlinux/extlinux.conf to refer to /dev/sda3 and edit the /dev/sda3/etc/fstab to mount /dev/sda2 as /swap and /dev/sda3 as /.  You’ll end up with a small swap space, and you’ll need to update and install software on that new hard drive since you won’t have copied the software from your µSD, but it’ll be a much faster process and less error prone.   I haven’t done it this way (yet), but I’ll try it on my next install and append to this if it’s as straightforward as I think it will be.
The instructions above assume you're installing on a µSD card, but a USB thumb drive or hard drive will work just fine.  If you use a USB drive, though, you'll still need a µSD card on which you can install the boot code for Fedora on the Pi -- a copy of the /dev/sda1 partition from the USB drive.  I have not (yet) been able to get the Pi-3B to boot off the USB drive directly, probably because the newly-implemented MSD boot mechanism is designed for Raspbian.