Thursday, September 21, 2023

Backup and restore a Linux installation

Recently, I upgraded my HP Omen 2017 laptop NVME SSD to a bigger one, the original SSD was a Toshiba SSD 256GB, PCIe 3.0, replaced by a Crucial P5 Plus 2TB NVME SSD PCIe 4.0. This kind of old laptop has PCIe 3.0 system but since PCIe is backward compatible against it's predecessor, it's still able to run PCIe 4.0 with of course at PCIe 3.0 speed. The old SSD has left around 40GB or 15% of empty spaces, despite how smart I manage to keep the empty spaces, I felt exhausted to keep fight on the limitation. I decided it's time to ugprade the old SSD, thinking of getting more bigger spaces, faster video editing and reduced gaming load time advantages, etc. Anyway, this HP Omen laptop also has a pre-installed 2.5" 1TB harddisk which I use to store larger files such as videos, games and backups.

Backup and restore Windows system image

As usual, this laptop come with pre-installed Windows 10, my upgrade procedure was by creating a system image from the old SSD then restore the image to the new SSD, I used a Seagate 1TB Expansion Portable Drive (USB interface) to store the image. After the system image backup and restore, I had to expand the main Windows partition to be bigger size then create (or move to be precisely) a new Windows recovery partition as the previous recovery partition was overlapped by the expansion.

Backup the root filesystem

The laptop is UEFI compatible and also Windows 10 system partition is EFI. When expanding the Windows partition, I left some gigabytes for Linux partitions, they are the EFI boot partition and root partition. I already have Fedora 38 installation on my other system so I thought to copy the Fedora install instead of install a fresh Fedora 38 then messing up with apps and settings I have to install or set later. My Fedora filesystem is pretty well organized, such as I don't store larger files, projects, etc. in the main partition, but mount other parititions and set the mount points into the root filesystem's directories such as /data mount point. This strategy allows me to keep root filesystem clean from contents that outside system scope. To backup my another Fedora root filesystem, I booted the system from a USB stick drive that has Fedora 38 live install, then attach my Seagate USB drive and run these following commands as superuser.

Create external USB drive mount point and mount it, if not already mounted by desktop manager.

mkdir /mnt/external_drive
mount /dev/sdb1 /mnt/external_drive

In these following commands, the root partition is /dev/nvme0n1p2 and the EFI partition /dev/nvme0n1p1. The root filesystem is EXT4 and the EFI is FAT16.

Mount EFI filesystem and make the tarball.

mkdir /mnt/efi
mount /dev/nvme0n1p1 /mnt/efi
cd /mnt/efi
tar cf /mnt/external_drive/efi.tar .

Mount root filesystem and make the tarball.

mkdir /mnt/rootfs
mount /dev/nvme0n1p2 /mnt/rootfs
cd /mnt/rootfs
tar --same-owner --same-permissions -cf /mnt/external_drive/rootfs.tar .

Note that --same-owner and --same-permissions arguments are necessary to preserve files's (and directories) owner and permission upon restoration (that is, extracting the tarball). For the EFI partition, we don't need to preverse owner and permission since it is a FAT16 filesystem. Optionally, compression can be applied with longer tar's time caveat.

Unmount all mount points.

umount /mnt/rootfs
umount /mnt/efi
umount /mnt/external_drive

Restore the root filesystem

Now I have Fedora 38 root filesystem and EFI tarballs from another system and ready to be restored in the HP Omen laptop. I still need the Fedora 38 Live USB stick to boot and of course the Seagate Portable Drive. I created 2 additional partitions in the new SSD, first is the EFI partition (FAT16) and second is Linux root partition (EXT4). I don't go into detail how to create these partitions and formatting them, the tools are just gparted and mkfs. The HP Omen's EFI partition is /dev/nvme0n1p4 and Linux root partition is /dev/nvme0n1p5, the Seagate drive is /dev/sdc1.

Here all the commands, basically they are just the same as in backup section, only differs in partition name.

mkdir /mnt/external_drive
mount /dev/sdc1 /mnt/external_drive

mkdir /mnt/efi
mount /dev/nvme0n1p4 /mnt/efi
tar -xf /mnt/external_drive/efi.tar -C /mnt/efi

mkdir /mnt/rootfs
mount /dev/nvme0n1p5 /mnt/rootfs
tar -xf /mnt/external_drive/rootfs.tar -C /mnt/rootfs

Looks good so far, I have almost-working Fedora Linux on my HP Omen laptop, but not too close.

SELinux issue

Fedora Linux runnig Security-Enhanced Linux (SELinux), that means all processes and files are labeled in a way the represents security-relevant information. This information is called the SELinux context. That means, when we copy all these root filesystem files into a new filesystem, it could make the SELinux context also change and might result files access or process problems. To fix this, we can disable SELinux security operation mode momentary then enable that later. The SELinux configuration is located in /etc/selinux/config.

SELINUX=disabled

I also wrote a post in the past that cover this SELinux issue.

Boot setting

Now the HP Omen laptop has 2 EFI partitions, one is for Windows and another for Fedora Linux. In this post, I don't want to cover how to adjust the new Linux boot setting, basically all I did was changing root partition location in GRUB's config and Linux EFI's files. I didn't make GRUB as the default boot manager replacing Windows boot manager, I just hit a startup key that allow me to select which EFI partition to boot. Maybe it's not a convenient solution but I'll use this Fedora Linux on my HP Omen occasionally.

Saturday, August 26, 2023

Build Flutter app on Linux host

I have been busy developing a Flutter app. When come to CI/CD processes, I still rely on online repository CI/CD pipeline such as BitBucket Pipeline, GitHub Actions or even better all-in-one Flutter built-in functionality such as Codemagic especially for Android APK build. While they already offer complete CI/CD processes, I still think why not to use my own server and deploy a Flutter build system there.

Server requirements

I'm running a DigitalOcean droplet, 4 vCPUs, 8GB RAM running Ubuntu 20.04 LTS 64bit. Since the Flutter Android build process uses a huge amount of resources, especially RAM, I don't think this can be achieved by a system running RAM less than 8GB at least based on what I have tested. With 4GB RAM, after running build for a while, the build processes will be terminated by Linux Out-Of-Memory (OOM) killer or the amount of available RAM is simply not enough.

After server resource requirements above, there are also some build requirements for Linux that listed on Flutter - Linux installation page, of course that includes Git.

Install location

All build components will be located in my home directory inside "dev" directory or ~/dev. The reason I'm doing this because I don't want to mix any of them inside any root sub-directories and to simplify whole build process steps.

Install Flutter

To install Flutter, we can do by either using Snap or manual install, here I choose manual install. When this post is written, Flutter has reached version 3.13. Get the Flutter latest version zip and extract it into ~/dev.

cd ~/dev 
tar xf flutter_linux_3.13.5-stable.tar.xz
mv flutter_linux_3.13.5-stable flutter
rm flutter_linux_3.13.5-stable.tar.xz

Add Flutter path into PATH environment variable. We can do this either in ~/.bashrc (local) or /etc/profile (system-wide). But again, I don't want to make any system change, so I just append these lines at the end of my .bashrc file.

FLUTTER_BIN="/home/ardhan/dev/flutter/bin"
export PATH="$PATH:$FLUTTER_BIN"

Install Android SDK

As we build on a remote host and working on command based tools only, we don't need Android Studio to have all the Android SDK components as described on Flutter getting started page. We just need the all the basic SDK including sdkmanager which is included in command-line-tools package of Android SDK. Head to Android Studio download page, download command line tools for Linux and put it also inside ~/dev directory (latest file is commandlinetools-linux-10406996_latest.zip). We're installing Android SDK in ~/dev/android-sdk.

cd ~/dev
unzip commandlinetools-linux-10406996_latest.zip
mkdir android-sdk
mv android-sdk-cmdline-tools android-sdk/cmdline-tools
rm commandlinetools-linux-10406996_latest.zip

Now to add android-cmdline-tools/bin directory in PATH, do the same thing we added PATH on Flutter above. Add these following lines at end of ~/.bashrc

ANDROID_SDK_CMDLINE_TOOLS_BIN="/home/ardhan/dev/android-sdk/cmdline-tools/bin"
export PATH="$PATH:$FLUTTER_ROOT:$ANDROID_SDK_CMDLINE_TOOLS_BIN"

Then we might need start a new shell or just simply logout and re-login to activate the new PATH. Now, we can get all the Android build tools with sdkmanager command line tool. We set ~/dev/android-sdk directory as our Android SDK build tools location, and run sdkmanager to get list of available packages.

sdkmanager --sdk_root=/home/ardhan/dev/android-sdk --list

There are too many Android build packages and we may wonder which packages and versons to have. But don't worry, sdkmanager will automatically resolve all the package dependencies. We just need to collect informations about the version of our existing build tools, platform sdk, etc. in your local Android project. If you are using Android Studio, go to File | Settings | Appearance & Behavior | System Settings | Android SDK to view packages that are installed, or you can also use sdkmanager tool that should be located in your local Android SDK.

As you can see from the screenshot above, some major installed packages are

  • SDK Platforms: Android 11.0 (R)
  • SDK Tools: Android SDK Build-Tools 34, Android SDK Command-line Tools (latest), Android Emulator

The SDK platform and SDK build tools are the most required packages for build process. We can skip Android SDK Command-line Tools as we already installed it and for Android Emulator, we obviously don't need an emulator in a server-like host. Now, lets run sdkmanager to install the required SDK tools, the --sdk_root option tells sdkmanager where to install the packages, then you might asked to accept some licenses.

sdkmanager --sdk_root=/home/ardhan/dev/android-sdk \
  --install "platforms;android-11"
sdkmanager --sdk_root=/home/ardhan/dev/android-sdk \
  --install "build-tools;34.0.0"

You might asked to accept some licenses, you know what you to do right?. After they have been installed, add ANDROID_SDK_ROOT environment variable in ~/.bashrc.

ANDROID_SDK_ROOT=/home/ardhan/dev/android-sdk
export ANDROID_SDK_ROOT=$ANDROID_SDK_ROOT

Install OpenJDK

And of course we'll need a JDK (Java Development Kit), Android build uses OpenJDK which is the open source implementation of Java. Choosing Java version to work correctly is a bit tricky since specific Gradle (The Android build system) version also depends on a specific JDK version. If you are unsure about both Gradle and JDK version you are currently use, just follow the version of Gradle and Java your local computer. You can do this by executing Gradle's build script inside "android" directory of your Flutter project. These scripts are gradlew for Linux and gradlew.bat for Windows. Anyway, here my Gradle script output under Windows 10.

C:\Projects\app\android>gradlew.bat --version
------------------------------------------------------------
Gradle 7.4
------------------------------------------------------------
Build time: 2022-02-08 09:58:38 UTC
Revision: f0d9291c04b90b59445041eaa75b2ee744162586
Kotlin: 1.5.31
Groovy: 3.0.9
Ant: Apache Ant(TM) version 1.10.11 compiled on July 10 2021
JVM: 11.0.11 (Microsoft 11.0.11+9)
OS: Windows 10 10.0 amd64

That output shows my OpenJDK is version 11 (Microsoft build) and Gradle version is 7.4. Actually you don't need to have the same version of OpenJDL as listed above. Gradle has a Java Compability Matrix table shows compability between various Gradle and Java versions.

My Flutter project uses Gradle 7.4 and according to the compability table above, Gradle 7.3 compatible with Java 17, that means Gradle 7.4 should also work with Java 17. So here I choose OpenJDK version 17. But of course, we can still use OpenJDK 11, I just wanted to play with the newer version. Head to OpenJDK Archive to download it, There I choose OpenJDK 17.0.2 (build 17.0.2+8) as the latest build of version 17, download the Linux 64bit version and put into ~/dev directory and extract.

cd ~/dev
tar xzf openjdk-17.0.2_linux-x64_bin.tar.gz
mv openjdk-17.0.2_linux-x64_bin openjdk-17.0.2

Then again, add this OpenJDK binaries into PATH and create a new JAVA_HOME environment variable, this JAVA_HOME is required for every Java installation. At these new lines into ~/.bashrc.

JAVA_BIN="/home/ardhan/dev/jdk-17.0.2/bin"
JAVA_HOME="/home/ardhan/dev/jdk-17.0.2"
export PATH="$PATH:$FLUTTER_BIN:$ANDROID_SDK_CMDLINE_TOOLS_BIN:$JAVA_BIN"
export JAVA_HOME=$JAVA_HOME

Final .bashrc

So to recap, my final ~/.bashrc shall look like this, remember these new lines were appended at end of the file.

FLUTTER_BIN="/home/ardhan/dev/flutter/bin"
ANDROID_SDK_CMDLINE_TOOLS_BIN="/home/ardhan/dev/android-sdk/cmdline-tools/bin"
ANDROID_SDK_ROOT="/home/ardhan/dev/android-sdk"
JAVA_BIN="/home/ardhan/dev/jdk-17.0.2/bin"
JAVA_HOME="/home/ardhan/dev/jdk-17.0.2"
    
export PATH="$PATH:$FLUTTER_BIN:$ANDROID_SDK_CMDLINE_TOOLS_BIN:$JAVA_BIN"
export ANDROID_SDK_ROOT=$ANDROID_SDK_ROOT
export JAVA_HOME=$JAVA_HOME

Building

To test Flutter Android build, we might need to re-login to shell to apply the new ~/.bashrc. Then head to your Flutter project and issue these flutter commands.

cd ~/my-app
flutter clean
flutter build apk

Flutter might download additional required packages, tools, asked to accept some licenses etc. If something went wrong, check the message for more informations, also consult with flutter doctor command. In the future, I would like to write another post about how this build process can be automated, such as what online build pipelines do, creating a git repository with its githook to trigger automatic build.

Tuesday, February 05, 2019

Hello Write

It's been almost 2 years ago since I wrote a post in this blog, there are many things happened, most of them are life and or just some random stuff. While I've been busy with life, I couldn't spend my times to write, and probably I was just too lazy to write, or even didn't care to write anymore.

My families in hometown are doing fine, we built a new house in our farm, my crazy uncle lives there, baby-sitting the farm. After his retirement, my dad used to stay there before and after the new house construction, due to his eyes and old man health issues, he's now spending most of his time in our family house watching sports, politics and praying. My mother as always, she's strong, if her right knee isn't pain, she'll be as super busy as my sisters. She also has retired a couple years ago. My bro is okay, just like me, he loves and always hungry for fishing, he'll definitely give anything for it.

About me, where to start, work? work went great, I'm still managing my own venture. We have plenty new clients, they are awesome and came from variety range of industries, RFID, medical equipment, military, public services, and maybe a south American lottery enterprise. I might withdraw my full time job position soon, and become fully focused on my own realm. I have lost one of my best "friend", we have been together since 2010 and felt terribly lost for that. But just like everyone says, life must go on.

I got 2 additional motorcycles since 2016, Honda sport bike and a BMW. I have been enjoying them a lot, had many street rides, adventures and YouTube vlogs. I recently much into mountain biking (MTB) after watching Red Bull rampages, well, it's literally my old shit but thing getting more technical now. I almost completely overhauled my bro's old and cheap bike, replaced almost part of it, drivetrain, wheels, crankset, gearset (you name it) and left the frame only. Oh yeah, also 2 new bikes, one was built by myself. I'm not sure how I can ride them all. How about being a rockstar? well, I don't have to write it explicitly.

I have had plenty of travels, mostly domestic though, Wakatobi, Maluku and northern Sumatra. Diving, fishing, motor trailing, making new friends, and of course, vlog-ing.

Saturday, May 14, 2016

Linux Distribution Kernel Upgrade

In this post I want to write some steps to upgrade Distribution's Linux kernel from the source (yeah, not by specific distribution package manager), this assuming all required build tools has been installed and configured properly, consult to the distribution's resources to build a kernel from source.

I'm using Linux Mint 13 32bit (codename Maya, a Ubuntu 12.04 LTS derivative) as the host, running kernel version 3.13.0 which I steal from Linux Lite 2.4. Here I upgrade it to kernel version 4.4.10 (LTS). These steps here are not limited to Ubuntu derivate distributions, it can be for other distribution such as Debian, Fedora, Arc Linux and the others.

Get the kernel source

Firstly, download and unpack the kernel source, here I download it directly from kernel.org and pick an XZ archive version. We use /project/linux as our base working directory.

$ cd /project/linux
$ wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.4.10.tar.xz
$ tar xJf linux-4.4.10.tar.xz


Apply current kernel configuration

To match the current kernel configuration and to avoid too much dives in kernel configuration, we will reuse a configuration file that is very likely shipped by the distribution we are using, usually it is located inside the /boot directory, copy this file into our kernel source tree as .config

$ cd linux-4.4.10 
$ make mrproper
$ cp /boot/config-`uname -r` .config 

Building the kernel

Issue this to enter the kernel build configuration, for any errors that may occurred, again please refer to distribution documentation describing a system preparation to build kernel from source.

$ make menuconfig

The text based (ncurses) configuration menu appears, since we already have a .config file, we don't *really* care to configure some options again but let the configuration it self adjust and adapt against our existing configuration file. Exit from the configuration menu and confirm to save the configuration. Now we are ready to build the kernel by issuing

$ make

The build system should now begin to compile, various build messages or compiler warnings appear. The build time depends on hardware speed (CPU, RAM, disk I/O) and how many kernel's components we enabled in configuration. When the build has been finished, install all kernel's modules with superuser privilege

$ sudo make modules_install
$ cp arch/x86/boot/bzImage /project/linux/vmlinuz-4.4.10
$ cp .config /project/linux/config-4.4.10

This will install kernel modules inside /lib/modules/4.4.10 directory and copy the new kernel image as vmlinuz-4.4.10-generic inside our base working directory. For reference, we ought to copy configuration file for our future reference, of course like what we are doing now.

Rebuild initial ramdisk image

Since our distribution uses an initial ramdisk or so called initrd then we need to rebuild it to contain our new kernel's modules, we will use the existing initrd as our base.

Anyway, this is the most interesting part of upgrading process, the initrd file is a cpio archive and was gz compressed so we will use zcat and cpio utilities to extract and uncompress it. Assuming cpio and zcat are installed, issue these following commands

$ cd /project/linux
$ mkdir initrd
$ cd initrd
$ zcat /boot/initrd.img-`uname -r` | cpio -i

In above commands, we've created a directory named initrd as our initrd's files location after the extraction, we only need to replace old (current) kernel modules to our new modules. Ideally, by initrd nature which only provide basic init system, utilities and kernel modules need to boot, we only need to replace each module that are inside the old kernel, but since I'm too lazy to pick each one, I just copy full module directory here. Make it is easy, the downsize is just getting the final initrd file much bigger, don't worry about that. Issue these following commands to copy.

$ cd lib/modules
$ rm -rf `uname -r`
$ cp -r /lib/modules/4.4.10 .

And its time to rebuild the initial ramdisk image, first we back to our initrd base directory and optionally make sure all initrd files are owned by Linux's UID 0 (root) then archive and compress it into a new initrd image. Switch to root and issue

# cd /project/linux/initrd
# chown -R root:root *
# find . | cpio -o -H newc | gzip -9 > /project/linux/initrd.img-4.4.10

Now, we should have these files in the base working directory

vmlinuz-4.4.10
initrd.img-4.4.10
config-4.4.10

We may need to copy or move these files into /boot directory if we like. And that's all, we can now test to boot the new kernel by modifying the GRUB boot loader configuration that usually in /boot/grub/grub.cfg (GRUB v2), see GRUB manual for that.

Thursday, May 12, 2016

Back To Write Again

It's been a long way since my last update in 2013, many thing was happened, I was too busy with rocking in, life, code, GTA V and all the for-nothing stuff. Now I'm back (hopefully with a great consistency) to write again.


The web technology has been advanced so fucking great so does the Blogger, it has many improvements such as the cool editor interface, personally I like it and want to switch from my old favorite half syntax HTML editor. Also, I may need to change this blog template since its begin look old to me then add a nice commenting system like Disqus. That's all for now, there are tons of stuff in my brain that open for fire.