Linux Distribution Kernel Upgrade

In this post I want to write some steps to upgrade Distribution's Linux kernel from the source (yeah, not by specific distribution package manager), this assuming all required build tools has been installed and configured properly, consult to the distribution's resources to build a kernel from source.

I'm using Linux Mint 13 32bit (codename Maya, a Ubuntu 12.04 LTS derivative) as the host, running kernel version 3.13.0 which I steal from Linux Lite 2.4. Here I upgrade it to kernel version 4.4.10 (LTS). These steps here are not limited to Ubuntu derivate distributions, it can be for other distribution such as Debian, Fedora, Arc Linux and the others.

Get the kernel source

Firstly, download and unpack the kernel source, here I download it directly from kernel.org and pick an XZ archive version. We use /project/linux as our base working directory.

$ cd /project/linux
$ wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.4.10.tar.xz
$ tar xJf linux-4.4.10.tar.xz


Apply current kernel configuration

To match the current kernel configuration and to avoid too much dives in kernel configuration, we will reuse a configuration file that is very likely shipped by the distribution we are using, usually it is located inside the /boot directory, copy this file into our kernel source tree as .config

$ cd linux-4.4.10 
$ make mrproper
$ cp /boot/config-`uname -r` .config 

Building the kernel

Issue this to enter the kernel build configuration, for any errors that may occurred, again please refer to distribution documentation describing a system preparation to build kernel from source.

$ make menuconfig

The text based (ncurses) configuration menu appears, since we already have a .config file, we don't *really* care to configure some options again but let the configuration it self adjust and adapt against our existing configuration file. Exit from the configuration menu and confirm to save the configuration. Now we are ready to build the kernel by issuing

$ make

The build system should now begin to compile, various build messages or compiler warnings appear. The build time depends on hardware speed (CPU, RAM, disk I/O) and how many kernel's components we enabled in configuration. When the build has been finished, install all kernel's modules with superuser privilege

$ sudo make modules_install
$ cp arch/x86/boot/bzImage /project/linux/vmlinuz-4.4.10
$ cp .config /project/linux/config-4.4.10

This will install kernel modules inside /lib/modules/4.4.10 directory and copy the new kernel image as vmlinuz-4.4.10-generic inside our base working directory. For reference, we ought to copy configuration file for our future reference, of course like what we are doing now.

Rebuild initial ramdisk image

Since our distribution uses an initial ramdisk or so called initrd then we need to rebuild it to contain our new kernel's modules, we will use the existing initrd as our base.

Anyway, this is the most interesting part of upgrading process, the initrd file is a cpio archive and was gz compressed so we will use zcat and cpio utilities to extract and uncompress it. Assuming cpio and zcat are installed, issue these following commands

$ cd /project/linux
$ mkdir initrd
$ cd initrd
$ zcat /boot/initrd.img-`uname -r` | cpio -i

In above commands, we've created a directory named initrd as our initrd's files location after the extraction, we only need to replace old (current) kernel modules to our new modules. Ideally, by initrd nature which only provide basic init system, utilities and kernel modules need to boot, we only need to replace each module that are inside the old kernel, but since I'm too lazy to pick each one, I just copy full module directory here. Make it is easy, the downsize is just getting the final initrd file much bigger, don't worry about that. Issue these following commands to copy.

$ cd lib/modules
$ rm -rf `uname -r`
$ cp -r /lib/modules/4.4.10 .

And its time to rebuild the initial ramdisk image, first we back to our initrd base directory and optionally make sure all initrd files are owned by Linux's UID 0 (root) then archive and compress it into a new initrd image. Switch to root and issue

# cd /project/linux/initrd
# chown -R root:root *
# find . | cpio -o -H newc | gzip -9 > /project/linux/initrd.img-4.4.10

Now, we should have these files in the base working directory

vmlinuz-4.4.10
initrd.img-4.4.10
config-4.4.10

We may need to copy or move these files into /boot directory if we like. And that's all, we can now test to boot the new kernel by modifying the GRUB boot loader configuration that usually in /boot/grub/grub.cfg (GRUB v2), see GRUB manual for that.

Scatter/gather I/O operation

This is a common way to do scatter/gather I/O (sometime called vectored I/O), it has great performance over large chunk of data. The readv() and writev() function are atomic means these system call only issued for once to transfer multiple data into the kernel.

unsigned len;
long iovcnt, iovcnt_max;

iovcnt_max = sysconf(_SC_IOV_MAX);
/* Determine how many part of data to be written */

while (len > 0) {
  iovcnt = len >= iovcnt_max ? iovcnt_max : len;
  struct iovec iov[iovcnt];
  /* Initialize iov's members, process with readv() or writev() */
  len -= iovcnt;
}

The above code first determine maximum allowed size of iovcnt, in Linux this value is 1024 but we should not rely on this but using system specific sysconf() function, after that the code determine struct iovec array and declare it as VLA and loop to write or read data until all data has been processed.

Beside to clone the kernel development tree directly from its git repository, I want to create repository from a stable kernel and its patches, I wil

Beside to clone the kernel development tree directly from its git repository,
I want to create repository from a stable kernel and its patches, I will have
commit for each subversion patch. The last stable kernel at this time is 2.6.38.

First download the 2.6.38 first release (not the subversion releases) named
linux-2.6.38.tar.bz2, extract it with (my favourite bzip2 - tar style)

$ bzip2 -dc linux-2.6.38.tar.bz2 | tar xf -

Initialize git repository, add files and do initial commit.

$ cd linux-2.6.38
$ git init && git add .
$ git commit -m 'Linux 2.6.38'

Time to download the patches, please note that the kernel patch system is not a
increment patch, this means when we got version 2.6.38.3 patch this must be
applied to the first version in this case 2.6.38, not any of its subversion, eg.
2.6.38.2 or 2.6.38.1. At this time, I have these following patch files

patch-2.6.38.1.bz2
patch-2.6.38.2.bz2

After we got out desired patches, so now lets create another branch for patching
purposes, still at our master branch.

$ git branch 2.6.38
$ git branch 2.6.38.1
$ git checkout 2.6.38.1

Issue patch command with (also my favourite bzip2 - patch command style)

$ bzip2 -dc /path_to/patch-2.6.38.1.bz2 | patch -p1
$ git commit -m 'Linux 2.6.38.1'

Now we have 2.6.38.1 version at 2.6.38.1 branch, then lets create 2.6.38.2
branch for 2.6.38.2 patch as well, remember that we must have first version to
apply a patch, so create this branch based the first version branch.

$ git checkout 2.6.38
$ git branch 2.6.38.2
$ bzip2 -dc /path_to/patch-2.6.38.2.bz2 | patch -p1
$ git commit -m 'Linux 2.6.38.2'

Lets describe what we have in our repository, we already have 4 branches with
its branch's name as its version, except for master branch, it has 2.6.38
version. I want to have each version has its commit on the master branch, but I
don't have increment path for version 2.6.38.1 to 2.6.38.2, it is impossible to
just merging 2.6.38.1 and 2.6.38.2 branch, so we must result a diff file from
version 2.6.38.1 to 2.6.38.2, those commands simply do the magic.

$ git checkout 2.6.38.2
$ mkdir /path_to/2.6.38.2
$ cp -r . /path_to/2.6.38.2/
$ git checkout 2.6.38.1
$ diff -ur . /patch_to/2.6.38.2 | bzip2 -zc > /path_to/patch-inc-2.6.38.2.bz2

The 2.6.38.1 to 2.6.38.2 patch is now generated in bz2 format, this an
increament patch, we can now apply this patch to the master branch.

$ git checkout master
$ bzip2 -dc /path_to/patch-inc-2.6.38.2.bz2 | patch -p1
$ git commit -m 'Linux-2.6.38.2'
$ git log

As you can see the master branch log is now have increment version for each
commit, this will be helpfull if we want to trace patches for every subversion.

Local Kernel Repository

Beside to clone the kernel development tree directly from its git repository, I want to create repository from a stable kernel and its patches, I will have commit for each subversion patch. The last stable kernel at this time is 2.6.38.

First download the 2.6.38 first release (not the subversion releases) named linux-2.6.38.tar.bz2, extract it with (my favourite bzip2 - tar style)

$ bzip2 -dc linux-2.6.38.tar.bz2 | tar xf -

Initialize git repository, add files and do initial commit.

$ cd linux-2.6.38
$ git init && git add .
$ git commit -m 'Linux 2.6.38'

Time to download the patches, please note that the kernel patch system is not a increment patch, this means when we got version 2.6.38.3 patch this must be applied to the first version in this case 2.6.38, not any of its subversion, eg. 2.6.38.2 or 2.6.38.1. At this time, I have these following patch files

patch-2.6.38.1.bz2
patch-2.6.38.2.bz2

After we got out desired patches, so now lets create another branch for patching purposes, still at our master branch.

$ git branch 2.6.38
$ git branch 2.6.38.1
$ git checkout 2.6.38.1

Issue patch command with (also my favourite bzip2 - patch command style)

$ bzip2 -dc /path_to/patch-2.6.38.1.bz2 | patch -p1
$ git commit -m 'Linux 2.6.38.1'

Now we have 2.6.38.1 version at 2.6.38.1 branch, then lets create 2.6.38.2 branch for 2.6.38.2 patch as well, remember that we must have first version to apply a patch, so create this branch based the first version branch.

$ git checkout 2.6.38
$ git branch 2.6.38.2
$ bzip2 -dc /path_to/patch-2.6.38.2.bz2 | patch -p1
$ git commit -m 'Linux 2.6.38.2'

Lets describe what we have in our repository, we already have 4 branches with its branch's name as its version, except for master branch, it has 2.6.38 version. I want to have each version has its commit on the master branch, but I don't have increment path for version 2.6.38.1 to 2.6.38.2, it is impossible to just merging 2.6.38.1 and 2.6.38.2 branch, so we must result a diff file from version 2.6.38.1 to 2.6.38.2, those commands simply do the magic.

$ git checkout 2.6.38.2
$ mkdir /path_to/2.6.38.2
$ cp -r . /path_to/2.6.38.2/
$ git checkout 2.6.38.1
$ diff -ur . /patch_to/2.6.38.2 | bzip2 -zc > /path_to/patch-inc-2.6.38.2.bz2

The 2.6.38.1 to 2.6.38.2 patch is now generated in bz2 format, this an increament patch, we can now apply this patch to the master branch.

$ git checkout master
$ bzip2 -dc /path_to/patch-inc-2.6.38.2.bz2 | patch -p1
$ git commit -m 'Linux-2.6.38.2'
$ git log

As you can see the master branch log is now have increment version for each commit, this will be helpfull if we want to trace patches for every subversion.

Improving the TCT Tool

This previous post describes how TCT Tool was written, It is using POSIX Message Queue to exchange data. Basic on work, TCT Tool's message queue implementation almost work, only the main problem is the sender could read it's sent data, this sound weird because the TCT Tool would read it's data that intended to be read by the TCT. On the Toll, this problem hung up the TCT.

I change the IPC implementation using UNIX Domain Socket, first with datagram socket, the problem was just like the message queue, the TCT Tool receive it's sent data. The TCT Tool was not work until I use reliable stream socket, here the source code.

ARM based Notebook

This is maybe a ARM week for me, Trio my office friend told me about this notebook, produced by an Indonesian made Notebook vendor ELEVO. It is using ARM9-533 Mhz processor comes with 10" display, 128 MB RAM, 2 GB Nano Flash storage, 802.11g Network Adapter, Built-in camera, 3 USB port, 1 SD card port. The more exciting thing is it's prices, sold only for about Rp.1.398.000. The ELEVO ARM based notebook series come with 2 model, the R7 and R10, the difference between them is only the display size.

I'm very interesting to buy the R10, yes it is! it is based on ARM board, it is a `litle' unique processor for notebook, isn't it?. The bundled OS is Windows CE 6.0 but i want to make it Linux heh.

UNIX exec family system call

This codes creating pipe for IPC, fork another process (child) and executing fortune with some cookies arguments. It also redirect child's stdout to it's writer pipe to be read by parent's pipe. exec() family calls never return except for errors, if this happen it is very bad ;p. exec() will replace it's calling process with a new image, since the kernel is not know where to continue after invoking exec() then the child just exit. Just to make sure you don't write codes after that exec() except for error checking, don't care how cool your codes, it just not work. Trust me heh?.

http://pastebin.com/raw.php?i=pceQuWjc

ulimit

Do you want to enable realtime priority or change maximum memory locking with non root user?, the /etc/security/limits.conf store this limitation, for list of limitation issue a uname -a command, the name of limitation is described in limits.conf file. This is a example to enable realtime priority and change maximum locked memory to user ajhwb:

ajhwb    -    rtprio     60
ajhwb    -    memlock    512

Now save the file, if the affected user is currently login, logout first and re-login. Changes are made after next login. You may want to test by write a simple code that switch to a realtime scheduler and priority or you can use chrt utility, also test maximum locked memory more than 64Kb as standard definition. There are many you can change in the limits.conf, have fun!

CONFIG_PREEMPT_RT kernel patch

On rt-users kernel mailing lists Linus said:

"Controlling a laser with Linux is crazy, but everyone in this room is crazy in his own way. So if you want to use Linux to control an industrial welding laser, I have no problem with your using PREEMPT_RT" -- Linus Torvalds

As you may know that the standard build Linux kernel is always preempt high priority user-mode process when kernel-mode process is executing, this bad for time critical application. Realtime application need precise timing for it's work. Let say you control a motor machine, it's use many sensors to determine it's rotor position, you need to know each sensor's state by reading the sensor device. With a high speed motor, you will need to read sensor state for example every 1 milisecond and do something as soon as sensor state change. Imagine what will happen when the kernel-mode process is running and delaying your realtime application while you ask the machine to stop? of course disaster.

In 2.6 kernel version, there are some preemption setting introduced such as CONFIG_PREEMPT_VOLUNTARY and CONFIG_PREEMPT, but these are not meet a realtime application that need very small latencies. The PREEMPT_RT patch convert Linux to be fully preemptible kernel, this done with:

• Making in-kernel locking-primitives (using spinlocks) preemptible though reimplementation with rtmutexes.


• Critical sections protected by i.e. spinlock_t and rwlock_t are now preemptible. The creation of non-preemptible sections (in kernel) is still possible with raw_spinlock_t (same APIs like spinlock_t).


• Implementing priority inheritance for in-kernel spinlocks and semaphores. For more information on priority inversion and priority inheritance please consult Introduction to Priority Inversion.


• Converting interrupt handlers into preemptible kernel threads: The RT-Preempt patch treats soft interrupt handlers in kernel thread context, which is represented by a task_struct like a common user space process. However it is also possible to register an IRQ in kernel context.


• Converting the old Linux timer API into separate infrastructures for high resolution kernel timers plus one for timeouts, leading to user space POSIX timers with high resolution.

Main Maintainer: Ingo Molnar, Thomas Gleixner
Website: http://rt.wiki.kernel.org/index.php/Main_Page
Files: http://www.kernel.org/pub/linux/kernel/projects/rt/

GCC -g option

Glynn Clements answer an email in the linux-c-programming mailing list to someone who ask about GCC -g option

Most the time I compile my application without the -g option due to performance reasons.

The -g switch has absolutely no effect upon performance. It simply causes and additional section to be added to the resulting binary. When the program is run normally (i.e. not under gdb), that section won't be mapped. The only downside to -g is that it increases the size
of the file.

However: debug information isn't necessarily much help if you compile with optimisation enabled, as the resulting machine code will bear little resemblance to the original source code. Statements will be re-ordered, many variables will be eliminated, etc.

Problem is that when it hits some bug and dumps core, this is not very useful because there is hardly any information in it. Is there some way to get some useful information out of the core file. For example one of my program crashed and with gdb I see the following: At least I know that the bug is in my function start_process. But is there some way to find out at what line it happened?

It isn't meaningful to talk about a "line" in the source code if you compile with optimisation enabled. However, you can tell gdb to disassemble the machine code for a particular function, and you can print the values contained in registers or at specific memory locations. Working out what that information means in terms of the source code is something which needs to be done manually.

Glynn Clements is an active member in the Linux's kernel mailing list, i love the way he answer a question.

The Linux proc Filesystem

/proc/cpuinfo - CPU informations
/prco/meminfo - Physical and virtual memory informations
/proc/sys/net/ipv4/ip_forward - IP forwarding setting (respect on protocol version)
/proc/sys/vm/overcommit_memory - Virtual memory overcommit setting

iptables

iptables is a administration tool for IPv4 packet filtering and NAT, it's mainly in UNIX like operating system including, of course Linux. Here my iptables commands collection based on my own purposes:

- List applied filters
#iptables -L

- Blocking specific IP address
#iptables -I INPUT -s <address> -j DROP

- Unblocking specific IP address
#iptables -D INPUT -s <address> -j DROP

Linux Termios

I have been writing TCT smartcard reader module, using communication protocol that shipped with the reader, the reader actually provide API for Linux, the API is good, but not so good. It's closed code means we can't do more hacks. One thing that was a big issue which force me to write the module is API's ability to determine timeout I/O operation. For example, we send command to the reader, but let say the reader power has fail or something error occurred then the API will wait and block for uncertain of times because there are no respond from the reader.

Then i write the module from scratch, i got some weird problems, the reader will not reply for certain type of byte (character), 0xD (CR), 0xA (NL), 0x11 and 0x13. This problem has made me crazy and temporary ask me to use simple ASCII protocol. The termios structure contains input/output flags that determine serial communication characteristic.

By default (and at least i solve this problem, thanks you all), Linux serial API perform CR to NL and NL to CR for both input and output, so when i write a CR, it was actually converted to NL and vice versa, this translation also inherited for output. To remove this setting, do bit operation with control setting as follow:

struct termios opt;
/* open device and read current setting */
opt.c_iflag &= ~(INLCR | INCRNL);
opt.c_oflag &= ~(ONLCR | ONCRNL);

Now the CR-NL and NL-CR translation has been resolved, what now? i got another headaches, the same problem with character 0x11 and 0x13 on input, after do some man tcsetattr i see Linux also interpreting this as a special character, the XON/XOFF option flag has been enable by default, it's serial line hardware flow control, so i disable this in input control setting:

opt.c_iflag &= ~(IXON | IXOFF);

...and everything is fine, there are no unresponsive actions by the reader caused by invalid data sequence, hey.. i just beginner serial communication programmer, and thanks to the ACR-120 reader, thanks for let me know this issues.

ETHTOOL: socket-level I/O control calls

The TCT machine that we developed should has network capability, this used for acquiring and retrieving data from the database server (PostgreSQL server). I made a low level utility today, share the memory through POSIX shared memory in order to be read by the TCT machine, the purposes of this utlity is to provide network status such as plugged/unplugged network cable, unreached server and notify if the database server is running or not.

The socket-level I/O control call that kernel provide using ioctl system call require root permission. So i create this as another process (like the daemon) and shared it's status memory area, to know the server connection i simply forking ping utility, pass some options to pinging the server. At last, to probe the database service running state i just connect to the database server via it's running port.

And this is the core codes, i'm pointing to the kernel's socket-level I/O control calls, and call them in userspace (kernel-2.6.27):

#include <sys/ioctl.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <net/if.h>
#include <arpa/inet.h>
#include <stdio.h>
#include <string.h>
#include <linux/ethtool.h>
#include <linux/sockios.h>

int main (void)
{
  int s, ret;
  struct ethtool_value edata;
  struct ifreq ifr;

  s = socket (AF_INET, SOCK_STREAM, IPPROTO_TCP);
  if (s == -1)
    {
      perror ("socket");
      return -1;
    }
  edata.cmd = ETHTOOL_GLINK;
  strncpy (ifr.ifr_name, "eth0", sizeof(ifr.ifr_name) - 1);
  ifr.ifr_data = (char *) &edata;

  ret = ioctl (s, SIOCETHTOOL, &ifr);
  if (ret == -1)
    {
      perror ("ioctl");
      return -1;
    }
  printf ("link is %s\n", edata.data ? "up" : "down");
  close (s);

  return 0;
}

AutoFS: The linux's kernel automounter

Last night i began to build CSD and CSR development system, use Debian Lenny with Linux's kernel-2.6.26. The CSD and CSR cartridge has a USB disk to write some useful infos such as cartridge's usage, cartridge id, and a list of smartcards data that filled in the cartridge.

Autofs or the kernel automounter is actually a additional module to linux kernel, it's work by mounting a filesystem when they are accessed and unmounting them when they are no longer in use.

I use autofs version 4 fetched from a Debian repo server, write some autofs's configuration files.

/etc/auto.master
# The fs should be mounted on /mnt directroy
# and i'll need 3 secs timeout before autofs unmount the fs
/mnt /etc/auto.mnt --timeout=3

/etc/auto.mnt
disk -fstype=auto,rw,nosuid,nodev :/dev/sda

Restart autofs daemon and write a small demonstration codes, i know when the disk already mounted or not by polling a file in the disk with open() system call (ENOENT error code), but the real problem is: how does my codes receive a kernel's event when disk is plugged.

PCM-1028 Multi Serial Ports

It's been 2 days i worked on TCT boards, PCM-4386 SBC from Advantech. Use Debian 5.0 (Lenny) with Linux kernel-2.6.26, good news that Lenny's kernel already built with extended 8520 / 16550 serial ports support to use with PCM-1028 multi-serial board, i only need to pass kernel parameters:

8250.nr_uarts=12

since the total port will be 12 ports. I got difficulties when installing the PCM-1028, have been tried many configurations:

- Use independent IRQ address for each port as default setting provide. (not work, mouse device was not work, possibility is IRQ conflict).
- Change interrupt address and base address, still not work too.
- Share one IRQ to all ports. Seem to work, but...

The data is not valid, i can't receive or send valid characters through minicom and my own codes. I assumed a baud base issue, so i change to the PCM-1028 maximum supported baud base. Not work, i only receive dumb characters, i don't believe what i see, i don't believe the serial cable, so i switch another serial cable, and fucking shit, the characters received correctly. I admit, the old cable fuck me in 2 days!

This is a example serial configuration on TCT board:

/dev/ttyS4 uart 16550 port 0x200 irq 7 baud_base 921600 skip_test

Note that Debian store setserial config file to it's own location, /var/lib/setserial/autoserial.conf, other setting just follow it's default configuration.

Updated on Thursday, May 14 2009 11:16 AM
Oww, the PCM-4386 already assigned IRQ 7 to the parallel port 0x378. I change IRQ sharing to 11.

Kernel: Compiling a simple module

Here a simple for beginner to compile their first kernel module:

1. Create a simple Makefile:

obj-m += mymodule.o

2. Compile and link the module, notice you should replace your kernel source:

$make -C /usr/src/linux SUBDIRS=$PWD modules

3. Use insmod or rmmod to insert and remove your module into kernel space.
4. Watch your module log message in the kernel ring buffer via dmesg or tailing /var/log/messages only if you did some debugging info.

Update on 09/09/2011

Another method is to create complete Makefile script for example

KERNEL_DIR = /usr/src/linux
SOURCE_DIR = /project/cpu

obj-m += cpu.o

all:
    make -C ${KERNEL_DIR} SUBDIRS=${SOURCE_DIR} modules

clean:
    make -C ${KERNEL_DIR} SUBDIRS=${SOURCE_DIR} clean

Have many fun...

Linux Kernel Project

What you need to care in linux kernel development

* The kernel does not have access to the C library.
* The kernel is coded in GNU C.
* The kernel lacks memory protection like user-space.
* The kernel cannot easily use floating point.
* The kernel has a small fixed-size stack.
* Because the kernel has asynchronous interrupts, is preemptive, and supports SMP, synchronization and concurrency are major concerns within the kernel.
* Portability is important.

Building RTAI Linux

Today i build RTAI Linux based system on Intel Celeron(M) processor @ 1Ghz, it is a single board computer for use in TCT (Toll Collecting Terminal). I use RTAI version 3.6 with Kernel 2.6.23 on Debian Lenny. Since Debian Lenny comes with GNU C Compiler version 4.3, the Kernel and RTAI was failed to build - I know, the Kernel actually forbid to use except GCC version 4.1 -. Then i move to Fedora 8 with GCC version 4.1. Here my steps:

1. Patch the current Kernel.
#patch -p1 < /tmp/rtai/base/arch/i386/patches/hal-2.6.23-rtai-patches-x.x-x.patch

2. Customizing and compile the current Kernel.
#cd /tmp/kernel-2.6.23
#make menuconfig
#make bzImage

3. Compile RTAI (set Kernel and installation directory, in this sample i use /rtai for the installation directory).
#cd /tmp/rtai
#make menuconfig
#make
#make install
#tar cjvf rtai-tct.tar.bz2 /rtai

4. Copy the compiled Kernel and RTAI into the host machine.
5. Run test suite.

to be continued...

RTAI
Linux Kernel Project

RTAI on TCT

Stuck lagi, RTOS diminta oleh client kami (katanya dalam spec). So, patch RTAI, compile kernel. Distro yang digunakan mungkin Debian Lenny dengan direcfb-gtk. TCT harus punya GUI untuk interaksi. Shit!

Realtime Application Interface