Tag Archives: bash

LFS 6.8 (Part 12): Preparing to Build the LFS System

Just like I broke up building the LFS toolchain into several parts, I will break up building the LFS system.

Right now I have my temporary toolchain, but to start building the LFS system, I need to make a few changes to get it ready. If I break something major, I will have to start from here with the known-good backup of my temporary toolchain (which I just made). As it is, I’m ecstatic that I’ve made it this far with no troubles.

Preliminary Speak

The LFS book recommends that first-time LFS builders (like me!) should build without optimizations, since those speed gains can come with problems in compilation and bugs in the “optimized” software. Will I go back and try optimization later? Possibly, but not very likely unless I’m testing on a faster system (it took me 30 hours to get this far, so I’m feeling conservative).

The book also emphasizes that I should build all the packages in order, so “no program accidentally acquires a path referring to /tools hard-wired into it.”

Virtual Kernel File Systems

The kernel uses several virtual (RAM-hosted) filesystems, but they are mounted as though they are actually on disk; therefore, I need to create their host directories.

The kernel also requires a few virtual device nodes (those things I mostly don’t understand that crowd /dev) in order to boot at all, so I need to create a few manually.

Most of /dev, though, isn’t populated for the LFS system that I will chroot into. Since /dev is normally filled during boot, and the LFS chroot has never booted, I have no valid system information yet, and if I tried to build anything, the software would be baffled by the complete lack of hardware. I got around this by using a bind mount (a mount that mirrors an existing mount point or directory) of my build environment’s (the LFS live CD) /dev. This even allows me to add/remove hardware on the fly, since the changes in /dev will be mirrored to the chrooted LFS build environment.

Finally, I mount a few more virtual kernel file systems that are empty, but that the kernel will need later, including /dev/pts, /dev/shm, /proc, and /sys.

Package Management Speak

LFS doesn’t include any package management scheme, as the goal of LFS is to teach about Linux’s inner workings. The LFS book does discuss various solutions for package management, along with their strengths and weaknesses, but leaves their implementation up to the user.

Almost as a side note, the LFS book mentions that LFS can be handy for use on multiple computers with similar hardware, and can be deployed with only a little modification. I don’t really know why anyone would do this in the real world; I’d rather use a Linux system with package management of some sort.

Entering the Chroot Environment

So with the temporary toolchain ready and the virtual kernel file systems mounted and mirrored, I am ready to chroot into the $LFS directory and start building the final LFS system!

The actual chroot command is pretty long, but is fairly self-explanatory. The new chroot environment is quite similar to the live CD environment, except that it is tuned to make building with the temporary toolchain more painless (so the system uses the newly built software right away automatically).

Creating Directories

At this point, $LFS is pretty sparse, consisting only of the /dev, /source, and /tools directories (if memory serves). I needed a full directory tree, so I created one with the dozen-odd commands in the LFS book.

Ah, that’s better.

Creating Essential Files and Symlinks

A few final tweaks configure the LFS chroot environment to be ready to build! After adding a few symlinks, directories, and files, I started a new shell (thus fixing the I have no name! in the prompt) and created some log files.

Time to build LFS!

LFS 6.8 (Part 9): The First Part of the Rest of the Temporary Toolchain

The LFS temporary toolchain is going to take a while to cover completely, so I’m breaking it up into several parts for readability.

After building the compiler and C libraries, I need to build numerous small, but not insignificant, utilities for the temporary toolchain.

Tcl

Tcl is the Tool Command Language, a flexible scripting language that is commonly used on *nix systems.

Tcl, Expect, and DejaGNU are used in the test suites for the versions of Binutils and GCC I will build for the final LFS system. According to LFS, “[i]nstalling three packages for testing purposes may seem excessive, but it is very reassuring, if not essential, to know that the most important tools are working properly.” After the test suites, though, these programs are unnecessary, so I won’t be building them again for the final LFS system. To build Tcl, I first cd‘ed into the unix/ subdirectory of the Tcl source and ran configure and make there. I also ran the optional test suite, just because. Tcl installed to /tools (in the temporary toolchain, /mnt/lfs/tools).

Tcl was supposed to take about 25 minutes to build, but with the test suites it took a total of 38 minutes to build.

I made the Tcl libraries writable so I can strip debugging symbols at the end of the temporary toolchain build process (to recover extra space, and presumably speed things up a bit). Next, I installed Tcl’s private headers, since Expect needs them, and made a symlink to tclsh.

Expect

Expect is a popular extension to Tcl that makes it easy to script interactions with *nix programs (in fact, Expect is one of the main reasons people use Tcl in the first place).

Before compiling Expect, I ran a script to make it use the known-good /bin/stty; this prevents Expect from trying to use a (non-existent) /usr/local/bin/stty that can be present in the live CD build environment. I configured Expect to look in /tools/lib (in the temporary toolchain, /mnt/lfs/tools/lib) for the freshly compiled Tcl libraries, and to build without the included Expect scripts, since they are unneeded. I also ran the optional test suite, which experienced eleven failures, but the LFS book says not to worry about Expect test suite failures, since they apparently fail at random with no consequences (“under certain host conditions that are not within our control” to be exact). Expect is installed to /tools (in the temporary toolchain, /mnt/lfs/tools).

I expected Expect to build in 5 minutes, but instead Pressie expectorated Expect in 6 minutes.

DejaGNU

DejaGNU is a program testing framework written with Expect.

LFS uses DejaGNU 1.4.4, which was released in 2004. Naturally, GNU released DejaGNU 1.5 six days after the LFS 6.8 book was released 😛 .

However, I’m using version 1.4.4, so I applied a patch containing several fixes to the old version before compiling. DejaGNU is built and installed in a single step, so had to run the optional test suite after the installation rather than before. DejaGNU installed to /tools (in the temporary toolchain, /mnt/lfs/tools).

DejaGNU was supposed to build in less than 5 minutes, and Pressie delivered a shining build time of 2 minutes.

Ncurses

Ncurses is a handy set of libraries that lets programs define a text interface without having to rewrite the interface for every different terminal program. Think of it as a text counterpart to GTK+ or Qt.

Ncurses was fairly easy to compile, with a few options enabled that ensure it will be usable once I chroot into the final LFS system. I did not run the test suites for Ncurses since they can only be run once Ncurses is installed, and the LFS book did not provide explicit instructions for not screwing things up. Ncurses is installed to /tools (in the temporary toolchain, /mnt/lfs/tools).

Estimated build time was 34 minutes, but Ncurses built in a brisk 24 minutes. Way to go, Pressie!

Bash

Bash is the Bourne-Again SHell, a Swiss Army tool for interacting with your computer in a super-efficient way on the command line. There are seriously way too many features and capabilities in Bash for me to do them justice: just know that there are a lot.

Bash has a built-in malloc (memory allocation) function, but it apparently breaks a lot, so I compiled Bash to use Glibc’s more stable malloc. Temporary toolchain Bash is installed to /tools (in the temporary toolchain, /mnt/lfs/tools).

Bash was supposed to build in 25 minutes, and it only took 30 minutes.

Finally, I created a sh symlink to bash. Similar to the cc symlink for gcc, this allows for scripts to be generic, and for system administrators to use a program of their choice.

Bzip2

Bzip2 is a compression/decompression utility that is newer and better than Gzip, but older and not quite as good as Xz. It is one of the most commonly used compression methods for distributing source code (though Gzip is still pretty popular).

Bzip2 is installed to /tools (in the temporary toolchain, /mnt/lfs/tools).

I expected Bzip2 to compile in less than 5 minutes, and Pressie breezed through in only 3 minutes.

Coreutils

Coreutils is a giant wad of basic system utilities. I think the only reason they’re all together is that it was easier than having dozens of individual packages that would all have to be installed anyway.

I needed to tweak Coreutils to build the hostname program, which the Perl test suite will need later on. Coreutils installed to /tools (in the temporary toolchain, /mnt/lfs/tools).

Coreutils was supposed to build in 34 minutes, but took 1 hour 35 minutes. (I ran the optional test suite, which is probably why Pressie took so long. No errors yet!)

One of the tests failed in the (optional) Coreutils test suite, but I didn’t record which one. I hope this doesn’t come back to bite me….

Lastly, I manually installed this temporary toolchain version of su as su-tools according the the LFS book instructions. su is not truly installable yet, as I’m building and installing the temporary toolchain as the user lfs, and only root can install a su that can setuid root. The manually installed su-tools will be used in tests in the final LFS system, and I’ll be able to keep a fully functional su accessible in the LFS live CD build environment.

Diffutils

Diffutils is a collection of tools for comparing files and directories.

Diffutils installed to /tools (in the temporary toolchain, /mnt/lfs/tools).

Diffutils was supposed to build in less than 5 minutes, but took 18 minutes. (Again, running the optional test suite likely inflated the numbers.)

File

File is a very old, very solid *nix program for determining filetypes and reading files. According to its authors, File is used in every known BSD and every known Linux distribution — which is pretty cool!

File is also installed to /tools (in the temporary toolchain, /mnt/lfs/tools).

Since File is a fairly small program, it was projected to compile in only 10 minutes, though it actually took 7 minutes, even with the test suite!

Findutils

Findutils has several GNU programs to find files, as well as to generate and maintain a file database (to find files faster).

Findutils is installed to /tools (in the temporary toolchain, /mnt/lfs/tools).

Findutils was supposed to compile in 15 minutes, though it actually took 16 minutes with the test suite.

Gawk

Gawk is the GNU implementation of the AWK language. AWK lets you script the manipulation of text files.

Gawk is installed to /tools (in the temporary toolchain, /mnt/lfs/tools).

Gawk was supposed to take 10 minutes to build, and actually took 10 minutes. Hooray! Even the test suite didn’t slow Pressie down!

More to come!

So far, no major problems have stopped my progress in building LFS. Let’s hope my luck holds!

LFS 6.8 (Part 5): Version Checks and Final Preparations

Continuing my LFS adventures, it is time to check that the base system (in this case, the LFS 6.3-r2160 live CD) has the necessary tools for successfully compiling LFS.

There is a handy version check bash script on page xviii of the LFS book that I ran. I typed it in by hand, but it was satisfying work because I learned some interesting text processing tidbits in the process. It turns out that everything on the live CD is the exact version that LFS 6.8 requires, except that grep is version 2.5.1, but the book lists the minimum required version as 2.5.1a — hopefully everything will work anyway.

Next, I set up the $LFS variable in bash. The $LFS variable is important because it is used throughout the book (trust me, I’ve looked) while building and installing LFS. I haven’t deviated from the book yet, so I was able to type in all the commands verbatim.

Then I created the $LFS/tools directory, which I will use while I build the temporary toolchain. Then I created a /tools symlink on the live CD / (root directory) — this means the toolchain will install its files to /mnt/lfs/tools (aka $LFS/tools), even though thanks to the symlink, they think they installed them in /tools. That way, when the first toolchain is completed and ready to build the second (clean) toolchain, and I have to chroot into $LFS, the toolchain can still look at /tools and find everything it needs.

Next, I followed the instructions for creating a new user — lfs — and setting up a known-good bash environment. I then used the source command to enter into my newly created bash environment: spartan, but effective.

Finally, the LFS book addresses SBUs (Standard Build Units) and test suites before beginning the first toolchain. SBUs are a rough measure of how long a particular package will take to compile. Skipping ahead some, my SBU is 49 minutes and 5 seconds, a far cry from the 3-odd minutes of the “fastest” systems the LFS book speaks of. The book also recommends skipping the test suites that are usually run on the first toolchain, since they offer near-zero benefit at this point.

I am now ready to build the first LFS toolchain. Onward, ho!

LFS 6.8 (Part 1): The Linux From Scratch Book

I have been fascinated by the idea of building my own Linux system for years: I was browsing through some old files of mine when I found Linux From Scratch resources I’d downloaded in February 2005! Now in 2011, I am finally taking the plunge and building my own LFS.

To my mind, making my own LFS system is a logical step in the process of learning how to program — I need to understand my tools, if for no other reason than to understand what wheels don’t need reinventing. Before jumping into things, I read the official LFS book entirely through and learned two core lessons:

  1. Linux is very easy to homebrew (conceptually, at least): build a preliminary toolkit, then use that to build the main toolkit, then build your own distro from the main toolkit.
  2. Linux was very hard to understand (conceptually, at least) until I read the LFS book: “Linux” is really not at all like the monolithic Win32 base I used for over a decade! Rather, the Linux “operating system” is a collection of simple tools, which can perform complex tasks when used together.

It took more than four months to finish it, but the LFS book was far and away some of the finest technical documentation I have ever read. The authors of the LFS book went out of their way to fully explain even tedious and potentially “obvious” things, and I have a much stronger grasp of what the most essential Linux tools are (hint: bash!) thanks to their labors.

I am also amazed by how many core tools hail from the GNU project — I wonder if anyone from GNU ever sleeps.

I hope to learn much by doing, and will be writing about my progress as I go along. I am only planning on making the LFS build at this point, but I may also progress to BLFS in order to get more experience with the X windowing system, which has baffled me utterly from day one. Happy hacking!

Bonus: My favorite quote of the book: “false [d]oes nothing, unsuccessfully”.

Alternative To The “200 Lines Kernel Patch That Does Wonders” Which You Can Use Right Away: It Worked For Me!

This article is absolutely priceless! Ever since I first heard about the 200-ish line kernel patch that improves Linux desktop interactivity by, oh, an order of magnitude or better, I’ve been champing at the bit for K/Ubuntu to update their  kernels with the patch… but apparently the patch was predated by some bash-y proof-of-concept magic. Apparently, a few tweaks in your configuration files can do the same thing as this amazing patch for those people who don’t immediately custom-roll a kernel when cool patches go out (read, “most people”). I was even more excited to see that there was even a Ubuntu-tailored version that is a little more involved thanks to Ubuntu-y quirks, but my desktop experience has gone from a “meh, it’s okay” to a “whoa!”

The smoothness (no more lag!) and snappiness of my desktop is absolutely stunning — things that before I just took for granted as slow (like HD video when I do anything else in the background, file operations, that pause before menu and notifications are actually drawn) are now practically realtime. Applications themselves take just as long to load, but now everything doesn’t… take… forever… to… finally stop lagging so I can get to work. I would definitely recommend at least trying this out (of course, after a thorough backup — just in case)

Kernel developers +1!

What I Learned In Python This Week #2: Lists, and How To Manipulate Them

Tools I Used:

  • Geany
    • My favorite mini-IDE!
  • Python shell / bash shell
    • I use the Python shell to learn functions interactively, and the bash shell to execute the scripts I write.

Accomplishments:

  • Date Translator (date.v1-1-1.py)
    • Kind of a bug release, I guess — I fixed a problem where single-digit days were showing up with an unnecessary initial zero.
  • Domain Name Translator (domainname.v1-0-0.py)
    • A simple script that strips the domain name from a URL. Basically just a proof-of-concept to help me understand lists in Python a bit better.
  • Quotation Box (quotationbox.v1-0-0.py)
    • Takes a line of user input and centers it in a ASCII text box in their terminal.

Functions/Concepts I Learned:

  • Lists
    • Lists are essentially sequences of data that you can manipulate in any number of ways — and manipulating data is what programming is all about!
    • [], [:], and [::] let you index, slice, and choose step size of the slice respectively. These are tools that by themselves only return the value of a list, but can be used in conjunction with other functions can powerfully manipulate lists or parts of lists.
    • None is Python’s concept of, not zero, but nothing at all; it is a null or an unset value. This is useful for filling a list up that you will put data in later.
    • len(), min(), and max() let you check the length of a list, the smallest member in a list, and the largest member in a list respectively.
    • del lets you delete members from a list.
  • Tuples
    • I think tuples are a bit above me at this point, but I understand them to be similar to lists, only non-modifiable once created. Useful for dictionaries, or perhaps an internal checksum.
  • Methods
    • Methods are really awesome! They are like functions, only you tell methods to work only on specific parts of a sequence, rather than the whole. Several of these can be implemented using slices, but using methods is less ugly looking and much more human-readable.
    • .append adds a single data point to the end of a list, while .extend lets you add multiple data points to the end of a list (including adding other lists).
    • .count counts the occurrences of elements in a list, but does not examine sublists (i.e., for numbers = [1,2,1,1,2], numbers.count(2) will return 2, but for numbers = [1,2,1,[1,2]], it would return only 1).
    • .index returns the index value of the first occurrence of your search term in a given list, and .remove will search for and remove the first occurrence of the term. However, these only deal with the first occurrence — if the value appears multiple times in a list, you will need multiple .index/.remove operations to deal with them, or else some other Python function I don’t know about yet.
    • .pop is interesting because it “pops” (i.e., deletes) the last entry in a list out and returns the value. According to Hetland, this is a way to implement a stack, a word which I have heard a lot concerning programming, but have never really heard explained.
    • .reverse reverses the elements in a list (not just returning them reversed like the reversed function).
    • .sort sorts the elements in a list in ascending order (not just returning them sorted like the sorted function). You can use the cmp, key, and reverse arguments to sort by various other means than the default ascending order.
  • list() converts a sequence to a list, and tuple() similarly converts a sequence to a tuple.
  • cmp lets you compare two values, returning 0 for a match, 1 when the first value is larger, and -1 when the first value is smaller.

This week I finally got lists, and it was very rewarding; I feel much more confident to take on the rest of Python now. While I didn’t learn any cool new functions, I learned a lot about how to put lists to work for me, and I think it was worth it. Next week, I will try to get through Hetland’s chapter on strings, and I think it will be similarly devoid of new functions, but I’ll sure learn a lot about how to play around with strings!

What I Learned In Python This Week #1: First Impressions

My goal with the “What I Learned” series of posts is to give a rough overview of my skill progression in programming various languages (currently Python) that I can use both for personal edification and for a benchmark of my progress. (And yes, this post is more like “this month” than “this week”, but that’s what I get for coming up with ideas too late 😛 .)

Tools I Used:

  • Geany
    • Geany is a really nice text editor focused around Python development (kind of a mini-IDE).
  • IDLE
    • I haven’t used it much, but so far IDLE seems to be nicer than a plain Python shell because of highlighting and a potentially useful (but ugly) GUI interface. Probably has many cooler features that I will explore later.
  • Python shell / bash shell
    • I use the Python shell to use Python interactively (you know, like testing the print “Hello, World!” command to see how it works) and I use bash to execute my scripts.

Accomplishments:

  • Hello, World! (hello.py, hello.v1-1.py)
    • Hello, World! is my first program, and an exercise in using the print and raw_input functions of Python.
  • Date Translator (date.py, date.v1-1.py)
    • Date Translator is my second program, and is an exercise in more complex processing techniques by converting strings/integers and using lists.
  • A slightly better understanding of Python 😉

Functions/Concepts I Learned:

  • print, Strings, and Integers
    • The print function is linked (in my mind) with strings and integers because it is used to print strings and/or integers on your computer’s screen.
    • Strings are, well, strings of data. They can be text, special characters, numbers, command characters, or even a string made of other strings. Strings are the bread and butter of programming because of their versatility and usefulness for holding/representing data. You can use str, repr, and `` (backticks) to convert data to a string.
    • Integers are numbers. You can use basic math operators (+, -, /, and *) in Python by default, and you can load the math or cmath modules (more on modules later) to do more complex operations with integers. You can use int to convert numerical data in a string to an integer.
  • Variables
    • Variables are extremely powerful tools. They are somewhat like desktop links to a program or document in that they do not mean anything by themselves, but they refer to something else. This means their use is only limited by your imagination (and your precision in typing skills after n cups of coffee late at night 🙄 ).
  • input and raw_input
    • Get input from your user! Mostly raw_input is better because input acts like the user is at a Python shell and interprets their input like a Python statement, whereas raw_input just converts their input into a string.
  • Lists
    • Lists are basically sequences of data. You can use them to store preconfigured data or user input. I’ve yet to get a handle on them entirely, so that’s all I’m definitely sure of so far 😳 .
  • Algorithms/Programs
    • For the purposes of the small programs I’ve written so far, these are basically the same. An algorithm is the method/steps you use to solve a given problem; a program is a particular implementation in code of one or more algorithms.

I’ve both learned a lot and not much at the same time. These are basic concepts to computer programming, and are therefore very powerful tools, but I haven’t learned much about how to put them together to write programs much bigger than a few dozen lines… yet.

Right now I feel a lot like when I’m learning the alphabet for a foreign language: I’m learning basic, super-essential things, but it’s kinda boring to slog through them at first and (seemingly) not very useful because I don’t yet know the specific vocabulary, grammar, semantics, context, etc. which would let me know how to practically apply them. Oh well, soldier on!