Two Posts on the /usr Merge Relative to systemd That Were Filtered on debian-user

The debian-user list is not moderated, but sometimes they shut down threads that get out of hand.

I responded to two posts on a thread on the topic of the /usr merge, but my posts never made it to the list. I'm feeling stubborn, so I'm posting my posts here. But I don't want to mess with copyright issues, so I'll paraphrase what Hans and Jonathan said instead of quoting them.

I include links to their posts in the archives so you can decide if my interpretation of their posts is accurate.

[Now that I have the thing posted here and look back over it, I don't think I've interpreted Jonathan's post correctly, and I think I've dragged my responses out waaaaaay too long. (Leaving this here in case my posts ever make it through whatever tarpit the debian list server sent them to.)

Short version: So what if the recent /usr merge being pushed by the guys who bring us systemd is just another in a long line of attempts at removing a poorly understood organizational division? So what if we have big hard drives? 

The solution is not to abandon efforts to organize things. Rather, we should take a clue from such projects as Gobolinux (They aren't the only ones.) and go in the other direction -- even more separation. 

You should only read the following if you have time to waste.]

================================


(1) systemd and initrd and /usr [Questions by Hans.]

--------------------------------
Warning -- I do not use systemd.

> [Asking how to mount /usr and /usr/local to initrd]

I'll leave that for someone else.

> [The problem involves a mapped, encrypted /usr partition. There is a question as to why systemd requires /usr and /usr/local to be the same directory, flying in the face of traditional unix practices of putting important directories in their own partitions, with the comment that the tradition is as old as SCSI disk drive technology. Concern that systemd is intended to sweep away the traditionalprinciple of partition separation.]

Partially [I mean, about changing the partitioning practices]. These might help:

https://news.ycombinator.com/item?id=3519952
http://wiki.gentoo.org/wiki//usr_move

You can find more by searching the web for things like "/bin /usr/bin merge" or "/usr merge".

(For me, it's a very depressing tale, the lame leading the blind, and all of us being dragged into the ditch, so to speak, but that's supposedly my personal opinion.)

> [And what about /var and /home ?]

They weren't stupid enough to require /home in the same partition.
/var, I'm not sure about. [After checking, yes, /var can also remain separate, with some caveats.] Some of what I read indicates that when they were slapped in the face with their idiocy by the problems of logging when /var couldn't be mounted, they chose to ignore reality. Which is just as well, since they now aren't trying too hard to force everyone into the world of one big partition.

> [Request for explanation of reasons and methods.]

I'll leave that to someone who agrees with new policy.

==========================


(2) systemd and initrd and /usr [Response by Jonathan.]

--------------------------------
My reply to Hans didn't make it to the list. Wonder why. Wonder if my comments on Jonathan's response will make it. [Neither have made it as I post this.]

>> [Question about /usr and /usr/local and primary directories being mounted  separately, see above.]

> [Assertion that something about the portrayal of separation as traditional was wrong.]

I'd say partially wrong. Or partially right concerning some, less right concerning others. At least, I think Hans is talking indirectly about the sizes of hard drives.

Anyway, there were many people involved and they were not all doing the same things at the same time.

> [Assertion that the /usr split was derived from diskspace limitations.]

Or something. People like to organize things for some reason, and sometimes they use incidental divisions as their organizational dividers.

Consider the tendency during the '80s to try to use the accidental segmentation of the 8086 to organize software in a sort of modularization. Of course, if you didn't happen to be in the management sessions where this was promoted as a "feature" of the 8086, and/or the later sessions where the use of long pointers had to be explained, and/or the debugging sessions where coders tried to explain to managers that you basically had to load both parts of the segmented pointer on every use to avoid obscure bugs, you may not be familiar with the intended example.

But I've sure seen it a lot. Come to think of it, when inheriting a desk, I often used to leave the dividers the previous user of the desk left as they were, thinking it might save me the time of setting up my own dividers.

> [Implied suggestion that we have been stuck with the split since then ...]

I wonder why it would seem to have stuck, considering that (as you point out) merges have been attempted so many times.

It hasn't really "stuck", of course, except as one of several default practices.

> [... implied assertion that the purpose of /usr was for holding login user account home directories that there would otherwise not be room enough for, pointing to the name as evidence ...]

Uh, no. There were many different people doing many different things, remember.

The history you are familiar with seems to be one thing some people were doing.

I'm familiar with a different history. Others are familiar with other histories.

However, the point you make further down that the current use is full of historical and functional inconsistencies is well made. That you seem to suggest that the merge is a good thing, is something I'll take exception to.

> [..., assertion that there is no more need ...]

I think the real reasons have not at all vanished, have only become more pronounced.

> [... because disks are big enough now, and login directories are now pretty much moved to /home or /user or /u , and the split occurred way before SASI/SCSI anyway.]

You've lost me a little on your time line. Mixing the time line and (probably unintentionally) cherry-picking examples may be what leads you to the conflation of /usr and /Users.

> [Lamenting that some people don't understand history.]

Sad? Maybe.

Little to no awareness? Sure. I don't know anyone who knows the whole history.

There was lots of history, lots of different shops doing lots of different things.

We didn't have the internet, or even its predecessor, connecting most of the people who were using unix like we did in the 1980s and later. People got their information from the vendor or support group, from conferences (once a year, maybe), from magazines, and from private discussions and correspondence.

> [A reference to Stephen Coffin, in UNIX System V Release 4: The Complete Reference (1987) observing that /bin has become a link to /usr/bin , and /lib a link to /usr/lib in Unix System 5 Release 4, and that the change was part of changing the file system structure in Release 4.]
 
Which was a bit of a controversial release, and the book was also a bit controversial. Not even everyone who used release 4 thought that book was worth the paper it was printed on. Others thought it was wonderful, IIRC.
But such is the substance of history books (as opposed to the substance of history, itself).

> [Lamenting that some people think the idea of merging is of recent invention.]
 
Sad? I don't know. Not unusual, sure. Not exactly optimal, under some definitions of optimal, yeah.

Avoidable confusion? No.

Preventable confusion? No.

Should we all learn more from history? Absolutely.

> [Discussion of history of merges, The 2010 merge in Solaris 11 being pointed to as precedent to the recent /usr merge in the Linux community, but Solaris 11 being preceded by at least 20 years.]

Evidence that people who don't learn from history tend to repeat it, I guess?

> [Assertion that the motivation for the merge is that /usr is no longer on a separate volume, ...]
 
You mean, because we no longer need to keep our volumes small, I think.

> [... or at least will be mounted early enough that programs, libraries, and other resources in /usr will be available in time for the boot process to use them, ...]

Which early mounting is, as we all know, difficult to achieve in practice, because of the differences in vendor/distribution (both hardware and software) implementations.

> [... and then pointing out that the boot process in the initramfs must indeed explicitly mount /usr if it is in a separate partition.]

Because mount order and timing is so hard to enforce across all hardware/software combinations.

> [Declining to further explain, deferring to other's explanations:
 * https://fedoraproject.org/wiki/Features/UsrMove 
 * http://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/
]

Which is an approach to the problem taken by people with a specific goal, and ignores a lot of issues they consider side-issues. (If they acknowledge the issues at all.)

But I think they (and you?) are focusing on the size limitation motivation for the separation and ignoring the need for structure and organization.

> [Pointing out that the opinion was expressed as early as the early 1980s, that dividing things between the root file system and the /usr file system was a mess resulting from the historical accident ...]

Would it be wrong to say, rather, "... some people were under the impresssion that /usr was a mess caused by historical accident."?

> [Reference to a 1986 Unix system administration book by David Fielder ...]

David Fiedler, for those who want to look him up. (My fingers get crossed when I'm typing fast, too. :-/)

> [... and Bruce Hunter, who gave up on trying to explain the division, calling it a "hodge-podge" "overflow" "grab-bag". Also invoking a computer historian, Rob Landley, recently agreeing with Fiedler and Hunter:
* http://lists.busybox.net/pipermail/busybox/2010-December/074114.html
]
 
The nice thing about experts is you can usually find one to support the point of view you have. :-/

I look and find others that describe the organization as somewhat arbitrary, accidental, and so forth, but stay short of calling it miscellany.

> [Assertion that the /usr merge is not just Lennart Poettering's baby, neither merely a feature of systemd.]

You mean that it is not particularly original with the systemd project or with that project's founder, Lennart Poettering. And you're right about that. But it is one of the things that exists within the broader agenda of the loose group of projects that includes systemd.

> [Pointing out that people are on record as thinking it inconvenient and messy the year Poettering was born, and that failing to fix it is tantamount to failing to learn from history.]

Some people see a mess where others see incomplete structure. Or historical organization, as the case may be.

My timeline is a bit different from yours, by the way. From memory, and not properly consulting my records/sources:

** late 1960s ~ early 1970s:

In early unix systems, some integrators and end-users were separating binaries in /bin and /sbin by whether they were statically linked (and thus could be used early in the boot process) or not. /sbin was mounted earlier than /bin in many cases, but were on the same disk or partition in other cases. Also, if independent drives could be read concurrently, it was convenient to have executables and libraries spread across more than one drive.

It was said (by many) to be more convenient to always separate the binaries so you could mount them separately when you needed to, which, due to disk sizes those days, was not too rare.

[This is not a problem of size, but of organization, although the problem of organization can currently be ignored somewhat if the boot volume is large enough to contain both the root file system, with /bin and /lib , et. al., and /usr and all that is under that.]

(Note that dynamic linking was not available for the earliest unix systems. Also note that "vendor" in this context is most often a department within Bell, rather than Bell, itself.)

(Also note that many of the patterns and practices used in Unix were inherited from extant non-Unix OSses that we don't hear of much anymore, so what I'm describing here is naturally mixing part of unix history with the history of those other OSses.)

/usr was not present on all unix systems during much of this time.

Where it was present, it was, in fact, sometimes used as the home directory for user accounts, when the system integrator or whoever was setting it up wanted to collect all login directories under a single tree.

But it was more often intended as a place to localize all the customizations for a customer ("user", in the broader senses of end-user, most often a department within Bell). (If you're saying, "Wait! Wasn't that /opt or /usr/local?", slow down. You're jumping the gun. That was in a later iteration.)

(Sometimes, people who put login directories directly under /usr were chided for confusing "usr" for "user", in a possibly misguided effort to get them to understand the difference between "end-user" stuff and "user account" stuff.)

IIRC, there were floppy-only systems that had root, including /sbin, on one floppy, and /bin on another. And they might have /usr on a third.

** mid ~ late 1970s:

We actually had people talking about "mature unix systems" already. Heh.

Much of the root tree structure was becoming common, if not exactly standard -- /bin , /sbin , /lib , /include (!!), /etc , and so forth. Vendors had their own quirks, and directory names were not exactly reliable indications of what they contained.

But /usr , as a place for customization, was possibly closer to a standard practice than /sbin for your start-up binaries. Sometime during this time, /include tended to get moved to /usr/include, and parallels of /bin , /sbin and /lib under /usr , for less general, more end-user oriented stuff became fairly common.

Many hard disk manufacturers got their products stabilized, so systems running on floppy became somewhat rare. Disk size was becoming a non-issue, but the organizational structure was still useful.

/sbin , in particular, became viewed more as an "important system binaries" directory, rather than a "statically linked binaries" directory, but it was still common to arrange for /sbin to be on the first volume to be mounted, whether /bin was or not.

** late 1970s ~ mid 1980s:

This was where Shugart's SASI interface became something of a standard, and where IBM and others refined it to SCSI.

But we should note that neither of these interfaces were original in providing the ability to connect multiple drives. We should also note that the size of a system was increasing, so even with "huge" 20M hard drives, it was useful to physically split the file system and the file system, and the organizational divisions provided natural boundaries to split the volumes on. Thus, /usr as a place for integrators to use in customizing an installation became somewhat standard.

The content of /usr/bin , /usr/sbin , and /usr/lib became fairly standardized as well, partly because of the BSD work making once optional utility and application programs easier to get and install.

So now it became desirable to have another place to put the end-user customizations. /usr/local and /opt were among several directories that came to be commonly used for that purpose around this time. /home for the user account directories was not yet as common.

Remember that there were no hard-set rules about the structure of the file system tree. In particular, unix-like OSses (OS-9 is one example that comes quickly to mind) would tend to vary significantly from the mainstream practices. Also, many integrators and end-users would re-organize the tree as they saw fit, generally seeking more meaningful names.

Various efforts at merging the binary and library directories were tried, as well. Putting everything executable in /usr/bin and putting links in /bin and /sbin was just one of many attempts.

** mid 1980s ~ mid-first-decade of 21st century:

/bin , /sbin , /lib , and other parts directly under root (including some of /etc) have been the focus of unix industry standardization efforts, also in Linux.

Stuff under /usr has been the focus of distribution- or vendor-specific standardization, in both unix and Linux. And /home has become something close to a standard place for user-level login account home directories.

And various efforts to re-structure the tree continued to be tried, including merging.

Note that there is a natural organization seen here:

-- core stuff, things that all players in the industry want to rely on;

-- value-added stuff, things that individual vendors want to use to differentiate their offerings;

-- customization stuff, the applications that are the real reason for buying the computer.

Note also that Solaris, the captive OS of Sun and now Oracle does not need the first two levels. They are vendor and integrator. Apple's Mac OS and iOS, likewise. Google's Android, likewise.

** mid 1990s ~ present:

Linux-based systems generally started a step or so behind mainstream unix in standardization efforts and such, but ultimately took the lead in general-purpose systems [in many ways, not all]. The kernel developers borrowed lots of ideas from embedded systems and non-unix unix-like systems, including alternate file system structures.

The BSD derivatives have not [been] as popular as the Linux-based systems, but there is a lot of sharing, so there is a bit of convergence, a lot of similar experiments.

The three pseudo-layers of the file system tree that [I] specified above have continued to be convenient for the various distributions (as virtual vendors) to use, to show their stuff in. They have been traditionally reflected in the division of / , /usr , and /usr/local , or the several variations thereof.

Merging /bin with /usr/bin seems to me to be nothing less that pushing one set of integrations into the implied standard.

Unfortunately, in spite of the borrowings from embedded systems, one desktop-oriented file system layout now gets a lot of attention, and by the middle of the first [second, I meant] decade of the new century, has come to be seen as the one-and-only goal by a small, vocal, and inordinately influential group.

We missed the news. Linux on the desktop was Mac OS X. The fact that MkLinux was dropped for FreeBSD is only incidental. Linux and BSD are twins. We "won" the desktop. But there seem to be lots of engineers who wanted to be leading the charge, and they seem to want a re-match.

Okay, the last two paragraphs are somewhat speculative on my part.

The point I'm trying to make is that the /usr merge seems to me to be motivated by the assumption that the integrations thought to be appropriate for a certain desktop environment are necessarily appropriate and correct for all environments that use the Linux kernel. It's one of several such changes that are being pushed by a group of projects which includes systemd.

And, while I have seen noise about how the merge is more convenient to certain developers who take a certain class of approaches to desktop systems, for my purposes, I'd far rather see the approach taken by the GoboLinux project become the standard, if we have to have a single standard layout.

==========================

Not exactly a satisfactory exposition of the questions. It leaves the question dangling, of whether there is a better organization, and I think there is.

Gobolinux is heading in the right direction, but common practices in Linux application packages interfere with taking the next step. And completely effacing all organization is just going to make those bad programming habits worse.

Getting Fedora 17 running on my netbook.

Okay, okay. I do need Fedora to study for the LPIC 2.

(Looking really carefully at Mint, for when I'm done and the root partition fills up.)

So, getting Fedora updated to v. 17 --

Start with the netinstall image. Use Fedora 16 live USB utility to load it onto a USB thumb drive.

yum install liveusb-creator

if you need it. (You can get the utility for MSWindows, too. And see the Fedora wiki on it for more details and options. It looks like there's a tool for Debian, too, I need to check that.)

I had to run liveusb-creator from the command line with the option to write the MBR (as suggested when it wrote the image the first time). The first try without the fix failed, but the second try with the MBR fixed worked. The command line with the option looks like this:

liveusb-creator -m

or

liveusb-creator --reset-mbr

The MBR may have been the reason my attempts at dd-ing the image didn't work.

So much, so good. Boot the USB image, either by using the BIOS settings to give USB devices priority in the boot order, or, if your BIOS supports it, by watching the first few seconds of bootup and hitting the key that it says allows you to choose the boot media.

I installed the minimal because I really wanted to keep control of the process. Was very careful to select custom layout. (Do not select any thing else or you will lose your data, if you are trying to do a "fresh" install like I was doing.)

Only set it up to grab the old swap partition and /tmp partition and the old root partition. Was very, very careful not to touch the old /home partition. (Did I say that enough times?)

Should have copied out the /etc first. Should have gotten a list of installed packages from yum before I shot myself in the foot trying to upgrade from F16 to F17 the easy way that doesn't work.

Gave it a root password, but no other users. Didn't want to have clutter in /home, since that is where the old home partition will mount.

Next, I asked around, to see if splitting /usr off after a successful install might work, and, no, the boot process needs stuff they've gone and moved to /usr. (See my diatribe linked above about root filling up.) So I'm stuck with a small monolithic root partition.

So, (if /dev/sda10 is my /home partition) log in as root and (back up /dev/sda10 first and)

blkid /dev/sda10 > home_uuid.text

vi /etc/fstab home_uuid.text

and use the :next and :prev vi commands to ytl (yank) the uuid and paste it in appropriately for the new line for the /home partition (saved in /dev/sda10).

Write fstab out (":wq") and quit vi and

mount -a

to bring the /home partition in and check your fstab. (Make sure you ls -la /home to be sure you didn't mistake the old /var partition for the old /home partition.)

Now, "useradd" with the "-u" option for the old user-ids and "--home" option appropriately for assigning the old home directories that are now in place.

yum install gpm

for using the mouse in a virtual console. Very handy, especially the right-click trick, to save you from typos.

yum groupinstall "X Window System"

yum groupinstall "Xfce"

[Update 30 Dec.: Had to take a break and go shopping at this point last night.]

and, lo, and behold,

systemctl start graphical.target

starts your XFCE desktop.

ls -l /etc/systemd/system/default.target 

ls -l /lib/systemd/system/graphical.target

to make sure you see what you are doing.

rm /etc/systemd/system/default.target

(Use sudo if you are being smart and are logged in as a non-root member of the wheel group.) Link the one in /lib in where you just removed the old link to multi-user. (Why was that the obsolete run-level-3 on mine?) Don't get the order wrong, or you'll cause yourself a world of hurt. (I wonder if there is a gui tool for this to protect you from when you fall asleep at the keyboard.)

ln -s /lib/systemd/system/graphical.target \ /etc/systemd/system/default.target 

(One line is fine, in which case you don't need the "\". And notice the difference between /lib and /etc and which one the symbolic link you are replacing is under.)

And, now that I have done that, I have to go wading through all the cool tools I had installed and bring them back in, one by one, starting with Firefox and fbreader and gcc and hexedit and lazarus and ....

(No, wait, lazarus is down the road a bit.)

[Update 31 Dec.:]
Oh. I want to leave a note to myself, from a post on the fedora user list archives.

When moving stuff, like moving the /var directory from the new OS to the /var partition from the old OS, mounted temporarily on /mnt, rsync seems to be your friend:

rsync -avxHSAX /var/* /mnt

diff -r /var /mnt 

• -a => archive
• -v => verbose
• -x => don't cross file system boundaries
• -H => preserve hard links
• -S => do sparse files efficiently
• -A => preserve ACLs
• -X => preserve extended attributes

and that seems (as shown by the recursive diff) to handle the file links better than

cp -a

• -a => archive or same as -dR --preserve=all

Theoretically, this is what I want to do when cloning or backing up a drive.

More stuff to play with study in my spare time that I never have.

(Mustn't forget to umount the partition before mkfs and, oh, best to boot from a live USB or CD so the OS isn't playing around and writing in the drive one is trying to copy as one tries to copy it. And such things.)

And one more thing. fixfiles -fF relabel is best not to do directly.

touch .autorelabel

and reboot, instead.

/usr merge and the tail wagging the dog

Fedora 16 17 is will probably be the end of the line for me.

Really wanted to keep my hand in, there, since I still need to take the LPIC level 2. But there are a few guys in there who are all gung ho over ripping out the flooring and restructuring the entire Linux file system layout to match their vision of layout of some other so-called operating system.

Microsoft envy. And hubris. Hubris without utility.

Here's what I'm fighting:

• /sda1: some boot-up partition for MSWindows. 7.
• /sda2: the usual "C:" partition with "System" and "Program Files", etc. 
• /sda3: the usual "D:" partition, except it contained a bunch of stuff for the bundled utilities, install packages and the like. If it weren't for the program to update my wireless router, I'd junk the whole of the MSWindows system.
• /sda4: the "restore" partition. Doesn't work, for what it's worth. Just breaks anything you installed since the last time you tagged it.

I suppose the C: partition had to be a MiSeryDOS basic partition. Would have been nice if they could have put it in a "logical" (i. e., nested in the extended) partition. But, noooo, Microsoft doesn't have any motivation for making it easy to install any other OS on the box for dual-booting, especially not a "competitor's" OS.

D: and the restore partition --No, there is no valid technical reason not to put those in a logical partition. Not that they (Lenovo) have any motivation to make it easy to install an alternate OS on the box, either, especially with the kickback from Microsoft. ("Volume" discounts only available to "A" rated integrators and OEMs, where "A" includes deliberately getting in the way of Microsoft's competition with games like these.)

Not that it would have helped. The restore function is hard-wired, against all technical reason, to the vendor installed partition layout. So the end result would be the same.

I described the above and how I originally installed Fedora 16 on the box elsewhere.

The way I did that, /sda4 ended up at 14.5 gigabytes. I little on the large side for a bot partition, but it has to be the boot partition because the boot partition has to be a basic partition, and there are only four of those, and /sda3 has to be a basic partition so it can be the enclosing extended partition for the other partitions I cut for the install: /usr, /home, /var, /tmp .

Now it's Fedora 16's end-of-life. Time to update to Fedora 17.

But.

Some hot-heads (apparently spearheaded by a group at freedesktop.org) got a bee in their bonnets about how "Everything ought to be in /usr. There is no technical reason to separate them now."

Makes me want to swear. Yes there are plenty of technical reasons, security, stability, and simply having a bit of division of purpose.

And Microsoft envy is not a good reason for changing that.

Sure, we no longer have the sort of situation they had in the early days when drives were small and you wanted to have all the stuff you needed to boot up in one set of directories and all the rest in another. Right?

lvm in /usr instead of /bin is a pain, yeah.

Plymouth in /usr is liveable, but it would be nice not to have to tell newbs why their cool start-up screens don't get loaded at startup. Yeah. I understand. But it's not a technical reason.

And ACLs (including the curious perversion of the concept that someone calls securelinux or something like that) mitigate the need to separate classes of binaries and libraries. Sure. I understand that you think such things.

You're wrong, but I understand that you think such things.

No, vendors never do stupid things like I described above, right?

RIGHT?

ARE YOU GUYS LISTENING?

No. You're not. I'm just a nameless slob, roadkill on your way to inventing reason for code churn and the accompanying job security.

I've lost about a month, total, to your need to prove how smart you are. A month of my time that I could have been writing bug reports on outstanding bugs. But it means nothing to you guys, because I'm a nameless slob out here in never-never-land.

When I hit the 15G limit on the root partition because /usr is packed and I can't put it on another partition because of your hubris, I'm gone.

Debian's engineers are smarter for the time being, but, if they roll over, shoot, configuring openbsd from scratch is going to take less time than figuring out work-arounds to your vanities.