Two Posts on the /usr Merge Relative to systemd That Were Filtered on debian-user

The debian-user list is not moderated, but sometimes they shut down threads that get out of hand.

I responded to two posts on a thread on the topic of the /usr merge, but my posts never made it to the list. I'm feeling stubborn, so I'm posting my posts here. But I don't want to mess with copyright issues, so I'll paraphrase what Hans and Jonathan said instead of quoting them.

I include links to their posts in the archives so you can decide if my interpretation of their posts is accurate.

[Now that I have the thing posted here and look back over it, I don't think I've interpreted Jonathan's post correctly, and I think I've dragged my responses out waaaaaay too long. (Leaving this here in case my posts ever make it through whatever tarpit the debian list server sent them to.)

Short version: So what if the recent /usr merge being pushed by the guys who bring us systemd is just another in a long line of attempts at removing a poorly understood organizational division? So what if we have big hard drives? 

The solution is not to abandon efforts to organize things. Rather, we should take a clue from such projects as Gobolinux (They aren't the only ones.) and go in the other direction -- even more separation. 

You should only read the following if you have time to waste.]

================================


(1) systemd and initrd and /usr [Questions by Hans.]

--------------------------------
Warning -- I do not use systemd.

> [Asking how to mount /usr and /usr/local to initrd]

I'll leave that for someone else.

> [The problem involves a mapped, encrypted /usr partition. There is a question as to why systemd requires /usr and /usr/local to be the same directory, flying in the face of traditional unix practices of putting important directories in their own partitions, with the comment that the tradition is as old as SCSI disk drive technology. Concern that systemd is intended to sweep away the traditionalprinciple of partition separation.]

Partially [I mean, about changing the partitioning practices]. These might help:

https://news.ycombinator.com/item?id=3519952
http://wiki.gentoo.org/wiki//usr_move

You can find more by searching the web for things like "/bin /usr/bin merge" or "/usr merge".

(For me, it's a very depressing tale, the lame leading the blind, and all of us being dragged into the ditch, so to speak, but that's supposedly my personal opinion.)

> [And what about /var and /home ?]

They weren't stupid enough to require /home in the same partition.
/var, I'm not sure about. [After checking, yes, /var can also remain separate, with some caveats.] Some of what I read indicates that when they were slapped in the face with their idiocy by the problems of logging when /var couldn't be mounted, they chose to ignore reality. Which is just as well, since they now aren't trying too hard to force everyone into the world of one big partition.

> [Request for explanation of reasons and methods.]

I'll leave that to someone who agrees with new policy.

==========================


(2) systemd and initrd and /usr [Response by Jonathan.]

--------------------------------
My reply to Hans didn't make it to the list. Wonder why. Wonder if my comments on Jonathan's response will make it. [Neither have made it as I post this.]

>> [Question about /usr and /usr/local and primary directories being mounted  separately, see above.]

> [Assertion that something about the portrayal of separation as traditional was wrong.]

I'd say partially wrong. Or partially right concerning some, less right concerning others. At least, I think Hans is talking indirectly about the sizes of hard drives.

Anyway, there were many people involved and they were not all doing the same things at the same time.

> [Assertion that the /usr split was derived from diskspace limitations.]

Or something. People like to organize things for some reason, and sometimes they use incidental divisions as their organizational dividers.

Consider the tendency during the '80s to try to use the accidental segmentation of the 8086 to organize software in a sort of modularization. Of course, if you didn't happen to be in the management sessions where this was promoted as a "feature" of the 8086, and/or the later sessions where the use of long pointers had to be explained, and/or the debugging sessions where coders tried to explain to managers that you basically had to load both parts of the segmented pointer on every use to avoid obscure bugs, you may not be familiar with the intended example.

But I've sure seen it a lot. Come to think of it, when inheriting a desk, I often used to leave the dividers the previous user of the desk left as they were, thinking it might save me the time of setting up my own dividers.

> [Implied suggestion that we have been stuck with the split since then ...]

I wonder why it would seem to have stuck, considering that (as you point out) merges have been attempted so many times.

It hasn't really "stuck", of course, except as one of several default practices.

> [... implied assertion that the purpose of /usr was for holding login user account home directories that there would otherwise not be room enough for, pointing to the name as evidence ...]

Uh, no. There were many different people doing many different things, remember.

The history you are familiar with seems to be one thing some people were doing.

I'm familiar with a different history. Others are familiar with other histories.

However, the point you make further down that the current use is full of historical and functional inconsistencies is well made. That you seem to suggest that the merge is a good thing, is something I'll take exception to.

> [..., assertion that there is no more need ...]

I think the real reasons have not at all vanished, have only become more pronounced.

> [... because disks are big enough now, and login directories are now pretty much moved to /home or /user or /u , and the split occurred way before SASI/SCSI anyway.]

You've lost me a little on your time line. Mixing the time line and (probably unintentionally) cherry-picking examples may be what leads you to the conflation of /usr and /Users.

> [Lamenting that some people don't understand history.]

Sad? Maybe.

Little to no awareness? Sure. I don't know anyone who knows the whole history.

There was lots of history, lots of different shops doing lots of different things.

We didn't have the internet, or even its predecessor, connecting most of the people who were using unix like we did in the 1980s and later. People got their information from the vendor or support group, from conferences (once a year, maybe), from magazines, and from private discussions and correspondence.

> [A reference to Stephen Coffin, in UNIX System V Release 4: The Complete Reference (1987) observing that /bin has become a link to /usr/bin , and /lib a link to /usr/lib in Unix System 5 Release 4, and that the change was part of changing the file system structure in Release 4.]
 
Which was a bit of a controversial release, and the book was also a bit controversial. Not even everyone who used release 4 thought that book was worth the paper it was printed on. Others thought it was wonderful, IIRC.
But such is the substance of history books (as opposed to the substance of history, itself).

> [Lamenting that some people think the idea of merging is of recent invention.]
 
Sad? I don't know. Not unusual, sure. Not exactly optimal, under some definitions of optimal, yeah.

Avoidable confusion? No.

Preventable confusion? No.

Should we all learn more from history? Absolutely.

> [Discussion of history of merges, The 2010 merge in Solaris 11 being pointed to as precedent to the recent /usr merge in the Linux community, but Solaris 11 being preceded by at least 20 years.]

Evidence that people who don't learn from history tend to repeat it, I guess?

> [Assertion that the motivation for the merge is that /usr is no longer on a separate volume, ...]
 
You mean, because we no longer need to keep our volumes small, I think.

> [... or at least will be mounted early enough that programs, libraries, and other resources in /usr will be available in time for the boot process to use them, ...]

Which early mounting is, as we all know, difficult to achieve in practice, because of the differences in vendor/distribution (both hardware and software) implementations.

> [... and then pointing out that the boot process in the initramfs must indeed explicitly mount /usr if it is in a separate partition.]

Because mount order and timing is so hard to enforce across all hardware/software combinations.

> [Declining to further explain, deferring to other's explanations:
 * https://fedoraproject.org/wiki/Features/UsrMove 
 * http://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/
]

Which is an approach to the problem taken by people with a specific goal, and ignores a lot of issues they consider side-issues. (If they acknowledge the issues at all.)

But I think they (and you?) are focusing on the size limitation motivation for the separation and ignoring the need for structure and organization.

> [Pointing out that the opinion was expressed as early as the early 1980s, that dividing things between the root file system and the /usr file system was a mess resulting from the historical accident ...]

Would it be wrong to say, rather, "... some people were under the impresssion that /usr was a mess caused by historical accident."?

> [Reference to a 1986 Unix system administration book by David Fielder ...]

David Fiedler, for those who want to look him up. (My fingers get crossed when I'm typing fast, too. :-/)

> [... and Bruce Hunter, who gave up on trying to explain the division, calling it a "hodge-podge" "overflow" "grab-bag". Also invoking a computer historian, Rob Landley, recently agreeing with Fiedler and Hunter:
* http://lists.busybox.net/pipermail/busybox/2010-December/074114.html
]
 
The nice thing about experts is you can usually find one to support the point of view you have. :-/

I look and find others that describe the organization as somewhat arbitrary, accidental, and so forth, but stay short of calling it miscellany.

> [Assertion that the /usr merge is not just Lennart Poettering's baby, neither merely a feature of systemd.]

You mean that it is not particularly original with the systemd project or with that project's founder, Lennart Poettering. And you're right about that. But it is one of the things that exists within the broader agenda of the loose group of projects that includes systemd.

> [Pointing out that people are on record as thinking it inconvenient and messy the year Poettering was born, and that failing to fix it is tantamount to failing to learn from history.]

Some people see a mess where others see incomplete structure. Or historical organization, as the case may be.

My timeline is a bit different from yours, by the way. From memory, and not properly consulting my records/sources:

** late 1960s ~ early 1970s:

In early unix systems, some integrators and end-users were separating binaries in /bin and /sbin by whether they were statically linked (and thus could be used early in the boot process) or not. /sbin was mounted earlier than /bin in many cases, but were on the same disk or partition in other cases. Also, if independent drives could be read concurrently, it was convenient to have executables and libraries spread across more than one drive.

It was said (by many) to be more convenient to always separate the binaries so you could mount them separately when you needed to, which, due to disk sizes those days, was not too rare.

[This is not a problem of size, but of organization, although the problem of organization can currently be ignored somewhat if the boot volume is large enough to contain both the root file system, with /bin and /lib , et. al., and /usr and all that is under that.]

(Note that dynamic linking was not available for the earliest unix systems. Also note that "vendor" in this context is most often a department within Bell, rather than Bell, itself.)

(Also note that many of the patterns and practices used in Unix were inherited from extant non-Unix OSses that we don't hear of much anymore, so what I'm describing here is naturally mixing part of unix history with the history of those other OSses.)

/usr was not present on all unix systems during much of this time.

Where it was present, it was, in fact, sometimes used as the home directory for user accounts, when the system integrator or whoever was setting it up wanted to collect all login directories under a single tree.

But it was more often intended as a place to localize all the customizations for a customer ("user", in the broader senses of end-user, most often a department within Bell). (If you're saying, "Wait! Wasn't that /opt or /usr/local?", slow down. You're jumping the gun. That was in a later iteration.)

(Sometimes, people who put login directories directly under /usr were chided for confusing "usr" for "user", in a possibly misguided effort to get them to understand the difference between "end-user" stuff and "user account" stuff.)

IIRC, there were floppy-only systems that had root, including /sbin, on one floppy, and /bin on another. And they might have /usr on a third.

** mid ~ late 1970s:

We actually had people talking about "mature unix systems" already. Heh.

Much of the root tree structure was becoming common, if not exactly standard -- /bin , /sbin , /lib , /include (!!), /etc , and so forth. Vendors had their own quirks, and directory names were not exactly reliable indications of what they contained.

But /usr , as a place for customization, was possibly closer to a standard practice than /sbin for your start-up binaries. Sometime during this time, /include tended to get moved to /usr/include, and parallels of /bin , /sbin and /lib under /usr , for less general, more end-user oriented stuff became fairly common.

Many hard disk manufacturers got their products stabilized, so systems running on floppy became somewhat rare. Disk size was becoming a non-issue, but the organizational structure was still useful.

/sbin , in particular, became viewed more as an "important system binaries" directory, rather than a "statically linked binaries" directory, but it was still common to arrange for /sbin to be on the first volume to be mounted, whether /bin was or not.

** late 1970s ~ mid 1980s:

This was where Shugart's SASI interface became something of a standard, and where IBM and others refined it to SCSI.

But we should note that neither of these interfaces were original in providing the ability to connect multiple drives. We should also note that the size of a system was increasing, so even with "huge" 20M hard drives, it was useful to physically split the file system and the file system, and the organizational divisions provided natural boundaries to split the volumes on. Thus, /usr as a place for integrators to use in customizing an installation became somewhat standard.

The content of /usr/bin , /usr/sbin , and /usr/lib became fairly standardized as well, partly because of the BSD work making once optional utility and application programs easier to get and install.

So now it became desirable to have another place to put the end-user customizations. /usr/local and /opt were among several directories that came to be commonly used for that purpose around this time. /home for the user account directories was not yet as common.

Remember that there were no hard-set rules about the structure of the file system tree. In particular, unix-like OSses (OS-9 is one example that comes quickly to mind) would tend to vary significantly from the mainstream practices. Also, many integrators and end-users would re-organize the tree as they saw fit, generally seeking more meaningful names.

Various efforts at merging the binary and library directories were tried, as well. Putting everything executable in /usr/bin and putting links in /bin and /sbin was just one of many attempts.

** mid 1980s ~ mid-first-decade of 21st century:

/bin , /sbin , /lib , and other parts directly under root (including some of /etc) have been the focus of unix industry standardization efforts, also in Linux.

Stuff under /usr has been the focus of distribution- or vendor-specific standardization, in both unix and Linux. And /home has become something close to a standard place for user-level login account home directories.

And various efforts to re-structure the tree continued to be tried, including merging.

Note that there is a natural organization seen here:

-- core stuff, things that all players in the industry want to rely on;

-- value-added stuff, things that individual vendors want to use to differentiate their offerings;

-- customization stuff, the applications that are the real reason for buying the computer.

Note also that Solaris, the captive OS of Sun and now Oracle does not need the first two levels. They are vendor and integrator. Apple's Mac OS and iOS, likewise. Google's Android, likewise.

** mid 1990s ~ present:

Linux-based systems generally started a step or so behind mainstream unix in standardization efforts and such, but ultimately took the lead in general-purpose systems [in many ways, not all]. The kernel developers borrowed lots of ideas from embedded systems and non-unix unix-like systems, including alternate file system structures.

The BSD derivatives have not [been] as popular as the Linux-based systems, but there is a lot of sharing, so there is a bit of convergence, a lot of similar experiments.

The three pseudo-layers of the file system tree that [I] specified above have continued to be convenient for the various distributions (as virtual vendors) to use, to show their stuff in. They have been traditionally reflected in the division of / , /usr , and /usr/local , or the several variations thereof.

Merging /bin with /usr/bin seems to me to be nothing less that pushing one set of integrations into the implied standard.

Unfortunately, in spite of the borrowings from embedded systems, one desktop-oriented file system layout now gets a lot of attention, and by the middle of the first [second, I meant] decade of the new century, has come to be seen as the one-and-only goal by a small, vocal, and inordinately influential group.

We missed the news. Linux on the desktop was Mac OS X. The fact that MkLinux was dropped for FreeBSD is only incidental. Linux and BSD are twins. We "won" the desktop. But there seem to be lots of engineers who wanted to be leading the charge, and they seem to want a re-match.

Okay, the last two paragraphs are somewhat speculative on my part.

The point I'm trying to make is that the /usr merge seems to me to be motivated by the assumption that the integrations thought to be appropriate for a certain desktop environment are necessarily appropriate and correct for all environments that use the Linux kernel. It's one of several such changes that are being pushed by a group of projects which includes systemd.

And, while I have seen noise about how the merge is more convenient to certain developers who take a certain class of approaches to desktop systems, for my purposes, I'd far rather see the approach taken by the GoboLinux project become the standard, if we have to have a single standard layout.

==========================

Not exactly a satisfactory exposition of the questions. It leaves the question dangling, of whether there is a better organization, and I think there is.

Gobolinux is heading in the right direction, but common practices in Linux application packages interfere with taking the next step. And completely effacing all organization is just going to make those bad programming habits worse.