'Top Level Packages' Script for FreeBSD

A BOM-like approach to ports dependencies

One of the problems I've run into with FreeBSD is the difficulties I often have when upgrading the installed 'packages' or 'ports' using the FreeBSD ports system. I really like the fact that everything is designed to COMPILE FROM SOURCE, which means that if you install a library, the things you need for a program to use it are installed along with the library itself.

But for every UP side, there's a yin/yang type of DOWN side. The DOWN side in the case of the FreeBSD ports system is the TIME it takes to build everything from source, and that changing an insignificant library that the universe depends on requires, well, RE-BUILDING EVERYTHING THAT DEPENDS UPON IT to ensure binary compatibility. That can be pretty time-consuming (2 to 3 days of compiling, based on my last update).

Still, if you want SUPER RELIABILITY, building from source is probably your best bet. It minimizes the chances of binary incompatibility, assuming that the applications themselves aren't buggy.

I have found, however, that when I want to upgrade after a LONG time of 'not upgrading' (due to the machine down-time involved) that it's better to DUMP EVERYTHING and re-install from scratch. And to figure out what "top level" ports I need to install in order to drive dependencies for all of the libraries and other support stuff, I came up with a script that easily lets me list them.


Script: top.level.packages.sh

#!/bin/sh

  if test -e /tmp/p2.txt ; then rm /tmp/p2.txt ; fi
  if test -e /tmp/p3.txt ; then rm /tmp/p3.txt ; fi
  if test -e /tmp/p4.txt ; then rm /tmp/p4.txt ; fi

  if test -e /tmp/p3a.txt ; then rm /tmp/p3a.txt ; fi
  if test -e /tmp/p3c.txt ; then rm /tmp/p3c.txt ; fi

  echo generating initial list of ports not depended on...

  pkg_info | awk '{ print $1 }' >/tmp/p1.txt

  for xx in `cat /tmp/p1.txt` ; do 
    pkg_info -qR $xx > /tmp/p1a.txt
    echo $xx `wc -l /tmp/p1a.txt | awk '{ print "_" $1 "_" }'` >>/tmp/p2.txt
    rm /tmp/p1a.txt
  done

  grep "_0_" /tmp/p2.txt | awk '{ print $1 }' > /tmp/p3.txt

  rm /tmp/p1.txt
  rm /tmp/p2.txt

  echo determine build plus run depends for each top-level port

  for xx in `cat /tmp/p3.txt` ; do
    yy=`pkg_info -o $xx | grep /`
    cd /usr/ports/${yy}
    make run-depends-list >>/tmp/p3a.txt
    make build-depends-list >>/tmp/p3a.txt
    make all-depends-list >>/tmp/p3a.txt
  done

  sort -u /tmp/p3b.txt
#  echo line count `wc -l /tmp/p3b.txt`

  echo generating port names from run/build depends list

  for xx in `cat /tmp/p3b.txt` ; do
    vv=$(echo $xx | sed 's/\/usr\/ports\///g') ; pkg_info -q -O $vv >>/tmp/p3c.txt
  done

  sort -u /tmp/p3d.txt
  sort -u /tmp/p3e.txt

#  echo line counts `wc -l /tmp/p3d.txt` "," `wc -l /tmp/p3e.txt`

  echo generate actual list of top-level ports

  diff /tmp/p3e.txt /tmp/p3d.txt | grep '<' | awk '{ print $2 }' >/tmp/p4.txt

  rm /tmp/p3.txt /tmp/p3a.txt /tmp/p3b.txt /tmp/p3c.txt /tmp/p3d.txt /tmp/p3e.txt

#  echo line count: `wc -l /tmp/p4.txt`

  echo list of top-level ports placed into /tmp/p4.txt



How it works

The script uses a straightforward approach that you would use if you were attempting to 'level' a bill of material. First of all, you want to find out which ports/packages have nothing that depend on them. Using the FreeBSD application 'pkg_info' I can query for anything that has an empty "packages that depend on this one" list. So far so good. Unfortunately, this list is not completely accurate, so I have to do some additional filtering.

To account for 'run dependencies' vs 'build dependencies', which do not always show up as 'dependencies' from the pkg_info application, I grab lists of run dependencies, build dependencies, and all dependencies from the top-level ports list I just created. The list is sorted and 'uniqued' so that there's only one entry for each package/port. This becomes a comprehensive list of "things that are depended upon" which will be automatically installed if "the thing that depends on them" (i.e. a top-level port) is installed.

Once I have the list of things 'depended upon', I want to remove ANY entry from the 'top level' list that has a corresponding entry in the 'dependend upon' list. This will give me a complete list of ports that have NOTHING depending on them. The easiest way to do this is to use the 'diff' application. You first sort the two lists alphabetically, and then use 'diff' to determine which entries ONLY exist in the 'top level' list by searching for the '<' and using 'awk' to only print out the package/port name.

The Final Steps

The script is simple, and writes the trimmed list of top-level ports into /tmp/p4.txt . You can change this as you see fit. But the list of packages, being interesting and all, cannot be DIRECTLY used to re-install ports from scratch. To do THAT, I execute another command to get the 'origin' directory for each installed port:

pkg_info -o `cat /tmp/p4.txt` | grep -v '^$' | grep -v ':$' >top.level.ports.txt

This gives me a list of the 'top level ports' as a set of directories, which I can then pass to an application such as 'portinstall'.

The Overall Process

The process itself takes a bit of time and discipline to set up properly. But if you do it THIS way, you should get good results, even if you haven't updated your ports in YEARS. You may find that 'top level' port directories have changed, though, so a subsequent manual install would be necessary. The 'portinstall' program will tell you when something can't be found, so you can use its output to assist you.

  1. Obtain a list of top-level ports:
    1. run 'top.level.packages.sh' to get a list of top level packages.
    2. run the 'pkg_info' command, above, writing the data into 'top.level.ports.txt'.
      NOTE: Typically I will place this file into a convenient directory, such as /tmp, /root or /usr/ports, so when you see this file name mentioned later on without a qualifying path, please refer to the FULL path instead (k-thx).
  2. Make sure you have 'ports-mgmt/portupgrade' already installed. You will most likely need it.
  3. Update the ports tree (this must be done AFTER getting the list of 'top level ports'). On FreeBSD, you can use the 'portsnap' utility.
    Other methods also exist, see the FreeBSD Handbook for more
  4. At this point you will probably want to download distributions and configure any options. Having an 'option' dialog box pop up while you're sleeping means HOURS of lost 'wall time' during the installation process. A good way to do this is to use the following command:
    portupgrade -c -n -O `cat top.level.ports.txt`
    portupgrade -F -O `cat top.level.ports.txt`
    
    The first command will configure all of the options. The second will fetch all of the additional source files. NOTE 1: If you build a NEW ports tree by first moving the old one (a good idea, so it's easier to 'go back to what you had'), you could use 'cp -Rl' to create symlinks to all of the old source tarballs in the new /usr/ports/distfiles directory tree. NOTE 2: If you can NOT install portupgrade for some reason, you can try uninstalling everything FIRST, and then do this step AFTER you get portupgrade installed with the new ports collection. It should still work, but you would have fewer options available for 'fall back'.
  5. Backups. You might want to back up certain files first, such as Samba 'TDB' files, just in case a deinstall removes THEM as well. If you're not sure, do a complete system backup just to be safe.
  6. Uninstall EVERYTHING by using
    pkg_delete "*"
    
  7. Now that you have NO ports installed, you must re-install things in the proper order.
    1. Install Perl. Determine which version you want. Typically it will be the newest one. At the time of this writing, the newest is /usr/ports/lang/perl5.16
    2. Install 'ports-mgmt/portupgrade'
      If you had to postpone the two 'portupgrade' commands from step 4, do them now, substituting 'portinstall' for 'portupgrade'.
    3. Install anything that you need a specific version for, if the default will give you the wrong version. In my case, I have to install 'subversion16' to deal with incompatibilities with version 1.7 with 'rapidsvn' (a GUI tool I happen to like). YMMV.
    4. Finally, install EVERYTHING ELSE with the following command (as root, assume 'csh' shell):
    portinstall `cat top.level.ports.txt` |& tee /usr/ports/install.log
      
  8. "Let her rip". It could take several days, fewer if you use 'FORCE_MAKE_JOBS' in make.conf

The reason you want to use the 'tee' command piped with stderr is that you want ALL OF THE ERROR MESSAGES to be in your log file in case something goes awry. For ME, something ALWAYS does. Pass 1 will reveal those ports that, for some reason, didn't build or install the way they should have, and 'portinstall' will then put a list of those ports (some of which are skipped due to missing dependencies) at the end of the log file. From this list you can manually install (and fix) problems with ports that would not build. Typically I just have to re-build them with everything else 'already installed properly'. Other times I may have to disable an option or two to prevent build issues, then continue forward with the rest of them. It's intuitively obvious that you could EDIT THE LIST OF PORTS based on the log output, then pass THOSE to 'portinstall' so you can continue forward. But if you want, you could STILL pass the original list to 'portinstall' again (it will ignore things that have already been installed, but you wanted to save TIME, right?).

FINALLY, you might want to check to see if there are ports that weren't installed because their 'origin' directory is no longer the same. I found that 'open office' wasn't reinstalled, so I did a subsequent build. Again, YMMV.


©2013 by Stewart~Frazier Tools, Inc. - all rights reserved
Last Update: 5/06/2013

Back to S.F.T. Inc. main page