Network file system AFS

If you have your home directory in AFS, or in some other way work with directories in AFS, you should know what is described here about AFS.

You find out if a certain directory is in AFS by issuing the command "df directory". If it is an AFS file system, AFS is written in the column to the far left. If you want to know if your home directory is in AFS, you can type "df ~".

Differences compared to UFS (Unix File System)

The following are the most important differences which AFS features compared to normal UNIX file systems:

File access works differently

The access rights are governed at the directory level by access control lists, one for each directory. An access list indicates who can do what with all the files in the directory. Access control lists are described in section D.2.

Output from the command ls does not show file protection

For each file, only the first three file protection codes shown by the command ls are used. In the AFS system, these codes show whether the file is protected for all who have these rights through the access control list fore directory.

A home directory problem with file protection

The system with file protection at the directory level causes a problem in directories which need to include both private and public files. This is primarily a problem pertaining to the top level of a user's home directory. How to solve this problem is described in section D.3.

A Kerberos ticket is required

The use of AFS files requires that you have authenticated yourself with the security system Kerberos: You need a valid Kerberos ticket for AFS. You get one automatically when you log on, but sometimes you need to get one manually. One such case is when you log on to another computer using SSH or telnet. Often, the tickets are not forwarded to the other computer, so you need to get a new one. Another case is when the ticket ceases to be valid, normally after 10 hours. You use the command kauth to identify yourself to Kerberos and get an AFS ticket. The command klist is used to show what Kerberos ticket you have.

Quota is used

Each AFS volume has a limit for the disk space it is allowed to use. This is called quota. Quota can be used in ordinary UNIX file systems, but so far this has not been used at KTH. KTH ITA handles changes to the quota for a volume. The command

fs listquota folder_name

shows the quota for the volume (storage unit) where the directory is and shows how large a part of the quota that is being used at the moment. (listquota can be abbreviated lq.)

The last backup is available on line for about 24 hours

Every night, your files are backed up to tape. At the same time, a snapshot of every volume is created, and this "frozen" volume is available during the next day. In your home directory, the backup is available under the directory OldFiles. If you make an error - remove a file, change a file's contents by mistake - and discover this the same day, you can get yesterday's version from the directory OldFiles.

In other directories, you use can the fs exa command to find out which volume the directory is stored on. Then you issue the command

fs mkm directory volumename.backup

to access the backup volume. After you copy the files, you use

fs rmm directory

to unmount the backup volume.

Access control lists

Each directory in AFS is associated with an access control list, ACL, that defines who is allowed to do what with the files in that directory. There are seven different actions for a directory and the files within, for which AFS governs the rights.

An example of an ACL:

ran rlidwka
system:administrators rlidwka
system:anyuser l
bg rlidwk
ran:friends rl

The line "ran rlidwka" indicates that the owner (the user "ran" in this example) has full access rights. "system:administrators rlidwka" shows that the special group system:administrators has the same rights (the members of this group can always get whatever rights they want, so it is meaningless to remove them). "system:anyuser l" tells us that the group system:anyuser, which includes all users, only has the right to list the files in the directory. In the fourth line the user bg gets all rights, except the right to change the ACL. "ran:friends rl" (or "ran:friends read") in the last line gives the members of the user owned group ran:friends the right to list and read the files in the directory. (See section D.4. about groups of your own.)

Commands for dealing with access control lists

fs listacl directory

Shows the ACL for the given directory (listacl can be abbreviated la)

fs setacl directory user/group rights

Gives a certain right to a certain group of the indicated directory. (Abbreviated sa)

fs setacl directory user/group none

Revokes all rights for the user/group.

fs help

List all fs commands

How to work around the problem that you cannot have different file protection for different files in the same directory

Since the protection of files in AFS is done on the directory level, you cannot have different file protection for different files in the same directory. For example, you cannot have a file that only you can read in a directory that is readable to any user. Usually this does not matter, you just put the files in different directories, but in some cases you have a problem. Some initialization files (e.g. .bashrc, .tschrc) in your home directory need to be publicly readable. If these files are not public, remote (or even local) login will not function normally.

At the same time you need secret files. The solution is to give list access to system:anyuser and then create a public subdirectory where you put non-secret files. For each file that needs to be publicly readable from the protected home directory, you then create symbolic links to the public subdirectory. In this way, programs access the non-secret files through the links in the read protected directory.

An example of commands used:

Go to the home directory:

cd

Create a sub directory:

mkdir Public

Make the subdirectory publically readable:

fs setacl Public system:anyuser read

Move the files that should be publically readable to that directory:

mv .bash_profile .bashrc .modules Public

Create links

ln -s Public/.bash_profile; ln -s Public/.bashrc; [etc.]

Personal access groups

In AFS, each user has the possibility to create personal groups of users in order to create ACLs in a convenient way. By adding a certain group to an ACL, all members in that group are given the same access rights at the same time. In the ACL above ran:friends is an example of a personal group. The name of personal groups always starts with the name of the owner followed by a colon.

Commands for dealing with groups in the AFS system

pts creategroup owner:group

Creates a group. The owner should be your username (creategroup can be abbreviated cg)

pts adduser user owner:group

Adds the user to the group (adduser can be abbreviated ad)

pts delete owner:group

Delete the group (delete can be abbreviated del)

pts removeuser user owner:group

Remove the user from the group (removeuser can be abbreviated rem)

pts membership owner:group

Show the members of the group (membership can be abbreviated m)

pts help

Show all pts commands

Most users' files at KTH are stored on network file systems. Typically, the disk space available is limited by quotas. There are many tools available to see what your quota is, and to find files that are taking up space.

AFS

AFS (Andrew File System) is used for home directories on many workstations and servers at KTH, in particular on Unix-like operating systems—Ubuntu and RedHat Linux, Solaris, macOS, etc. The home directories' AFS volumes use a disk quota. Students' home directories are typically limited to 3GiB, but you can receive more disk space if needed, as long as it is for study-related usage.

To see how much space you are using, open a terminal window and use

fs lq

What is using the space?

Ubuntu

If your quota is near the limit, you can use Ubuntu's Disk Usage Analyzer (under Applications and then Accessories) to see what is taking up space. Once it starts, press "Scan Home" to analyze your home directory.

Note that configuration files are often hidden, or in hidden directories—that is, the file name has a leading period/full stop. To show hidden files, use Ctrl-H in the file manager, or ls -a in a terminal.

Can't log in?

If the grapical environment (Gnome) does not start, you can use a console login to clear some space. See the FAQ  for more information on console login. You can then use command-line tools to check disk usage. These tools can also be used when logged in to a shell server with SSH . See the example below for more information.

Mac

To get an overview for all storage locations, open

(apple menu), About This Mac, “Storage” tab and then “Manage…”

The left column will show different storage locations. For each, you can sort by size, “kind” (data type), etc. and remove files if needed.

You can also use command-line tools, see the example below.

.OldFiles does not count

Note that the directory .OldFiles does not count as part of your quota. It actually links to a backup directory, a copy of your home directory from yesterday, which is a separate AFS volume. (You can't modify the files there, and it will be updated again when the daily backup is made, sometime during the night.)

Need more space?

Full quotas are automatically increased for active accounts, but the increase is only a few percent so that logging in won't be blocked. But if the space is filled by e.g. a browser cache, you will likely fill up the quota again next time you log in. Also, the quota scans only run once every night.

Students

If your course work requires a lot of space, check with your course responsible. Sometimes, large files can be stored in course directories so that students don't need copies in their home directory. If each student must store large files, or if an application requires a lot of space, the course leader can request additional space for all who are registered on the course. However, late registrants may have been left out; if you think this applies to you, contact your course responsible. If they have already requested larger quotas, contact IT support , but please be sure to refer to their ticket ID.

Command-line tools (example)

You can use du (see manualsidan för du(1) (extern länk) ) with some options to show MiB, then sort by size. But note that it is not AFS aware—it won't recognize AFS mountpoints (like OldFiles) as separate filesystems. To check all directories (trailing /):

$ du -sm */ .[a-z]*/ | sort -n | tail
147     .mozilla/
197     .vscode/
532     .OLD_gnome_20190201T165603/
903     .matlab/
987     .local/
1408    .config/
1712    .ros/
1826    Library_kth_20190617 
2276    .cache/
4418    OldFiles/

OldFiles can be ignored, see above.

The .OLD_gnome_xxx directory has been created by gnome-cleanup ( see FAQ ). But as it is more than a year old, it can probably be deleted using rm. You can use the -r (--recursive) option to delete entire directories—but make sure you do not accidentally delete too much! There is a huge difference between removing a* (all files starting with a) and a * (the file a, then all other files).

The Library_kth_2019… directory was used for settings in an older version of macOS. Old Library directories were renamed as the macOS used in computer classrooms after 2019 was no longer compatible, and migrating the settings on login would have taken a very long time. If you no longer need the old settings, the directory can be removed.

To see what's in other large directories:

$ du -sm .config/* .matlab/* .ros/* .cache/* | sort -n | tail -12
11      .cache/thunderbird
183     .matlab/R2020a
231     .matlab/R2018a
248     .matlab/R2015b
252     .matlab/R2017a
268     .cache/google-chrome
520     .config/google-chrome
706     .cache/pip
783     .config/Code
967     .local/lib
1130    .cache/mozilla
1711    .ros/log

The old Matlab directories can be removed. As of 2020–2021 only R2020A is available in the standard environment.

The Python pip cache can be cleared, and if you are no longer using personal python packages (sub directories to ~/.local/lib), you may be able to remove them as well.

The browser caches for Firefox (~/.cache/mozilla) and Google Chrome can be cleared. However, if you delete the files the browser may still try to use the cached data, showing broken web pages. It is better to clear from within the browser:

• Firefox: Edit » Preferences » Privacy & Security » Cookies and Site Data » Clear Data...
• Google Chrome: Ctrl+Shift+Del will open Clear browsing data. Be very careful to make sure only Cached images and files are marked, using Time range: All time. Click Clear data. You do not want to delete your history and saved passwords!

The other directories, such as logs from ROS Melodic, or Visual Studio Code's ~/.vscode and ~/.config/Code, may or may not be needed—you will need to decide for yourself.