{"id":48,"date":"2018-08-10T10:37:38","date_gmt":"2018-08-10T08:37:38","guid":{"rendered":"https:\/\/www.kth.se\/blogs\/pdc\/?p=48"},"modified":"2019-03-28T11:53:32","modified_gmt":"2019-03-28T10:53:32","slug":"skip-the-configuration-get-to-the-cluster-docker-way","status":"publish","type":"post","link":"https:\/\/www.kth.se\/blogs\/pdc\/2018\/08\/skip-the-configuration-get-to-the-cluster-docker-way\/","title":{"rendered":"Skip the configuration, get to the cluster: Docker way"},"content":{"rendered":"

Sometimes it can be a daunting task to get all the Kerberos and SSH configurations right on your first attempt at using the PDC systems.\u00a0Nearly every day PDC Support receives a number of help requests and questions from researchers who have run into configuration problems. So PDC has introduced an alternative way of logging in to our clusters by using Docker containers with pre-configured Kerberos and SSH files.<\/p>\n

What is Docker?<\/h2>\n

Docker is a tool used to deploy and run single or many applications within what are known as containers. Containers are packed with all the parts that are needed to run an application, such as the actual application, any relevant tools, libraries or other necessary information. Each Docker container is delivered as a single package with all the necessary material included. This means that the person using the Docker container does not have to worry about installing any new tools\/libraries or configuring them before using them. Since everything is preloaded, the applications inside the containers are ready-to-use and can be executed regardless of any customized settings on the host machine.<\/p>\n

Later in this blog article, we will talk about a Docker container that we have developed for the purpose of logging in to PDC clusters.<\/p>\n

<\/p>\n

Why not use a Virtual Machine (VM)?<\/h2>\n

In a way, Docker is a like a Virtual Machine (VM), but the difference is that instead of installing a whole new operating system, each Docker container only contains the parts of the operating system that are required by the applications in that particular container. Thus, unlike VMs, which each have an entire guest operating system, each Docker container runs as an isolated process in user space on the host operating system.\u00a0 This saves a lot of memory and makes it easier to use a container rather than a new operating system. Thus, Docker containers enjoy the resource isolation and allocation benefits of VMs but are much more portable and efficient.<\/p>\n

\"\"
Differences between Docker and Virtual Machines visualized (Source: docker.com\/resources\/what-container)<\/figcaption><\/figure>\n

Here we are focusing on using Docker to log in to PDC supercomputers, which is a relatively small task. Hence, we use the lighter Docker containers instead of actual VMs.<\/p>\n

How to get started with Docker?<\/h2>\n

For logging in to our clusters here at PDC, we have created our own Docker container. To use it, you need to have Docker Toolbox installed on your system. You can find the instructions for installing Docker Toolbox on the operating system that you are currently using via the following links.<\/p>\n

Windows:\u00a0https:\/\/docs.docker.com\/toolbox\/toolbox_install_windows\/<\/a>
\nUbuntu:\u00a0https:\/\/docs.docker.com\/install\/linux\/docker-ce\/ubuntu\/<\/a>
\nMac:\u00a0https:\/\/docs.docker.com\/docker-for-mac\/install\/#install-and-run-docker-for-mac<\/a><\/p>\n

When you have installed Docker Toolbox on your system, if you are using Windows or a Mac, you will find that you have a new terminal application, called Docker Terminal, installed on your system.\u00a0You can then use Docker Terminal for loading any Docker containers that you want to use. If you have a Ubuntu system, you can simply use Docker commands in your regular terminal. The docker installation links shown above for Windows and Mac are for the Docker toolbox versions only – these are lighter legacy versions of Docker CE. If you do not mind the huge size of the actual Docker CE software (~500MB), or if you will be performing more advanced tasks, you can find the links for installing the full Docker CE software in the ‘Get Docker’ section of https:\/\/docs.docker.com<\/a>.<\/p>\n

A hub for the containers<\/h2>\n

Docker also provides a personal storage place for containers known as Docker Hub<\/a>.\u00a0 It is a cloud-based registry service which allows you to link to code repositories, build your own images, test them and store manually pushed images. In fact, we have created one container in our PDC repository on Docker Hub that we will be using in the upcoming steps.<\/p>\n

Alright, can we login now?<\/h2>\n

Once you have installed Docker in your system, you can simply open the Docker Terminal on Windows or Mac, or your regular terminal on Ubuntu, and enter the command below to launch the container that we have created for you. Note: Make sure your system is connected to the Internet before you do this.<\/p>\n

docker run -it pdcsupport\/pdclogin:latest<\/pre>\n
The -it<\/code> flag in the command means that you want to run the Docker application interactively. If that flag is not used, the process will run in the background and has to be accessed separately later. pdcsupport<\/code> is our Docker username, pdclogin<\/code> is the name of a repository and latest<\/code>\u00a0is the tag name given to our latest container in that repository.<\/p>\n
Once you have successfully loaded the PDC container, you can log in to any of the PDC host machines using two simple commands, as shown in the following example. You will need to give your username and the cluster you want to log in to.<\/p>\n
kinit -f <username>@NADA.KTH.SE\r\nssh <username>@<hostname>.pdc.kth.se<\/pre>\n
Your screen output should look similar to the following image.<\/p>\n
\"\"
Logging in to Beskow with Docker<\/figcaption><\/figure>\n
Now you should be logged in and able to use our machines. Isn\u2019t that easy!<\/p>\n
Basically, kinit<\/code> is a Kerberos command used to create a ticket which is stored locally in your system. It is then forwarded to the host machine when you try logging in using the ssh<\/code> command. If you want to learn more about how kinit<\/code> and ssh<\/code> work at PDC, you can find some information here<\/a>.<\/p>\n
File sharing between Docker and host machines<\/h2>\n
Sometimes you will need to transfer files from your local system to the PDC host machine. To do this with Docker, you need to map a folder in your local system to a folder inside the PDC container. To do this use the -v<\/code>\u00a0flag when launching the PDC container.<\/p>\n
docker run -it \\\r\n    -v <path-to-the-local-folder>:\/root\/<new-folder> \\\r\n    pdcsupport\/pdclogin:latest<\/pre>\n
Note: Use the full path to your local folder rather than the relative path.<\/p>\n
You can now see your files inside the PDC Docker container in the path root\/<new-folder><\/code>. Any changes that are made to the folder in the local system will be reflected inside the <new-folder><\/code>\u00a0in the Docker container and vice versa.<\/p>\n
Now that your container has access to your data, meaning the files inside the specified folder, you can transfer data between your local system and the PDC host machines.\u00a0 To copy any file from the Docker container to the PDC host machine, you can use the scp<\/code>\u00a0command, which uses the ssh protocol for secure transfers.<\/p>\n
# Copy a single file from local system to PDC host machine\r\nscp root\/<fileToCopy> \\\r\n    <username>@t04n28.pdc.kth.se:\/afs\/pdc.kth.se\/home\/u\/user\/<\/span>\r\n<\/pre>\n
Please note the highlighted section of the command above. The path to your home folder in PDC’s AFS file system is actually\u00a0\/afs\/pdc.kth.se\/home\/<first-letter-of-your-username>\/<your-username><\/code>\u00a0however, in this article, we will just write\u00a0\u00a0\/u\/user\/<\/code>\u00a0for the last two parts of the path to make things easier to read. You can find out more about file systems and data management at PDC\u00a0here<\/a>.<\/p>\n
To copy entire folders you can use the -r<\/code> flag as shown below.<\/p>\n
#Copy a directory from local system to PDC host machine\r\nscp -r<\/span> root\/<dirToCopy> \\\r\n    <username>@t04n28.pdc.kth.se:\/afs\/pdc.kth.se\/home\/u\/user\/<dir> \r\n<\/pre>\n
Bear in mind that you need to be in your Docker container, not on the PDC host machines, when using these file transfer commands. The general command usage is scp -r <source> <destination><\/code>. If you want to copy the files the other way around, that is, from a PDC cluster to your system, you still need to be in the Docker container and use the command in the following way.<\/p>\n
#Copy a directory from PDC host machine to local system\r\nscp -r <username>@t04n28.pdc.kth.se:\/afs\/pdc.kth.se\/home\/u\/user\/<dir> \\\r\n    root\/<dir><\/pre>\n
Some things to remember<\/h2>\n
    \n
  1. These procedures should be considered as an intermediate platform to make it easier for you to log in, and transfer files, from your local computer to a PDC system.<\/li>\n
  2. The Docker container used at PDC is based on a lightweight version of Ubuntu 16.04 which only has basic built-in commands. You can however download our Docker image, and install other tools that you need in it.<\/li>\n<\/ol>\n
    Further Reading<\/h2>\n
    If you would like to know more about Docker and how to use it, follow this link.<\/a><\/p>\n
    If you encounter any difficulties when logging in during any of the above steps, please check our troubleshooting page here<\/a>.<\/p>\n<\/div>","protected":false},"excerpt":{"rendered":"
    Sometimes it can be a daunting task to get all the Kerberos and SSH configurations right on your first attempt at using the PDC systems.\u00a0Nearly every day PDC Support receives a number of help requests and questions from researchers who have run into configuration problems. So PDC has introduced an alternative way of logging in […]<\/p>\n","protected":false},"author":1158,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[6],"tags":[16,15,17],"class_list":["post-48","post","type-post","status-publish","format-standard","hentry","category-environment-management","tag-container","tag-docker","tag-virtual-machine"],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p9W9Im-M","_links":{"self":[{"href":"https:\/\/www.kth.se\/blogs\/pdc\/wp-json\/wp\/v2\/posts\/48","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.kth.se\/blogs\/pdc\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.kth.se\/blogs\/pdc\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.kth.se\/blogs\/pdc\/wp-json\/wp\/v2\/users\/1158"}],"replies":[{"embeddable":true,"href":"https:\/\/www.kth.se\/blogs\/pdc\/wp-json\/wp\/v2\/comments?post=48"}],"version-history":[{"count":37,"href":"https:\/\/www.kth.se\/blogs\/pdc\/wp-json\/wp\/v2\/posts\/48\/revisions"}],"predecessor-version":[{"id":388,"href":"https:\/\/www.kth.se\/blogs\/pdc\/wp-json\/wp\/v2\/posts\/48\/revisions\/388"}],"wp:attachment":[{"href":"https:\/\/www.kth.se\/blogs\/pdc\/wp-json\/wp\/v2\/media?parent=48"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.kth.se\/blogs\/pdc\/wp-json\/wp\/v2\/categories?post=48"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.kth.se\/blogs\/pdc\/wp-json\/wp\/v2\/tags?post=48"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}