Lab #2: Diving into command-line

Labs: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14.

In this lab we will start on learning the most effective way to control your Linux machine – via command-line interface.

Do not forget that the Before class reading is mandatory and there is a quiz that you are supposed to complete before coming to the labs.

This reading is before the second lab.

MemTotal:        7899128 kB
MemFree:          643052 kB
MemAvailable:    1441284 kB
Buffers:          140256 kB
Cached:          1868300 kB
SwapCached:            0 kB
Active:           509472 kB
Inactive:        5342572 kB
Active(anon):       5136 kB
Inactive(anon):  5015996 kB
Active(file):     504336 kB
Inactive(file):   326576 kB
...

Fire up your terminal, please

For this (and almost all other labs) we will be working in the terminal. Please, locate this program in your environment and start it. Depending on your environment, it will be either Terminal, Console, or perhaps even Shell (although, technically, shell is the program running inside a terminal emulator).

We recommend you spend some time configuring the look of your terminal, such as having a nice font family and a reasonable font size. You will be spending quite a lot of time with it, so make the experience nice. Below are some possibilities of what you might get :-).

You will see something like [intro@localhost ~] and a blinking cursor after that. This is called a prompt and if you see it, it means you can enter your commands.

The prompt is displayed by your shell which is an interpreter of the commands you enter. The shell is actually a full-fledged programming language, but in this lab we will use it to launch very simple commands only.

Type uptime and start this command by submitting it with <Enter>. Until you hit <Enter>, you can easily edit the command. Shortcuts such as <Ctrl>-<Arrow> for jumping over words works too.

As we already mentioned, the experience is somewhat similar to an interactive Python session (editing etc.).

Quick copy-paste (and forceful program termination)

Whenever you select a text in the terminal with your mouse, it is automatically copied. This text then can be inserted by simply clicking the middle mouse-button (or the wheel).

Note that the well-known <Ctrl>-C and <Ctrl>-V combinations do not work in the shell as <Ctrl>-C is used to forcefully terminate a program. However, <Ctrl>-<Shift>-C usually works.

Note that these are actually two distinct clipboards – the special one bound to middle mouse-button and the one bound to <Ctrl>-C (<Ctrl>-<Shift>-C) and <Ctrl>-V. In graphical applications, <Ctrl>-C and <Ctrl>-V work as usual.

Closing the terminal

To close the terminal, you can simply close the whole window (e.g., via mouse) but you can also type exit or hit <Ctrl>-D on an empty line. Because we are moving away from needing mouse (in a sense), you should prefer <Ctrl>-D ;-).

Debugging issues

When running programs in a terminal, never paste their output as a screenshot. Instead, select the text (including the command you have run) and paste where needed.

For pasting into our Forum enclose the text in the fenced block ``` to preserve the monospace font.

```
ls nonexistent
ls: cannot access 'nonexistent': No such file or directory
```

Navigating through the filesystem

We will start with simple navigation through the file system. Two basic commands will get you through.

Directory listing with ls

The ls command lists files in the current directory.

Executing ls shall produce something like this:

Desktop    Downloads  Music     Public     Videos
Documents  gif.md     Pictures  Templates

Now run ls -l. That is, ls and -l separated by a space. Here we are calling the program ls and giving it an extra argument, -l. Because the argument starts with a dash, it is actually a so-called option (or switch) that instructs ls to modify its behaviour. Now ls prints something like this:

total 4
drwxr-xr-x. 1 intro intro    0 Feb 10 13:43 Desktop
drwxr-xr-x. 1 intro intro    0 Feb 10 13:43 Documents
drwxr-xr-x. 1 intro intro    0 Feb 10 13:43 Downloads
-rw-r--r--. 1 intro intro 1022 Jan  9 18:13 gif.md
drwxr-xr-x. 1 intro intro    0 Feb 10 13:43 Music
drwxr-xr-x. 1 intro intro    0 Feb 10 13:43 Pictures
drwxr-xr-x. 1 intro intro    0 Feb 10 13:43 Public
drwxr-xr-x. 1 intro intro    0 Feb 10 13:43 Templates
drwxr-xr-x. 1 intro intro    0 Feb 10 13:43 Videos

The -l turned on the so-called long mode where more details about each file are printed.

We will return to the meaning of some of the columns later on, deciphering the columns for the last modification time and the file size is straightforward and sufficient for the moment.

Changing working directory with cd

The cd command allows us to change the working (current) directory. It takes one argument – the directory we want to switch to.

Thus, cd Documents would move us to the Documents directory.

Execute ls here. What is the output? Answer.

How would you move back to the parent directory? Answer.

What will do the following command?

cd .

Notice that the command prompt changed whenever you switched to a different directory.

By default, it shows only the last component of the path. To show the full (absolute) path, we need to run pwd.

It will show something like

/home/intro/Videos

Tab completion

Typing long filenames can be cumbersome and making typos is annoying. Shell offers tab completion to help you with this.

For this example, we assume you just launched your terminal and ls prints Desktop Documents Downloads Templates etc.

If we want to change to directory Templates, start typing cd Te and hit <Tab>. Unless there is another filename (directory) starting with Te, the name shall be completed for you and should read the full cd Templates/.

Submitting the command with <Enter> would switch you to the directory as we would expect. Try it and come back to this directory again.

Now, let us switch to Documents directory. For this example, type cd Do and press <Tab>. There are two directories with this prefix: Documents and Downloads. Because the shell cannot know which one you want, it does nothing.

However, pressing <Tab> for the second time shows the possible matches and after typing c (the next letter), <Tab> can finish the completion.

Tab completion is an extremely powerful feature that saves hundreds of keystrokes and makes your interaction with the shell much faster.

As an exercise, what happens if you type cd and hit <Tab>? Answer.

Type just c (as in cd) and hit <Tab>. What happens? Answer.

Home directory

You probably noticed that when you start your terminal, the directory name you see there is just a ~ even though it should read intro (or your username on that particular machine) as that is the last component from pwd.

However, the path /home/intro is your home directory and has a special shortcut of tilde ~.

Futhermore, if you just run command cd without any extra arguments, it will change the directory back to your home.

Text user interface tools

While the use of purely command-line tools such as uptime, ls or cd is cool and extremely useful for scripts, there are also occasions where a more interactive approach is faster.

In this sense, Linux typically offers three layers you can choose from. From a fully graphical one called Graphical User Interface (GUI), over a tool with a Text-based User Interface (TUI) to a pure Command-Line Interface (CLI). Every of these can be useful, depending on the circumstances.

By a textual user interface we mean what is offered by Midnight commander or Ranger.

Midnight commander

Run mc and navigate through the files as you have done with ls and cd.

The numbers at the bottom refer to your functional keys for typical file operations (e.g., F5 copies the file).

Note that in a typical setup, MC offers two panels with file listing, you switch between them via <Tab> and, by default, copying is done to the directory in the other panel.

MC is a quite powerful tool as it can inspect file archives, show files on a remote machine, etc.

We will briefly mention the most important things that you can do with it. Do try them :-)

• <Insert> allows you to select multiple files for deletion/copying.
• <F3> displays file contents.
• <F4> offers simple text editor with syntax highlighting.
• <+> allows you to enter a filename mask to select multiple files at once (we will talk about this more later in the Wildcards section).
• <Ctrl>-o hides the panels and temporarily switches you back to shell. Perfect for running commands without leaving MC.

You can quit MC with <F10> or via a menu (activated by <F9>). Note that some terminals capture <F10> to activate their window menu (but this behaviour can be tuned in Preferences of the terminal application).

Ranger

Ranger is a Vim-inspired file manager for the console. It brings some well-known key bindings from the Vim realm together with tabs pages.

Navigation

• j - Move down
• k - Move up
• h - Move to the parent directory
• l - Open file or move to directory
• gg - Go to the top of the list
• G - Go to the bottom of the list
• gh - cd ~
• gm - cd /media
• gr - cd /
• q - Quit Ranger

Working with Files

• zh - View hidden files
• cw - Rename current file
• <space> - Select current file
• yy - Yank (copy) file (or selected files)
• dd - Mark file (or selected files) for cut operation file
• pp - Paste yanked or cut file(s)
• dD - Delete file (or selected files)

See more on Ranger: A terminal file manager.

Editing file contents

You probably noticed that the Development submenu contains several graphical text editors that you can use to edit the source code. However, it is also possible to edit files in TUI editors.

If you are asking why to learn another editor (if you are already happy with some of the graphical ones), here is the answer. On some machines, you may not have access to GUI at all. Recall that we talked about remote access earlier: in that case you will have only TUI available (and you will often need to edit files on the remote machine). Some users thus never use GUI editors at all, the reasoning is that it is much better to learn (and customize) one editor properly, and that editor is a TUI-based one.

On our disk, you will find Emacs, Joe, mcedit and Vim.

Each has its own advantages and it is up to you which one you will choose. Note that mcedit is probably the closest to an editor you may know from other systems. joe is a small one, but perfectly suitable for script editing that we will be doing the most. Both emacs and vim are extremely powerful tools that can do much more than just edit files. However, they require a bit of time investment before you can start using them effectively.

If you are new to Linux, we would recommend you to use mcedit (either using it directly or when editing files in Midnight commander) and come back to the other ones later on for a final decision of THE text editor of your choice.

All of these editors can be launched from the command-line, giving it the filename to edit as a parameter (e.g., mcedit quiz.md).

Shell wildcards

For the following you will need to have the same list of files as we have.

Please, download this archive and unpack its contents. If you want to download it from the command line, you can use wget URL, otherwise use whatever browser you like. Use Midnight commander to copy the unpacked content to your home directory.

You should see directory nswi177-lab02 on your disk.

So far, we used ls to display all files in a directory. If we are interested in only a subset, we can specifically name them on the command line.

Move to the directory where you have unpacked the nswi177-lab02.tar.gz. You should see the following files:

a/ b/ c/ one.txt two.txt three.txt four.txt

If we want to list only details about the text files, we can execute

ls -l one.txt two.txt three.txt four.txt

-rw-r--r-- 1 intro intro 0 Mar  3 13:38 four.txt
-rw-r--r-- 1 intro intro 0 Mar  3 13:38 one.txt
-rw-r--r-- 1 intro intro 0 Mar  3 13:38 three.txt
-rw-r--r-- 1 intro intro 0 Mar  3 13:38 two.txt

Doing that for more files would not be very elegant, but the shell offers so called wildcards to specify multiple files at once. Thus, the same output can be obtained by running

ls -l *.txt

It is essential to note that ls (or any other program for that matter) will receive the expanded list of files – finding the matching files is done by the shell, not by individual programs. Thus for the above example, from inside ls there is no way of distinguishing whether the user used the full list or the *.txt wildcard. You will experiment with this in one of the next labs where we will talk about accessing these parameters in your favorite programming language. For developers, it means that they do not need to care about implementing the wildcard expansion themselves. The program would always receive a list of existing filenames, not a wildcard.

By the way – is the last sentence completely correct? What happens if we run ls -l *.txxxt? Answer.

How would you print all files starting with the letter t? Answer.

If we would like to print only information about files starting with either o or f with .txt extension, we would use.

ls [of]*.txt

If we want to print files that end with any of the letters from a to f, we could use

ls *[a-f].txt

Try it in the a subdirectory.

Note that the files are sorted alphabetically when specified via wildcards.

Switch back to your home directory.

And now list all files/directories starting with D (recall that Linux is case-sensitive). You might be surprised because a straightforward ls D* would actually list the contents in these directories. It is perfectly expectable, because ls Documents is supposed to print a list of files in that directory. If we do not want ls to descend into directories, we can add -d option to prevent that.

What happens when you specify a file that does not exist? And what if only some of the specified files do not exist?

Recall that filenames starting with dot . are hidden. These are by default not listed by ls. If you want to see these files too, you have to either name them explicitly or use the -a option.

Try it in the nswi177-lab02 directory.

What hidden files are in your home directory? Answer.

Again: it is not a security measure, just a way to make the listing less cluttered.

Exploring file contents

We have already mentioned text editors and MC to look into files when working in the terminal. They are not the only options.

Text files

The simplest way to dump the contents of any file is to call program called cat. Its arguments are filenames to print. The name cat has nothing to do with the mammal but refers to the middle of the word concatenate as it can be used to actually concatenate files.

Move to the b subdirectory. Executing cat 000.txt will show the contents of 000.txt on the screen.

How would you show the contents of all files in this directory? Answer.

Binary files

If we want to dump binary files (such as images), it is usually better to dump their bytes in hexadecimal.

hexdump utility can be used for that.

We will always use it with -C switch to print hexdump and ASCII characters next to each other. The dump of the GIF file looks like this:

hexdump -C c/sample.gif

00000000  47 49 46 38 39 61 0a 00  0a 00 91 00 00 ff ff ff  |GIF89a..........|
00000010  ff 00 00 00 00 ff 00 00  00 21 f9 04 00 00 00 00  |.........!......|
00000020  00 2c 00 00 00 00 0a 00  0a 00 00 02 16 8c 2d 99  |.,............-.|
00000030  87 2a 1c dc 33 a0 02 75  ec 95 fa a8 de 60 8c 04  |.*..3..u.....`..|
00000040  91 4c 01 00 3b                                    |.L..;|
00000045

Unprintable values (e.g., smaller than 32) are replaced with a dot.

File system hierarchy

By now, you can navigate the file system in a TUI manager or via cd and ls. Open two terminals next to each other and perform the same actions in both.

Do not be afraid to actually display contents of the files we mention here. hexdump -C is really a great tool.

/boot contains the bootloader for loading the operating system. You would rarely touch this directory once the system is installed.

/dev is a very special directory where hardware devices have their file counterparts. You will probably see there a file sda or nvme0 that represents your hard (or SSD) drive. Unless you are running under a superuser (more about that later), you will not have access to these files, but if you would hexdump them, you would see the bytes as they are on the actual hard drive. And writing to this file would overwrite the data on your drive!

This directory also contains several special but very useful files for software development.

/dev/urandom returns random bytes indefinitely. It is probably internally used inside your favorite programming language to implement its random() function. Try to run hexdump on this file (and recall that <Ctrl>-C will terminate the program once you are tired of the randomness).

/etc/ contains system-wide configuration. Typically, most programs in unix systems are configured via text files. The reasoning is that an administrator needs to learn only one tool – a good text editor – for system management. The advantage is that most configuration files have support for comments and it is possible to comment even on the configuration. For an example of such a configuration file, you can have a look at /etc/systemd/system.conf to get the feeling.

Perhaps the most important file is passwd that contains a list of user accounts. Note that it is a plain text file where each row represents one record and individual attributes are simply separated by a colon :. Very simple to read, very simple to edit, and very simple to understand. In other words, the KISS principle in practice.

/home contains home directories for normal user accounts (i.e., accounts for real – human – users).

/lib and /usr contain dynamic libraries, applications, and system-wide data files.

/var is for volatile data. If you would install a database or a web server on your machine, its files would be stored here.

/tmp is a generic location for temporary files. This directory is automatically cleaned at each reboot, so do not use it for permanent storage.

/proc is a virtual file system that allows controlling and reading of kernel (operating system) settings. For example, the file /proc/meminfo contains quite detailed information about RAM usage.

Again, /proc/* are not normal files, but virtual ones. Until you read them, their contents do not exist physically anywhere.

Manual pages

We have seen that the ls behaviour can be modified with -a, -d, and -l. hexdump has -C. Do you know that uptime accepts -s? And that cat takes -n to print line numbers?

It is virtually impossible to remember all of this. Luckily, Linux contains so-called manual pages (or just manpages) that describe the available options for (almost) each program that you have on your system.

Execute man cmd to access a manual for the cmd program (substitute cmd for the actual command name). Use arrows for scrolling and q to quit the manual. You can search inside the page with / (slash) key.

Manual pages are organized into sections and you can specify the section number as part of the man execution, e.g., man 3 printf opens a help page for printf() function in the C language because that is the contents of section 3. Note that man printf would show you the contents of printf manual from section 1, i.e., the shell command.

Open man man to see the full list of sections. Briefly, 1 is for shell commands, 3 is for library calls, and 4 and 5 are used for specific files (e.g., man 5 proc launches the manual page for the whole /proc directory).

Note that manual pages are also available on-line, hence you can study your favourite commands even without access to your Linux machine.

Typical options

Many of the options are more-or-less standardized across multiple programs and are worth remembering.

Almost all GNU programs that you will have on your machine will print a small help when executed with --help. Try it for ls or cd.

--version could be used to print the version and copyright information of the executed program. Sometimes -v or -V works as well.

--verbose or --debug (sometimes -v or -d) launch the program in verbose mode where the program prints in more detail what it is doing.

--dry-run (sometimes -n) executes the program without performing actual changes (e.g., it can print which files would be removed without actually deleting any of them).

--interactive (sometimes -i) will typically cause the program to ask for interactive confirmation of destructive actions.

-- could be used to terminate the list of options if you have filenames starting with a dash. For a classical example, move into the d subdirectory of nswi177-lab02 and list information about a file named -a. Then check your result and try again using the -- delimiter. Answer.

Caveats (file names with spaces in them)

If you create a file called file with spaces.txt and then execute

ls file with spaces.txt

you will receive

ls: cannot access 'file': No such file or directory
ls: cannot access 'with': No such file or directory
ls: cannot access 'spaces.txt': No such file or directory

because the space (or tab) is used as a delimiter between parameters. Hence, ls was actually looking for three files.

If you would use tab completion, your command would be completed with escape characters.

ls file\ with\ spaces.txt

Note that the output would typically look like this:

'file with spaces.txt'

because using apostrophes (or quotes) is another way to specify that the space is a literal character and not a separator.

We will mention this again when talking about scripts, but it is something to remember: spaces in filenames can cause unexpected surprises and it is better to avoid such naming.

And yes, it is possible to create a file named ' ' (i.e., space) and show its contents with cat " " but it is not a very sensible idea to do so. It is similar to creating files starting with a dash – it is possible, there are ways to bypass the issues (e.g., using -- delimiter) but it is just simpler to avoid these issues.

Work effectively

Do not be afraid of running multiple terminals next to each other. Use one to navigate with ls and cd, use the other one for Midnight commander to mirror your actions.

Open another one with a manual page for the command you are using.

Most desktop environments allow you to create multiple workspaces or desktops. Then, each workspace has its own list of opened windows, windows opened on other workspaces are not visible. This can reduce the clutter significantly and – with proper keyboard shortcuts – speed up your work.

Running Python scripts from the command-line

We will talk about this in greater detail in the following lab, for now you can use the following command to actually run your Python script:

python3 path_to_your_python_script.py

Graded tasks (deadline: Mar 6)

02/tree.py (30 points)

Write a Python script that prints a tree of files in the current directory.

Assuming the following files exists:

README.md
01/factor.py
02/wildcards.md
02/before.md
02/temp/file.txt
bin/run_tests.sh
.gitlab-ci.yml

The script would print the following (notice how directories are handled, and the ordering).

01/
    factor.py
02/
    before.md
    temp/
        file.txt
    wildcards.md
README.md
bin/
    run_tests.sh

Hint: you should consider using os.scandir() as a starting point of your implementation. Note that the documentation actually contains quite useful example to start with. Experienced developers may notice that os.scandir is preferred over older os.walk for performance reasons (PEP 471).

Special handling is needed in the following cases:

• Hidden files are not printed at all.
• Symbolic links are completely ignored (use follow_symlinks=False for the is_file and similar functions).

You can safely assume that sorted() is the right function for sorting and that the directory structure would not change during script execution.

Do not forget to copy the first line from 01/factor.py (wait for next labs for the proper explanation) and prepare your code for modules (i.e., the line with __name__).

02/wildcards.md (30 points)

Copy the following fragment to GitLab and fill in the answers.

Note that this task is not fully checked by GitLab as it would reveal the answers.

Get this archive first:

https://d3s.mff.cuni.cz/f/teaching/nswi177/202122/labs/nswi177-task02.tar.gz

Get contents of all files in subdirectory `login` that
start with a decimal digit, ends with `z.txt` and the middle letter is
any letter (i.e., A to Z, not numbers) from your GitLab login
(in lowercase).

For example, if your login is `johndoe`, you should paste contents from files
`0jz.txt`, `1ez.txt` but not from `ajz.txt` or `2wz.txt` or
`0jx.txt`.

Sort the list of files alphabetically before getting their content, duplicate
letters should be ignored (i.e., use wildcards naturally and you will be fine).


**Q1** Paste the contents of the files here.

**[A1]** ... **[/A1]**


**Q2** Insert here the wildcard pattern that you have used.

**[A2]** ... **[/A2]**

02/uptime.txt (20 points)

Explain in your own words what does the following command print.

cut -d ' ' -f 1 /proc/uptime

Recall that you can use man command to get information about given command, /proc filesystem is described in the fifth section.

The checks done by GitLab merely tests presence of this file.

02/architecture.sh (20 points)

Paste into this file (as its only content!) a command that prints what hardware architecture your computer has (for most of you, it will be x86_64).

Hint: learn about the uname command.

Learning outcomes

Conceptual knowledge

Conceptual knowledge is about understanding the meaning and context of given terms and putting them into context. Therefore, you should be able to …

• explain cases where command-line interface is better than a graphical one

• explain what is a terminal and what is a shell

• explain what is a path (of a specific file), what is relative and absolute path

• explain how shell filename wildcards work

• describe basic top-level directories and important files on a typical Linux system

• explain how the directory tree is formed by mounting individual subsystems

• explain difference between a normal file, directory and a symlink

Practical skills

Practical skills is usually about usage of given programs to solve various tasks. Therefore, you should be able to …

• start and customize a preferred terminal emulator

• navigate through filesystem in a terminal via TUI tools or via CLI commands

• view contents of text files (use cat)

• view contents of binary files (as hexadecimal dumps using hexdump)

• use built-in manual pages

• use clipboards available on a Linux system

• use tab-completion for effective writing of filenames

• use irregularly named files

• use cd, ls (basic options)

• use mc or similar manager (basic operations)

ls -- -a

cd ..

ls t*

ls -a ~
ls -d ~/.*

cat *

cat [0-9][0-9][0-9].txt

cd ..

ls -- -a

ls t*

ls -a ~
ls -d ~/.*

cat *

cat [0-9][0-9][0-9].txt