CMPUT 379: Operating System Concepts
Department of Computing Science
January 2005

Assignment #3: A Distributed Directory Nanny
(Sockets, Pipes, and Signals)

Due Date: Wednesday, March 30, 2005
On-line submission using the astep program before 9 P.M. on due date.

What if my Assignment #2 was incomplete?

If you were unable to complete Assignment #2 and would like a solution to build upon for this assignment, you may purchase an assignment for a penalty of -20% (i.e., 20/100) off your mark for this assignment. You cannot arbitrarily adopt another student's (or anyone else's) code; that is considered plagiarism. This purchased assignment is not guaranteed to be bug free, but we will provide a solution of reasonable quality. Please contact your instructor if you would like to do this. Of course, read this assignment description first.

Assignments in this Course:

All three assignments in CMPUT 379 this term have been related to each other. By the end of this assignment, you will have built a sophisticated system that uses concurrency, signals, and various forms of IPC (including across workstations) to accomplish a non-trivial task.

Overview:

In this assignment, you will be extending and improving the dirnanny program from Assignment #2. So far, the two versions of dirnanny that you have developed have been limited to monitoring directories on a single Unix workstation. There are many environments, such as our instructional laboratories or a cluster of workstations in a research laboratory, in which we would like to monitor directories on a number of different workstations. Of course, we could start up a separate dirnanny on each workstation, but that would be more difficult for the systems administrators to manage. For example, if the contents of the configuration file changes, that would require a separate SIGHUP to be sent to each dirnanny. Also, in order for the sysadmins to monitor the activity of dirnanny on the different workstations, multiple output files may have to be inspected. For more than a handful of workstations, this administration overhead is too high.

We will address this problem with a (simplified) system that uses a client-server software architecture to centralize the configuration over a number of workstations, to centralize the output of the activities of all the processes, and to centralize and coordinate their clean exit.

In this assignment, you must write two different programs:

1. A client dirnanny (which is a modified form of the dirnanny from the previous assignment). Each workstation will have at most one client dirnanny per user. A client dirnanny that starts execution, before it proceeds, must first make sure that any previously running client dirnnany of the same user on the same host, is terminated. Also, the client must use fork() to create child processes that monitor the directories that need to be monitored on the specific  host. The name of your executable for the client must be dirnanny.client
2. A server dirnanny (which can also be a modified form of the dirnanny from the previous assignment). There is only one server  per user. A server dirnanny that starts execution, before it proceeds, must first make sure that any previously running server dirnanny of the same user on the same host, is terminated. The server centralizes the configuration information, it centralizes the logging of activities related to directory monitoring, and it centralizes the clean exit of all related processes. The name of your executable for the server must be dirnanny.server

Only the server dirnanny reads the configuration file and the file itself has a similar format as in Assignment #2 with some notable changes described below. Client dirnanny processes are given the configuration information by communicating with the server process using an Internet socket. The client dirnanny is responsible for creating child processes on the workstation on which it monitors directories.

Only the server dirnanny ever prints anything to stdout. Client processes send messages to the server describing the activity on the system (e.g., which files of a monitored directory have been deleted) and it is the server that actually outputs to stdout and log file. Similarly, the child processes of the clients never print anything to stdout. They send messages to their parent (the clients) about the activity on the system and the parent passes the information along to the server. Use of the stderr by the client and/or children is allowed only in extreme cases where exceptional situations prohibit the program from continuing its normal execution.

Input/Output and Behavior Specification:

When the dirnanny.server starts up, it must output to stdout its PID, the name of the workstation it is running on, and the port number used by the server to accept new socket connections (see the discussion of Internet sockets below) in the format shown in the following example:

Note that the port number is the very last item printed in the line. The contents of the configuration file might be:
 

Note that the syntax has changed with respect to (a) the total run time,  and (b) the directory name. The total run field that we found in previous assignments, that is, the first line in the configuration file, has been removed. The server process, once started, never terminates unless it is sent a SIGINT. In addition, the directory to monitor is now given as a combination of the hostname and the directory. That is, a directory path is not enough information to characterize a directory in a unique fashion, we need the hostname too. Note also that a host may have multiple monitored directories.

The above configuration file is just an example. Your program must work with all possible configuration files that follow the specified format. In fact, as part of your testing, you will want to experiment with different configuration files. If your server program detects an error in the configuration file, it must print an error message to stderr and then exit.

After the server process has been started, client processes can be started on a number of workstations. For this assignment, you can safely assume that there will never be more than 32 clients. The client program takes exactly two command-line arguments, which tell the client where the server is located. For example, if the server is started as shown above, then each client would be started from the command line using:

When the client process starts up, it must make a socket connection with the server (using the information provided on the command line). This socket will remain open for the entire lifetime of the client process. Using the socket connection, the server will tell each client which directories to monitor and their frequency and policy attributes, as indicated in the configuration file. Note that the clients never read the configuration file directly.

Each client, with the help of child processes, monitors the relevant directories on a given workstation. When files in a directory are removed, appropriate information is sent from the child to the client, and from client to the server process. The server process then outputs an appropriate message to the log file. Note that the server process is the only one with access to the logfile. As in the previous assignments, the location of the global logfile is determined by an environment variable DIRNANNYLOGS seen by dirnanny.server. If the environment variable is not set, then the default location $HOME/.dirnanny/ is used. The logfile from a new run of dirnanny is free to clobber the previously existing logfile. The filename of the logfile is dnnylog.global. The messages stored in the logfile that originated at the clients are extended to include hostname information, so they are of the form:
 

Messages to indicate the start of communication with a client and monitoring of directories on each client (similar to the messages in the previous assignments but with indication of the hostname on which they apply) are also to be written in the logfile. Again, note that only the server process generates  output. The child process that removed a file informs its parent (which is the client process) which, in turn, informs the server process. If the directory to monitor does not currently exist, the child, as in the previous assignments, will continue the monitoring, and it will pass a message, that will be recorded in the logfile, that the corresponding directory is currently not present.

At any time, the user is allowed to change the configuration file to add, remove, or change any line in the file. The user is allowed to send a hangup signal (SIGHUP) to the server process which forces dirnanny.server to re-read the configuration file (which has the same filename). Then, the server must print to the stdout (and  to the logfile):
 
 

Since only the server reads the configuration file, then only the server needs to be sent the SIGHUP. Then, the server must inform all of the client processes of the new configuration information. As with the previous assignment, the various child processes (and, now, client processes) must be reconfigured without any exiting/re-starting unless the number of directories (or hosts) being monitored is reduced.

Also, the user can send an interrupt signal (SIGINT) to the server dirnanny.server process which forces dirnanny.server to close any open files that it might have, ask each of the client processes to kill their children and exit, and free up all resources (e.g., memory). In fact, SIGINT to the server is the only means of terminating the server process (and subsequently the clients and their children) through user intervention. The client processes must free up any of their resources in order to make a clean exit. In essence, an interrupt signal to the server is used to cleanly exit from the server, the clients, and all of the child processes. Once all of the cleaning up has been completed, the server dirnanny.server process prints the following to stdout (and  to the logfile), including a count of the number of files removed, and exits:
 
 

Again, note that aside from the output specified above, there should be no other output from your program (except for what goes to the logfile that is). You may, of course, insert printf() output for your own debugging purposes, but those lines of code should be removed. Marks will be deducted for  verbose output in the version submitted for marking.

Required Design:

As with Assignment #2, when the program dirnanny.server is started from the command line, it must read the configuration file. Then, your system checks all the directories specified in the configuration file. If the corresponding directory does not exist, it is not considered an error. Instead, dirnanny.server reports the situation to stdout.

As in Assignment #2, the parent process communicates with the child processes using a Unix pipe. The parent dirnanny.client process should never fork a new child process if an existing child process is available to do the monitoring. That is, the number of children of the client are adjusted to the number of directories monitored at the particular host. An increase in the number of directories monitored must trigger the creation of additional children. Similarly, the reduction of the number of directories being monitored,  must trigger the termination of some children.

The only time a child process should exit is if the parent process explicitly tells the child to exit with a message on the pipe. The termination of a child is necessary when we reduce the number of monitored directories at a particular host, but also when the whole application is terminating. In the latter case, the parent process tells its children to exit  if it has been explicitly told to exit by the server process because the server process has received a SIGINT signal. All terminations at the client+children side are message-induced, and not triggered by signals.

The child process should execute the exact same program as dirnanny.client (i.e., you should not use exec()). The parent communicates with the child process (and vice versa) using Unix pipes. Basically, the child dirnanny.client process executes an infinite loop. It exits from the loop only if the parent tells it to do so. The children of dirnanny.client also execute an infinite loop, allowed only to exit when the parent tells them to. (By the way, the server also executes an infinite loop, allowed only to exit when the user intervenes with a SIGINT.)

While in the loop, the child should be in a position to be assigned, when instructed by the parent to do so, to monitor a different directory (with the corresponding  frequency and policy). Note that the server process has to (potentially) manage and communicate with a number of client processes. Also, each client  process has to (potentially) manage and communicate with a number of child processes. In addition, each client process has to communicate with the server. Since it is never acceptable to have the server or the clients block when reading data from a socket or pipe, you must use the select() (or poll()) system call to multiplex the I/O. By using select()(or poll()), you can detect whether or not there is actually a message to be read on a given file descriptor. Therefore, when you call read(), you can be sure that you will never block. Again, note that the client has both socket and pipe file descriptors. But, the select() (or poll())system call can deal with both types of IPC mechanisms. In addition, you should also use select() (or poll()) in the child processes.

The following figure is an example of the locations where processes run and the communication between them:

You must write a Makefile for your program. When someone types make, your Makefile should build the executable programs dirnanny.server and dirnanny.client. When someone types make clean, your Makefile should remove the executables, all .o files (if any), and all core files (if any).

Hints:

1. Be sure to read the newsgroup on a regular basis.
2. You may want to learn about the following C/Unix library  functions: fork(), popen(), pclose(), sleep() (see man 3c sleep), pipe(), read(), write(), socket(), bind(), listen(), accept(), connect(), select(), setsockopt(), gethostbyname(), gethostname(), htons(), signal() or sigaction()
3. Remember, the communication between client and server must be based on Internet sockets, (i.e. AF_INET domain stream sockets).
4. A couple of good tutorials on network programming using sockets are available, including one from Jim Frost and Beej's Guide. If you intend to read both, read Frost's first.
5. Once UNIX has assigned an available port number to your server's socket, you can determine the port number assigned as being the sin_port field in the struct sockaddr_in structure which is passsed to getsockname( ).

Other Implementation Details:

It is IMPERATIVE that your program properly deallocates ALL dynamic memory in a correct fashion (i.e., using free()) before your program terminates, or else your assignment will LOSE marks. To check that your program properly allocates and de-allocates ALL dynamic memory it uses, you must use the MEMWATCH package, which is simple to do.

Using MEMWATCH:

This term, we will use Version 2.71 (stable) of MEMWATCH. You can download the package yourself.

The TAs will expect that your files have been compiled with the header file memwatch.h and file memwatch.c in your working directory. MEMWATCH also has a README and FAQ.

In all of your source files (either directly or indirectly), you must add the directive

#include "memwatch.h"

and when you compile, you must compile memwatch.c along with your source file with the variables MEMWATCH and MW_STDIO defined.  As an example:

gcc -Wall -DMEMWATCH -DMW_STDIO main.c memwatch.c

When you run your program, if you get a message in your output that reads something like:

MEMWATCH detected 5 anomalies

it means you have not de-allocated dynamic memory properly.  In particular, this message indicates that 5 allocated structures have not been de-allocated. You should also check the MEMWATCH log file for any reports. If your assignment is not properly compiled with MEMWATCH enabled or if MEMWATCH reports that your memory allocation/deallocation was incorrect, then you will lose marks.

What to hand in:

All elements are to be handed in on-line via the astep program. Use the command:

unix-prompt% astep -c c379 -p as3 submit.tar

All of the following must be packaged into a tar file with the name submit.tar. Information about tar is available from the manual page (see man tar). For example, tar cvf submit.tar Makefile main.c my.h is an archetypal command; be very, very careful of the tar cvf submit.tar part. Before you submit, make sure your tar file works from within a fresh directory.

1. A README file (ASCII text is fine) for your assignment with: (1) your name, (2) student number, (3) Unix id, (4) lecture section, (5) instructor's name, (6) lab section, and (7) TA's name clearly labelled. Marks will be deducted if any of these items are missing. The README file must also include a short description of your program, as well as a description of the relevant commands to build (e.g. make all) and how to execute your programs including command line parameters.

2. A report in HTML file format, in a file called report.html, describing the design, implementation, and testing of your assignment. The report should contain no more than 1,000 words. (If /usr/local/bin/dehtml < report.html | wc -w is greater than 1,050 (i.e., 1,000 + a small margin; 1,051 is too many words), then marks will be deducted.) You do not need to repeat any information contained in this assignment description. I recommend you spend 25% of your report on an overview of your assignment, 50% on your design and implementation, and 25% on how you tested your program, and some concluding remarks. Note the emphasis on testing your program.

3. Your source code file(s) for dirnanny.server and dirnanny.client including all header files. Do NOT submit any MEMWATCH files, as the TA will use his/her own fresh copy of that code, but the use of MEMWATCH should be enabled in your code and Makefile.

4. Your Makefile.

NOTE: Do not submit files or test data not described above. Only submit what is requested and what is required to compile your program (except, of course, the MEMWATCH files).

Also, make sure that your program does not produce any debugging or extraneous output during normal execution. Only the requested output should be generated. Marks will be deducted for incorrect and other unrequested output. It is acceptable to have output to report an actual error.

Marking:

The assignment is worth 15% of your final mark in the course. This is an individual assignment. Do not work in groups. Review the Course Outline (extracted below) on this matter.

The assignment itself will be marked as follows: 20% for your report (clarity, technical accuracy, completeness, thoroughness of the testing, etc.), 60% for the correctness of the program when we test it using CSC 225, using gcc, and 20% for the quality of the implementation (design, modularity, good software engineering, coding style, useful and appropriate comments, etc.).

If your source code, as submitted, does not compile and run (using the submitted Makefile) on the CSC 225 workstations using gcc, you will receive a mark of zero for correctness. Review the Course Outline (extracted below) on this matter.

All that you have learned about good technical communication (e.g., for your report) will apply. All that you have learned about good programming style and comments in your code will apply. Having correct code is important, but good style, design, and documentation are also important. We cannot provide an exhaustive list of what we will look for, but an incomplete list includes: a comment for each source code file, a comment for each procedure/function, a comment for each significant (global or local) variable, good choice of names/identifiers, proper modularity (e.g., do NOT put all/most of the code in main()) etc.

Hints:

You may also want to learn about the following Unix programs: ps, grep, kill

Learn about system call select().

Further hints may be given later on in the newsgroup, if warranted. Be sure to read the newsgroup on a regular basis.

Cleaning up runaway processes is good etiquette when using a shared computer. For this course, it is a necessity. Make sure you know how to use the ps and kill commands.

Important Extracts from the Course Outline:

The University of Alberta is committed to the highest standards of academic integrity and honesty. Students are expected to be familiar with these standards regarding academic honesty and to uphold the policies of the University in this respect. Students are particularly urged to familiarize themselves with the provisions of the Code of Student Behaviour and avoid any behaviour which could potentially result in suspicions of cheating, plagiarism, misrepresentation of facts and/or participation in an offence. Academic dishonesty is a serious offence and can result in suspension or expulsion from the University. (GFC 29 SEP 2003)

NOTE: All assignments must be completed individually. Some high-level discussion of concepts between students is allowed. Do not work in groups. Do not share or discuss specific code in any way with other students; seek help from your TAs or instructor on these matters. Do not post code fragments longer than about 5 lines of code to the newsgroup. Note that we may use automated tools, such as MOSS, to detect potential cases of plagiarism. Note the definition of plagiarism and cheating in the Code of Student Behaviour

VERY IMPORTANT: Your programming assignments, as submitted, must work on the department's laboratory machines in CSC 225 (Linux, uj01 to uj19 ) and with the gcc compiler. All testing will be done on these machines using gcc. We also recommend that you use the gdb debugger. A program that does not work in CSC 225 with gcc, even if it works on a different Unix-like machine (e.g., other versions Linux or BSD) or compiler, will be considered incorrect. It is your responsibility to double check your tar files on the lab workstations before submitting them. Any mistakes in the above procedures, Makefiles, missing files, improper pathnames, and ``last minute changes'' to the files that prevent proper compilation will result in a mark of zero for correctness (approximately 60% of the total marks for each assignment). If you find an error in your submission, you can use the Late Policy (see below) to correct the mistake.

LATE POLICY: All programming assignments must be submitted electronically before 9 P.M. on the due date. (Note that even 1 second past 9 P.M. will be considered late.) Though not advised, it is possible to submit assignments late, with a penalty. The penalty for being late 1 day (i.e., up to 24 hours) is 10% of the maximum possible mark. Similarly, the late penalty for 2 days (i.e., more than 24 and up to 48 hours) is 20% of the maximum possible mark. No assignments will be accepted after 2 days past the deadline, except under extraordinary conditions and only with the approval of the instructor in advance.