These guidelines provide detailed examples for using Ninf-G. In addition to the instructions on Ninf-G given in Chapters 2 to 9, also see the sample programs.
Ninf-G software helps the development and execution of parallel task application working on a computational Grid consisting of multiple clusters. Ninf-G was developed to utilize various Grid middleware like the Globus Toolkit. In order to exercise this tutorial, the required software has to be properly installed and ready for use in all the machines (clusters). Required software includes a batch scheduler (for clusters), the Globus Toolkit (optional), SSH (optional) and Ninf-G. An outline of the batch scheduler, the Globus Toolkit and Ninf-G is given below.
These functions are important for the management and use of large-scale clusters, and in fact most large-scale clusters have introduced batch systems and require users to always use the batch system. Portable Batch System (PBS) and Sun Grid Engine (SGE) are widely used as batch systems.
Globus Toolkit is software package that provides UNIX commands and API of C and Java to implement some of the functions necessary for the development and execution of Grid software and applications. Currently, Globus Toolkit Version 4 is released and Ninf-G Version 5 supports Globus Toolkit Version 4.
In general in Pre-WS GRAM, Globus Gatekeeper runs on the entrance host of each cluster and sets a jobmanager, which activates jobs on the computation hosts with help of a batch system such as PBS or SGE, as a default jobmanager. In other words, unless a jobmanager is explicitly specified, jobs are activated on the computation hosts through the batch system. The computation hosts are utilized through Globus Gatekeeper on the entrance host and the jobmanager, since Globus Gatekeeper is not running on the computation hosts.
Ninf-G is the supporting software for the development and execution of programs with the Grid Remote Procedure Call (GridRPC) on Grid. Here we explain the Ninf-G Version 5 (Ninf-G5). Ninf-G provides simple, yet powerful, client-server-based frameworks for programming on the Grid. Ninf-G has been designed so that it provides 1) high performance in a large-scale computational Grid, 2) rich functionalities which are required to compensate for the heterogeneity and unreliability of a Grid environment, and 3) an API which supports easy development and execution of Grid applications. For the details of Ninf-G, see Chapt. 2 "How to use Ninf-G" of this document and "Ninf-G Users Manual" (in the package, or Web page).
Ninf-G allows you to carry out the following tasks.
Simply speaking, Ninf-G does not provide the following functions. The following are the functions that should be set, if necessary, in the programs using Ninf-G.
In Chapter 3 and the following chapters, the procedure for actual use of Ninf-G is described based on the sample programs. Firstly, we rewrite a sequential C program of non Ninf-G type to a program that uses Ninf-G, and gradually sophisticate the utilization of Ninf-G.
In this and the following chapters, use the sample programs to check the procedure.
We define the technical terms that will be used in this and the following chapters and explain the terms necessary for using Ninf-G.
The user is the one that creates Ninf-G based programs.
RPC function means in Ninf-G a function that actually carries out numerical computation. It is executed on a computational server and requires computational resources (CPU time and memory) on the server for the execution. The user should create an RPC function and can use the functions that other users made (share the RPC functions). (Users should set a function that requires the computational resources as an RPC function.)
It is a program or a process that calls RPC functions and controls the entire computation. The Ninf-G client program should be created by users.
Ninf-G Executable is a program to perform numerical computation as an RPC function. It is activated when a function handle is created from the Ninf-G client. When activated, it waits for start request for the execution (computation) of the RPC function.
There are two types of Ninf-G Executable: One that can define only one RPC function and does not retain the function call state i.e. does not retain the result of the previous function call, and the other that can define more than one RPC function (method) and retain the result of the previous function call. The latter is called the Ninf-G remote object.
A machine on which users execute the Ninf-G client program
A machine on which users execute the Ninf-G server program
It is grpc_function_handle_t type data prepared
or created by the Ninf-G client to call the RPC function.
It is an abstraction of the connection (communication path)
between the Ninf-G remote object and the Ninf-G client.
A computational server name needs to be given for creating a function handle. A function handle has one-to-one correspondence with the Ninf-G Executable process activated on the computational server.
It is grpc_object_handle_t_np type data prepared
or created by the Ninf-G client to make the Ninf-G remote object
and call a method function.
It is an abstraction of the connection (communication path) between
the Ninf-G remote object and the Ninf-G client.
A computational server name needs to be given for creating an object handle. An object handle has one-to-one correspondence with the Ninf-G Executable process activated on the computational server.
IDL is an abbreviation of Interface Description Language and defines, in an IDL file, the input/output type and order for the RPC function call
In Ninf-G IDL, object files (ending with .o) and others can be specified to make a link for the actual computational processing procedure and execution of the function described in IDL.
The Ninf-G server program and NRF file are created based on this user-created IDL file.
NRF file is a file containing information of a Ninf-G Executable, such as path and arguments of the Ninf-G Executable.
NRF file is created when the IDL file is compiled and the Ninf-G Executable is created on the server machine. Ninf-G Client retrieves information of Ninf-G Executable from the NRF file when its attribute is specified in Client Configuration file.In the following command line explanation, there are lines starting with "%" or "$". The symbols mean a command prompt on various shells. If the line starts with "%", the command is executed on csh (C shell) or tcsh. If the line starts with "$", the command is executed on sh (Bourne shell) or bash.
In this document, the following operating environment is assumed.
The above machine names are just tentative and do not exist. Replace them with the actual machine names users use to edit and execute the samples.
The sample programs are listed below.
The directory of each sample program is presented on the right side of the symbol ":".
Users should prepare the following files to use Ninf-G and the sample programs include them.
Before the execution of the sample programs, make the following modification.
Change "example.org" in the file to the host name of the server actually used.
You need to set user environments for Ninf-G prior to the execution of Ninf-G applications. In addition, if you use the Globus Toolkit, you need to set user environments for the Globus Toolkit. The configuration of Globus Toolkit and Ninf-G is necessary on every login, although you do not have to set manually every time if you write the procedure in advance in the file, such as .cshrc, .login, .profile or .bashrc that the shell loads on each login.
If the Invoke Server GT4py (WS GRAM) or GT2c (Pre-WS GRAM) is used, Globus Toolkit must be available.
For the user environment of Globus Toolkit, see Globus Toolkit manual. Configuration of Globus Toolkit Version 4, for example, should be set in the following steps.
$ GLOBUS_LOCATION=[directory where Globus Toolkit is installed]
$ export GLOBUS_LOCATION
$ . $GLOBUS_LOCATION/etc/globus-user-env.sh
% setenv GLOBUS_LOCATION [directory where Globus Toolkit is installed]
% source $GLOBUS_LOCATION/etc/globus-user-env.csh
User configuration of Ninf-G should be set in the following steps.
$ NG_DIR=[directory where Ninf-G is installed]
$ export NG_DIR
$ . $NG_DIR/etc/ninfg-user-env.sh
% setenv NG_DIR [directory where Ninf-G is installed]
% source $NG_DIR/etc/ninfg-user-env.csh
Before running Ninf-G Client, preparation is required on the client machine.
If the Invoke Server GT4py or GT2c is used, Globus proxy certificate is required.
% grid-proxy-init
'Enter passphrase'
grid-proxy-init is not necessary for all
the executions of the client programs.
The command does not have to be executed again until
the created proxy certificate will be expired.
You can confirm the status of your proxy certificate by
grid-proxy-info command.
You should test the Globus GRAM. If the remote WS GRAM or Pre-WS GRAM server (for example, server01.example.org) is available, the following test should be succeeded.
(for WS GRAM server)
% globusrun-ws -submit -F https://server01.example.org:8443/ -s -c /bin/hostname
(for Pre-WS GRAM server)
% globus-job-run server01.example.org /bin/hostname
If the command is successful, the command outputs the hostname of the server.
If Invoke Server SSH is used, remote shell execution without password or passphrase must be possible.
For example, SSH Agent is used for this case.
(for sh or bash)
$ eval `ssh-agent`
$ ssh-add
'Enter passphrase'
(for csh or tcsh)
% eval `ssh-agent -c`
% ssh-add
'Enter passphrase'
The ssh-agent command does not have to be executed again during the agent is alive and the user does not logout.
If a SSH daemon in the remote server (for example, server01.example.org) is available, the following test should be succeeded.
% ssh server01.example.org /bin/hostname
If the test is successful, the command outputs the hostname of the server without entering password.
The sample program which does not utilize Ninf-G is available in the "serial" directory.
This sample program is a sequential program to compute the circle ratio using the Monte Carlo method. In the Monte Carlo method, a dot is randomly placed within a square whose side length is 1. Check if the distance between the origin and the dot is shorter than 1. The circle ratio can be obtained by repeating these steps a number of times.
The circle ratio is derived from the following equation:
Circle ratio
= the number of dots whose distance from the origin is shorter than 1
/ total number of dots x 4
In this method, the computation precision of the circle ratio varies with the number of dots. The number of dots should be increased to improve the precision.
Placing dots within the square requires some computational capacity, and more capacity is needed to achieve a higher precision result. Ninf-G uses RPC to improve the utilization of the computational capacity.
In the simple execution of the programs created by not utilizing Ninf-G, pi_serial.c, only one CPU can be used as computation power or resource. However Ninf-G allows simultaneous/parallel use of multiple computational servers and can produce more dots in the same computation time. Consequently, a more precise result could be obtained.
A function suitable for RPC is one that can start computation
only with the information provided as input,
just like pi_trial() function used in this program.
Execute the program in the following steps.
% cd serial
% make
The execution file "pi_serial" is created.
% ./pi_serial 100000
The computed circle ratio is displayed as a result. Here 100000 is the number of dots and you may enter an arbitrary number.
The first example uses a synchronous call.
A server program is invoked by a client program. The client and the server run on different machines.
| Sample program | : | 1server |
| Server machine | : | server01.example.org (Step 5.1) |
| Client machine | : | client.example.org (Step 5.2) |
Change to the sample program directory before proceeding.
% cd 1server
Login on the server machine and go to the directory 1server. Follow the below steps.
% ng_gen pi.idl
% make -f pi.mak
(The above steps, i.e. server setup (1) and (2), are given in Makefile.
So the execution of the 'make stub' command will also follow the same
procedure.)
The NRF file "pi.server01.example.org.nrf" that contains the server machine name is created simultaneously. This file is used by Ninf-G client.
% ng_cc -o pi_client_1server pi_client_1server.c
(The above steps, i.e. client setup (1), are given in Makefile.
So the execution of the 'make pi_client_1server' command will
also follow the same procedure.)
The file "pi.server01.example.org.nrf" in the directory on server01.example.org, where Ninf-G Executable is compiled, must be copied to the client machine client.example.org.
% scp server01.example.org:/path/to/pi.server01.example.org.nrf .
(In this case the command scp is used for copying.
It is a program included in SSH
(Secure SHell: Remote operation command with coded communication path).
Here scp copies the remote file into
the current directory. )
Edit client.conf file.
(Use the text editor like vi and emacs command)
#.
% ./pi_client_1server 10000 server01.example.org
pi_client_1server receives the number of dots and the server host name in the arguments.
It should be noted that a single activation of a job requires more than several seconds if the Globus Toolkit is used. Most of the time is caused by the overhead of the Globus Toolkit such as authentication and authorization by GSI and process invocation via the Globus jobmanager. Job activation corresponds to the creation of a function handle for Ninf-G, and therefore the creation of a function handle takes more than several seconds.
This means, even if the actual RPC calculation time is less than 1 second, the execution of the client program takes more than several seconds. But the overhead can be negligible for large-scale applications.
Also, this waiting time varies depending on the batch system and queuing system of jobs. In some scheduling of the batch system, the activation of jobs may not start immediately and may have to wait until other jobs end.
1server/pi_client_1server executed in this chapter is a program modified from the one explained in Chapter 4.
The Ninf-G server program for calculating PI is
defined in the IDL file, pi.idl.
To be more precise, the program is modified so that
the pi_trial() function can be used as
an RPC function and executed through Ninf-G.
Also in the modification, the pi_trial() function is called
from an IDL file and the RPC function pi_trial()
defined in the IDL is called from the Ninf-G client program.
A procedure for calling the defined RPC function
pi_trial() is also added in the Ninf-G client program.
The added procedure calls the following GridRPC API functions.
grpc_initialize(), grpc_finalize()
grpc_function_handle_init(),
grpc_function_handle_destruct()
grpc_call()
This GridRPC APIs enable RPC model computation with Ninf-G.
With these steps, it is confirmed that the Ninf-G client program activated on client.example.org requests server01.example.org to execute PI library function.
The client machine and the server machine can be located at geographically separated locations. As seen here, the computation of using remote computation resources over the network can be performed with Ninf-G.
Next, the operation of Ninf-G is tested with the API of asynchronous call.
Two server programs are invoked by a client program. The client and the server run on different machines. Parallel computation is realized by distributing the job to the two computational servers.
| Sample program | : | 2servers |
| Server machine 1 | : | server01.example.org (Step 6.1) |
| Server machine 2 | : | server02.example.org (Step 6.1) |
| Client machine | : | client.example.org (Step 6.2) |
Move to the sample program directory before proceeding.
% cd 2servers
Login each server machine and follow the steps in 5.1. (Since Makefile contains the rule, the execution of the make command will create the server program.)
Modify pi_client_1server.c, used for test in Chapter 5, for executing asynchronous computation. (Modified source file is pi_client_2servers.c.)
Login the client machine and make an operation.
% ng_cc -o pi_client_2servers pi_client_2servers.c
The files "pi.server01.example.org.nrf" and "pi.server02.example.org.nrf" in the directory on server01.example.org and server02.example.org, where Ninf-G Executable is compiled, must be copied to the client machine client.example.org.
% scp server01.example.org:/path/to/pi.server01.example.org.nrf . % scp server02.example.org:/path/to/pi.server02.example.org.nrf .
Edit client.conf file.
#
for both servers.
% ./pi_client_2servers 10000 server01.example.org server02.example.org
pi_client_2servers receives the number of dots and the server host name in the arguments.
With these steps, it is confirmed that the usage of the API for asynchronous GridRPC call on the Ninf-G client program.
The API for synchronous GridRPC call used in the previous Chapter is a blocking call and the API will not return until the server completes the computation and the client receives the results from the server. Therefore, the Ninf-G client can do nothing until the API will return.
The asynchronous GridRPC call is a non-blocking call and
it does not wait for the end of computation by the server.
Therefore, the Ninf-G client is able to call other functions
immediately after calling grpc_call_async() with
regardless of the status of the computation by the server.
For example, you can call grpc_call_async() for
another server after calling grpc_call_async().
It is required to call waiting functions such as
grpc_wait() or its variants
(e.g. grpc_wait_all()) for waiting the end of
the computation and receive the result from the server.
Asynchronous GridRPC call is usually used for the implementation of task-parallel applications, i.e. calling multiple asynchronous GridRPC calls simultaneously on different servers enables to implement task-parallel applications.
In this program, the computational servers share the dots, whose total number is specified in the command line argument, and carry out the computation in parallel.
Therefore the computation time of this program with two servers is smaller than that with a single server. If the number of the dots is of the order of 10,000, the computation ends immediately and the effect of parallelization is not clear. However if the computation takes a long time such as one day, multiple server computation reduces the computation time significantly. The more computational servers we have, the smaller the computation time is.
A cluster system usually has multiple computational servers (computational nodes). The jobs submitted to the job submission node of the cluster system are allocated to each computational node and executed in the system.
To use more than one computational node from Ninf-G, a function handle needs to be created for each of the nodes.
The creation of one function handle takes at least several seconds, and that of more than one function handle takes time proportional to the number of the handles to be created. Also, a Globus Toolkit jobmanager process is activated in the entrance host every time the request for creating a function handle is submitted. So the entrance host will be overloaded if more than several tens or more than several hundreds of function handles are created in the simple way, i.e. by making the handle one by one. To prevent the overload, Ninf-G has a function to create more than one function handle simultaneously when it receives a single request for the creation of the handles. With this function of Ninf-G, the creation time is saved and a large number of (several tens to several hundreds of) function handles can be created.
A test on using this function is conducted in this chapter.
| Sample program | : | array |
| Server machine | : | server01.example.org (Step 7.1) |
| Client machine | : | client.example.org (Step 7.2) |
Move to the sample program directory before proceeding.
% cd array
Login the server machine (cluster management node) and make operations following the steps in 5.1.
pi_client_2servers.c, used as the sample in Chapter 6,
is modified with API grpc_function_handle_array_init_np()
to create more than one function handle simultaneously.
(The source file obtained by the modification is pi_client_array.c.)
grpc_function_handle_init()
is changed to a single call for
grpc_function_handle_array_init_np().
% ng_cc -o pi_client_array pi_client_array.c
The file "pi.server01.example.org.nrf" in the directory on server01.example.org, where Ninf-G Executable is compiled, should be copied to the client machine client.example.org.
% scp server01.example.org:/path/to/pi.server01.example.org.nrf .
Edit client.conf file.
#.
% ./pi_client_array 10000 server01.example.org 4
pi_client_array receives arguments of the number of dots, server hostname, and the number of handles.
"4" at the end of the command line represents the number of function handles to be created. Here the number is set to 4, although it is not necessarily 4. If the cluster system you use is shared with other users, pay attention to the number of the created handles and be sure to use the system insofar as it is allowed.
This program uses "Simultaneous creation function of function handles" to suppress the overhead in the creation of function handles and keep the number of the jobmanagers activated in the entrance host as small as possible. This function is needed for the usage of a large size cluster system with tens to hundreds processors.
Ninf-G Executable is state-less. Namely, even when the RPC function is called repeatedly with the same function handle, the state in the previous call is not retained in Ninf-G Executable. So, data needs to be sent for each computation even in the repeated computations with the same data.
Ninf-G provides Ninf-G Executable, as a remote object, capable of retaining states. More than one function (method) can be defined in the remote object, and the object provides a function for Ninf-G client to call these methods.
For use of the remote object capability, object handles,
not function handles, should be created.
Object handles are the grpc_object_handle_t_np type data
and abstraction of the connection (communication path)
between Ninf-G remote object and Ninf-G client.
Once an object handle is created,
the corresponding remote object retains the state
until the object handle is discarded.
It can also call various methods to change the state
for the object handle.
For example when the input data of RPC is extremely large and the same input data is used in the call for every RPC, the remote object capability can be effectively utilized. Once an initialization method sending the large input data is called just one time after each handle is created, and saved in Ninf-G Executable, the large input data need not to be transferred again in the subsequent calls for RPC and the system performance is improved.
The following is another example of the implementation of methods in Ninf-G class.
Arguments are given. The given arguments are stored in the data holding area.
The data in the holding area is used to execute Computation 1. The result is stored in the data holding area and not returned.
The result in the data holding area is used to execute Computation 2. The final result of Computation 2 is returned to the client.
| Sample program | : | object |
| Server machine | : | server01.example.org (Step 8.1) |
| Client machine | : | client.example.org (Step 8.2) |
Move to the sample program directory before proceeding.
% cd object
The IDL file pi.idl used in Chapters 5-7 is changed to an object. (The source file obtained in the modification is pi_object.idl) Edit pi_object.idl.
% ng_gen pi_object.idl
% make -f pi_object.mak
pi_client_2servers.c, used as an example in Chapter 6, is modified to a form that executes computation using Ninf-G remote object capability. (The source file obtained in the modification is pi_client_object.c)
% ng_cc -o pi_client_object pi_client_object.c
The file "pi.server01.example.org.nrf" in the directory on server01.example.org, where Ninf-G Executable is compiled, should be copied to the client machine client.example.org.
% scp server01.example.org:/path/to/pi_object.server01.example.org.nrf .
Edit client.conf file.
#.
% ./pi_client_object 10000 server01.example.org
pi_client_object receives arguments of the number of dots and server hostname.
Parallel programs written in MPI can be set as RPC functions in Ninf-G. This feature allows the programming to perform fine-grained parallel processing with MPI on a server machine that are called in remote procedure call, in addition to the master-worker type parallel task programming mentioned in the above.
| Sample program | : | mpi |
| Server machine | : | server01.example.org (Step 9.1) |
| Client machine | : | client.example.org (Step 9.2) |
Move to the sample program directory before proceeding.
% cd mpi
The IDL file pi.idl used in Chapters 5-7 is modified to a form that uses MPI. (The modified source file is pi_mpi.idl)
Edit pi_mpi.idl.
A code for exchanging the given arguments and computation results through MPI communication is added to carry out computation at each rank of MPI.
% ng_gen pi_mpi.idl
% make -f pi_mpi.mak
pi_client_1server.c used as an example in Chapters 5 is modified to a form that uses MPI for executing computation. (The source file obtained in the modification is pi_client_mpi.c)
grpc_call_async().
% ng_cc -o pi_client_mpi pi_client_mpi.c
The file "pi.server01.example.org.nrf" in the directory on server01.example.org, where Ninf-G Executable is compiled, should be copied to the client machine client.example.org.
% scp server01.example.org:/path/to/pi_mpi.server01.example.org.nrf .
Edit client.conf file.
#.
If the Invoke Server SSH is used, detailed MPI configuration must be specified by the user, although it is configured by a system administrator for the Globus Toolkit WS GRAM and Pre-WS GRAM. For example, MPI command path, a command argument specifying the machine file must be specified by client configuration file.
A sample Client Configuration file (mpi/client.conf) is for Invoke Server SSH and MPI configuration. See 4.4.3 Invoke Server SSH section on Ninf-G Users Manual for details.
% ./pi_client_mpi 10000 server01.example.org
pi_client_mpi receives arguments of the number of dots and server hostname. The number of CPU used here is set in mpi_runNoOfCPUs in the section SERVER of the configuration file as described in (4). There are several other ways in Ninf-G to set the number of CPU for MPI. For details see the Ninf-G Users Manual.
MPI is famous as a parallel programming model in existing parallel systems. It has the advantage that existing MPI programs can be executed on a Grid in their original forms with Grid-compatible MPI implementation such as MPICH-G2. On the other hand, it is difficult to ensure co-allocation in a Grid, i.e. to undertake simultaneous activation of all MPI processes at the start of the program execution, although MPI requires the co-allocation. Some functions such as process spawning may help the co-allocation to a certain extent, although in principle the processors and the number of processes MPI uses are determined statically and the dynamical property of the Grid is not entirely compatible with MPI. It is one of the most attractive features of Grids to carry out a large size application in a large-scale environment for a long time, while it is important to pay attention to what should be done in case of system failures, e.g. hardware failure of computation hosts. However with MPI, all program executions are terminated if an error occurs in a single process. In this sense, MPI is not completely fault-tolerant.
GridRPC is a programming model that can circumvent the above problem. GridRPC does not always require the co-allocation, and resources can be rather easily added/released dynamically by the dynamical creation/destruction of function handles. Also, an error in a computational host has no influence on the computation executed in other hosts, and therefore one can easily deal with failures by terminating the troubled computation or by sending the request again.
MPI is a popular programming model for parallel computation. Under the current circumstances, however, even the programs that are not suited for MPI are often written with MPI. The most suitable programming model should be selected according to the characteristics of the implemented application. To implement an application that executes more than one independent computation with distributed resources on Grid, GridRPC should be used. (At least MPI should not be used.) Both advantages of GridRPC and MPI can be utilized for the implementation of a large-scale application by combining them to use MPI for computation with frequent detailed communication and Ninf-G for computation with infrequent communication.
Past questions and cautions are listed below.
Q. Creating a function handle with
grpc_function_handle_init() API requires the server's name.
Can I omit the domain name in the server's name,
or do I have to use the domain name?
A. The domain name is necessary.
When creating a function handle and client configuration file, you have to use FQDN (Fully Qualified Domain Name) to set the server's name and cannot use "localhost". If the server's name is not properly specified with FQDN, Ninf-G API is terminated with an error.
For example when you use a server of the domain name server.example.org, you cannot specify the server's name simply as "server" but have to use "server.example.org".
Q. I made and executed an Ninf-G based client program, but it hasn't executed properly. It seems the execution of Ninf-G API failed. What is the cause?
A. Ninf-G has a customizable function for log output.
By making log-related settings in the client configuration file and the server configuration file, you get the output of the information on the error in the execution of Ninf-G API as well as on debugging. The output can be set to a file.
With this log function you can check the execution status of Ninf-G API and trace the cause of the failure in the execution of API.
For details of log function settings, see Ninf-G Users Manual in the package or Web page.
(Log output setting for client: Section 4.3.5)
(Log output setting for server: Section 3.3.2, 4.3.10)
(Log output setting for External Module: Section 4.3.5)
The log output messages will appear in the following order.
"Date and time:Client/Server:Host name:Log level: ID of each data structure:Name of Ninf-G internal function:Log message"
Every Ninf-G API returns the value of grpc_error_t
that presents success or failure of the execution.
User's programs should be written to include a function
for checking these values.
If the computation function executed with Ninf-G Executable results in Segmentation fault, the coreDumpSize attribute of the section <SERVER> in the client configuration file might be enabled.
"Request to Invoke Server XXX(0) failed" mean?Q. The function/object handle creation is terminated with an error. What is the cause? In the error log, the following log message appeared.
"QueryFeatures: Request to Invoke Server XXX(0) failed"
A. The Invoke Server XXX may exit immediately after the invocation for some reason. To investigate this problem, set the client configuration file invoke_server_log attribute on <CLIENT> section and run the Ninf-G Client again.
The error reason might appear in the Invoke Server log file.
For example, Invoke Server GT2c will exit immediately if the
grid-proxy-init was not performed in advance.
"Unexpected job_done" mean?grpc_call() or grpc_call_async() terminated with an error. What is the cause? In the error log, the following log message appeared.
...:Error: ... : Invoke Server XXX(0) Job failed.
...:Error: ... : Unexpected job_done was occurred.
A. The error message "Unexpected job_done"
indicates an error in Invoke Server XXX.
To investigate this problem, set the invoke_server_log attribute
on <CLIENT> section and run the Ninf-G Client again.
Q. The function/object handle creation terminated with an error. What is the cause? In the error log, the following log message appeared.
...:Error: ... : Can't get information for hostname="..." classname="..." from the Information Services.
A. The Information Service couldn't find the Ninf-G Executable information. Setting loglevel to Warning may output more detailed information. information_service_log attribute on <CLIENT> section might be also effective.
Check the following items.
Q. When I used Ninf-G, a file "gram_job_mgr_[numerics].log" was created in the server's home directory. What is this file?
A. This file is a log file from GRAM, a job execution module of Globus Toolkit Ninf-G uses.
GRAM activates jobmanager process in the same timing of job execution. The jobmanager process detects the completion and executes forced termination by monitoring the job.
The jobmanager eliminates the log file if the job ends normally, but does not if any error occurs or the job execution is canceled. In the latter case, the log file of GRAM remains in the home directory.
If Ninf-G client is terminated by e.g. ^C key
after grpc_function_handle_init() API creates
a function handle and before
grpc_function_handle_destruct() destructs the handle,
the job corresponding to the handle is cancelled.
A file gram_job_mgr-[numerics].log remains in the home directory of the server as a result of such abnormal terminations.
You may delete the file if you do not need to find the cause of the job termination.
Q. I would like to create a Ninf-G based program, I don't know how to write. Is a sample program available?
A. Ninf-G based sample programs are available in the Ninf-G package.
Ninf-G can be downloaded from the web page:
http://ninf.apgrid.org/packages/welcome.shtml
Download Ninf-G package from the web page.
The package contains sample programs.
| samples/pi | : | Monte Carlo computation for obtaining circle ratio |
| samples/add | : | Addition of arrays |
| samples/pi_wait_any | : | Computation with the use of asynchronous GridRPC API. |
| samples/pi_mpi | : | Computation with the use of MPI |
| diag | : | Exhaustive test to check if Ninf-G can be used with no problem. |
Q. After making a function handle, I would like to use it for more than a few days. What should I be aware of?
A. If you use Invoke Server GT4py or GT2c,
pay attention to the valid term of the Globus temporary certificate.
The command grid-proxy-init generates
the temporary certificate with the valid term
set to 12 hours unless given specifically.
There are two ways to use function handles for more than 12 hours.
-valid option of grid-proxy-init command.
Q. It is difficult to debug Ninf-G Executable because it is activated remotely. Is there any convenient method of debugging?
A. Ninf-G has a debugging function for Ninf-G Executable.
Set debug_busyLoop, debug, and coreDumpSize in the section <SERVER> of the client configuration file. For details see the Users Manual.
The log output function is enabled also in the debugging operation. Both client and Ninf-G Executable can output a log and can check every change in the operation status of Ninf-G API.
Q. The following message appeared in the log output in Ninf-G client. Is there any problem?
"... heartbeat timeout warning (60 seconds) occurred ..."
A. It does not always mean a problem.
Heartbeat is a Ninf-G function to detect if the Ninf-G Executable process is running with no problems, if the system is not freezing, and if the network has any problems.
Ninf-G Executable sends the heartbeat periodically to the client. If the network or Ninf-G Executable freezes, the heartbeat cannot be transferred from Ninf-G Executable to the client.
If the heartbeat does not reach the Ninf-G client within a certain period of time, the client considers that the Ninf-G Executable (function handle) has a problem and cannot be used, and puts out an error. Then the function handle cannot be used any more. The RPC in execution returns with an error.
For the sake of this function, the Ninf-G client can detect failure without freezing, and the users can make a trouble-resistant, fault-tolerant program.
Suppose Ninf-G Executable transmits the heartbeat every 60 seconds. The warning of the heartbeat appears if the Ninf-G client does not receive the heartbeat for 60 seconds. So, the warning appears even when a slight delay of Ninf-G Executable processing causes the delay in periodic heartbeat transmission. It is therefore not certain that this warning status should be determined as a failure.
In the default setting, the interval of the periodic heartbeat transmission is 60 seconds and the failure judgment is made after the client does not received the heartbeat for 60 seconds x 5 =300 seconds. (For details see the Users Manual.)
If the following message is displayed after the heartbeat warning, it means the recovery of the heartbeat transmission.
"... heartbeat revived again ..."
Q. I want to use Ninf-G. Is a shared memory function available in Ninf-G? Is it possible to share the variables in Ninf-G Executable?
A. Ninf-G does not have a shared memory function. A variable sharing function for Ninf-G Executable is also not available.
Q. Can I set the compiler and linker?
A. Yes you can. Compiler, Linker and others are available in IDL file syntax.
Q. I made a function handle but the job is not executed
immediately and grpc_call() API
and grpc_call_async() API stop.
How could this happen?
A. It can happen.
For example, if the cluster that the job goes to already has many users and other user's jobs are preferentially executed, your job has to wait until the other jobs end. So the execution of jobs does not always starts immediately.
If you want to place a limit on the execution time for your job, set the job_startTimeout attribute in the section <SERVER> of the client configuration file.
Q. What is the workDirectory attribute that can be set in the client configuration file?
A. workDirectory is used to specify the directory in which the Ninf-G Executable process operates.
In default Ninf-G Executable operates in the directory in which the Ninf-G Executable program is installed. Use workDirectory to change this directory.
If a relative file name is specified in an RPC function or computational function to open a file, the file is searched starting from the workDirectory and opened.
In case of a Segmentation fault due to a computational function failure, a core file, which is a memory image at abnormal termination, is sometimes created. This core file also goes to the directory specified in the workDirectory.
The absence of the specified workDirectory causes an error. So you need to be careful.
You may need to be careful if the job submission node and the job execution computational node are different in the cluster system. Due to the restriction of Globus Toolkit, an error occurs if the workDirectory exists in the computational node but not in the job submission node.
Q. Character strings are put out from
printf() or fprintf()
in IDL files or computational functions,
but the output result is not sent to the client.
There is a setting "redirect_outerr true" in the section
<SERVER> in the client configuration file.
Why isn't it sent?
A. Ninf-G uses the stdout/stderr transfer function provided by each Invoke Server (for example, Globus Toolkit GASS for Invoke Server GT2c) to send stdout and stderr of the computational functions.
Output is sometimes not transferred when the job is cancelled. In case of Ninf-G, jobs are cancelled for safety if an error such as abnormal disconnection occurs in the connection to the corresponding handle.
Output is also not transferred if Ninf-G client
is terminated by e.g. exit() before
the execution of grpc_function_handle_destruct().
Q. Login to the job submission node (e.g. server01.example.org), which is a designated host for the creation of function handles, is not permitted in my cluster system, and I cannot compile Ninf-G Executable in server01.example.org. So the NRF file for server01.example.org is not created. How can I make the file?
A. NRF file is a text file and users can rewrite the content.
For example, you can obtain a NRF file for server01.example.org in the following way.
First, compile the IDL file for RPC on a host (ex. client.example.org) other than server01.example.org. Then a NRF file for client.example.org is created.
Then, modify the obtained NRF file with a text editor. Modify all the lines in the NRF file that contain the host name. (Change client.example.org to server01.example.org.)
The file name of the NRF file also contains the host name, although it has no influence on the client execution. To prevent confusion, change the file name to server01.example.org.
There is another way to create NRF files. If the compilation host and the target host share the same environment, (including executable path of the Ninf-G Executable directory), with the following steps you can create NRF files. This is because the Ninf-G tries to get the local hostname by NG_HOSTNAME environment variable, firstly.
% ng_gen target.idl
% env NG_HOSTNAME=server01.example.org make -f target.mak
The NRF file created in the above steps should be specified in the Ninf-G client configuration file.
Users are fully responsible for the modification of the NRF files.