External Module Developer's Manual

This document describes how to develop a Ninf-G External Module.

• 1. Introduction
• 2. External Module
  • 2.1 Detailed overview of External Module
  • 2.2 Protocol of External Module
    • 2.2.1 Protocol Overview
    • 2.2.2 Request
    • 2.2.3 Reply
    • 2.2.4 Notify
• 3. Invoke Server
  • 3.1 Introduction
    • 3.1.1 Overview of a typical client application
    • 3.1.2 Requirements for underlying middleware
    • 3.1.3 Implementation Overview
    • 3.1.4 Execution flow
  • 3.2 Specification of Invoke Server
    • 3.2.1 Details of Invoke Server
    • 3.2.2 Protocol between a Ninf-G Client and Invoke Server
      • 3.2.2.1 Overview
      • 3.2.2.2 Request
      • 3.2.2.3 Reply
      • 3.2.2.4 Notify
      • 3.2.2.5 JOB_CREATE Request
  • 3.3 How to specify Invoke Server
    • 3.3.1 How to specify Invoke Server
    • 3.3.2 How to pass information to Invoke Server
    • 3.3.3 Polling interval
    • 3.3.4 Logfile
    • 3.3.5 Maximum number of jobs per Invoke Server
    • 3.3.6 How to specify the path of Invoke Server
  • 3.4 Miscellaneous information
    • 3.4.1 Job Timeout
    • 3.4.2 Redirect stdout/stderr implemented using files
• 4. Communication Proxy
  • 4.1 Introduction
    • 4.1.1 Execution flow
  • 4.2 Specification of Communication Proxy
    • 4.2.1 Details of Communication Proxy
    • 4.2.2 Protocol between a Ninf-G Client (Ninf-G Executable) and Communication Proxy
      • 4.2.2.1 Overview
      • 4.2.2.2 Request
      • 4.2.2.3 Reply
      • 4.2.2.4 Notify
• 5. Information Service
  • 5.1 Introduction
  • 5.2 Specification of Information Service
    • 5.2.1 Details of Information Service
    • 5.2.2 Protocol between a Ninf-G Client and Information Service
      • 5.2.2.1 Overview
      • 5.2.2.2 Request
      • 5.2.2.3 Reply
      • 5.2.2.4 Notify

1. Introduction

Ninf-G implements a function called "External Module." External Module abstracts the functions of Ninf-G and implements the function as a separate process. These functions include (1) invocation of jobs (called "Invoke Server"), (2) communication between Ninf-G Client and Ninf-G Executable (called "Communication Proxy"), (3) Information services of Ninf-G Executable (called "Information Service"). Implementation of these functions depends on the underlying grid middleware. Abstracting the difference of the implementations, External Module provides the same interface for Ninf-G. In order to adopt a new grid middleware, it is required to implement External Module by the grid middleware.

Any user is able to implement an External Module to support a new grid middleware.

2. External Module

2.1 Detailed overview of External Module

Invoke Server, Communication Proxy, and Information Service are External Modules. Some features are common for all External Modules though each External Module has its own feature.

Common feature of External Module is as follows.

• External Module is invoked by Ninf-G Client or Ninf-G Executable.

  • Invoked by Ninf-G Client:

    Invoke Server, Client Communication Proxy, Information Service

  • Invoked by Ninf-G Executable:

    Remote Communication Proxy

• Invoke Server and Communication Proxy are External Modules both of which is related with jobs. In these External Modules, the maximum number of jobs per External Module can be limit by users. If the number of jobs exceeds the limit, a new External Module is invoked.

• Ninf-G Client (Ninf-G Executable) and External Module communicate using three pipes which are created by the Ninf-G Client (Ninf-G Executable) when the External Module is invoked. The details of External Module communication protocol is described in Section 2.2.

• There are two types of finalization of External Module. Ninf-G Client (1) do not wait or (2) wait, for the termination of External Module after the Ninf-G Client sends an EXIT request to the External Module.

• External Module is implemented as a executable or script file which should be located in the ${NG_DIR}/bin directory. It can be located in another directory if External Module is supplied with an absolute path to the executable file.

• Ninf-G Client invokes several processes for the same type of External Modules. For example, if the API grpc_config_file_read_np() is called, a request for creating a function/object handle or a information query is processed by a newly invoked process. Outstanding External Module is only used to handle existing function/object handles.

• Log file for External Module can be specified as an optional argument of the External Module command.

  Example:

  
  -l [Log file name]
  

  If this option is specified, External Module outputs logs to the file specified by this argument. Otherwise, logs are not recorded.

• External Module must not exit by unexpected optional argument of the command.

2.2 Protocol of External Module

2.2.1 Protocol Overview

A Ninf-G Client(Ninf-G Executable) and External Module exchange three types of messages: Request, Reply, and Notify. A Request message is sent from a Ninf-G Client(Ninf-G Executable) to External Module. Reply and Notify messages are sent from External Module to the Ninf-G Client(Ninf-G Executable). The Ninf-G Client(Ninf-G Executable) assumes that a Reply message must be returned from External Module when the Ninf-G Client (Ninf-G Executable) sends a Request message. A Notify message is used to send messages from External Module to the Ninf-G Client(Ninf-G Executable) asynchronously. Three different pipes are used for sending these three types of message.

All messages are sent as plain text. The Return code (<RET>) is 0x0d0a. The Return code is a delimiter that determines the unit of messages.

2.2.2 Request

Request have two types, one is described by one line and the other one is described by multiple lines. Each request has respective type.

In one line type request, parameters for the request is delimited by space character, and the parameter cannot include space characters. The request is expressed as follows:

<Request Name> <Parameter> <Parameter> ...<RET>

On multiple line type request, request name is on the first line. Following lines are expressed as attribute name, space character and attribute value. The attribute name cannot include space character. The end of the request is expressed as "request name _END." The request is expressed as follows:

<Request Name><RET> 
<Attribute Name> <Attribute Value><RET> 
<Attribute Name> <Attribute Value><RET> 
... 
<Request Name>_END<RET> 

There is one common request on External Module. The QUERY_FEATURES request is prepared to query features implemented on the specific External Module. This request is issued just after the External Module process invocation. This request is a one line type request, and have no parameter.

2.2.3 Reply

Reply also has two types, one is described by one line and the other one is described by multiple lines. The each reply has respective type.

In one line type reply, return success or failure as a parameter followed by the return value. On successful, "S" is returned followed by a return value. On failure, "F" is returned followed by an error message. The reply is expressed as follows:

S <Return Value> | F <Error Message><RET>

On multiple line type reply, "SM" is returned on the first line. The failure cannot be expressed. Following lines are expressed as attribute name, space character and attribute value. The attribute name cannot include space character. The end of the reply is expressed as "REPLY_END." The request is expressed as follows:

SM<RET> 
<Attribute Name> <Attribute Value><RET> 
<Attribute Name> <Attribute Value><RET> 
... 
REPLY_END<RET> 

The QUERY_FEATURES reply is a multiple line type reply. On the reply, there are 3 type of information. "protocol_version", "feature" and "request" are replied.

• protocol_version

  This specifies the protocol version of External Module. Invoke Server, Communication Proxy, Information Service have respective protocol version. This information can return only one line.

• feature

  This specifies the extended features that the External Module implements. One feature is on single line and multiple lines can be returned.

• request

  This specifies all the requests, that the External Module implements and is responsible. One request is on single line, and multiple lines can be returned.

2.2.4 Notify

Notify also has two types as same as Request.

In one line type request, the Notify is expressed as follows:

<Notify Name> <Parameter> <Parameter> ...<RET> 

In multiple line type request, the Notify is expressed as follows:

<Notify Name><RET> 
<Attribute Name> <Attribute Value><RET> 
<Attribute Name> <Attribute Value><RET> 
... 
<Notify Name>_END<RET> 

3. Invoke Server

3.1 Introduction

A Ninf-G Client invokes a Ninf-G Executable on the server machine when a function requiring initialization of function/object handles, such as grpc_function_handle_init(), is called. Ninf-G, Version 2, implements the remote process invocation using the Globus Toolkit's Pre-WS GRAM feature. Implemented using the Globus API, the invocation mechanism has been embedded in Ninf-G. In order to utilize other systems, such as WS GRAM, UNICORE, or Condor for remote process invocation, Ninf-G, Version 5, implements the invocation mechanism as a separate module called "Invoke Server." This design enables users and developers to implement and add a new Invoke Server that can utilize any job invocation mechanism.

Ninf-G Version 5.0.0 includes the following Invoke Servers:

• Invoke Server for WS GRAM, implemented in Python (GT4py)
• Invoke Server for Pre-WS GRAM, implemented in C (GT2c)
• Invoke Server for SSH, implemented in C (SSH)
• Invoke Server for Condor, implemented in Java (Condor)
• Invoke Server for NAREGI Super Scheduler, implemented in Java (NAREGISS)
• Invoke Server for WS GRAM, implemented in Java (GT4java)
• Invoke Server for UNICORE, implemented in Java (UNICORE)

3.1.1 Overview of a typical client application

Here is a typical flow of a Ninf-G Client application:

• (1) grpc_initialize()

  Initializes data structures used by the Ninf-G Client.

• (2) grpc_function_handle_init()

  Creates a function/object handle which requests remote process invocation. The request will be processed and a Ninf-G Executable will be created on the server machine. When the Ninf-G Executable is created, it connects to the Ninf-G Client to establish a TCP connection between the Ninf-G Executable and the Ninf-G Client.

• (3) grpc_call() or grpc_call_async()/grpc_wait_any()

  Calls the remote function, i.e. (3.1) the Ninf-G Client sends arguments to the Ninf-G Executable, (3.2) the Ninf-G Executable performs some form of computation, and (3.3) the Ninf-G Executable sends the results to the Ninf-G Client.

• (4) grpc_function_handle_destruct()

  Requests the Ninf-G Executable to terminate its process. If an error occurs during the termination, the Ninf-G Client requests the Invoke Server to kill the Ninf-G Executable.

• (5) grpc_finalize()

  Frees the data structures used by the Ninf-G Client.

Invoke Server is required to implement initialization and finalization of the function/object handles which are described in steps (2) and (4).

3.1.2 Requirements for underlying middleware

The only requirement for underlying middleware is that the middleware must be capable of remote process invocation. Examples of such middleware include the Globus Toolkit Pre-WS GRAM, Globus Toolkit WS GRAM, UNICORE, Condor, and SSH.

3.1.3 Implementation overview

Invoke Server is an adapter for the underlying middleware and it handles requests from a Ninf-G Client. Invoke Server analyzes and processes the request sent from the Ninf-G Client and replies to the Ninf-G Client. For example, if Invoke Server receives a JOB_CREATE request from the Ninf-G Client, Invoke Server creates a Job ID, returns the Job ID to the Ninf-G Client, and invokes the job processes called for in the request.

Invoke Server can be implemented using any language. The details of the protocol existing between the Ninf-G Client and Invoke Server are described in Section 3.2

3.1.4 Execution flow

This section describes a sample RPC flow to a server called serverA via the Invoke Server, IS_SAMPLE.

• (Prerequisite)

• (1) A client configuration file that describes that Invoke Server IS_SAMPLE is used for RPC to serverA must be prepared.

• (grpc_function_handle_init())

• (2) The Ninf-G Client requests Invoke Server IS_SAMPLE to create a function/object handle.

• (3) The first time IS_SAMPLE is required to create a function/object handle, the IS_SAMPLE process is spawned by the Ninf-G Client on the same machine. ${NG_DIR}/bin/ng_invoke_server.IS_SAMPLE is a command for spawning an IS_SAMPLE process.

• (4) The Ninf-G Client and IS_SAMPLE communicate using three pipes (stdin, stdout, and stderr). QUERY_FEATURES request and reply are performed at first.

• (5) When grpc_function_handle_init() is called, the Ninf-G Client sends JOB_CREATE request to IS_SAMPLE, followed by the required information (e.g., the hostname and port number of the remote server), and JOB_CREATE_END.

• (6) When IS_SAMPLE receives JOB_CREATE request, IS_SAMPLE returns "S" to the Ninf-G Client, which indicates that the request has been received by the Invoke Server.

• (7) IS_SAMPLE generates a new Job ID that corresponds to the Request ID that was transferred with the JOB_CREATE request, and notifies the Job ID to the Ninf-G Client. Then, IS_SAMPLE invokes the remote processes (Ninf-G Executable) on serverA using its underlying middleware.

• (8) The Ninf-G Client waits for the reply from IS_SAMPLE, and notify of Job ID. When the Ninf-G Client receives the reply and Job ID, it resumes the execution without waiting for actual job invocation on serverA.

• (grpc_call())

• (9) When the Ninf-G Executable is invoked on serverA, it connects to the Ninf-G Client using Globus IO. The connection is used for communication (e.g., argument transfers from the Ninf-G Client to the Ninf-G Executable) between the Ninf-G Client and the Ninf-G Executable. IS_SAMPLE does nothing for grpc_call(). If the underlying middleware for IS_SAMPLE returns an error on remote process invocation, IS_SAMPLE must notify the Ninf-G Client that the job invocation has failed.

• (grpc_function_handle_destruct())

• (10) When grpc_function_handle_destruct() is called, the Ninf-G Client requests the Ninf-G Executable to exit the process. This communication is carried out between the Ninf-G Client and the Ninf-G Executable. The Ninf-G Client does not wait for the Ninf-G Executables to be terminated.

• (11) When the Ninf-G Executable exits the process, the job status managed by IS_SAMPLE should be changed to DONE, and IS_SAMPLE notifies the Ninf-G Client of the change in job status to DONE.

• (12) The Ninf-G Client sends a JOB_DESTROY request to IS_SAMPLE.

• (13) IS_SAMPLE returns "S" to the Ninf-G Client when it receives the JOB_DESTROY request.

• (14) IS_SAMPLE returns DONE to the Ninf-G Client if the state of the corresponding job is DONE. Otherwise, IS_SAMPLE cancels the job and notifies the Ninf-G Client of the change in status to DONE when the cancellation is completed and the status of the job actually becomes DONE.

• (grpc_finalize())

• (15) When grpc_finalize() is called, the Ninf-G Client sends an EXIT request to IS_SAMPLE.

• (16) IS_SAMPLE returns "S" to the Ninf-G Client when it receives the EXIT request. The pipes between IS_SAMPLE and Ninf-G Client (stdin, stdout, stderr) are closed after it.

• (17) IS_SAMPLE cancels all jobs and wait the termination of all jobs, and exit.

• (18) When the Ninf-G Client receives an "S" from IS_SAMPLE, it continues its execution, and does not wait for the termination of all jobs.

The following figure illustrates the interaction between the Ninf-G Client, Invoke Server, and the Ninf-G Executable.

Figure 1: Interaction between the Ninf-G Client, Invoke Server and the Ninf-G Executable

3.2 Specification of Invoke Server

This section describes a detailed overview of Invoke Server and the protocol existing between a Ninf-G Client and Invoke Server.

3.2.1 Details of Invoke Server

1. Invoke Server is invoked when a Ninf-G Client initializes a function/object handle on the remote server which Ninf-G Client is configured to use with the Invoke Server.

2. The maximum number of jobs per Invoke Server is limited. If the number of jobs exceeds the limit, a new Invoke Server is invoked.

3. Invoke Server exits the process if it receives an EXIT request from the Ninf-G Client. This request is sent when the Ninf-G Client calls grpc_finalize(). Invoke Server also exits the process if it is managing the maximum number of jobs and all jobs are terminated.

4. Ninf-G Client does not wait for the termination of Invoke Server after the Ninf-G Client sends an EXIT request to Invoke Server.

5. If the Ninf-G Client exits abnormally, the pipes will be disconnected. When Invoke Server detects that the pipes have been disconnected, Invoke Server must cancel all jobs and exit the process.

6. The file names used with Invoke Server must follow the naming convention of "ng_invoke_server" + suffix, where the suffix corresponds to rules for the underlying middleware used for remote process invocation.

3.2.2 Protocol between a Ninf-G Client and Invoke Server

3.2.2.1 Overview

A common feature of External Module protocol is described in Section 2.2. A protocol specific to Invoke Server is described in the following sections.

3.2.2.2 Request

Five Request messages, JOB_CREATE, JOB_STATUS, JOB_DESTROY, EXIT, and QUERY_FEATURES are supported.

1. JOB_CREATE
   • Format

     JOB_CREATE <Request ID><RET> hostname .....<RET> port .....<RET> ... (snip) JOB_CREATE_END<RET>

   • Explanation

     This request is used to create and invoke a new job. Required information for job invocation is described as a set of attributes that is transferred along with a JOB_CREATE request. The details of these attributes are described in Section 3.2.2.6. JOB_CREATE is the only request that is described using multiple lines. All the other requests can be described with a single line.

     A Ninf-G Client transfers a Request ID to Invoke Server. Invoke Server generates a unique Job ID and returns it to the Ninf-G Client. The Job ID is used by the Ninf-G Client to specify the job.

     When Invoke Server receives a JOB_CREATE request, it must send a Reply message to the Ninf-G Client. Then, Invoke Server generates a unique Job ID and notifies the Ninf-G Client of the Job ID. Finally, Invoke Server requests job invocation on remote servers via the underlying middleware used with the Invoke Server.

2. JOB_STATUS
   • Format

     JOB_STATUS <Job ID><RET>

   • Explanation

     This request queries Invoke Server on the status of jobs. The current version of Ninf-G and prior does not use this JOB_STATUS request.

3. JOB_DESTROY
   • Format

     JOB_DESTROY <Job ID><RET>

   • Explanation

     This request is used to terminate and destroy jobs. Invoke Server cancels all jobs if it receives this request and the corresponding jobs are not completed. When Invoke Server confirms that all jobs are cancelled, it sends DONE to the Ninf-G Client.

4. EXIT
   • Format

     EXIT<RET>

   • Explanation

     This request is used to terminate Invoke Server. If Invoke Server receives this EXIT request, it must cancel all outstanding jobs and wait for their termination.

3.2.2.3 Reply

Invoke Server must send a Reply message to a Ninf-G Client if Invoke Server receives a Request message from that Ninf-G Client.

• JOB_CREATE, JOB_DESTROY, EXIT

  The reply to JOB_CREATE, JOB_DESTROY, and EXIT messages is:

  [S | F <Error String>]<RET>

  where S is sent in case of Success. Otherwise, F is returned, followed by <Error String>.

• JOB_STATUS

  The reply to a JOB_STATUS request is:

  [S <Status> | F <Error String>]<RET>

  Where <Status> is denoted as:

  
  <Status> : [PENDING | ACTIVE | DONE | FAILED]
  

  Each status indication indicates the status such that:

  • PENDING : the Ninf-G Executable is waiting for invocation.
  • ACTIVE : the Ninf-G Executable is already invoked.
  • DONE : the Ninf-G Executable is already done.
  • FAILED : the Ninf-G Executable exited abnormally.

• QUERY_FEATURES

  The reply to a QUERY_FEATURES request is described in 2.2.3. Ninf-G Version 5.0.0 Invoke Server has protocol version 2.0.

  Following features can be returned.

  • STAGING_AUTH_NUMBER

    If the Invoke Server implements the staging of Simple Auth Number by file, this feature must be returned.

  • STAGING_COMMUNICATION_PROXY

    If the Invoke Server implements the staging of Remote Communication Proxy, this feature must be returned.

3.2.2.4 Notify

A Notify message is used to send an asynchronous message from Invoke Server to a Ninf-G Client. Two types of Notify message are provided.

1. CREATE_NOTIFY
   • Format

     CREATE_NOTIFY <Request ID> [S <Job ID> | F <Error String>]<RET>

   • Explanation

     This is used to notify the Ninf-G Client of the Job ID. A Job ID is case sensitive and cannot include invisible characters.

2. STATS_NOTIFY
   • Format

     STATUS_NOTIFY <Job ID> <Status> <String><RET>

     
     <Status> : [PENDING | ACTIVE | DONE | FAILED]
     

   • Explanation

     This message is used to send notification that the status of a job has been changed.

     <String> can be any string, and the <String> is stored in an output log. It should be noted that the status of job can be changed from PENDING to DONE.

3.2.2.6 JOB_CREATE Request

This section describes the details of a JOB_CREATE Request.

• Format

  JOB_CREATE <Request ID><RET> hostname .....<RET> port .....<RET> ... (snip) JOB_CREATE_END<RET>

  Attributes are placed between JOB_CREATE<RET> and JOB_CREATE_END<RET>. Only one attribute can occupy one line and one line must include one and only one attribute. Attributes can be placed in any order. There are two types of attributes, mandatory attributes and optional attributes. Invoke Server must return an error if mandatory attributes are not included. Any unknown optional attributes must be ignored.

• Attributes

  The following is a list of attributes supported by Ninf-G. Some of these attributes are provided for the Globus Toolkit's Pre-WS GRAM and WS-GRAM. Any new attribute can be defined using the Client configuration file <SERVER> section "invoke_server_option" attribute.

  name	mandatory	meanings
  hostname	yes	Host name of the server
  port	yes	Port number
  jobmanager	no	Job Manager
  subject	no	Subject of the GRAM
  client_name	yes	Host name of the Ninf-G Client
  executable_path	yes	Path of the Ninf-G Executable
  backend	yes	Backend of the remote function (e.g., MPI)
  count	yes	Number of Ninf-G Executables
  staging	yes	A flag indicating if staging is used or not
  auth_number	no	Simple Auth Number
  argument	yes	Arguments for the Ninf-G Executable
  work_directory	no	Working directory of the remote function
  tmp_dir	no	temporary files directory
  redirect_enable	yes	A flag indicating redirection of stdout/stderr
  stdout_file	no	file name of stdout
  stderr_file	no	file name of stderr
  communication_proxy_staging	no	stage the Communication Proxy or not
  communication_proxy_path	no	path of the Communication Proxy
  environment	no	Environment variables
  status_polling	yes	Interval of status polling
  refresh_credential	yes	Interval of credential refresh
  max_time	no	Maximum execution time
  max_wall_time	no	Maximum wall clock time
  max_cpu_time	no	Maximum CPU time
  queue_name	no	Name of the queue
  project	no	Name of the project
  host_count	no	Number of executables per host
  min_memory	no	Minimum size of requested memory
  max_memory	no	Maximum size of requested memory
  rsl_extensions	no	RSL extension

  Detailed description

  • hostname [hostname]

    Host name of the server machine.

  • port [number]

    The server port number on which the server is listening. The default value is depend on underlying middleware.

  • jobmanager [jobmanager name]

    The job manager used on the server machine.

  • subject [subject]

    The certificate subject of the resource manager contact.

  • client_name [client name]

    The host name of the client machine.

  • executable_path [path to the executable]

    Absolute path of the Ninf-G Executable. The path represents a remote path if staging is off. Otherwise, the path represents a local path.

  • backend [backend]

    The method for launching the Ninf-G Executable is specified as backend. The value is NORMAL, MPI, or BLACS. If MPI or BLACS is specified, the Ninf-G Executable must be invoked via the mpirun command.

  • count [number]

    The number of Ninf-G Executables to be invoked. If the backend is MPI or BLACS, count means the number of nodes.

  • staging [true/false]

    The value is true if staging is on and Invoke Server must transfer the Ninf-G Executable file from the local machine to the remote machine.

  • auth_number [number]

    The value to be used as a simple auth number of Ninf-G. The Invoke Server must write this number to file and stage to the server.

    In addition, the Invoke Server must add the "--authNumberFile=[remote auth number file]" argument to the Ninf-G Executable to tell the Simple Auth Number.

    If this attribute can be treated, the Invoke Server must tell the Ninf-G Client by sending feature STAGING_AUTH_NUMBER on QUERY_FEATURES request's reply.

  • argument [argument]

    An argument for the Ninf-G Executable is specified using this attribute. This attribute can specify one argument only, and multiple arguments must be specified one by one, by using this attribute for each one. The arguments must be passed to the Ninf-G Executable as arguments.

    Example:

    
    argument --connectbackAddress=...
    argument --authNumber=...
    argument --contextID=...
    

  • work_directory [directory]

    This attribute specifies the directory in which the Ninf-G Executable is invoked.

  • tmp_dir [directory]

    The directory in which temporal files are placed.

  • redirect_enable [true/false]

    This attribute is set to true if the stdout/stderr of the Ninf-G Executable has been requested to be transferred to the Ninf-G Client.

  • stdout_file [filename]

    If redirect_enable is set to true, this attribute specifies the name of the output file for stdout. Invoke Server must output the stdout to this file. The Ninf-G Client reads this file as an output file and writes the contents of the file to the stdout of the Ninf-G Client.

  • stderr_file [filename]

    If redirect_enable is set to true, this attribute specifies the name of the output file of the stderr. Invoke Server must output the stderr to this file. The Ninf-G Client reads this file as an output file and writes the contents of the file to the stderr of the Ninf-G Client.

  • communication_proxy_staging [true/false]

    The value is true if the staging of Remote Communication Proxy is enabled and Invoke Server must transfer the Remote Communication Proxy file from the local machine to the server machine. Staged executable file path on the server machine must be passed to Ninf-G Executable by --communicationProxyPath argument. This argument must be added by the Invoke Server.

    Otherwise, staging of Remote Communication Proxy is not performed.

    If this attribute can be treated, the Invoke Server must tell the Ninf-G Client by sending feature STAGING_COMMUNICATION_PROXY on QUERY_FEATURES Reply.

  • communication_proxy_path [path]

    This attribute specifies the staged executable file path of Remote Communication Proxy on the local machine. Invoke Server must stage the file specified by this attribute if the staging of Remote Communication Proxy is set to true.

  • environment [ENV=VALUE]

    The environment variable for the Ninf-G Executable is passed using this attribute. The environment variable and its value are connected by =. Only the variable is specified if it does not take a value. Multiple environment variables must be specified one by one.

  • status_polling [interval]

    Invoke Server may need to check the status of jobs by polling the status of existing jobs. This attribute specifies the interval of the polling. The value is in seconds, and if it is not specified, the default value 0 is passed.

  • refresh_credential [interval]

    This attribute specifies the interval for refreshing credentials. The value is in seconds, and if it is not specified, the default value 0 is passed.

  • max_time [time]

    This attribute specifies the maximum time of the job.

  • max_wall_time [time]

    This attributes specifies the maximum wall clock time of the job.

  • max_cpu_time [time]

    This attribute specifies the maximum cpu time of the job.

  • queue_name [queue]

    This attribute specifies the name of the queue to which the Ninf-G Executable should be submitted.

  • project [projectname]

    This attribute specifies the name of the project.

  • host_count [number of nodes]

    This attribute specifies the number of nodes.

  • min_memory [memory size (MB)]

    This attribute specifies the minimum requirements for the memory size of the job.

  • max_memory [memory size (MB)]

    This attribute specifies the maximum memory size of the job.

  • rsl_extensions [RSL extension]

    This attribute can be used to specify the RSL extension which is available for the Globus Toolkit's WS GRAM.

3.3 How to specify Invoke Server

3.3.1 How to specify Invoke Server

Invoke Server is specified by using the invoke_server attribute in the <SERVER> section.

invoke_server [type]

Type specifies the type of the Invoke Server, such as GT4py or UNICORE.

3.3.2 How to pass information to Invoke Server

Invoke Server may require options for its execution. Such options can be specified by an option attribute in the <INVOKE_SERVER> section or by an invoke_server_option attribute in the <SERVER> section.

option [String]
invoke_server_option [String]

Multiple attributes can be specified in the <SERVER> or <INVOKE_SERVER> sections.

3.3.3 Polling interval

Invoke Server must check the status of jobs, and this may be implemented using polling. The polling interval can be specified by the status_polling attribute in the <INVOKE_SERVER> section.

status_polling [interval (seconds)]

3.3.4 Logfile

The filename of the Invoke Server's execution log can be specified by the invoke_server_log attribute in the <CLIENT> section.

invoke_server_log [filename]

If this attribute is specified, Invoke Server outputs logs to a file with the specified filename and file type of that Invoke Server.

The log_filePath attribute in the <INVOKE_SERVER> section can be used to specify a log file for a specific Invoke Server.

log_filePath [Log file name]

3.3.5 Maximum number of jobs per Invoke Server

The maximum number of jobs per Invoke Server can be limited by the max_jobs attribute in the <INVOKE_SERVER> section. If the number of requested jobs exceeds this value, the Ninf-G Client invokes a new Invoke Server and requests that Invoke Server to manage the new jobs.

max_jobs [maximum number of jobs]

3.3.6 How to specify the path of Invoke Server

If Invoke Server is not located in a pre-defined directory, the path attribute in <INVOKE_SERVER> can be used to specify the path of the Invoke Server.

path [path of the Invoke Server]

3.4 Miscellaneous Information

3.4.1 Job Timeout

The Job Timeout function is managed by the Ninf-G Client. Invoke Server is not responsible for the timeout.

3.4.2 Redirect stdout/stderr implemented using files

Redirect stdout/stderr is implemented using files.

• The Ninf-G Client passes the filename to Invoke Server as an attribute for the JOB_CREATE request.
• Invoke Server outputs the stdout/stderr of the Ninf-G Executable to the file.
• The Ninf-G Client outputs the contents of the file to the stdout/stderr.

4. Communication Proxy

4.1 Introduction

Communication Proxy mediates communications between Ninf-G Executable and Ninf-G Client.

Ninf-G Client communicates with Ninf-G Executables. At this time, if Communication Proxy is not used, direct TCP/IP connection from the Ninf-G Executable to the Ninf-G Client is performed.

If the Communication Proxy is used, following connection is performed.

1. Ninf-G Executable invokes the Remote Communication Proxy and connects to it.

2. Remote Communication Proxy connects to the Client Communication Proxy.

3. Client Communication Proxy connects to the Ninf-G Client.

About connection of 2., any the communication method and Grid Middleware can be used to implement a Communication Proxy.

In communications between Ninf-G Client and Client Communication Proxy, and in communications between Ninf-G Executable and Remote Communication Proxy, a socket is used as Ninf-G protocol while the External Module protocol (request, reply and notify) is communicated with three pipes.

Ninf-G Version 5.0.0 includes the following Communication Proxy:

• Communication Proxy for Globus Toolkit XIO, implemented in C (GT)

4.1.1 Execution flow

Following is the flow of connection from Ninf-G Executable to Ninf-G Client via Communication Proxy.

1. If the Client Communication Proxy is not invoked, Ninf-G Client invokes the Client Communication Proxy and sends INITIALIZE Request to the Client Communication Proxy.

2. The Client Communication Proxy receives INITIALIZE Request and initializes itself. Then returns INITIALIZE Reply to the Ninf-G Client.

3. The Ninf-G Client sends PREPARE_COMMUNICATION Request, before requesting Invoke Server to invoke job related to function/object handle.

4. The Client Communication Proxy returns the Reply, and prepares communication.

5. The Client Communication Proxy sends COMMUNICATION_REPLY Notify to Ninf-G Client after the preparation of the communication is completed. At this time, COMMUNICATION_REPLY attributes sent to Ninf-G Client includes the information to connect to the Client Communication Proxy(We note "CCP connect info" here).

6. The Ninf-G Client requests to the Invoke Server to invoke a Ninf-G Executable on the server. "CCP connect info" is also send to the Invoke Server.

7. By the Invoke Server, a Ninf-G Executable on the server is invoked. The Ninf-G Executable invokes a Remote Communication Proxy.

8. The Ninf-G Executable sends INITIALIZE Request to the Remote Communication Proxy with "CCP connect info."

9. The Ninf-G Executable waits the INITIALIZE Reply from the Remote Communication Proxy.

10. The Remote Communication Proxy prepares the connection from the Ninf-G Executable, and the address information to connect to the Remote Communication Proxy is returned to the Ninf-G Executable by INITIALIZE Reply.

11. The Ninf-G Executable connects to the address which was returned from the Remote Communication Proxy.

12. The Remote Communication Proxy connects to the address which is described in "CCP connect info", by respective way of Communication Proxy defines.

4.2 Specification of Communication Proxy

4.2.1 Details of Communication Proxy

• Client Communication Proxy and Remote Communication Proxy is invoked when a Ninf-G Client initializes a function/object handle on the remote server which Ninf-G Client is configured to use with the Communication Proxy.

• The maximum number of jobs per Client Communication Proxy is limited. If the number of jobs exceeds the limit, a new Client Communication Proxy is invoked. There may have several connections on one job.

• Remote Communication Proxy exits when the Ninf-G Executable is exiting.

• Ninf-G Client (Ninf-G Executable) waits the termination of Communication Proxy after the Ninf-G Client sends an EXIT request to Communication Proxy.

• The file names used with Communication Proxy must follow the naming convention of "ng_client_communication_proxy" + suffix for Client Communication Proxy, "ng_remote_communication_proxy" + suffix for Remote Communication Proxy , where the suffix corresponds to rules for the underlying middleware used for communication.

4.2.2 Protocol between a Ninf-G Client (Ninf-G Executable) and Communication Proxy

4.2.2.1 Overview

A common feature of External Module protocol is described in Section 2.2. A protocol specific to Communication Proxy is described in the following sections.

4.2.2.2 Request

Four Request messages, INITIALIZE, PREPARE_COMMUNICATION, EXIT and QUERY_FEATURES are supported.

1. INITIALIZE
   • Format

     INITIALIZE<RET> listen_port ...<RET> buffer_size ...<RET> ... (snip) INITIALIZE_END<RET>

   • Explanation

     This request is used to initialize the Communication Proxy after the invocation. Ninf-G Client and Ninf-G Executable send this request to Communication Proxy only once. INITIALIZE request after the first time INITIALIZE causes a failure. This request allows to take time to initialize and not return immediately. This request is a multiple line type request.

     INITIALIZE request attributes are different between Client Communication Proxy and Remote Communication Proxy.

     • Client Communication Proxy

       Following attributes are passed from Ninf-G Client.

       Any attributes not shown in the following table can be specified by specifying Ninf-G Client configuration file <CLIENT_COMMUNICATION_PROXY> section option attribute. These attributes are used for any reason for each Communication Proxy.

       name	mandatory	multiple	meanings
       listen_port	yes	no	port number of Ninf-G Client
       buffer_size	yes	no	buffer size

       • listen_port [number]

         This specifies the port number on which the Ninf-G Client is listening.

       • buffer_size [size]

         This specifies the buffer size of Client Communication Proxy. The units are in bytes.

     • Remote Communication Proxy

       Following attributes are passed from Ninf-G Executable.

       Any attributes not shown in the following table can be specified by specifying communication_proxy_option attribute in <SERVER> section of Ninf-G Client configuration file. These attributes are used for any reason for each Communication Proxy.

       In addition, Remote Communication Proxy receives any attributes which is decided by each Communication Proxy for the following purpose. This attributes may be described by multiple lines.

       • The information to connect to Client Communication Proxy

         This information is generated by COMMUNICATION_REPLY on Client Communication Proxy.

       name	mandatory	multiple	meanings
       hostname	yes	no	hostname of Ninf-G Client
       buffer_size	yes	no	buffer size
       tcp_nodelay	yes	no	TCP_NODELAY
       tcp_connect_retryCount	yes	no	retry count
       tcp_connect_retryBaseInterval	yes	no	retry base interval
       tcp_connect_retryIncreaseRatio	yes	no	retry increase ratio
       tcp_connect_retryRandom	yes	no	random retry

       • hostname [hostname]

         This specifies the hostname of Ninf-G Client.

       • buffer_size [size]

         This specifies the buffer size of Remote Communication Proxy. The units are in bytes.

       • tcp_nodelay [true/false]

         This specifies whether or not to set TCP_NODELAY. If the value is true, TCP_NODELAY must be set to all connections to Remote Communication Proxy.

         Note: If the TCP/IP is not used for communication between Remote Communication Proxy and Client Communication Proxy, the value is ignored.

       • tcp_connect_retryCount [count]

         This specifies the maximum number of retries for establishing a connection.

       • tcp_connect_retryBaseInterval [interval]

         This specifies the base interval time for the first retry. The value is in seconds. This value is used as the maximum interval time for the first retry.

       • tcp_connect_retryIncreaseRatio [ratio]

         This specifies the increase ratio which is used to calculate the maximum interval time between retries. The maximum interval time is calculated by multiplying this value and the maximum interval time for the last retry. For the first retry, the value of tcp_connect_retryBaseInterval is used as the maximum interval time.

         The value is greater than 1.0.

       • tcp_connect_retryRandom [true/false]

         This specifies a flag that specifies whether a random value is used or not for the interval time. If the value is true, the interval time between retries is set randomly between 0.0 seconds to the maximum interval time. If the value is false, the maximum interval time is used as the interval time.

2. PREPARE_COMMUNICATION
   • Format

     PREPARE_COMMUNICATION<RET> request_id 1<RET> tcp_nodelay true<RET> tcp_connect_retryCount 4<RET> tcp_connect_retryBaseInterval 4<RET> tcp_connect_retryIncreaseRatio 2.0<RET> tcp_connect_retryRandom true<RET> ... (snip) PREPARE_COMMUNICATION_END<RET>

   • Explanation

     This request is used to command the Client Communication Proxy to prepare communication for Remote Communication Proxy, when a function handle is creating. This request is issued before the job invocation request for Invoke Server. To prepare communication, it will take time. Thus, report to Ninf-G Client that the communication is ready, is told not by reply, but of a COMMUNICATION_REPLY notify. This request is only issued to Client Communication Proxy. This request is a multiple line type request.

     Following attributes are passed from Ninf-G Client.

     Any attributes not shown in the following table can be specified by specifying Ninf-G Client configuration file <SERVER> section communication_proxy_option attribute. These attributes are used for any reason for each Communication Proxy.

     name	mandatory	multiple	meanings
     request_id	yes	no	Request ID
     tcp_nodelay	yes	no	TCP_NODELAY
     tcp_connect_retryCount	yes	no	retry count
     tcp_connect_retryBaseInterval	yes	no	retry base interval
     tcp_connect_retryIncreaseRatio	yes	no	retry increase ratio
     tcp_connect_retryRandom	yes	no	random retry

     • request_id [id]

       This specifies the ID to identify the request. This ID is used by the COMMUNICATION_REPLY notify, when the communication become ready.

     • tcp_nodelay [true/false]

       This specifies whether or not to set TCP_NODELAY. If the value is true, TCP_NODELAY must be set to connections related to this request on the Client Communication Proxy.

       Note: If the TCP/IP is not used for communication between Remote Communication Proxy and Client Communication Proxy, the value is ignored.

     • tcp_connect_retryCount [count]

       This specifies the maximum number of retries for establishing a connection.

     • tcp_connect_retryBaseInterval [interval]

       This specifies the base interval time for the first retry. The value is in seconds. This value is used as the maximum interval time for the first retry.

     • tcp_connect_retryIncreaseRatio [ratio]

       This specifies the increase ratio which is used to calculate the maximum interval time between retries. The maximum interval time is calculated by multiplying this value and the maximum interval time for the last retry. For the first retry, the value of tcp_connect_retryBaseInterval is used as the maximum interval time.

       The value is greater than 1.0.

     • tcp_connect_retryRandom [true/false]

       This specifies a flag that specifies whether a random value is used or not for the interval time. If the value is true, the interval time between retries is set randomly between 0.0 seconds to the maximum interval time. If the value is false, the maximum interval time is used as the interval time.

3. EXIT
   • Format

     EXIT<RET>

   • Explanation

     This request is used to exit the Communication Proxy. This request is a one line type request and have no parameter.

4.2.2.3 Reply

Communication Proxy must send a Reply message to a Ninf-G Client (Ninf-G Executable) if Communication Proxy receives a Request message from that Ninf-G Client (Ninf-G Executable).

• INITIALIZE
  • Client Communication Proxy

    The reply to INITIALIZE messages is:

    [S | F <Error String>]<RET>

    where S is sent in case of Success. Otherwise, F is returned, followed by <Error String>.

  • Remote Communication Proxy

    The reply to INITIALIZE messages is:

    SM<RET> address [address-string]<RET> REPLY_END<RET>

  where SM is sent in case of Success. Otherwise, F is returned, followed by <Error String>.

  [address-string] is the address, which the Ninf-G Executable must connect to. This address is TCP or the local socket address. This address may not be Remote Communication Proxy.

  The address form is as follows:

  • TCP

    
    ng_tcp://[hostname]:[port-number] 
    

    Ninf-G Executable must connect to the hostname of [hostname], and the port number of [port-number] by TCP.

  • local socket

    
    ng_local://[path-to-local-socket] 
    

    Ninf-G Executable must connect to the [path-to-local-socket] by local socket (UNIX domain socket).

• PREPARE_COMMUNICATION, EXIT

  The reply to PREPARE_COMMUNICATION and EXIT messages is:

  [S | F <Error String>]<RET>

  where S is sent in case of Success. Otherwise, F is returned, followed by <Error String>.

• QUERY_FEATURES

  The reply to a QUERY_FEATURES request is described in 2.2.3. Ninf-G Version 5.0.0 Communication Proxy has protocol version 1.0.

  Communication Proxy have no extended features.

4.2.2.4 Notify

A Notify message is used to send an asynchronous message from Communication Proxy to a Ninf-G Client. One type of Notify message is provided.

1. COMMUNICATION_REPLY
   • Format

     COMMUNICATION_REPLY<RET> request_id 1<RET> result [S | F]<RET> ... (snip) COMMUNICATION_REPLY_END<RET>

   • Explanation

     This Notify tells the Ninf-G Client that the preparation, which was requested by PREPARE_COMMUNICATION Request, has been completed. This Notify is only used on Client Communication Proxy. This Notify is a multiple line type Notify.

     Following attributes are passed from Client Communication Proxy.

     Any attributes not shown in the following table can be passed, for the following purpose.

     • The information to connect to Client Communication Proxy

       This information is used by Remote Communication Proxy to connect to Client Communication Proxy. This information is first passed to Ninf-G Client, then Ninf-G Executable, and finally passed to Remote Communication Proxy. Multiple attributes can be passed. If the preparation of communication failed, this attributes cannot be passed.

     name	mandatory	multiple	meanings
     request_id	yes	no	Request ID
     result	yes	no	result
     message	no	no	message

     • request_id [id]

       This specifies the Request ID passed by PREPARE_COMMUNICATION Request.

     • result [S|F]

       This specifies the preparation of communication was successful or not. S is returned on success, F is returned on failure.

     • message [error message]

       This specifies the error cause message. This attribute is only specified on failure case.

5. Information Service

5.1 Introduction

Information Service searches and retrieves the Ninf-G Executable information. Ninf-G Executable information is required when a function handle is creating. Information Service is a External Module, which enable to search and retrieve Ninf-G Executable information, without depending on specific Grid Middleware.

Ninf-G Version 5.0.0 includes the following Information Service:

• Information Service for NRF, implemented in C (NRF)
• Information Service for Globus Toolkit WS MDS, implemented in Java (MDS4)

5.2 Specification of Information Service

5.2.1 Details of Information Service

• Information Service is invoked when a Ninf-G Client searches a Ninf-G Executable information for new function/object handle.

• Ninf-G Client wait for the termination of Information Service after the Ninf-G Client sends an EXIT request to Information Service.

• The file names used with Information Service must follow the naming convention of "ng_information_service" + suffix, where the suffix corresponds to rules for the underlying middleware used for information query.

5.2.2 Protocol between a Ninf-G Client (Ninf-G Executable) and Information Service

5.2.2.1 Overview

A common feature of External Module protocol is described in Section 2.2. A protocol specific to Information Service is described in the following sections.

5.2.2.2 Request

Four Request messages, QUERY_REMOTE_EXECUTABLE_INFORMATION, CANCEL_QUERY, EXIT and QUERY_FEATURES are supported.

1. QUERY_REMOTE_EXECUTABLE_INFORMATION
   • Format

     QUERY_REMOTE_EXECUTABLE_INFORMATION<RET> hostname ...<RET> classname ...<RET> source ...<RET> ... (snip) QUERY_REMOTE_EXECUTABLE_INFORMATION_END<RET>

   • Explanation

     This Request is used to query the Ninf-G Executable information. The searched information is returned to Ninf-G Client by REMOTE_EXECUTABLE_INFORMATION_NOTIFY Notify. This Request is a multi line type Request.

     Following attributes are passed from Ninf-G Client.

     Any attributes not shown in the following table can be specified by specifying Ninf-G Client configuration file <INFORMATION_SOURCE> section option attribute. These attributes are used for any reason for each Information Service.

     name	mandatory	multiple	meanings
     hostname	yes	no	hostname
     classname	yes	no	classname
     source	yes	yes	source

     • hostname [hostname]

       This specifies the hostname of query target.

     • classname [classname]

       This specifies the classname of query target.

     • source [source string]

       This specifies the source of Ninf-G Executable information. The meaning and expression of source can be defined by respective Information Service. The Information Service must search the information from the specified sources. This attribute can specify one source only, and multiple sources can be specified one by one.

2. CANCEL_QUERY
   • Format

     CANCEL_QUERY <Query ID><RET>

   • Explanation

     This request is used to cancel the ongoing query of the Ninf-G Executable information. The argument <Query ID> is the ID issued by QUERY_REMOTE_EXECUTABLE_INFORMATION Reply. This Request is a one line type Request.

3. EXIT
   • Format

     EXIT<RET>

   • Explanation

     This request is used to exit the Information Service. This request is a one line type request and have no parameter.

5.2.2.3 Reply

Information Service must send a Reply message to a Ninf-G Client if Information Service receives a Request message from that Ninf-G Client.

• QUERY_REMOTE_EXECUTABLE_INFORMATION

  The reply to QUERY_REMOTE_EXECUTABLE_INFORMATION messages is:

  [S <Query ID> | F <Error String>]<RET>

  where S is sent in case of Success. Otherwise, F is returned, followed by <Error String>.

  Query ID is the ID to identify the query. A Query ID is case sensitive and cannot include invisible characters.

• CANCEL_QUERY, EXIT

  The reply to CANCEL_QUERY and EXIT messages is:

  [S | F <Error String>]<RET>

  where S is sent in case of Success. Otherwise, F is returned, followed by <Error String>.

• QUERY_FEATURES

  The reply to a QUERY_FEATURES request is described in 2.2.3. Ninf-G Version 5.0.0 Information Service has protocol version 1.0.

  Information Service have no extended features.

5.2.2.4 Notify

A Notify message is used to send an asynchronous message from Information Service to a Ninf-G Client. One type of Notify message are provided.

1. REMOTE_EXECUTABLE_INFORMATION_NOTIFY
   • Format

     REMOTE_EXECUTABLE_INFORMATION_NOTIFY<RET> query_id ...<RET> result [S | F]<RET> remote_executable_information <?xml version="1.0" encoding="utf-8 ?><RET> remote_executable_information <RemoteExecutableInformation><RET> remote_executable_information <hostName>...</hostName><RET> ... (snip) remote_executable_information </class><RET> remote_executable_information </RemoteExecutableInformation><RET> REMOTE_EXECUTABLE_INFORMATION_NOTIFY_END<RET>

   • Explanation

     This Notify is used to return the information, queried in advance from the Ninf-G Client. The returned query result is success or failure, and if the result is success, XML form of information is returned. This Notify is a multi line type Notify.

     The XML information, that must be returned to Ninf-G Client is as follows:

     • The returned information is Remote Executable Information.
     • Following namespaces must be defined.

       
       rei=http://ninf.apgrid.org/2006/12/NinfGRemoteInformation
       rci=http://ninf.apgrid.org/2006/12/RemoteClassInformation 
       

     • Remote Executable Information is expressed as XML document. root element is <rei:RemoteExecutableInformation>.
     • <rei:hostName> text in <rei:RemoteExecutableInformation> must be hostname of specified query target.
     • "name" attribute of <rci:class> in <rei:RemoteExecutableInformation> must be classname of specified query target.

     This XML document is written in NRF file, which is generated when the Ninf-G Executable is compiled.

     Following attributes are passed from Information Service.

     name	mandatory	multiple	meanings
     query_id	yes	no	Query ID
     result	yes	no	Query result
     remote_executable_information	no	yes	information
     error_message	no	no	error message

     • query_id [id]

       This specifies the query ID, which is generated by QUERY_REMOTE_EXECUTABLE_INFORMATION Reply.

     • result [S|F]

       This specifies the result of the query. If the query is successful, S is returned, otherwise, F is returned.

     • remote_executable_information [string]

       This specifies the Remote Executable Information. This information is XML document, which have multiple lines. This attribute can specify one line of XML document only, and multiple lines must be specified one by one, by using this attribute for each one. The attribute value cannot include a return character code.

       This attribute is only specified if the query is successful.

     • error_message [message]

       This specifies the error message, when the query fail.

       This attribute is only specified if the query fail.