    .SH "DESCRIPTION"
    \fBsacctmgr\fR is used to view or modify Slurm account information.
    The account information is maintained within a database with the interface
    being provided by \fBslurmdbd\fR (Slurm Database daemon).
    This database can serve as a central storehouse of user and
    computer information for multiple computers at a single site.
    Slurm account information is recorded based upon four parameters
    that form what is referred to as an \fIassociation\fR.
    These parameters are \fIuser\fR, \fIcluster\fR, \fIpartition\fR, and
    \fIaccount\fR. \fIuser\fR is the login name.
    \fIcluster\fR is the name of a Slurm managed cluster as specified by
    the \fIClusterName\fR parameter in the \fIslurm.conf\fR configuration file.
    \fIpartition\fR is the name of a Slurm partition on that cluster.
    \fIaccount\fR is the bank account for a job.
    The intended mode of operation is to initiate the \fBsacctmgr\fR command,
    add, delete, modify, and/or list \fIassociation\fR records then
    commit the changes and exit.

    .TP "7"
    \f3Note: \fP\c
    The content's of Slurm's database are maintained in lower case. This may
    result in some \f3sacctmgr\fP output differing from that of other Slurm
    commands.

    .SH "OPTIONS"

    .TP
    \fB\-h\fR, \fB\-\-help\fR
    Print a help message describing the usage of \fBsacctmgr\fR.
    This is equivalent to the \fBhelp\fR command.

    .TP
    \fB\-i\fR, \fB\-\-immediate\fR
    commit changes immediately without asking for confirmation.

    .TP
    \fB\-n\fR, \fB\-\-noheader\fR
    No header will be added to the beginning of the output.

    .TP
    \fB\-p\fR, \fB\-\-parsable\fR
    Output will be '|' delimited with a '|' at the end.

    .TP
    \fB\-P\fR, \fB\-\-parsable2\fR
    Output will be '|' delimited without a '|' at the end.

    .TP
    \fB\-Q\fR, \fB\-\-quiet\fR
    Print no messages other than error messages.
    This is equivalent to the \fBquiet\fR command.
    This is equivalent to the \fBverbose\fR command.

    .TP
    \fB\-V\fR , \fB\-\-version\fR
    Display version number.
    This is equivalent to the \fBversion\fR command.

    .SH "COMMANDS"

    .TP
    \fBadd\fR <\fIENTITY\fR> <\fISPECS\fR>
    Add an entity.
    Identical to the \fBcreate\fR command.

    .TP
    \fBassociations\fR
    Use with show or list to display associations with the entity.

    .TP
    \fBclear\fR stats
    Clear the server statistics.

    .TP
    \fBcreate\fR <\fIENTITY\fR> <\fISPECS\fR>
    Add an entity.
    Identical to the \fBadd\fR command.

    .TP
    \fBdelete\fR <\fIENTITY\fR> where <\fISPECS\fR>
    Delete the specified entities.

    .TP
    \fBdump\fR <\fIENTITY\fR> <\fIFile=FILENAME\fR>
    Dump cluster data to the specified file. If the filename is not specified
    it uses clustername.cfg filename by default.

    .TP
    \fBhelp\fP
    Display a description of sacctmgr options and commands.

    .TP
    \fBlist\fR <\fIENTITY\fR> [<\fISPECS\fR>]
    Display information about the specified entity.
    By default, all entries are displayed, you can narrow results by
    specifying SPECS in your query.
    Identical to the \fBshow\fR command.

    .TP
    \fBload\fR <\fIFILENAME\fR>
    Load cluster data from the specified file. This is a configuration file
    generated by running the sacctmgr dump command. This command does
    not load archive data, see the sacctmgr archive load option instead.
    \fBshow\fR <\fIENTITY\fR> [<\fISPECS\fR>]
    Display information about the specified entity.
    By default, all entries are displayed, you can narrow results by
    specifying SPECS in your query.
    Identical to the \fBlist\fR command.

    .TP
    \fBshutdown\fR
    Shutdown the server.

    .TP
    \fBversion\fP
    Display the version number of sacctmgr.

    .SH "INTERACTIVE COMMANDS"

    \fBNOTE:\fP
    All commands listed below can be used in the interactive mode, but \fINOT\fP
    on the initial command line.

    .TP
    \fBexit\fP
    Terminate sacctmgr interactive mode.
    Identical to the \fBquit\fR command.

    .TP
    \fBquiet\fP
    Print no messages other than error messages.

    .TP
    \fBquit\fP
    Terminate the execution of sacctmgr interactive mode.
    Identical to the \fBexit\fR command.

    .TP
    \fBverbose\fP
    Enable detailed logging.
    This includes time\-stamps on data structures, record counts, etc.
    This is an independent command with no options meant for use in
    interactive mode.

    .TP
    \fB!!\fP
    Repeat the last command.

    .SH "ENTITIES"

    .TP
    \fIaccount\fP
    A bank account, typically specified at job submit time using the
    \fI\-\-account=\fR option.
    These may be arranged in a hierarchical fashion, for example
    The \fIClusterName\fR parameter in the \fIslurm.conf\fR configuration
    file, used to differentiate accounts on different machines.

    .TP
    \fIconfiguration\fP
    Used only with the \fIlist\fR or \fIshow\fR command to report current
    system configuration.

    .TP
    \fIcoordinator\fR
    A special privileged user usually an account manager or such that can
    add users or sub accounts to the account they are coordinator over.
    This should be a trusted person since they can change limits on
    account and user associations inside their realm.

    .TP
    \fIevent\fR
    Events like downed or draining nodes on clusters.

    .TP
    \fIfederation\fP
    A group of clusters that work together to schedule jobs.

    .TP
    \fIjob\fR
    Used to modify specific fields of a job: Derived Exit Code and
    the Comment String.

    .TP
    \fIqos\fR
    Quality of Service.

    .TP
    \fIResource\fP
    Software resources for the system. Those are software licenses shared
    among clusters.

    .TP
    \fIRunawayJobs\fR
    Used only with the \fIlist\fR or \fIshow\fR command to report current
    jobs that have been orphanded on the local cluster and are now
    runaway.  If there are jobs in this state it will also give you an
    option to "fix" them.
    NOTE: You must have an \fBAdminLevel\fR of at least \fBOperator\fR to preform
    this.

    .TP
    \fIstats\fR
    Used with \fBlist\fR or \fBshow\fR command to view server statistics.
    Accepts optional argument of \fBave_time\fR or \fBtotal_time\fR to sort on those
    fields. By default, sorts on increasing RPC count field.

    being considered for being allocated resources.
    If starting a job would cause any of its group limit to be exceeded,
    that job will not be considered for scheduling even if that job might preempt
    other jobs which would release sufficient group resources for the pending
    job to be initiated.

    .TP
    \fIDefaultQOS\fP=<default qos>
    The default QOS this association and its children should have.
    This is overridden if set directly on a user.
    To clear a previously set value use the modify command with a new value of \-1.

    .TP
    \fIFairshare\fP=<fairshare number | parent>
    Number used in conjunction with other accounts to determine job
    priority. Can also be the string \fIparent\fR, when used on a user
    this means that the parent association is used for fairshare.  If
    Fairshare=parent is set on an account, that account's children will be
    effectively reparented for fairshare calculations to the first parent
    of their parent that is not Fairshare=parent.  Limits remain the same,
    only it's fairshare value is affected.  To clear a previously set
    value use the modify command with a new value of \-1.

    .TP
    \fIGraceTime\fP=<preemption grace time in seconds>
    Specifies, in units of seconds, the preemption grace time
    to be extended to a job which has been selected for preemption.
    The default value is zero, no preemption grace time is allowed on
    this QOS.

    NOTE: This value is only meaningful for QOS PreemptMode=CANCEL.

    .TP
    \fIGrpTRESMins\fP=<TRES=max TRES minutes,...>
    The total number of TRES minutes that can possibly be used by past,
    present and future jobs running from this association and its children.
    To clear a previously set value use the modify command with a new
    value of \-1 for each TRES id.

    NOTE: This limit is not enforced if set on the root
    association of a cluster.  So even though it may appear in sacctmgr
    output, it will not be enforced.

    ALSO NOTE: This limit only applies when using the Priority Multifactor
    plugin.  The time is decayed using the value of PriorityDecayHalfLife
    or PriorityUsageResetPeriod as set in the slurm.conf.  When this limit
    is reached all associated jobs running will be killed and all future
    jobs submitted with associations in the group will be delayed until
    they are able to run inside the limit.

    .TP
    \fIGrpTRESRunMins\fP=<TRES=max TRES run minutes,...>
    Resource plugin.

    .TP
    \fIGrpJobs\fP=<max jobs>
    Maximum number of running jobs in aggregate for
    this association and all associations which are children of this association.
    To clear a previously set value use the modify command with a new value of \-1.

    .TP
    \fIGrpJobsAccrue\fP=<max jobs>
    Maximum number of pending jobs in aggregate able to accrue age priority for this
    association and all associations which are children of this association.
    To clear a previously set value use the modify command with a new value of \-1.

    .TP
    \fIGrpSubmitJobs\fP=<max jobs>
    Maximum number of jobs which can be in a pending or running state at any time
    in aggregate for this association and all associations which are children of
    this association.
    To clear a previously set value use the modify command with a new value of \-1.

    .TP
    \fIGrpWall\fP=<max wall>
    Maximum wall clock time running jobs are able to be allocated in aggregate for
    this association and all associations which are children of this association.
    To clear a previously set value use the modify command with a new value of \-1.

    NOTE: This limit is not enforced if set on the root association of a
    cluster.  So even though it may appear in sacctmgr output, it will not
    be enforced.

    ALSO NOTE: This limit only applies when using the Priority Multifactor
    plugin.  The time is decayed using the value of PriorityDecayHalfLife
    or PriorityUsageResetPeriod as set in the slurm.conf.  When this limit
    is reached all associated jobs running will be killed and all future
    jobs submitted with associations in the group will be delayed until
    they are able to run inside the limit.

    .TP
    \fIMaxTRESMins\fP=<max TRES minutes>
    Maximum number of TRES minutes each job is able to use in this association.
    This is overridden if set directly on a user.
    Default is the cluster's limit.
    To clear a previously set value use the modify command with a new
    value of \-1 for each TRES id.

    .TP
    \fIMaxTRES\fP=<max TRES>
    Maximum number of TRES each job is able to use in this association.
    This is overridden if set directly on a user.
    Default is the cluster's limit.
    To clear a previously set value use the modify command with a new
    \fIMaxJobsAccrue\fP=<max jobs>
    Maximum number of pending jobs able to accrue age priority at any given time for
    the given association.
    This is overridden if set directly on a user.
    Default is the cluster's limit.
    To clear a previously set value use the modify command with a new value of \-1.

    .TP
    \fIMaxSubmitJobs\fP=<max jobs>
    Maximum number of jobs which can this association can have in a
    pending or running state at any time.
    Default is the cluster's limit.
    To clear a previously set value use the modify command with a new value of \-1.

    .TP
    \fIMaxWall\fP=<max wall>
    Maximum wall clock time each job is able to use in this association.
    This is overridden if set directly on a user.
    Default is the cluster's limit.
    <max wall> format is <min> or <min>:<sec> or <hr>:<min>:<sec> or
    <days>\-<hr>:<min>:<sec> or <days>\-<hr>.
    The value is recorded in minutes with rounding as needed.
    To clear a previously set value use the modify command with a new value of \-1.

    NOTE: Changing this value will have no effect on any running or
    pending job.

    .TP
    \fIPriority\fP
    What priority will be added to a job\'s priority when using this association.
    This is overridden if set directly on a user.
    Default is the cluster's limit.
    To clear a previously set value use the modify command with a new value of \-1.

    .TP
    \fIQosLevel\fP<operator><comma separated list of qos names>
    Specify the default Quality of Service's that jobs are able to run at
    for this association.  To get a list of valid QOS's use 'sacctmgr list qos'.
    This value will override its parents value and push down to its
    children as the new default.  Setting a QosLevel to '' (two single
    quotes with nothing between them) restores its default setting.  You
    can also use the operator += and \-= to add or remove certain QOS's
    from a QOS list.

    Valid <operator> values include:
    .RS
    .TP 5
    \fB=\fR
    Set \fIQosLevel\fP to the specified value. \fBNote:\fR the QOS that can be used
    at a given account in the hierarchy are inherited by the children of that account.
    By assigning QOS with the \fB=\fR sign only the assigned QOS can be used by the
    account and its children.
    .TP
    \fICluster\fP=<cluster>
    Specific cluster to add account to.  Default is all in system.

    .TP
    \fIDescription\fP=<description>
    An arbitrary string describing an account.

    .TP
    \fIName\fP=<name>
    The name of a bank account.
    Note the name must be unique and can not be represent different bank
    accounts at different points in the account hierarchy.

    .TP
    \fIOrganization\fP=<org>
    Organization to which the account belongs.

    .TP
    \fIParent\fP=<parent>
    Parent account of this account. Default is the root account, a top
    level account.

    .TP
    \fIRawUsage\fP=<value>
    This allows an administrator to reset the raw usage accrued to an
    account.  The only value currently supported is 0 (zero).  This is a
    settable specification only - it cannot be used as a filter to list
    accounts.

    .TP
    \fIWithAssoc\fP
    Display all associations for this account.

    .TP
    \fIWithCoord\fP
    Display all coordinators for this account.

    .TP
    \fIWithDeleted\fP
    Display information with previously deleted data.
    .P
    NOTE: If using the WithAssoc option you can also query against
    association specific information to view only certain associations
    this account may have.  These extra options can be found in the
    \fISPECIFICATIONS FOR ASSOCIATIONS\fP section.  You can also use the
    general specifications list above in the \fIGENERAL SPECIFICATIONS FOR
    ASSOCIATION BASED ENTITIES\fP section.

    .SH "LIST/SHOW ACCOUNT FORMAT OPTIONS"

    .TP
    when using the WithCoordinator option.)
    .P
    NOTE: If using the WithAssoc option you can also view the information
    about the various associations the account may have on all the
    clusters in the system. The association information can be filtered.
    Note that all the accounts in the database will always be shown as filter only
    takes effect over the association data. The Association format fields are
    described in the \fILIST/SHOW ASSOCIATION FORMAT OPTIONS\fP section.

    .SH "SPECIFICATIONS FOR ASSOCIATIONS"

    .TP
    \fIClusters\fP=<comma separated list of cluster names>
    List the associations of the cluster(s).

    .TP
    \fIAccounts\fP=<comma separated list of account names>
    List the associations of the account(s).

    .TP
    \fIUsers\fP=<comma separated list of user names>
    List the associations of the user(s).

    .TP
    \fIPartition\fP=<comma separated list of partition names>
    List the associations of the partition(s).
    .P
    NOTE: You can also use the general specifications list above in the
    \fIGENERAL SPECIFICATIONS FOR ASSOCIATION BASED ENTITIES\fP section.

    \fBOther options unique for listing associations:\fP

    .TP
    \fIOnlyDefaults\fP
    Display only associations that are default associations

    .TP
    \fITree\fP
    Display account names in a hierarchical fashion.

    .TP
    \fIWithDeleted\fP
    Display information with previously deleted data.

    .TP
    \fIWithSubAccounts\fP
    Display information with subaccounts.  Only really valuable when used
    with the account= option.  This will display all the subaccount
    associations along with the accounts listed in the option.

    .TP
    from the parent).

    .SH "LIST/SHOW ASSOCIATION FORMAT OPTIONS"

    .TP
    \fIAccount\fP
    The name of a bank account in the association.

    .TP
    \fICluster\fP
    The name of a cluster in the association.

    .TP
    \fIDefaultQOS\fP
    The QOS the association will use by default if it as access to it in
    the QOS list mentioned below.

    .TP
    \fIFairshare\fP
    Number used in conjunction with other accounts to determine job
    priority. Can also be the string \fIparent\fR, when used on a user
    this means that the parent association is used for fairshare.  If
    Fairshare=parent is set on an account, that account's children will be
    effectively reparented for fairshare calculations to the first parent
    of their parent that is not Fairshare=parent.  Limits remain the same,
    only it's fairshare value is affected.

    .TP
    \fIGrpTRESMins\fP
    The total number of TRES minutes that can possibly be used by past,
    present and future jobs running from this association and its children.

    .TP
    \fIGrpTRESRunMins\fP
    Used to limit the combined total number of TRES minutes used by all
    jobs running with this association and its children.  This takes into
    consideration time limit of running jobs and consumes it, if the limit
    is reached no new jobs are started until other jobs finish to allow
    time to free up.

    .TP
    \fIGrpTRES\fP
    Maximum number of TRES running jobs are able to be allocated in aggregate for
    this association and all associations which are children of this association.

    .TP
    \fIGrpJobs\fP
    Maximum number of running jobs in aggregate for
    this association and all associations which are children of this association.

    .TP

    .TP
    \fIID\fP
    The id of the association.

    .TP
    \fILFT\fP
    Associations are kept in a hierarchy: this is the left most
    spot in the hierarchy.  When used with the RGT variable, all
    associations with a LFT inside this LFT and before the RGT are
    children of this association.

    .TP
    \fIMaxTRESMins\fP
    Maximum number of TRES minutes each job is able to use.

    .TP
    \fIMaxTRES\fP
    Maximum number of TRES each job is able to use.

    .TP
    \fIMaxJobs\fP
    Maximum number of jobs each user is allowed to run at one time.

    .TP
    \fIMaxJobsAccrue\fP
    Maximum number of pending jobs able to accrue age priority at any given time.

    .TP
    \fIMaxSubmitJobs\fP
    Maximum number of jobs pending or running state at any time.

    .TP
    \fIMaxWall\fP
    Maximum wall clock time each job is able to use.

    .TP
    \fIQos\fP
    Valid QOS\' for this association.

    .TP
    \fIParentID\fP
    The association id of the parent of this association.

    .TP
    \fIParentName\fP
    The account name of the parent of this association.

    .TP
    \fIPartition\fP
    The name of a partition in the association.

    was assigned, added or filtered in the hierarchy.

    .TP
    \fIRGT\fP
    Associations are kept in a hierarchy: this is the right most
    spot in the hierarchy.  When used with the LFT variable, all
    associations with a LFT inside this RGT and after the LFT are
    children of this association.

    .TP
    \fIUser\fP
    The name of a user in the association.

    .SH "SPECIFICATIONS FOR CLUSTERS"

    .TP
    \fIClassification\fP=<classification>
    Type of machine, current classifications are capability and capacity.

    .TP
    \fIFeatures\fP=<comma separated list of feature names>
    Features that are specific to the cluster. Federated jobs can be directed to
    clusters that contain the job requested features.

    .TP
    \fIFederation\fP=<federation>
    The federation that this cluster should be a member of. A cluster can only be a
    member of one federation at a time.

    .TP
    \fIFedState\fP=<state>
    The state of the cluster in the federation.
    .br
    Valid states are:
    .RS
    .TP
    \fIACTIVE\fP
    Cluster will actively accept and schedule federated jobs.

    .TP
    \fIINACTIVE\fP
    Cluster will not schedule or accept any jobs.

    .TP
    \fIDRAIN\fP
    Cluster will not accept any new jobs and will let existing federated jobs
    complete.

    .TP
    \fIDRAIN+REMOVE\fP
    Cluster will not accept any new jobs and will remove itself from the federation
    once all federated jobs have completed. When removed from the federation, the
    .TP
    \fIRPC\fP=<rpc list>
    Comma separated list of numeric RPC values.

    .TP
    \fIWithFed\fP
    Appends federation related columns to default format options
    (e.g. Federation,ID,Features,FedState).

    .TP
    \fIWOLimits\fP
    Display information without limit information. This is for a smaller
    default format of Cluster,ControlHost,ControlPort,RPC
    .P
    NOTE: You can also use the general specifications list above in the
    \fIGENERAL SPECIFICATIONS FOR ASSOCIATION BASED ENTITIES\fP section.

    .SH "LIST/SHOW CLUSTER FORMAT OPTIONS"

    .TP
    \fIClassification\fP
    Type of machine, i.e. capability or capacity.

    .TP
    \fICluster\fP
    The name of the cluster.

    .TP
    \fIControlHost\fP
    When a slurmctld registers with the database the ip address of the
    controller is placed here.

    .TP
    \fIControlPort\fP
    When a slurmctld registers with the database the port the controller
    is listening on is placed here.

    .TP
    \fIFeatures\fP
    The list of features on the cluster (if any).

    .TP
    \fIFederation\fP
    The name of the federation this cluster is a member of (if any).

    .TP
    \fIFedState\fP
    The state of the cluster in the federation (if a member of one).

    .TP
    \fIFedStateRaw\fP

    .TP
    \fINodeNames\fP
    The current Nodes associated with the cluster.

    .TP
    \fIPluginIDSelect\fP
    The numeric value of the select plugin the cluster is using.

    .TP
    \fIRPC\fP
    When a slurmctld registers with the database the rpc version the controller
    is running is placed here.

    .TP
    \fITRES\fP
    Trackable RESources (Billing, BB (Burst buffer), CPU, Energy, GRES, License,
    Memory, and Node) this cluster is accounting for.

    .P
    NOTE: You can also view the information about the root association for
    the cluster.  The Association format fields are described
    in the \fILIST/SHOW ASSOCIATION FORMAT OPTIONS\fP section.

    .SH "SPECIFICATIONS FOR COORDINATOR"

    .TP
    \fIAccount\fP=<comma separated list of account names>
    Account name to add this user as a coordinator to.
    .TP
    \fINames\fP=<comma separated list of user names>
    Names of coordinators.
    .P
    NOTE: To list coordinators use the WithCoordinator options with list
    account or list user.

    .SH "SPECIFICATIONS FOR EVENTS"

    .TP
    \fIAll_Clusters\fP
    Get information on all cluster shortcut.

    .TP
    \fIAll_Time\fP
    Get time period for all time shortcut.

    .TP
    \fIClusters\fP=<comma separated list of cluster names>
    List the events of the cluster(s).  Default is the cluster where the
    command was run.

    .TP
    \fIEvent\fP=<OPT>
    Specific events to look for, valid options are Cluster or Node,
    default is both.

    .TP
    \fIMaxTRES\fP=<OPT>
    Max number of TRES affected by an event.

    .TP
    \fIMinTRES\fP=<OPT>
    Min number of TRES affected by an event.

    .TP
    \fINodes\fP=<comma separated list of node names>
    Node names affected by an event.

    .TP
    \fIReason\fP=<comma separated list of reasons>
    Reason an event happened.

    .TP
    \fIStart\fP=<OPT>
    Period start of events.  Default is 00:00:00 of previous day, unless
    states are given with the States= spec events.  If this is the case
    the default behavior is to return events currently in
    the states specified.

    Valid time formats are...
    .sp
    HH:MM[:SS] [AM|PM]
    .br
    MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
    .br
    MM/DD[/YY]\-HH:MM[:SS]
    .br
    YYYY\-MM\-DD[THH:MM[:SS]]

    .TP
    \fIStates\fP=<comma separated list of states>
    State of a node in a node event.  If this is set, the event type is
    set automatically to Node.

    .TP
    \fIUser\fP=<comma separated list of users>
    Query against users who set the event.  If this is set, the event type is
    set automatically to Node since only user slurm can perform a cluster event.

    .SH "LIST/SHOW EVENT FORMAT OPTIONS"

    Period when event ended.

    .TP
    \fIEvent\fP
    Name of the event.

    .TP
    \fIEventRaw\fP
    Numeric value of the name of the event.

    .TP
    \fINodeName\fP
    The node affected by the event.  In a cluster event, this is blank.

    .TP
    \fIReason\fP
    The reason an event happened.

    .TP
    \fIStart\fP
    Period when event started.

    .TP
    \fIState\fP
    On a node event this is the formatted state of the node during the event.

    .TP
    \fIStateRaw\fP
    On a node event this is the numeric value of the state of the node
    during the event.

    .TP
    \fITRES\fP
    Number of TRES involved with the event.

    .TP
    \fIUser\fP
    On a node event this is the user who caused the event to happen.

    .SH "SPECIFICATIONS FOR FEDERATION"

    .TP
    \fIClusters\fP[+|\-]=<comma separated list of cluster names>
    List of clusters to add/remove to a federation. A blank value (e.g. clusters=)
    will remove all federations for the federation. NOTE: a cluster can only be a
    member of one federation.

    .TP
    \fIName\fP=<name>
    The name of the federation.

    .TP
    \fICluster\fP
    Name of the cluster that is a member of the federation.

    .TP
    \fIFedState\fP
    The state of the cluster in the federation.

    .TP
    \fIFedStateRaw\fP
    Numeric value of the name of the FedState.

    .TP
    \fIIndex\fP
    The index of the cluster in the federation.

    .SH "SPECIFICATIONS FOR JOB"

    .TP
    \fIDerivedExitCode\fP
    The derived exit code can be modified after a job completes based on
    the user's judgment of whether the job succeeded or failed. The user
    can only modify the derived exit code of their own job.

    .TP
    \f3Comment\fP
    The job's comment string when the AccountingStoreJobComment parameter
    in the slurm.conf file is set (or defaults) to YES.  The user can only
    modify the comment string of their own job.

    .TP
    The \fIDerivedExitCode\fP and \f3Comment\fP fields are the only fields
    of a job record in the database that can be modified after job
    completion.

    .SH "LIST/SHOW JOB FORMAT OPTIONS"

    The \fBsacct\fR command is the exclusive command to display job
    records from the Slurm database.

    .SH "SPECIFICATIONS FOR QOS"
    \fBNOTE:\fR The group limits (GrpJobs, GrpNodes, etc.) are tested when a job is
    being considered for being allocated resources.
    If starting a job would cause any of its group limit to be exceeded,
    that job will not be considered for scheduling even if that job might preempt
    other jobs which would release sufficient group resources for the pending
    job to be initiated.

    .TP
    \fIFlags\fP
    Used by the slurmctld to override or enforce certain characteristics.
    any jobs submitted with this QOS that fall below the UsageThreshold
    will be held until their Fairshare Usage goes above the Threshold.
    .TP
    \fINoReserve\fP
    If this flag is set and backfill scheduling is used, jobs using this QOS will
    not reserve resources in the backfill schedule's map of resources allocated
    through time. This flag is intended for use with a QOS that may be preempted
    by jobs associated with all other QOS (e.g use with a "standby" QOS). If this
    flag is used with a QOS which can not be preempted by all other QOS, it could
    result in starvation of larger jobs.
    .TP
    \fIPartitionMaxNodes\fP
    If set jobs using this QOS will be able to
    override the requested partition's MaxNodes limit.
    .TP
    \fIPartitionMinNodes\fP
    If set jobs using this QOS will be able to
    override the requested partition's MinNodes limit.
    .TP
    \fIOverPartQOS\fP
    If set jobs using this QOS will be able to
    override any limits used by the requested partition's QOS limits.
    .TP
    \fIPartitionTimeLimit\fP
    If set jobs using this QOS will be able to
    override the requested partition's TimeLimit.
    .TP
    \fIRequiresReservaton\fP
    If set jobs using this QOS must designate a reservation when submitting a job.
    This option can be useful in restricting usage of a QOS that may have greater
    preemptive capability or additional resources to be allowed only within a
    reservation.
    .TP
    \fINoDecay\fP
    If set, this QOS will not have its GrpTRESMins,
    GrpWall and UsageRaw decayed by the slurm.conf PriorityDecayHalfLife or
    PriorityUsageResetPeriod settings.  This allows a QOS to provide aggregate
    limits that, once consumed, will not be replenished automatically.  Such a
    QOS will act as a time-limited quota of resources for an association
    that has access to it.  Account/user usage will still be decayed for
    associations using the QOS.  The QOS GrpTRESMins and
    GrpWall limits can be increased or the QOS RawUsage value reset to 0
    (zero) to again allow jobs submitted with this QOS to be queued (if
    DenyOnLimit is set) or run (pending with QOSGrp{TRES}MinutesLimit
    or QOSGrpWallLimit reasons, where {TRES} is some type of trackable resource).
    .TP
    \fIUsageFactorSafe\fP
    If set, and \fIAccountingStorageEnforce\fR includes \fISafe\fR, jobs will only
    be able to run if the job can run to completion with the \fIUsageFactor\fR
    applied.
    .RE

    is reached no new jobs are started until other jobs finish to allow
    time to free up.

    .TP
    \fIGrpTRES\fP
    Maximum number of TRES running jobs are able to be allocated in aggregate for
    this QOS.

    .TP
    \fIGrpJobs\fP
    Maximum number of running jobs in aggregate for this QOS.

    .TP
    \fIGrpJobsAccrue\fP
    Maximum number of pending jobs in aggregate able to accrue age priority for this
    QOS.

    .TP
    \fIGrpSubmitJobs\fP
    Maximum number of jobs which can be in a pending or running state at any time
    in aggregate for this QOS.

    .TP
    \fIGrpWall\fP
    Maximum wall clock time running jobs are able to be allocated in aggregate for
    this QOS. If this limit is reached submission requests will be denied and the
    running jobs will be killed.
    .TP
    \fIID\fP
    The id of the QOS.

    .TP
    \fIMaxTRESMins\fP
    Maximum number of TRES minutes each job is able to use.

    .TP
    \fIMaxTRESPerAccount\fP
    Maximum number of TRES each account is able to use.

    .TP
    \fIMaxTRESPerJob\fP
    Maximum number of TRES each job is able to use.

    .TP
    \fIMaxTRESPerNode\fP
    Maximum number of TRES each node in a job allocation can use.

    .TP
    \fIMaxTRESPerUser\fP
    Maximum number of TRES each user is able to use.

    .TP
    \fIMaxJobsPerUser\fP
    Maximum number of jobs each user is allowed to run at one time.

    .TP
    \fIMinPrioThreshold\fP
    Minimum priority required to reserve resources when scheduling.

    .TP
    \fIMinTRESPerJob\fP
    Minimum number of TRES each job running under this QOS must request.
    Otherwise the job will pend until modified.

    .TP
    \fIMaxSubmitJobsPerAccount\fP
    Maximum number of jobs pending or running state at any time per account.

    .TP
    \fIMaxSubmitJobsPerUser\fP
    Maximum number of jobs pending or running state at any time per user.

    .TP
    \fIMaxWall\fP
    Maximum wall clock time each job is able to use.

    .TP
    \fIName\fP
    Name of the QOS.

    .TP
    \fIPreempt\fP
    Other QOS\' this QOS can preempt.

    .TP
    \fIPreemptMode\fP
    Mechanism used to preempt jobs of this QOS if the clusters \fIPreemptType\fP
    is configured to \fIpreempt/qos\fP.  The default preemption mechanism
    is specified by the cluster\-wide \fIPreemptMode\fP configuration parameter.
    Possible values are "Cluster" (meaning use cluster default), "Cancel",
    "Checkpoint" and "Requeue".  This option is not compatible with
    PreemptMode=OFF or PreemptMode=SUSPEND (i.e. preempted jobs must be removed
    from the resources).

    .TP
    \fIPreemptExemptTime\fP
    Specifies a minimum run time for jobs of this QOS before they are considered for
    preemption. This QOS option takes precedence over the global
    \fIPreemptExemptTime\fP. Setting to -1 disables the option, allowing another
    QOS or the global option to take effect. Setting to 0 indicates no minimum run
    time and supersedes the lower priority QOS (see \fIOverPartQOS\fP) and/or the
    global option in slurm.conf.

    .TP
    .TP
    \fIUsageThreshold\fP
    A float representing the lowest fairshare of an association allowable
    to run a job.  If an association falls below this threshold and has
    pending jobs or submits new jobs those jobs will be held until the
    usage goes back above the threshold.  Use \fIsshare\fP to see current
    shares on the system.

    .TP
    \fIWithDeleted\fP
    Display information with previously deleted data.

    .SH "LIST/SHOW QOS FORMAT OPTIONS"

    .TP
    \fIDescription\fP
    An arbitrary string describing a QOS.

    .TP
    \fIGraceTime\fP
    Preemption grace time to be extended to a job which has been
    selected for preemption in the format of hh:mm:ss.  The default
    value is zero, no preemption grace time is allowed on this partition.
    NOTE: This value is only meaningful for QOS PreemptMode=CANCEL.

    .TP
    \fIGrpTRESMins\fP
    The total number of TRES minutes that can possibly be used by past,
    present and future jobs running from this QOS.
    To clear a previously set value use the modify command with a new
    value of \-1 for each TRES id.
    NOTE: This limit only applies when using the Priority Multifactor
    plugin.  The time is decayed using the value of PriorityDecayHalfLife
    or PriorityUsageResetPeriod as set in the slurm.conf.  When this limit
    is reached all associated jobs running will be killed and all future jobs
    submitted with this QOS will be delayed until they are able to run
    inside the limit.

    .TP
    \fIGrpTRES\fP
    Maximum number of TRES running jobs are able to be allocated in aggregate for
    this QOS.
    To clear a previously set value use the modify command with a new
    value of \-1 for each TRES id.

    .TP
    \fIGrpJobs\fP
    Maximum number of running jobs in aggregate for this QOS.
    To clear a previously set value use the modify command with a new value of \-1.

    .TP
    this QOS.
    To clear a previously set value use the modify command with a new value of \-1.
    NOTE: This limit only applies when using the Priority Multifactor
    plugin.  The time is decayed using the value of PriorityDecayHalfLife
    or PriorityUsageResetPeriod as set in the slurm.conf.  When this limit
    is reached all associated jobs running will be killed and all future jobs
    submitted with this QOS will be delayed until they are able to run
    inside the limit.

    .TP
    \fIMaxTRESMins\fP
    Maximum number of TRES minutes each job is able to use.
    To clear a previously set value use the modify command with a new
    value of \-1 for each TRES id.

    .TP
    \fIMaxTRESPerAccount\fP
    Maximum number of TRES each account is able to use.
    To clear a previously set value use the modify command with a new value of \-1
    for each TRES id.

    .TP
    \fIMaxTRESPerJob\fP
    Maximum number of TRES each job is able to use.
    To clear a previously set value use the modify command with a new
    value of \-1 for each TRES id.

    .TP
    \fIMaxTRESPerNode\fP
    Maximum number of TRES each node in a job allocation can use.
    To clear a previously set value use the modify command with a new value of \-1
    for each TRES id.

    .TP
    \fIMaxTRESPerUser\fP
    Maximum number of TRES each user is able to use.
    To clear a previously set value use the modify command with a new value of \-1
    for each TRES id.

    .TP
    \fIMaxJobsPerAccount\fP
    Maximum number of jobs each account is allowed to run at one time.
    To clear a previously set value use the modify command with a new value of \-1.

    .TP
    \fIMaxJobsPerUser\fP
    Maximum number of jobs each user is allowed to run at one time.
    To clear a previously set value use the modify command with a new value of \-1.

    .TP
    \fIMaxSubmitJobsPerAccount\fP
    Maximum number of jobs pending or running state at any time per account.

    .TP
    \fIMinPrioThreshold\fP
    Minimum priority required to reserve resources when scheduling.
    To clear a previously set value use the modify command with a new value of \-1.

    .TP
    \fIMinTRES\fP
    Minimum number of TRES each job running under this QOS must request.
    Otherwise the job will pend until modified.
    To clear a previously set value use the modify command with a new
    value of \-1 for each TRES id.

    .TP
    \fIName\fP
    Name of the QOS.  Needed for creation.

    .TP
    \fIPreempt\fP
    Other QOS\' this QOS can preempt.
    Setting a Preempt to '' (two single
    quotes with nothing between them) restores its default setting.  You
    can also use the operator += and \-= to add or remove certain QOS's
    from a QOS list.

    .TP
    \fIPreemptMode\fP
    Mechanism used to preempt jobs of this QOS if the clusters \fIPreemptType\fP
    is configured to \fIpreempt/qos\fP.  The default preemption mechanism
    is specified by the cluster\-wide \fIPreemptMode\fP configuration parameter.
    Possible values are "Cluster" (meaning use cluster default), "Cancel",
    "Checkpoint" and "Requeue".  This option is not compatible with
    PreemptMode=OFF or PreemptMode=SUSPEND (i.e. preempted jobs must be removed
    from the resources).

    .TP
    \fIPriority\fP
    What priority will be added to a job\'s priority when using this QOS.
    To clear a previously set value use the modify command with a new value of \-1.

    .TP
    \fIUsageFactor\fP
    A float that is factored into a job’s TRES usage (e.g. RawUsage, TRESMins,
    TRESRunMins). For example, if the usagefactor was 2, for every TRESBillingUnit
    second a job ran it would count for 2. If the usagefactor was .5, every second
    would only count for half of the time. A setting of 0 would add no timed usage
    from the job.

    The usage factor only applies to the job's QOS and not the partition QOS.

    If the \fIUsageFactorSafe\fR flag \fIis\fR set and
    \fIAccountingStorageEnforce\fR includes \fISafe\fR, jobs will only be

    Default is 1. To clear a previously set value use the modify command with a new
    value of -1.

    .SH "SPECIFICATIONS FOR RESERVATIONS"

    .TP
    \fIClusters\fP=<comma separated list of cluster names>
    List the reservations of the cluster(s).  Default is the cluster where the
    command was run.

    .TP
    \fIEnd\fP=<OPT>
    Period ending of reservations. Default is now.

    Valid time formats are...
    .sp
    HH:MM[:SS] [AM|PM]
    .br
    MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
    .br
    MM/DD[/YY]\-HH:MM[:SS]
    .br
    YYYY\-MM\-DD[THH:MM[:SS]]

    .TP
    \fIID\fP=<OPT>
    Comma separated list of reservation ids.

    .TP
    \fINames\fP=<OPT>
    Comma separated list of reservation names.

    .TP
    \fINodes\fP=<comma separated list of node names>
    Node names where reservation ran.

    .TP
    \fIStart\fP=<OPT>
    Period start of reservations.  Default is 00:00:00 of current day.

    Valid time formats are...
    .sp
    HH:MM[:SS] [AM|PM]
    .br
    MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
    .br
    MM/DD[/YY]\-HH:MM[:SS]
    .br
    YYYY\-MM\-DD[THH:MM[:SS]]

    .SH "LIST/SHOW RESERVATION FORMAT OPTIONS"
    \fIFlags\fP
    Flags on the reservation.

    .TP
    \fIID\fP
    Reservation ID.

    .TP
    \fIName\fP
    Name of this reservation.

    .TP
    \fINodeNames\fP
    List of nodes in the reservation.

    .TP
    \fIStart\fP
    Start time of reservation.

    .TP
    \fITRES\fP
    List of TRES in the reservation.

    .TP
    \fIUnusedWall\fP
    Wall clock time in seconds unused by any job.

    .SH "SPECIFICATIONS FOR RESOURCE"

    \fIClusters\fP=<name list>
    Comma separated list of cluster names on which specified resources are to be
    available.  If no names are designated then the clusters already
    allowed to use this resource will be altered.

    .TP
    \fICount\fP=<OPT>
    Number of software resources of a specific name configured on the system being
    controlled by a resource manager.

    .TP
    \fIDescriptions=\fP
    A brief description of the resource.

    .TP
    \fIFlags\fP=<OPT>
    Flags that identify specific attributes of the system resource.  At this time
    no flags have been defined.

    .TP
    \fIServerType\fP=<OPT>
    The type of a software resource manager providing the licenses. For example
    \fIServer\fP=<OPT>
    The name of the server serving up the resource.  Default is 'slurmdb' indicating
    the licenses are being served by the database.

    .TP
    \fIType\fP=<OPT>
    The type of the resource represented by this record.  Currently the only valid
    type is License.

    .TP
    \fIWithClusters\fP
    Display the clusters percentage of resources.  If a resource hasn't
    been given to a cluster the resource will not be displayed with this flag.

    .P
    NOTE:  Resource is used to define each resource configured on a system
    available for usage by Slurm clusters.

    .SH "LIST/SHOW RESOURCE FORMAT OPTIONS"

    .TP
    \fICluster\fP
    Name of cluster resource is given to.

    .TP
    \fICount\fP
    The count of a specific resource configured on the system globally.

    .TP
    \fIAllocated\fP
    The percent of licenses allocated to a cluster.

    .TP
    \fIDescription\fP
    Description of the resource.

    .TP
    \fIServerType\fP
    The type of the server controlling the licenses.

    .TP
    \fIName\fP
    Name of this resource.

    .TP
    \fIServer\fP
    Server serving up the resource.

    .TP
    \fIType\fP
    Type of resource this record represents.

    .TP
    \fIPartition\fP
    Partition job ran on.

    .TP
    \fIState\fP
    Current State of the job in the database.

    .TP
    \fITimeStart\fP
    Time job started running.

    .TP
    \fITimeEnd\fP
    Current recorded time of the end of the job.

    .SH "SPECIFICATIONS FOR TRANSACTIONS"

    .TP
    \fIAccounts\fP=<comma separated list of account names>
    Only print out the transactions affecting specified accounts.

    .TP
    \fIAction\fP=<Specific action the list will display>

    .TP
    \fIActor\fP=<Specific name the list will display>
    Only display transactions done by a certain person.

    .TP
    \fIClusters\fP=<comma separated list of cluster names>
    Only print out the transactions affecting specified clusters.

    .TP
    \fIEnd\fP=<Date and time of last transaction to return>
    Return all transactions before this Date and time.  Default is now.

    .TP
    \fIStart\fP=<Date and time of first transaction to return>
    Return all transactions after this Date and time.  Default is epoch.

    Valid time formats for End and Start are...
    .sp
    HH:MM[:SS] [AM|PM]
    .br
    MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
    .br
    MM/DD[/YY]\-HH:MM[:SS]
    .br
    YYYY\-MM\-DD[THH:MM[:SS]]

    .TP
    \fIActor\fP

    .TP
    \fIInfo\fP

    .TP
    \fITimeStamp\fP

    .TP
    \fIWhere\fP
    .P
    NOTE: If using the WithAssoc option you can also view the information
    about the various associations the transaction affected.  The
    Association format fields are described
    in the \fILIST/SHOW ASSOCIATION FORMAT OPTIONS\fP section.

    .SH "SPECIFICATIONS FOR USERS"

    .TP
    \fIAccount\fP=<account>
    Account name to add this user to.

    .TP
    \fIAdminLevel\fP=<level>
    Admin level of user.  Valid levels are None, Operator, and Admin.

    .TP
    \fICluster\fP=<cluster>
    Specific cluster to add user to the account on.  Default is all in system.

    .TP
    \fIDefaultAccount\fP=<account>
    Identify the default bank account name to be used for a job if none is
    specified at submission time.

    .TP
    \fIDefaultWCKey\fP=<defaultwckey>
    Identify the default Workload Characterization Key.

    .TP
    \fIName\fP=<name>
    Name of user.

    .TP
    \fINewName\fP=<newname>
    Use to rename a user in the accounting database

    .TP
    \fIPartition\fP=<name>
    Partition name.

    .TP
    \fIWithCoord\fP
    Display all accounts a user is coordinator for.

    .TP
    \fIWithDeleted\fP
    Display information with previously deleted data.
    .P
    NOTE: If using the WithAssoc option you can also query against
    association specific information to view only certain associations
    this user may have.  These extra options can be found in the
    \fISPECIFICATIONS FOR ASSOCIATIONS\fP section.  You can also use the
    general specifications list above in the \fIGENERAL SPECIFICATIONS FOR
    ASSOCIATION BASED ENTITIES\fP section.

    .SH "LIST/SHOW USER FORMAT OPTIONS"

    .TP
    \fIAdminLevel\fP
    Admin level of user.

    .TP
    \fIDefaultAccount\fP
    The user's default account.

    .TP
    \fICoordinators\fP
    List of users that are a coordinator of the account. (Only filled in
    when using the WithCoordinator option.)

    .TP
    \fIUser\fP
    The name of a user.
    .P
    NOTE: If using the WithAssoc option you can also view the information
    about the various associations the user may have on all the
    clusters in the system. The association information can be filtered.
    Note that all the users in the database will always be shown as filter only
    takes effect over the association data. The Association format fields are
    described in the \fILIST/SHOW ASSOCIATION FORMAT OPTIONS\fP section.

    .SH "LIST/SHOW WCKey"

    .TP
    \fIWCKey\fP
    Workload Characterization Key.

    .TP
    \fICluster\fP
    Specific cluster for the WCKey.
    The name of the trackable resource.  This option is required for
    TRES types BB (Burst buffer), GRES, and License.  Types CPU, Energy,
    Memory, and Node do not have Names.  For example if GRES is the
    type then name is the denomination of the GRES itself e.g. GPU.

    .TP
    \fIID\fP
    The identification number of the trackable resource as it appears
    in the database.

    .TP
    \fIType\fP
    The type of the trackable resource. Current types are BB (Burst
    buffer), CPU, Energy, GRES, License, Memory, and Node.

    .SH "TRES information"
    Trackable RESources (TRES) are used in many QOS or Association limits.
    When setting the limits they are comma separated list.  Each TRES has
    a different limit, i.e. GrpTRESMins=cpu=10,mem=20 would make 2
    different limits 1 for 10 cpu minutes and 1 for 20 MB memory minutes.
    This is the case for each limit that deals with TRES.  To remove the
    limit \-1 is used i.e. GrpTRESMins=cpu=-1 would remove only the cpu
    TRES limit.

    NOTE: When dealing with Memory as a TRES all limits are in MB.

    NOTE: The Billing TRES is calculated from a partition's TRESBillingWeights. It
    is temporarily calculated during scheduling for each partition to enforce
    billing TRES limits. The final Billing TRES is calculated after the job has been
    allocated resources. The final number can be seen in \fBscontrol show jobs\fP
    and \fBsacct\fP output.

    .SH "GLOBAL FORMAT OPTION"
    When using the format option for listing various fields you can put a
    %NUMBER afterwards to specify how many characters should be printed.

    e.g. format=name%30 will print 30 characters of field name right
    justified.  A \-30 will print 30 characters left justified.

    .SH "FLAT FILE DUMP AND LOAD"
    sacctmgr has the capability to load and dump Slurm association data to and
    from a file.  This method can easily add a new cluster or copy an
    existing clusters associations into a new cluster with similar
    accounts. Each file contains Slurm association data for a single
    cluster.  Comments can be put into the file with the # character.
    Each line of information must begin with one of the four titles; \fBCluster, Parent, Account or
    User\fP. Following the title is a space, dash, space, entity value,
    then specifications. Specifications are colon separated.  If any
    variable such as Organization has a space in it, surround the name with
    single or double quotes.

    To create a file of associations one can run
    Cluster= \- specify a different name for the cluster than that which is
    in the file.

    Quick explanation how the file works.

    Since the associations in the system follow a hierarchy, so does the
    file.  Anything that is a parent needs to be defined before any
    children.  The only exception is the understood 'root' account.  This
    is always a default for any cluster and does not need to be defined.

    To edit/create a file start with a cluster line for the new cluster

    \fBCluster\ \-\ cluster_name:MaxNodesPerJob=15\fP

    Anything included on this line will be the defaults for all
    associations on this cluster.  These options are as follows...
    .TP
    \fIGrpTRESMins=\fP
    The total number of TRES minutes that can possibly be used by past,
    present and future jobs running from this association and its children.
    .TP
    \fIGrpTRESRunMins=\fP
    Used to limit the combined total number of TRES minutes used by all
    jobs running with this association and its children.  This takes into
    consideration time limit of running jobs and consumes it, if the limit
    is reached no new jobs are started until other jobs finish to allow
    time to free up.
    .TP
    \fIGrpTRES=\fP
    Maximum number of TRES running jobs are able to be
    allocated in aggregate for this association and all associations which
    are children of this association.
    .TP
    \fIGrpJobs=\fP
    Maximum number of running jobs in aggregate for this
    association and all associations which are children of this association.
    .TP
    \fIGrpJobsAccrue\fP
    Maximum number of pending jobs in aggregate able to accrue age priority for this
    association and all associations which are children of this association.
    .TP
    \fIGrpNodes=\fP
    Maximum number of nodes running jobs are able to be
    allocated in aggregate for this association and all associations which
    are children of this association.
    .TP
    \fIGrpSubmitJobs=\fP
    Maximum number of jobs which can be in a pending or
    running state at any time in aggregate for this association and all
    associations which are children of this association.
    .TP
    \fIGrpWall=\fP
    Maximum time (not related to job size) children of this accounts jobs can run.
    .TP
    \fIQOS=\fP
    Comma separated list of Quality of Service names (Defined in sacctmgr).
    .TP

    Followed by Accounts you want in this fashion...

    .na
    \fBParent\ \-\ root\fP (Defined by default)
    .br
    \fBAccount\ \-\ cs\fP:MaxNodesPerJob=5:MaxJobs=4:FairShare=399:MaxWallDurationPerJob=40:Description='Computer Science':Organization='LC'
    .br
    \fBParent\ \-\ cs\fP
    .br
    \fBAccount\ \-\ test\fP:MaxNodesPerJob=1:MaxJobs=1:FairShare=1:MaxWallDurationPerJob=1:Description='Test Account':Organization='Test'
    .ad

    .TP
    Any of the options after a ':' can be left out and they can be in any order.
    If you want to add any sub accounts just list the Parent THAT HAS ALREADY
    BEEN CREATED before the account line in this fashion...
    .TP
    All account options are
    .TP
    \fIDescription=\fP
    A brief description of the account.
    .TP
    \fIGrpTRESMins=\fP
    Maximum number of TRES hours running jobs are able to
    be allocated in aggregate for this association and all associations
    which are children of this association.
    \fIGrpTRESRunMins=\fP
    Used to limit the combined total number of TRES minutes used by all
    jobs running with this association and its children.  This takes into
    consideration time limit of running jobs and consumes it, if the limit
    is reached no new jobs are started until other jobs finish to allow
    time to free up.
    .TP
    \fIGrpTRES=\fP
    Maximum number of TRES running jobs are able to be
    allocated in aggregate for this association and all associations which
    are children of this association.
    .TP
    \fIGrpJobs=\fP
    Maximum number of running jobs in aggregate for this
    association and all associations which are children of this association.
    .TP
    \fIGrpJobsAccrue\fP
    Maximum number of pending jobs in aggregate able to accrue age priority for this
    association and all associations which are children of this association.
    .TP
    .TP
    \fIFairShare=\fP
    Number used in conjunction with other associations to determine job priority.
    .TP
    \fIMaxJobs=\fP
    Maximum number of jobs the children of this association can run.
    .TP
    \fIMaxNodesPerJob=\fP
    Maximum number of nodes per job the children of this association can run.
    .TP
    \fIMaxWallDurationPerJob=\fP
    Maximum time (not related to job size) children of this accounts jobs can run.
    .TP
    \fIOrganization=
    Name of organization that owns this account.
    .TP
    \fI\fIQOS(=,+=,\-=)\fP
    Comma separated list of Quality of Service names (Defined in sacctmgr).
    .TP

    .TP
    To add users to a account add a line like this after a Parent \- line
    \fBParent\ \-\ test\fP
    .br
    .na
    \fBUser\ \-\ adam\fP:MaxNodesPerJob=2:MaxJobs=3:FairShare=1:MaxWallDurationPerJob=1:AdminLevel=Operator:Coordinator='test'
    .ad

    .TP
    All user options are
    .TP
    \fIAdminLevel=\fP
    Type of admin this user is (Administrator, Operator)
    .br
    \fBMust be defined on the first occurrence of the user.\fP
    .TP
    \fICoordinator=\fP
    Comma separated list of accounts this user is coordinator over
    .br
    \fBMust be defined on the first occurrence of the user.\fP
    .TP
    \fIDefaultAccount=\fP
    system wide default account name
    .br
    \fBMust be defined on the first occurrence of the user.\fP
    .TP
    \fIFairShare=\fP
    Number used in conjunction with other associations to determine job priority.
    .TP
    \fIMaxJobs=\fP
    Maximum number of jobs this user can run.
    .TP
    completely understand what you are doing.  For slurmdbd options see
    "man slurmdbd" for more information.
    Loading data into the database can be done from these files to either
    view old data or regenerate rolled up data.

    .SS archive dump
    Dump accounting data to file. Depending on options and slurmdbd configuration
    data may remain in database or be purged. This operation cannot be rolled back
    once executed. If one of the following options is not specified when sacctmgr
    is called, the value configured in slurmdbd.comf is used.

    .TP
    \fIDirectory=\fP
    Directory to store the archive data.
    .TP
    \fIEvents\fP
    Archive Events.  If not specified and PurgeEventAfter is set
    all event data removed will be lost permanently.
    .TP
    \fIJobs\fP
    Archive Jobs.  If not specified and PurgeJobAfter is set
    all job data removed will be lost permanently.
    .TP
    \fIPurgeEventAfter=\fP
    Purge cluster event records older than time stated in months.  If you
    want to purge on a shorter time period you can include hours, or days
    behind the numeric value to get those more frequent purges. (e.g. a
    value of '12hours' would purge everything older than 12 hours.)
    .TP
    \fIPurgeJobAfter=\fP
    Purge job records older than time stated in months.  If you
    want to purge on a shorter time period you can include hours, or days
    behind the numeric value to get those more frequent purges. (e.g. a
    value of '12hours' would purge everything older than 12 hours.)
    .TP
    \fIPurgeStepAfter=\fP
    Purge step records older than time stated in months.  If you
    want to purge on a shorter time period you can include hours, or days
    behind the numeric value to get those more frequent purges. (e.g. a
    value of '12hours' would purge everything older than 12 hours.)
    .TP
    \fIPurgeSuspendAfter=\fP
    Purge job suspend records older than time stated in months.  If you
    want to purge on a shorter time period you can include hours, or days
    behind the numeric value to get those more frequent purges. (e.g. a
    value of '12hours' would purge everything older than 12 hours.)
    .TP
    \fIScript=\fP
    Run this script instead of the generic form of archive to flat files.
    .TP
    \fISteps\fP
    Archive Steps.  If not specified and PurgeStepAfter is set

    .TP
    \fIFile=\fP
    File to load into database.
    .TP
    \fIInsert=\fP
    SQL to insert directly into the database.  This should be used very
    cautiously since this is writing your sql into the database.

    .SH "ENVIRONMENT VARIABLES"
    .PP
    Some \fBsacctmgr\fR options may be set via environment variables. These
    environment variables, along with their corresponding options, are listed below.
    (Note: commandline options will always override these settings)
    .TP 20
    \fBSLURM_CONF\fR
    The location of the Slurm configuration file.

    .SH "EXAMPLES"
    \fBNOTE:\fR There is an order to set up accounting associations.
    You must define clusters before you add accounts and you must add accounts
    before you can add users.

    .eo
    .br
    -> sacctmgr create cluster tux
    .br
    -> sacctmgr create account name=science fairshare=50
    .br
    -> sacctmgr create account name=chemistry parent=science fairshare=30
    .br
    -> sacctmgr create account name=physics parent=science fairshare=20
    .br
    -> sacctmgr create user name=adam cluster=tux account=physics fairshare=10
    .br
    -> sacctmgr delete user name=adam cluster=tux account=physics
    .br
    -> sacctmgr delete account name=physics cluster=tux
    .br
    -> sacctmgr modify user where name=adam cluster=tux account=physics set
      maxjobs=2 maxwall=30:00
    .br
    -> sacctmgr add user brian account=chemistry
    .br
    -> sacctmgr list associations cluster=tux format=Account,Cluster,User,Fairshare tree withd
    .br
    -> sacctmgr list transactions StartTime=11/03\-10:30:00 format=Timestamp,Action,Actor
    .br
    -> sacctmgr dump cluster=tux file=tux_data_file
    .br
    -> sacctmgr load tux_data_file
    .br

    .br
    wrong-> sacctmgr modify user name=adam set fairshare=10 cluster=tux
    .br

    .br
    This will produce an error as the above line reads modify user adam
    set fairshare=10 and cluster=tux.
    .br

    .br
    right-> sacctmgr modify user name=adam cluster=tux set fairshare=10
    .br
    right-> sacctmgr modify user name=adam set fairshare=10 where cluster=tux
    .br

    .br
    When changing qos for something only use the '=' operator when wanting
    to explicitly set the qos to something.  In most cases you will want
    to use the '+=' or '\-=' operator to either add to or remove from the
    existing qos already in place.
    .br

    .br
    If a user already has qos of normal,standby for a parent or it was
    explicitly set you should use qos+=expedite to add this to the list in
    this fashion.
    .br

    If you are looking to only add the qos expedite to only a certain
    account and or cluster you can do that by specifying them in the
    sacctmgr line.

    .br
    -> sacctmgr modify user name=adam set qos+=expedite
    .br

    .br
    > sacctmgr modify user name=adam acct=this cluster=tux set qos+=expedite
    .br

    .br
    Let's give an example how to add QOS to user accounts.
    List all available QOSs in the cluster.
    .br

    .br
    ->sacctmgr show qos format=name
          Name
    .br
      ---------
    .br
      zebra       root         normal
    .br
      zebra       root         normal
    .br
      zebra          g         normal
    .br
      zebra         g1         normal
    .br

    .br
    Add the QOS expedite to account G1 and display the result.
    Using the operator += the QOS will be added together
    with the  existing QOS to this account.
    .br

    .br
    ->sacctmgr modify account name=g1 set qos+=expedite
    .br

    .br
    ->sacctmgr show assoc format=cluster,account,qos
    .br
       Cluster    Account   QOS
    .br
       --------  --------  -------
    .br
       zebra       root         normal
    .br
       zebra       root         normal
    .br
       zebra          g         normal
    .br
       zebra         g1  expedite,normal
    .br

    .br
    Now set the QOS expedite as the only QOS for the account G and display
    the result. Using the operator = that expedite is the only usable
    QOS by account G
    .br

    .br
    ->sacctmgr modify account name=G set qos=expedite
    .br

    .br
    >sacctmgr show assoc format=cluster,account,user,qos
    .br
       Cluster    Account  QOS
    .br
      ---------   -------- -----
    .br
    .br
    ->sacctmgr add account banana parent=G
    .br

    .br
    ->sacctmgr show assoc format=cluster,account,qos
    .br
       Cluster    Account   QOS
    .br
      ---------  --------  -----
    .br
      zebra       root       normal
    .br
      zebra       root       normal
    .br
      zebra          g       expedite
    .br
      zebra     banana       expedite
    .br
      zebra         g1       expedite,normal
    .br

    An example of listing trackable resources
    .br

    .br
    ->sacctmgr show tres
    .br
          Type              Name      ID
    .br
    ---------- ----------------- --------
    .br
           cpu                          1
    .br
           mem                          2
    .br
        energy                          3
    .br
          node                          4
    .br
       billing                          5
    .br
          gres         gpu:tesla     1001
    .br
       license               vcs     1002
    .br
            bb              cray     1003
    .br

    .ec

    .SH "COPYING"
    WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
    FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more
    details.

    .SH "SEE ALSO"
    \fBslurm.conf\fR(5),
    \fBslurmdbd\fR(8)