How to Modify User Account Associations in Slurm Cluster Without Recreating Users


13 views

When administering Slurm clusters, a common frustration arises when trying to modify user-account associations. The intuitive command:

sacctmgr modify user where name=example set account=groupb

often fails with errors because Slurm's design requires users to be explicitly associated with accounts through a separate relationship table.

Slurm treats user-account relationships as separate entities from user records themselves. This design allows for:

  • Multiple account associations per user
  • Different QOS settings per association
  • Flexible permission management

Here are three working approaches to change account associations:

Method 1: Delete and Recreate (Basic)

sacctmgr delete user name=example account=groupa
sacctmgr add user name=example account=groupb

Method 2: Direct Association Modification (Recommended)

sacctmgr modify user name=example set defaultaccount=groupb
sacctmgr modify association where user=example and account=groupa set account=groupb

Method 3: Batch Modification (For Multiple Users)

sacctmgr modify user set defaultaccount=groupb where account=groupa
sacctmgr modify association set account=groupb where account=groupa

Always verify changes with:

sacctmgr show user example
sacctmgr show association user=example

Error: "Association not found"
Solution: First ensure the target account exists:

sacctmgr show account groupb

Error: "Permission denied"
Solution: Run as Slurm administrator or use sudo:

sudo sacctmgr [command]

Warning: Pending jobs
Solution: Either wait for jobs to complete or force requeue:

scontrol requeue $(squeue -u example -h -o "%i")

For production environments, use transaction mode to prevent partial updates:

sacctmgr -i modify user name=example set defaultaccount=groupb
sacctmgr -i modify association where user=example set account=groupb

Many administrators face frustration when attempting to modify user associations in Slurm using the intuitive sacctmgr modify command. The typical approach:

sacctmgr modify user where name=example set account=groupb

Often fails with errors because Slurm requires additional context for user-account relationships. This isn't immediately obvious from the documentation.

Slurm treats user-account relationships as first-class entities called "associations." When you run:

sacctmgr show associations

You'll see output like:

Cluster   Account    User     Partition   Share GrpJobs GrpTRES GrpSubmit
-------   -------    ----     ---------   ----- ------- ------- ---------
cluster1  groupa     example  batch       1     0       cpu=0   0
cluster1  groupb     -        batch       1     0       cpu=0   0

Notice that user associations are distinct from account definitions. The hyphen (-) under User indicates a pure account definition.

To properly move a user between groups, you must specify both the source and target associations:

sacctmgr modify user where name=example and account=groupa set account=groupb

Key points about this command:

  • Explicitly identifies the current association (name=example AND account=groupa)
  • Modifies only that specific association
  • Preserves all other user attributes

After modification, verify with:

sacctmgr show user example

Should show output like:

User      Def Acct     Admin
----      --------     -----
example   groupb       None

And confirm with:

sacctmgr show associations user=example

For cases where users have multiple associations (rare but possible), you might need:

sacctmgr modify user where name=example set defaultaccount=groupb
sacctmgr modify association where user=example and account=groupa set account=groupb

This two-step approach ensures both the default account and specific associations are updated.

If you receive "Nothing modified" errors:

  1. Verify both source and target accounts exist
  2. Check for typos in user/account names
  3. Ensure you have sufficient privileges
  4. Try the verbose flag: sacctmgr -v modify ...

Remember that Slurm associations are hierarchical. Changes may propagate to child accounts depending on your configuration.