Research Sponsored Accounts
All files in
cs-xh-admin.cs.private.uwaterloo.ca:/software/accounts-master/data/sponsors/Research represent resources sponsored by
researchers. In general each researcher has one file named with their userid.
All resources which researchers request to support their research should be
allocated via their appropriate file.
In some cases there are files representing groups of collaborating researchers (eg: PLG, Watform)
These notes relate to adding users to groups and servers using the Xhiered Accounts packages. This controls access to "xhiered" machines as well as those using the CS Active Directory for accounts management.
Required authorization
- Users who need to update accounts need to be in group "sponsor" and group "accounts" on cs-xh-admin.cs.private.uwaterloo.ca
- That is, in those groups in the CS-GENERAL AD
- Users who need to manipulate accounts, for example set passwords, in the cs-general region to be in the "accounts" group on cs-general.cs.private. Similarly for any other region.
Creating Truly New Users (Userids)
Before you can use the
sponsors data software
to create new accounts
for a userid, that userid must be present
in the file
/software/accounts-userids/data/Userids on the machine
cs-xh-admin.cs.private.uwaterloo.ca.
Often, but not always, that happens automatically.
If the
userinfo command gives results
for the userid, then the userid is appropriately present
in this file, and you can proceed.
(And use the
Id Number: indicated by the output as the id number to associate with the userid). If
userinfo gives no output for the particular user, more work is needed to get the userid into the file.
The
Id Number: will be a student id, employee id, HR id, employee id, or
WatIAM "P-number", in that order. In fact, you will see other obscure values
listed too. But only the one shown can be used. Users with a lengthy
history at UW sometimes have several other values shown as the
Other Ids:
value; you cannot use any of those values.
In most cases user names and identification numbers already exist,
precreated in
WatIAM. In other limited cases, the correct thing to do
is create a userid which really should, for various reasons, not be put
in WatIAM. And occasionally you need to do things to cause the WatIAM
account creation.
WatIAM account creation (precreation) is discussed at this
link
which requires appropriate authentication.
After the
WatIAM account has been created, you normally need to
wait until the next business day for the new
userid to be available for
sponsors data. Various technical personnel, including
Adrian Pepper,
can accelerate that process, however.
If you need to do that yourself, perhaps see
Discussion of the Userids File for Experts.
And bear in mind
Information about Userids in the Sponsors Data.
Creating Truly New Groups (gids)
If you are creating a new group to appear in the Active Directory (eg: users_researchgroup), you will need to follow the instructions in
CreatingGroupAccounts
Location of Accounts Sponsor files
The accounts files for CS are on the UNIX computer
cs-xh-admin.cs.private.uwaterloo.ca
cd /software/accounts-master/data/sponsors
| Directory/File Name | Purpose |
|---|
| Research/
|
|---|
Research/ | files under Research correspond to an individual researcher or research group.
Here resources,
or extra resources, for users specifically
affiliated with particular researchers can be sponsored.
These might be extra disk space (or theoretically printing) for a grad student, or entire accounts for associated or visiting researchers. |
Files here are organized by individual faculty name or by group. The logic used is somewhat arbitrary, but here is the basic idea:
Single - for billing a single person
Group - for group billing, where several faculty are responsible for a group of resources
Finding the correct sponsor file to update
Let's say you want to add user
uuuu to group
gggg for a professor, who is the supervisor (sponsor) for this student/visitor
It's a good idea to look where this user or group has been added previously, so try:
grep uuuu */*
grep gggg */*
If you see an obvious file to update, then do so, otherwise you will probably update the file with name corresponding to the professor's userid (eg: "mterry" for Michael Terry).
Note that the exact file chosen doesn't affect what resources are allocated - it is the how it is put into the sponsor's file that will matter (see below). But the
file chosen will affect the apparent reason for (sponsor of) the resources.
Run sponsor_resources before making changes
While it is not strictly necessary, it seems a very wise practice
to run the
sponsor_resources
command
before you begin to make any changes to the sponsors data.
You should ensure that
sponsor_resources is running to completion (that is, not encountering a fatal error)
and note whether there are any reported errors.
This will help you later distinguish errors you might introduce from
the output of this initial run.
If
sponsor_resources is failing on fatal errors, and you cannot see how to fix them, you will need to track down someone who can.
Similarly, this precautionary preliminary run of
sponsor_resources might help you detect the fact someone else is in the process
of making changes at the same time.
The following page gives details of the
sponsor_resources command.
Use RCS co command to prepare file for editing
All files in the sponsor data are now maintained using the
RCS set of utilities.
Before using your favourite appropriate text editor to make changes to
the file, you must do the following
cs-xh-admin% rcsdiff _filename_
cs-xh-admin% co -l _filename_
The
rcsdiff is not strictly-speaking necessary, but it is a good habit
to get into.
It should return no output, meaning that you will not obliterate anyone
else's changes.
Somewhat similarly, if the
co command fails, it may be because another
person already has the file locked; you may need to talk to them to
resolve that.
Note that before you do the above, the file
filename should not be
writeable by you; this process modifies the file so you are able to
make changes working as your own personal userid (without using super-user).
The file might or might not be owned by your personal userid, depending upon who last did make a change to the file.
adding the user to the sponsor file
The sponsors files are divided into sections, delimited by 8 equal-signs, ie: "========"
Following that is typically a "Class:" entry which then lists a billing or sponsorship code. These will appear, for instance, in the inventory under the "Sponsorship Code" drop box. The convention that has been used is that codes ending with "001" are for personal account entries - ie: when adding account for the sponsor themself. Codes ending in "002" are for research accounts where the Faculty member is sponsoring someone else, eg: a grad student or visitor. Most additions will go in the "002" Class.
Use RCS ci command to unlock the file so others can edit it
At this point you must use
the
RCS utility
ci to
save a record of your changes, and
unlock the file so others can make
changes, as follows.
cs-xh-admin% ci -u _filename_
You will be prompted for a description of your changes. Make it
reasonably brief, but meaningingful. A reference to an ST number
is often appropriate.
More information about editing sponsors files
For a tutorial on editing sponsors files see
SponsorsDataEditingTutorial.
Cautionary Notes About Editing the Sponsors Data
In general you should avoid making
easy-to-make mistakes.
Nearly all files found in the directory
/software/accounts-master/data/sponsors/
and all directories beneath it, will be processed by the
sponsor_resources command.
Exceptions are files whose name begins with
. (dot/period)
and files in sub-directories which sub-directories are named
RCS.
If a subdirectory name begins with
. (dot/period)
then everything beneath it will be ignored
(unless referred to explicitly by other files).
The upshot of that is that you cannot place arbitrary files in these
directories, or
sponsor_resources will stop working correctly.
When editing the
sponsors data files,
it can often be convenient to copy and modify previously existing lines
to create your new additions.
If you do that, make sure you correctly change all relevant
SponsorshipEnds
dates,
or remove them as appropriate.
Also make sure you delete or change any comments which are
irrelevant in the new context. It's better to leave no comments
than leave confusing comments.
Consider that the files are intended to help later readers understand
the sponsorship situation; use a few comment lines (
"#"), and,
in general, put a blank line before each
==== line.
Note that id numbers need to be associated with userids. This can
be done directly in the
AssignTo line as in
AssignTo: sgamgee:02020202
Or it can be done in the
Userids: section at the top of the file. Only one specific id number can
be used for a particular userid. See
IncludeSponsorsDataIdNumbers for details.
To help make sure we don't slip up and put a real
userid:studentid
in one of these pages, we by convention omit
:studentid in all
examples.
Finally, make sure you remember to use the
ci -u command so you leave the file
editable by others.
Run the sponsor_resources program to process the changes
Note: everyone who runs the
sponsor_resources command
should be aware that it is really just a portion of the
accounts-master command.
If only one procedure needed to be chosen, the correct thing to do is
run
accounts-master, not
sponsor_resources.
accounts-master does a superset of what
sponsor_resources does.
However, after editing sponsors data,
sponsor_resources
alone is usually sufficient.
sponsor_resources takes the data under
/software/accounts-master/data/sponsors and produces per-user requirements
in per-machine (actually per-region) files under
/software/accounts-master/data/resources.
Along the way, it might detect errors in the changes you made.
Fix any problems that are reported and keep rerunning
sponsor_resources
until all your errors have gone away.
In particular, never leave the sponsors data in a state where
sponsor_resources does not produce the three lines which begin with
FYI:.
(The first being
FYI: ... computings ...). The
error, warnings, notes line can actually have non-zero errors if you can assert they are not
a result of your changes. (Corollary; run
sponsor_resouces before you even start making changes so you can be sure what noise you did not cause).
Typically you can simply use the command
sponsor_resources with no arguments. However, it is often prudent to redirect standard output and error output.
Here is an example of a "bad run":
@cs-xh-admin[140]% sponsor_resources
Error: /software/accounts-master/data/sponsors/Research/Terry line 63: Userid 'bjlafren' is not a standard userid
FYI: 137970(29794) computings, 1292(1274) printers, 530(403) aliases, 0(0) ppps
FYI: 1 error, 0 warnings, (0 notes)
FYI: expired sponsorship entries: 17888 computings, 51 printers, 0 aliases, 0 ppps
handle_group (group_id=1): Found 2 group description lines, expected <= 1
handle_group (group_id=11): Found 2 group description lines, expected <= 1
found 157201 userinfos, 14368 distinct
administration = cscf
psql::2: NOTICE: truncate cascades to table "sponsor_billcode"
psql::2: NOTICE: truncate cascades to table "sponsor_class"
psql::2: NOTICE: truncate cascades to table "sponsor_member"
psql::2: NOTICE: truncate cascades to table "sponsor_computing"
psql::2: NOTICE: truncate cascades to table "sponsor_printing"
psql::2: NOTICE: truncate cascades to table "sponsor_computing_group"
psql::2: NOTICE: truncate cascades to table "sponsor_mailalias"
TRUNCATE TABLE
@cs-xh-admin[141]%
In this case,
sponsor_resources complained because we used the short version
of the userid ("bjlafren"), but it always requires the long version
(ie: "bjlafreniere").
The following is an example of a "good run":
@cs-xh-admin[143]% sponsor_resources
FYI: 137973(29797) computings, 1292(1274) printers, 530(403) aliases, 0(0) ppps
FYI: 0 errors, 0 warnings, (0 notes)
FYI: expired sponsorship entries: 17888 computings, 51 printers, 0 aliases, 0 ppps
handle_group (group_id=1): Found 2 group description lines, expected <= 1
handle_group (group_id=11): Found 2 group description lines, expected <= 1
found 157201 userinfos, 14368 distinct
administration = cscf
psql::2: NOTICE: truncate cascades to table "sponsor_billcode"
psql::2: NOTICE: truncate cascades to table "sponsor_class"
psql::2: NOTICE: truncate cascades to table "sponsor_member"
psql::2: NOTICE: truncate cascades to table "sponsor_computing"
psql::2: NOTICE: truncate cascades to table "sponsor_printing"
psql::2: NOTICE: truncate cascades to table "sponsor_computing_group"
psql::2: NOTICE: truncate cascades to table "sponsor_mailalias"
TRUNCATE TABLE
@cs-xh-admin[144]%
Above we have been assuming a small, non-fatal error directly related to the change you made and which you were able to correct.
In extreme cases ("an extremely bad run?") you might end up with something like...
@cs-xh-admin[145]% sponsor_resources
Fatal error: "/software/accounts-master/data/sponsors/Research/yuying" line 89: not SponsorshipStarts, SponsorshipEnds, Quotas, Groups, or AssignTo: Computing: serverus.cs
@cs-xh-admin[146]%
The
Fatal error diagnostic, corroborated by the lack of
FYI: ... computings, ...
indicate that the sponsors data is severely broken to the extent that
the data under
/software/accounts-master/data/resources will not have been
updated at all.
Intended changes will not have taken effect, and, unless someone fixes the
problem, future changes will not take effect either. (However, the
resource data will remain in the state left by the last
successful_sponsor_resources_ ).
Corollary: run
sponsor_resources
before you begin making changes and make sure the sponsors data was not already
in a state like this; if it is, try to track down the most recent change,
fix it if it is easy and obvious and in any case alert the person who
left the broken state (you may in any case need their help to make some
decisions related to the fix).
The diagnostics indicate which file you should look at. Frequently problems occur because of missing
==== lines. But also, having two such lines in a row, even if separated by comments and blank lines, is a guaranteed fatal error, although there's no real reason why it should be. (That is to say,
it ought to have been possible at one time to modify the grammar to expect the
==== token and recognize (and mostly ignore) an empty section,
but that was never done, and so
==== must always be followed
by one of the appropriate keywords as listed in the diagnostic.
Unavoidable extraneous noise
It would be nice if we could tell you to always keep modifying until you have
0 errors
but because of details how some automatic data is produced from IST information,
that is not always possible. Some things have reached the point where they are
actually unresolvable by us. This results in errors which must be ignored, even
though you must continue to look for your own errors, and especially for
the absence of
Fatal error:.
Consequently noise like the following is sometimes unavoidable,
and while things may need to be done to work around the indicated problems,
the majority of resource data will reflect the most recent changes made.
@cs-xh-admin[145]% sponsor_resources
Error: "/software/accounts-master/data/sponsors/REGISTRAR/cs" line 845: Userid 'ktfrog' should have id number 'hr999007' not '74123456'
Error: "/software/accounts-master/data/sponsors/REGISTRAR/cs" line 868: Userid 'ktfrog' should have id number 'hr999007' not '74123456'
FYI: 137974(29801) computings, 1294(1276) printers, 530(403) aliases, 0(0) ppps
FYI: 2 errors, 0 warnings, (0 notes)
FYI: expired sponsorship entries: 17888 computings, 51 printers, 0 aliases, 0 ppps
handle_group (group_id=1): Found 2 group description lines, expected <= 1
handle_group (group_id=11): Found 2 group description lines, expected <= 1
found 157201 userinfos, 14368 distinct
administration = cscf
psql::2: NOTICE: truncate cascades to table "sponsor_billcode"
psql::2: NOTICE: truncate cascades to table "sponsor_class"
psql::2: NOTICE: truncate cascades to table "sponsor_member"
psql::2: NOTICE: truncate cascades to table "sponsor_computing"
psql::2: NOTICE: truncate cascades to table "sponsor_printing"
psql::2: NOTICE: truncate cascades to table "sponsor_computing_group"
psql::2: NOTICE: truncate cascades to table "sponsor_mailalias"
TRUNCATE TABLE
@cs-xh-admin[146]%
Path to the "sponsor_resources" command
If you don't happen to have the maintenance commands in your path, the path to the sponsor_resources command is:
/software/accounts-master/maintenance/sponsor_resources
Run the userinfo program to verify your changes
Run the
userinfo command
before and after you make your changes
to verify that its output reflects your intended changes.
More details are
here.
Run the accounts-client program
accounts-client {hostname, eg:cs-general.cs.private} >& ~/hostname-date &
This will cause the desired changes to actually happen on the
appropriate machine (
hostname; what is described as "Computing:"
in the sponsor file, which may in turn affect a
region of machines).
/etc/passwd and
/etc/group file will be updated if necessary,
as will system quota files, and home directories will be created
for any newly-created users.
The diagnostic output from the job will be written to the given
filename in your home directory.
eg:
accounts-client softbase.cs >& ~/softbase-20100208 &
If your changes will cause changes on multiple
regions,
you will need to run the command for each.
eg:
accounts-client student.cs >& ~/student-20100208 &
accounts-client cs-general.cs.private >& ~/cs-general-20100208 &
If you run
accounts-client with no name, all known regions
are updated.
eg:
accounts-client >& ~/ac-all-20100208 &
That can take a long time to finish.
Path to the "accounts-client" command
If you don't happen to have the maintenance commands in your path, the path to the accounts-client command is:
/software/accounts-master/maintenance/accounts-client
Updating Directory Services Domains
Once the
sponsor_resources process has run and files under
/software/accounts-master/data/resources/computing/have been updated, CSCF Directory Services domains are subsequently updated by cron job. User and group information are revised based upon the information stored in specific files found in the
resources/computing subdirectory. At time of writing (Feb 2017), the Directory Services updates are run following the schedule below.
- CS-GENERAL domain: 20 minutes past the hour
- Using the cs-general.cs.private and serverus.cs files.
- CS-TEACHING domain: 50 minutes past the hour
- Using the cs-teaching.cs.private and canadenis.student.cs files.
To force a domain update to run immediately, run the following commands on the xhier admin machine (
cs-xh-admin) as
root user:
- For CS-GENERAL domain:# /software/local_cs-xh-admin.cs.private.uwaterloo.ca/servers/cscf_ad_domain_update cs.uwaterloo.ca
- For CS-TEACHING domain:# /software/local_cs-xh-admin.cs.private.uwaterloo.ca/servers/cscf_ad_domain_update student.cs.uwaterloo.ca
Documentation
The following traditional UNIX man page documentation
describes the sponsors data base in a technical fashion.
As you become familiar with how things work in general,
you might find this documentation good for checking specific
details.
This stopped working a while ago. For now see SponsorsDataAccountsDocumentationBig and search for the command in question. When I get time I'll change the following references to be simply that.
CF Web>Administration>Accounts>ResearchSponsoredAccounts (2018-02-15, AdrianPepper)
EditAttach
Copyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.