If you ever used a workload manager to schedule commercial software runs on a computing environment, you almost certainly faced the problem of software license competition: you really need to only let a job run when there are available licenses.
Many commercial job schedulers offer a “dynamic resource” mechanism to handle such cases and PBS Professional does as well. But, in addition, PBS Professional offers the ability to use a plug-in to handle corner cases and complex license requirements, not easily expressed in terms of resources.
A PBS Professional plug-in is a script associated to an event, and in the case of license scheduling we can leverage the “runjob” event: this happens when a job is found able to run on the available resources, just before it is effectively dispatched. “Runjob” is a good time to check for certain health status information on the nodes selected for running the job, to check for accounting information (has the user enough allocation units left?), and in our case, to check for license availability.
The flow would be:
1. Job is submitted asking for a certain amount of “scheduling resources” (ie. ncpus, memory) and holds also the necessary information to express the license needs (ie. license server pointer, needed amount of features) but license related resources are not used by the scheduler.
2. When scheduler finds there are enough scheduling resources available, job is dispatched.
3. Just before dispatch, the “runjob” event is evaluated: as the plug-in is able to look into job data, it reads the licensing information, connects to the license server and lets the job be dispatched only if enough licenses are available (taking care to also minimize race conditions, for instance throttling the amount of jobs dispatched per minute).
Of course this is not perfect (as the dynamic resource mechanism, anyway), but I found I could handle tricky commercial software license requests without placing load in the scheduling mechanism (for instance license related information is processed just when needed and not at each cycle) or using complex resource configurations.
A real world example of the above comes with the document describing how to integrate Schlumberger Simulators in a PBS Professional Computing Environment (Altair-PBS-Eclipse-Integraton-2012.pdf), which is available from http://www.pbsworks.com (Support -> Resource Library -> Technical Papers).