libslurm38-22.05.11-150300.7.9.1<>,b(e p9|V=z^—DE-/^ o"ieX\:$l)v #E$C:<?:,d  # 7 )AGPX \ ` h  P(8(9X:a>6@6F6G6H6I6X7Y7\7H]7P^7rb7~c8'd8e8f8l8u8v8w9dx9ly9tz9999:(Clibslurm3822.05.11150300.7.9.1Libraries for SLURMThis package contains the library needed to run programs dynamically linked with SLURM.e s390zl38 SUSE Linux Enterprise 15SUSE LLC SUSE-GPL-2.0-with-openssl-exceptionhttps://www.suse.com/System/Librarieshttps://www.schedmd.comlinuxs390x e @e ~5450b23bed76a269d5baddb895c52afd14564da066619c05826d353075d0996elibslurm.so.38.0.0rootrootrootrootslurm_22_05-22.05.11-150300.7.9.1.src.rpmlibslurmlibslurm.so.38()(64bit)libslurm38libslurm38(s390-64)@@@@@@@@@@@@@@@@@@@@@    /sbin/ldconfig/sbin/ldconfiglibc.so.6()(64bit)libc.so.6(GLIBC_2.10)(64bit)libc.so.6(GLIBC_2.2)(64bit)libc.so.6(GLIBC_2.2.4)(64bit)libc.so.6(GLIBC_2.27)(64bit)libc.so.6(GLIBC_2.3)(64bit)libc.so.6(GLIBC_2.3.2)(64bit)libc.so.6(GLIBC_2.3.4)(64bit)libc.so.6(GLIBC_2.4)(64bit)libc.so.6(GLIBC_2.7)(64bit)libc.so.6(GLIBC_2.9)(64bit)libdl.so.2()(64bit)libdl.so.2(GLIBC_2.2)(64bit)libm.so.6()(64bit)libm.so.6(GLIBC_2.2)(64bit)libpthread.so.0()(64bit)libpthread.so.0(GLIBC_2.2)(64bit)libpthread.so.0(GLIBC_2.3.2)(64bit)libresolv.so.2()(64bit)libresolv.so.2(GLIBC_2.2)(64bit)libresolv.so.2(GLIBC_2.9)(64bit)rpmlib(CompressedFileNames)rpmlib(FileDigests)rpmlib(PayloadFilesHavePrefix)rpmlib(PayloadIsXz)slurm_22_05-config3.0.4-14.6.0-14.0-15.2-1libslurmslurm_22_05-configslurm_22_05-config22.0522.05.994.14.3eL@ehy@ee@e[J@e'd c>@cRcIO@c.@c)@bb?b@ba@b@b@bzSbu boa@aa{@amaa]aSaD@a6a0``f@``Q@`4@`@_I@_I@_T___ _F@__c^א^Ǿ^^}^d@^_@^Y^W@^W@^V]^M#@^E:@^B@^:@^.^*@^*@^)^[^@^ @] @]@]]B]]](]@]x]W@]+@](v@]'$]#0@]#0@] @] @] @\v{\T4\R@\R@\I\U@\[@[@[@[@[[[.[r@[b@[`O@[`O@[\Z[E@[{@Z؄ZЛZJ@ZJ@ZZ@ZZZo Zo Z_:Z!D@ZY@Yp@Y˒Y@Ycl@Y-^Y]XDX@XBXXXk@XAb@X>@X43@Xx@X@X@V5UL@TVTVeich@suse.comeich@suse.comeich@suse.comeich@suse.comeich@suse.comeich@suse.comeich@suse.comeich@suse.comcgoll@suse.comeich@suse.comeich@suse.comeich@suse.combwiedemann@suse.comeich@suse.comcgoll@suse.comeich@suse.comcgoll@suse.comeich@suse.comcgoll@suse.comcgoll@suse.comcgoll@suse.comcgoll@suse.comeich@suse.comcgoll@suse.comcgoll@suse.comcgoll@suse.comcgoll@suse.comeich@suse.comcgoll@suse.comcgoll@suse.comcgoll@suse.comeich@suse.comcgoll@suse.comcgoll@suse.comaguerrero@suse.comeich@suse.comeich@suse.comaguerrero@suse.comaguerrero@suse.comeich@suse.comaguerrero@suse.comfbui@suse.comaguerrero@suse.comeich@suse.comeich@suse.comcgoll@suse.comeich@suse.comcgoll@suse.comeich@suse.comcgoll@suse.comeich@suse.comkasimir_@outlook.decgoll@suse.comeich@suse.comeich@suse.comeich@suse.comcgoll@suse.comcgoll@suse.comeich@suse.comeich@suse.comdimstar@opensuse.orgcgoll@suse.comeich@suse.comeich@suse.comeich@suse.comeich@suse.comcgoll@suse.comeich@suse.comeich@suse.comeich@suse.comeich@suse.comeich@suse.comeich@suse.comeich@suse.comeich@suse.comeich@suse.comcgoll@suse.comcgoll@suse.comeich@suse.comeich@suse.comeich@suse.comeich@suse.comeich@suse.comcgoll@suse.comeich@suse.comeich@suse.comeich@suse.comcgoll@suse.comcgoll@suse.comcgoll@suse.comeich@suse.comeich@suse.comeich@suse.comeich@suse.comeich@suse.comcgoll@suse.comeich@suse.comeich@suse.comeich@suse.comeich@suse.comeich@suse.comeich@suse.comeich@suse.comcgoll@suse.comeich@suse.comeich@suse.comeich@suse.comeich@suse.comeich@suse.comcgoll@suse.comcgoll@suse.comeich@suse.comcgoll@suse.comjjolly@suse.comcgoll@suse.comcgoll@suse.comeich@suse.comeich@suse.comjengelh@inai.dejjolly@suse.comjjolly@suse.comjjolly@suse.comeich@suse.comeich@suse.comeich@suse.comjengelh@inai.deeich@suse.comeich@suse.comeich@suse.comeich@suse.comeich@suse.comeich@suse.comeich@suse.comeich@suse.comeich@suse.comeich@suse.comscorot@free.frscorot@free.frscorot@free.frscorot@free.fr- Update to 22.05.11 to fix (CVE-2023-49933 - bsc#1218046, CVE-2023-49936 - bsc#1218050, CVE-2023-49937 - bsc#1218051, CVE-2023-49938 - bsc#1218053) * Security Fixes: + Prevent message extension attacks that could bypass the message hash. CVE-2023-49933. + Prevent NULL pointer dereference on `size_valp` overflow. CVE-2023-49936. + Prevent double-xfree() on error in `_unpack_node_reg_resp()`. CVE-2023-49937. + Prevent modified `sbcast` RPCs from opening a file with the wrong group permissions. CVE-2023-49938.- Add missing service file for slurmrestd (boo#1217711).- Explicitly create an Obsoletes: entry for each package version that is obsoleted by the present version. These are all published versions of the last two major releases as well as all minor versions of the present release lower than the current one (bsc#1216869 2nd part). This prevents the current version to upgrade a old Slurm version for which no upgrade path exists.- On SLE-12 exclude build for s390x. Mon Nov 6 07:38:00 UTC 2023 - Egbert Eich - Add missing dependencies to slurm-config to plugins package. These should help to tie down the slurm version and help to avoid a package mix (bsc#1216869).- Update to 22.05.10 with fix for (CVE-2023-41914, bsc#1216207) Fix filesystem handling race conditions that could lead to an attacker taking control of an arbitrary file, or removing entire directories' contents. - Removed: Fix-test-32.8.patch Fix-test-38.11.patch Fix-test-3.13.patch - Other Fixes: * Changes in Slurm 22.05.9 + Allocate correct number of sockets when requesting gres and running with `CR_SOCKET*`. + Fix handling of `--prefer` for job arrays. + Fix regression in 22.05.5 that causes some jobs that request `--ntasks-per-node` to be incorrectly rejected. + Fix slurmctld crash when a step requests fewer tasks than nodes. + Fix incorrect task count in steps that request `--ntasks-per-node` and a node count with a range (e.g. -N1-2). + Fix some valid step requests hanging instead of running. + slurmrestd - avoid possible race condition which would cause slurmrestd to silently no longer accept new client connections. + Fix GPU setup on CRAY systems when using the `CRAY_CUDA_MPS` environment variable. GPUs are now correctly detected in such scenarios. + Fix the job prolog not running for jobs with the interactive step (salloc jobs with `LaunchParameters=use_interactive_step` set in `slurm.conf`) that were scheduled on powered down nodes. The prolog not running also broke `job_container/tmpfs`, `pam_slurm_adopt`, and `x11` forwarding. + `task/affinity` - fix slurmd segfault when request launch task requests of type `--cpu-bind=[map,mask]_cpu:` have no `` provided. + `sched/backfill` - fix segfault when removing a `PLANNED` node from system. + `sched/backfill` - fix deleted planned node staying in planned node bitmap. + Fix nodes remaining as PLANNED after slurmctld save state recovery. + Fix regression in 22.05.0rc1 that broke `Nodes=ALL` in a NodeSet. + Fix incorrect memory constraint when receiving a job from 20.11 that uses cpu count for memory calculation. + `openapi/v0.0.[36-38]` - avoid possible crash from jobs submitted with argv. + `openapi/v0.0.[36-38]` - avoid possible crash from rejected jobs submitted with batch_features. + `srun` - fix regression in 22.05.7 that prevented slurmctld from connecting to an srun running outside of a compute node. * Changes in Slurm 22.05.8 + Fix potential deadlock at `slurmctld` startup when job has invalid qos. + Avoid unnecessary call to `clusteracct_storage_g_cluster_tres()` when pinging dynamic nodes. This avoids significant slowdowns for `slurmctld` when `ping_nodes()` calls all nodes to re-register. + `openapi/dbv0.0.3[6-8]` - fix segfault that could arrise from Slurm database connnection failing. + Fix regression introduced in 22.05.0rc1 when updating a `NodeName=` with `NodeAddr` and/or `NodeHostname` if the specified nodelist wasn't sorted. + `openapi/v0.0.38` - change type of nice field for job submissions to integer from string. + `openapi/v0.0.38` - add oversubscribe option to job submission properties. + `openapi/v0.0.38` - fix incorrect data used to populate "time/start" field for jobs. + `openapi/dbv0.0.38` - avoid dumping failure if preempt qos with level 0 is provided by `slurmdbd` query. + Avoid an fd leak when lib dir for sbcast fails to be created. + `common/slurmdbd_pack` - fix env and script hash when unpacking a 21.08 `dbd_job_start_msg_t`. + Fix a race between `job_container/tmpfs`, `cncu`, `x11` setup and adoption of pids in cray with `pam_slurm_adopt`. * Changes in Slurm 22.05.7 + Fix slurmctlds in a federation not recovering a lost connection until a slurmctld restart or a federation update. + Fix incorrectly rejecting jobs that request GRES on multiple sockets and `--gres-flags=enforce-binding`. + Fix job core selection with `--gres-flags=enforce-binding`: this fixes cases where enforce-binding was not respected. + openapi/v0.0.38 - set assoc_id resolution failure as a debug log instead of an error log. + Fix issues when running in FrontEnd mode. + Fix srun and salloc I/O code to set `keepaliveinterval`/`keepalivetime` on connections. + Fix overwrite race of `cont_id` in `cgroup/v2` when using `task/cgroup` with `proctrack/cray_aries`. + Fix node remaining allocated after a reconfig with a completing job that has an `EpilogSlurmctld` instance still running. + `openapi/dbv0.0.38` - fix a cast to a wrong type + `openapi/dbv0.0.38` - correct issues where modifying association fields were ignored. + `openapi/dbv0.0.38` - correct issue where 16 bit integers were not able to be unset. + `openapi/dbv0.0.38` - correct issue where 16 & 32 bit integers were not able to be unset. + Fix `scontrol show hostlist` to read off a pipe correctly. + Fix `scontrol reconfigure` with configless enabled to send updated configs to 21.08 nodes correctly. (22.05 nodes are unaffected.) + Fix assertion and locks for dynamic node registrations with rpc queueing. + `rest_auth/local` - avoid logging root access as an error but instead as a normal info log. + `slurmrestd` - switch users earlier on startup to avoid sockets being made as root. + `slurmrestd` - disable umask during socket creation as the authentication is handled by Slurm instead of file ACLs. + Fix `sbcast`, `srun --bcast`, sstat not working with cloud/dynamic nodes. + `sacctmgr` - Allow removal of user's default account when `AllowNoDefAcct=yes`. + Fix segfault caused by race condition when a reconfiguration is requested while slurmd initializes. + `openapi/v0.0.38` - avoid divison by zero causing slurmrestd to crash during diag queries. + Update default values for text/blob columns in the database if upgrading to MariaDB >= 10.2.1 from a pre-10.2.1 version or from MySQL to avoid `sacctmgr` failures. + Fix several minor memory leaks in the cgroup/v2 plugin. + Fix possible issue when enabling controllers with long names in cgroup/v2. + Fix potential segfault in slurmd when using `acct_gather_energy/ipmi`. + Fix controller being able to contact non-addressable step hosts. + Allow setting `SuspendTime` to -1 when used on a partition. + `openapi/v0.0.38` - fix `maximum_memory_per_node` description typo. + `openapi/v0.0.38` - add `maximum_memory_per_cpu` to `GET /partitions`. + `openapi/v0.0.38` - add `default_memory_per_node` to `GET /partitions`. + Fix dynamic node losing dynamic node status when changing state to `FUTURE`. + Fix adoption into most recent job in `pam_slurm_adopt` and `cgroup/v2`. + `gres/shard` - Fix `plugin_name` definition to print correctly in debug lines. * Changes in Slurm 22.05.6 + Fix a partition's DisableRootJobs=no from preventing root jobs from working. + Fix the number of allocated cpus for an auto-adjustment case in which the job requests `--ntasks-per-node` and `--mem` (per-node) but the limit is `MaxMemPerCPU`. + Fix `POWER_DOWN_FORCE` request leaving node in completing state. + Do not count magnetic reservation queue records towards backfill limits. + Clarify error message when `--send-libs=yes` or BcastParameters=send_libs fails to identify shared library files, and avoid creating an empty `_libs` directory on the target filesystem. + Fix missing `CoreSpec` on dynamic nodes upon slurmctld restart. + Fix node state reporting when using specialized cores. + Fix number of CPUs allocated if `--cpus-per-gpu` used. + Add flag `ignore_prefer_validation` to not validate `--prefer` on a job. + Fix `salloc/sbatch` `SLURM_TASKS_PER_NODE` output environment variable when the number of tasks is not requested. + Permit using wildcard magic cookies with X11 forwarding. + `cgroup/v2` - Add check for swap when running OOM check after task termination. + Fix deadlock caused by race condition when disabling power save with a reconfigure. + Fix memory leak in the dbd when container is sent to the database. + `openapi/dbv0.0.38` - correct `dbv0.0.38_tres_info`. + Fix node `SuspendTime`, `SuspendTimeout`, `ResumeTimeout` being updated after altering partition node lists with scontrol. + `jobcomp/elasticsearch` - fix `data_t` memory leak after serialization. + Fix issue where `*` wasn't accepted in gpu/cpu bind. + Fix `SLURM_GPUS_ON_NODE` for shared GPU gres (MPS, shards). + Add `SLURM_SHARDS_ON_NODE` environment variable for shards. + Fix srun error with overcommit. + Fix bug in core selection for the default cyclic distribution of tasks across sockets, that resulted in random task launch failures. + Fix core selection for steps requesting multiple tasks per core when allocation contains more cores than required for step. + `gpu/nvml` - Fix MIG minor number generation when GPU minor number (`/dev/nvidia[minor_number]`) and index (as seen in `nvidia-smi`) do not match. + Fix accrue time underflow errors after slurmctld reconfig or restart. + Surpress errant errors from prolog_complete about being unable to locate `node:(null)`. + Fix issue where shards were selected from multiple gpus and failed to allocate. + Fix step cpu count calculation when using `--ntasks-per-gpu=`. + Fix overflow problems when validating array index parameters in `slurmctld` and prevent a potential condition causing `slurmctld` to crash. + Remove dependency on json-c in slurmctld when running with power saving. Only the new `SLURM_RESUME_FILE` support relies on this, and it will be disabled if json-c support is unavailable instead.- Create a macro for upgrade dependency to ensure uniform handling.- Add missing Provides:, Conflicts: and Obsoletes: to slurm-cray, slurm-hdf5 and slurm-testsuite to avoid package conflicts (bsc#1208810). - Unify Obsoletes:. - Add dependency for the general plugin package to the AcctGatherProfile HDF5 plugin.- Test Suite fixes: * Update README_Testsuite.md. * Clean up left over files when de-installing test suite. * Adjustment to test suite package: for SLE mark the openmpi4 devel package and slurm-hdf5 optional. * Add -ffat-lto-objects to the build flags when LTO is set to make sure the object files we ship with the test suite still work correctly. * Improve setup-testsuite.sh: copy ssh fingerprints from all nodes.- updated to 22.05.5 - NOTE: Slurm validates that libraries are of the same version. Unfortunately, due to an oversight, we failed to notice that the slurmstepd loads the hash_k12 library only after a job has completed. This means that if the hash_k12 library is upgraded before a job finishes, the slurmstepd will load the new library when the job finishes, and will fail due to a mismatch of versions. This results in nodes with slurmstepd processes stuck indefinitely. These processes require manual intervention to clean up. There is no clean way to resolve these hung slurmstepd processes. The only recommended way to upgrade between minor versions of 22.05 with RPM’s or upgrades that replace current binaries and libraries is to drain the nodes of running jobs first. - Fixes a number of moderate severity issues, noteable are: * Load hash plugin at slurmstepd launch time to prevent issues loading the plugin at step completion if the Slurm installation is upgraded. * Update nvml plugin to match the unique id format for MIG devices in new Nvidia drivers. * Fix multi-node step launch failure when nodes in the controller aren't in natural order. This can happen with inconsistent node naming (such as node15 and node052) or with dynamic nodes which can register in any order. * job_container/tmpfs - cleanup containers even when the .ns file isn't mounted anymore. * Wait up to PrologEpilogTimeout before shutting down slurmd to allow prolog and epilog scripts to complete or timeout. Previously, slurmd waited 120 seconds before timing out and killing prolog and epilog scripts.- Do not deduplicate files of testsuite Slurm configuration. This directory is supposed to be mounted over /etc/slurm therefore it must not contain softlinks to the files in this directory. - Improve .a and .o file collection for test suite: find these files even if there are multiple ones in a single line.- Fix build for older product version.- Fix a potential security vulnerability in the test package (bsc#1201674, CVE-2022-31251).- make slurmtest.tar reproducible- Improve check for mpicc in testsuite package: if binary isn't found, don't crash. - Patch NOFILE Limit in the slurmd.service copy for the testsuite.- update to 22.05.2 with following fixes: * Fix regression which allowed the oversubscription of licenses. * Fix a segfault in slurmctld when requesting gres in job arrays.- Package the Slurm testsuite for QA purposes. NOTE: This package is not meant to be used for testing by the user but rather for testing by the maintainers to ensure the package is working properly. DO NOT report test suite failures unless you are able to confirm that the failure is really a bug. * Fixes for test suite: Keep-logs-of-skipped-test-when-running-test-cases-sequentially.patch Fix-test-21.41.patch Fix-test-38.11.patch Fix-test-32.8.patch Fix-test-3.13.patch Fix-test7.2-to-find-libpmix-under-lib64-as-well.patch * Add documentation: README_Testsuite.md - Allow log in as user 'slurm'. This allows admins to run certain priviledged commands more easily without becoming root.- update to 22.05.0 with following changes: - Support for dynamic node addition and removal - Support for native Linux cgroup v2 operation - Newly added plugins to support HPE Slingshot 11 networks (switch/hpe_slingshot), and Intel Xe GPUs (gpu/oneapi) - Added new acct_gather_interconnect/sysfs plugin to collect statistics from arbitrary network interfaces. - Expanded and synced set of environment variables available in the Prolog/Epilog/PrologSlurmctld/EpilogSlurmctld scripts. - New "--prefer" option to job submissions to allow for a "soft constraint" request to influence node selection. - Optional support for license planning in the backfill scheduler with "bf_licenses" option in SchedulerParameters. - removed file slurm-2.4.4-init.patch as sysvinit is now realy deprecated - removed file load-pmix-major-version.patch as fixed upstream - set environment variable SUSE_ZNOW to 0 in %build to avoid module load failures due to unresolved symbols as module take advantage of lazy bindings (bsc#1200030).- Add a comment about the CommunicationParameters=block_null_hash option warning users who migrate - just in case.- Update to 21.08.8 which fixes CVE-2022-29500 (bsc#1199278), CVE-2022-29501 (bsc#1199279), and CVE-2022-29502 (bsc#1199281). - Added 'CommunicationParameters=block_null_hash' to slurm.conf, please add this parameter to existing configurations.- Update to 21.08.7 with following changes: * openapi/v0.0.37 - correct calculation for bf_queue_len_mean in /diag. * Avoid shrinking a reservation when overlapping with downed nodes. * Only check TRES limits against current usage for TRES requested by the job. * Do not allocate shared gres (MPS) in whole-node allocations * Constrain slurmstepd to job/step cgroup like in previous versions of Slurm. * Fix warnings on 32-bit compilers related to printf() formats. * Fix reconfigure issues after disabling/reenabling the GANG PreemptMode. * Fix race condition where a cgroup was being deleted while another step was creating it. * Set the slurmd port correctly if multi-slurmd * Fix FAIL mail not being sent if a job was cancelled due to preemption. * slurmrestd - move debug logs for HTTP handling to be gated by debugflag NETWORK to avoid unnecessary logging of communication contents. * Fix issue with bad memory access when shrinking running steps. * Fix various issues with internal job accounting with GRES when jobs are shrunk. * Fix ipmi polling on slurmd reconfig or restart. * Fix srun crash when reserved ports are being used and het step fails to launch. * openapi/dbv0.0.37 - fix DELETE execution path on /user/{user_name}. * slurmctld - Properly requeue all components of a het job if PrologSlurmctld fails. * rlimits - remove final calls to limit nofiles to 4096 but to instead use the max possible nofiles in slurmd and slurmdbd. * Allow the DBD agent to load large messages (up to MAX_BUF_SIZE) from state. * Fix potential deadlock during slurmctld restart when there is a completing job. * slurmstepd - reduce user requested soft rlimits when they are above max hard rlimits to avoid rlimit request being completely ignored and processes using default limits. * Fix Slurm user commands displaying available features as active features when no features were active. * Don't power down nodes that are rebooting. * Clear pending node reboot on power down request. * Ignore node registrations while node is powering down. * Don't reboot any node that is power down. * Don't allow a node to reboot if it's marked for power down. * Fix issuing reboot and downing when rebooting a powering up node. * Clear DRAIN on node after failing to resume before ResumeTimeout. * Prevent repeating power down if node fails to resume before ResumeTimeout. * Fix federated cloud node communication with srun and cloud_dns. * Fix jobs being scheduled on nodes marked to be powered_down when idle. * Fix problem where a privileged user could not view array tasks specified by _ when PrivateData had the jobs value set. - Changes in Slurm 21.08.6 * Fix plugin_name definitions in a number of plugins to improve logging. * Close sbcast file transfers when job is cancelled. * scrontab - fix handling of --gpus and --ntasks-per-gpu options. * sched/backfill - fix job_queue_rec_t memory leak. * Fix magnetic reservation logic in both main and backfill schedulers. * job_container/tmpfs - fix memory leak when using InitScript. * slurmrestd / openapi - fix memory leaks. * Fix slurmctld segfault due to job array resv_list double free. * Fix multi-reservation job testing logic. * Fix slurmctld segfault due to insufficient job reservation parse validation. * Fix main and backfill schedulers handling for already rejected job array. * sched/backfill - restore resv_ptr after yielding locks. * acct_gather_energy/xcc - appropriately close and destroy the IPMI context. * Protect slurmstepd from making multiple calls to the cleanup logic. * Prevent slurmstepd segfault at cleanup time in mpi_fini(). * Fix slurmctld sometimes hanging if shutdown while PrologSlurmctld or EpilogSlurmctld were running and PrologEpilogTimeout is set in slurm.conf. * Fix affinity of the batch step if batch host is different than the first node in the allocation. * slurmdbd - fix segfault after multiple failover/failback operations. * Fix jobcomp filetxt job selection condition. * Fix -f flag of sacct not being used. * Select cores for job steps according to the socket distribution. Previously, sockets were always filled before selecting cores from the next socket. * Keep node in Future state if epilog completes while in Future state. * Fix erroneous --constraint behavior by preventing multiple sets of brackets. * Make ResetAccrueTime update the job's accrue_time to now. * Fix sattach initialization with configless mode. * Revert packing limit checks affecting pmi2. * sacct - fixed assertion failure when using -c option and a federation display * Fix issue that allowed steps to overallocate the job's memory. * Fix the sanity check mode of AutoDetect so that it actually works. * Fix deallocated nodes that didn't actually launch a job from waiting for Epilogslurmctld to complete before clearing completing node's state. * Job should be in a completing state if EpilogSlurmctld when being requeued. * Fix job not being requeued properly if all node epilog's completed before EpilogSlurmctld finished. * Keep job completing until EpilogSlurmctld is completed even when "downing" a node. * Fix handling reboot with multiple job features. * Fix nodes getting powered down when creating new partitions. * Fix bad bit_realloc which potentially could lead to bad memory access. * slurmctld - remove limit on the number of open files. * Fix bug where job_state file of size above 2GB wasn't saved without any error message. * Fix various issues with no_consume gres. * Fix regression in 21.08.0rc1 where job steps failed to launch on systems that reserved a CPU in a cgroup outside of Slurm (for example, on systems with WekaIO). * Fix OverTimeLimit not being reset on scontrol reconfigure when it is removed from slurm.conf. * serializer/yaml - use dynamic buffer to allow creation of YAML outputs larger than 1MiB. * Fix minor memory leak affecting openapi users at process termination. * Fix batch jobs not resolving the username when nss_slurm is enabled. * slurmrestd - Avoid slurmrestd ignoring invalid HTTP method if the response serialized without error. * openapi/dbv0.0.37 - Correct conditional that caused the diag output to give an internal server error status on success. * Make --mem-bind=sort work with task_affinity * Fix sacctmgr to set MaxJobsAccruePer{User|Account} and MinPrioThres in sacctmgr add qos, modify already worked correctly. * job_container/tmpfs - avoid printing extraneous error messages in Prolog and Epilog, and when the job completes. * Fix step CPU memory allocation with --threads-per-core without --exact. * Remove implicit --exact when --threads-per-core or --hint=nomultithread is used. * Do not allow a step to request more threads per core than the allocation did. * Remove implicit --exact when --cpus-per-task is used.- update to 21.08.5 with following changes: * Fix issue where typeless GRES node updates were not immediately reflected. * Fix setting the default scrontab job working directory so that it's the home of the different user (*u ) and not that of root or SlurmUser editor. * Fix stepd not respecting SlurmdSyslogDebug. * Fix concurrency issue with squeue. * Fix job start time not being reset after launch when job is packed onto already booting node. * Fix updating SLURM_NODE_ALIASES for jobs packed onto powering up nodes. * Cray - Fix issues with starting hetjobs. * auth/jwks - Print fatal() message when jwks is configured but file could not be opened. * If sacctmgr has an association with an unknown qos as the default qos print 'UNKN*###' instead of leaving a blank name. * Correctly determine task count when giving --cpus-per-gpu, --gpus and * -ntasks-per-node without task count. * slurmctld - Fix places where the global last_job_update was not being set to the time of update when a job's reason and description were updated. * slurmctld - Fix case where a job submitted with more than one partition would not have its reason updated while waiting to start. * Fix memory leak in node feature rebooting. * Fix time limit permanetly set to 1 minute by backfill for job array tasks higher than the first with QOS NoReserve flag and PreemptMode configured. * Fix sacct -N to show jobs that started in the current second * Fix issue on running steps where both SLURM_NTASKS_PER_TRES and SLURM_NTASKS_PER_GPU are set. * Handle oversubscription request correctly when also requesting * -ntasks-per-tres. * Correctly detect when a step requests bad gres inside an allocation. * slurmstepd - Correct possible deadlock when UnkillableStepTimeout triggers. * srun - use maximum number of open files while handling job I/O. * Fix writing to Xauthority files on root_squash NFS exports, which was preventing X11 forwarding from completing setup. * Fix regression in 21.08.0rc1 that broke --gres=none. * Fix srun --cpus-per-task and --threads-per-core not implicitly setting * -exact. It was meant to work this way in 21.08. * Fix regression in 21.08.0 that broke dynamic future nodes. * Fix dynamic future nodes remembering active state on restart. * Fix powered down nodes getting stuck in COMPLETING+POWERED_DOWN when job is cancelled before nodes are powering up.- updated to 21.08.4 which fixes (CVE-2021-43337) which is only present in 21.08 tree. * CVE-2021-43337: For sites using the new AccountingStoreFlags=job_script and/or job_env options, an issue was reported with the access control rules in SlurmDBD that will permit users to request job scripts and environment files that they should not have access to. (Scripts/environments are meant to only be accessible by user accounts with administrator privileges, by account coordinators for jobs submitted under their account, and by the user themselves.) - changes from 21.08.3: * This includes a number of fixes since the last release a month ago, including one critical fix to prevent a communication issue between slurmctld and slurmdbd for sites that have started using the new AccountingStoreFlags=job_script functionality.- Utilize sysuser infrastructure to set user/group slurm. For munge authentication slurm should have a fixed UID across all nodes including the management server. Set it to 120 - Limit firewalld service definitions to SUSE versions >= 15.- added service definitions for firewalld (JSC#SLE-22741)- update to 21.08.2 - major change: * removed of support of the TaskAffinity=yes option in cgroup.conf. Please consider using "TaskPlugins=cgroup,affinity" in slurm.conf as an option. - minor changes and bugfixes: * slurmctld - fix how the max number of cores on a node in a partition are calculated when the partition contains multi*socket nodes. This in turn corrects certain jobs node count estimations displayed client*side. * job_submit/cray_aries - fix "craynetwork" GRES specification after changes introduced in 21.08.0rc1 that made TRES always have a type prefix. * Ignore nonsensical check in the slurmd for [Pro|Epi]logSlurmctld. * Fix writing to stderr/syslog when systemd runs slurmctld in the foreground. * Fix issue with updating job started with node range. * Fix issue with nodes not clearing state in the database when the slurmctld is started with clean*start. * Fix hetjob components > 1 timing out due to InactiveLimit. * Fix sprio printing -nan for normalized association priority if PriorityWeightAssoc was not defined. * Disallow FirstJobId=0. * Preserve job start info in the database for a requeued job that hadn't registered the first time in the database yet. * Only send one message on prolog failure from the slurmd. * Remove support for TaskAffinity=yes in cgroup.conf. * accounting_storage/mysql - fix issue where querying jobs via sacct * -whole-hetjob=yes or slurmrestd (which automatically includes this flag) could in some cases return more records than expected. * Fix issue for preemption of job array task that makes afterok dependency fail. Additionally, send emails when requeueing happens due to preemption. * Fix sending requeue mail type. * Properly resize a job's GRES bitmaps and counts when resizing the job. * Fix node being able to transition to CLOUD state from non-cloud state. * Fix regression introduced in 21.08.0rc1 which broke a step's ability to inherit GRES from the job when the step didn't request GRES but the job did. * Fix errors in logic when picking nodes based on bracketed anded constraints. This also enforces the requirement to have a count when using such constraints. * Handle job resize better in the database. * Exclude currently running, resized jobs from the runaway jobs list. * Make it possible to shrink a job more than once.- moved pam module from /lib64 to /usr/lib64 which fixes boo#1191095 via the macro %_pam_moduledir- updated to 21.08.1 with following bug fixes: * Fix potential memory leak if a problem happens while allocating GRES for a job. * If an overallocation of GRES happens terminate the creation of a job. * AutoDetect=nvml: Fatal if no devices found in MIG mode. * Print federation and cluster sacctmgr error messages to stderr. * Fix off by one error in --gpu-bind=mask_gpu. * Add --gpu-bind=none to disable gpu binding when using --gpus-per-task. * Handle the burst buffer state "alloc-revoke" which previously would not display in the job correctly. * Fix issue in the slurmstepd SPANK prolog/epilog handler where configuration values were used before being initialized. * Restore a step's ability to utilize all of an allocations memory if --mem=0. * Fix --cpu-bind=verbose garbage taskid. * Fix cgroup task affinity issues from garbage taskid info. * Make gres_job_state_validate() client logging behavior as before 44466a4641. * Fix steps with --hint overriding an allocation with --threads-per-core. * Require requesting a GPU if --mem-per-gpu is requested. * Return error early if a job is requesting --ntasks-per-gpu and no gpus or task count. * Properly clear out pending step if unavailable to run with available resources. * Kill all processes spawned by burst_buffer.lua including decendents. * openapi/v0.0.{35,36,37} - Avoid setting default values of min_cpus, job name, cwd, mail_type, and contiguous on job update. * openapi/v0.0.{35,36,37} - Clear user hold on job update if hold=false. * Prevent CRON_JOB flag from being cleared when loading job state. * sacctmgr - Fix deleting WCKeys when not specifying a cluster. * Fix getting memory for a step when the first node in the step isn't the first node in the allocation. * Make SelectTypeParameters=CR_Core_Memory default for cons_tres and cons_res. * Correctly handle mutex unlocks in the gres code if failures happen. * Give better error message if -m plane is given with no size. * Fix --distribution=arbitrary for salloc. * Fix jobcomp/script regression introduced in 21.08.0rc1 0c75b9ac9d. * Only send the batch node in the step_hostlist in the job credential. * When setting affinity for the batch step don't assume the batch host is node 0. * In task/affinity better checking for node existence when laying out affinity. * slurmrestd - fix job submission with auth/jwt. - removed Fix-statement-condition-in-netloc-autoconf-macro.patch issue was fixed upstream- Fix-statement-condition-in-netloc-autoconf-macro.patch: Fix netloc check, reestablish netloc disable code. - Make configure arg '--with-pmix' conditional. - Move openapi plugins to package slurm-restd.- updated to 21.08.0, major changes: * A new "AccountingStoreFlags=job_script" option to store the job scripts directly in SlurmDBD. * Added "sacct -o SubmitLine" format option to get the submit line of a job/step. * Changes to the node state management so that nodes are marked as PLANNED instead of IDLE if the scheduler is still accumulating resources while waiting to launch a job on them. * RS256 token support in auth/jwt. * Overhaul of the cgroup subsystems to simplify operation, mitigate a number of inherent race conditions, and prepare for future cgroup v2 support. * Further improvements to cloud node power state management. * A new child process of the Slurm controller called "slurmscriptd" responsible for executing PrologSlurmctld and EpilogSlurmctld scripts, which significantly reduces performance issues associated with enabling those options. * A new burst_buffer/lua plugin allowing for site-specific asynchronous job data management. * Fixes to the job_container/tmpfs plugin to allow the slurmd process to be restarted while the job is running without issue. * Added json/yaml output to sacct, squeue, and sinfo commands. * Added a new node_features/helpers plugin to provide a generic way to change settings on a compute node across a reboot. * Added support for automatically detecting and broadcasting shared libraries for an executable launched with "srun --bcast". * Added initial OCI container execution support with a new --container option to sbatch and srun. * Improved "configless" support by allowing multiple control servers to be specified through the slurmd --conf-server option, and send additional configuration files at startup including cli_filter.lua. - minor changes: * If an overallocation of GRES happens terminate the creation of a job. * AutoDetect=nvml: Fatal if no devices found in MIG mode. * Print federation and cluster sacctmgr error messages to stderr. * Add --gpu-bind=none to disable gpu binding when using --gpus-per-task. * Handle the burst buffer state "alloc-revoke" which previously would not display in the job correctly. * Fix issue in the slurmstepd SPANK prolog/epilog handler where configuration values were used before being initialized. * Restored --gpu-bind=single: to check core affinity like * -gpu-bind=closest does. This removal of this behavior only was in rc2. * slurmd - Fix assert failure on initialization due to bad node name. * Fix error codes in cgroup/v1. * Don't destroy the memory step outside fini, which leads to a double destroy causing an error message. * Add support for lua 5.4. * Force cgroup.clone_children to 0 in slurm cgroup directories. This caused issues in task cpuset plugin in systems with it enabled by default. * Clear GRES HAS_TYPE flag when removing type name. * Environment flags in gres.conf now override flags set by AutoDetect. * Environment flags in gres.conf now apply to subsequent gres.conf lines where Environment flags are not set. * Set missing job_uid and job_gid members when preparing a kill_job_msg_t in abort_job_on_node(), abort_job_on_nodes() and kill_job_on_node(). * Fix swappiness not being set in cgroups. * Fix coordinators for new subaccounts. * Fix coordinators when adding existing users with PrivateData=users. * slurmctld - do not attempt to relinquish control to self. * openapi/v0.0.37 - Honor kill_on_invalid_dependency as job parameter. * Check max_gres when doing step allocation, fix for regression in rc2. * SPANK plugins are now required to match the current Slurm version, and must be recompiled for each new Slurm release. * node_features/helpers - add ExecTime configuration option. * srun - Fix force termination with -X. * On slurmctld restart set node typed GRES counts correctly. * Fix places where a step wasn't allocated in the slurmctld but wasn't ever removed from the job. * Fix step allocation memory when using --threads-per-core. * Fix step allocations to consume all threads on a core when using threads-per-core. * Add check to validate cpu request on a step if --threads-per-core is given and it is less than what the core on the node has in the allocation. * Fix issue where a step could request more gres than the job had and the step would hang forever. This bug was only introduced in 21.08.0rc2. * Only print \r\n for logging messages on stderr when --pty has been explicitly requested. * Relax check on SPANK plugins to only require Slurm major + minor versions to match. * job_container/tmpfs - delegate handling of /dev/shm to the extern step so new step launches will be attached correctly even after the slurmd process has been restarted. * Limit the wait time in proctrack_g_wait() to UnkillableStepTimeout instead of a hardcoded value of 256 seconds, and limit the delay between tests to a maximum of 32 seconds. * fatal() on start if using job_container/tmpfs without PrologFlags=Contian. * Load bf_when_last_cycle from job state only if protocol version >= 21.08. * Docs - remove man3 section entirely. * Set step memory when using MemPerGPU or DefMemPerGPU. Previously a step's memory was not set even when it requested *-mem-per-gpu and at least one GPU. * Add cli_filter.lua support in configless mode. * Check that the step requests at least as many gres as nodes. * Make job's SLURM_JOB_GPUS print global GPU IDs instead of MIG unique_ids. * Fix miscounting of GPU envs in prolog/epilog if MultipleFiles was used. * Support MIGs in prolog/epilog's CUDA_VISIBLE_DEVICES & co. * Add SLURM_JOB_GPUS back into Prolog; add it to Epilog. * Fix issue where the original executable, not the bcast'd version, was executed with 'srun *-bcast'. * sacct - print '-' header correctly for fields over 53-characters wide. * openapi/dbv0.0.37 - replace "REST" with "Slurm OpenAPI" for plugin_name. * openapi/v0.0.37 - replace "REST" with "Slurm OpenAPI" for plugin_name. * configless - fix segfault on 'scontrol reconfigure'. * Use FREE_NULL_LIST instead of list_destroy. * If we made are running an interactive session we need to force track_steps. * Disable OPOST flag when using --pty to avoid issues with Emac. * Fix issue where extra bonus core was allocated in some situations. * Avoid putting gres with count of 0 on a TRES req/alloc. * Fix memory in requested TRES when --mem-per-gpu is used. * Changed ReqMem field in sacct to match memory from ReqTRES. * Changed --gpu-bind=single: to no longer check core affinity like * -gpu-bind=closest does. This consequently affects --ntasks-per-gpu. * slurmrestd - add v0.0.37 OpenAPI plugin. * slurmrestd/v0.0.37 - rename standard_in -> standard_input. * slurmrestd/v0.0.37 - rename standard_out -> standard_output. * Changed the --format handling for negative field widths (left justified) to apply to the column headers as well as the printed fields. * Add LimitFactor to the QOS. A float that is factored into an associations [Grp|Max]TRES limits. For example, if the LimitFactor is 2, then an association with a GrpTRES of 30 CPUs, would be allowed to allocate 60 CPUs when running under this QOS. * slurmrestd - Pass SLURM_NO_CHANGE_IN_DATA to client as 403 (Not Modified). * slurmrestd/v0.0.37 - Add update_time field to Jobs query to allow clients to only get jobs list based on change timestamp. * Reset job eligible time when job is manually held. * Add DEBUG_FLAG_JAG to improve logging related to job account gathering. * Convert logging in account_gather/common to DEBUG_FLAG_JAG. * Add more logging for jag_common_poll_data() when prec_extra() called. * slurmrestd/v0.0.37 - add API to fetch reservation(s) info. * Catch more errors in task/cgroup initalization and cleanup to avoid allowing jobs to start when cgroups failure to configure correctly. * Fix cgroup ns detection when using containers (e.g. LXC or Docker). * Reset job's next_step_id counter to 0 after being requeued. * Make scontrol exit with non-zero status after failing to delete a partition or reservation. * Make NtasksPerTRES optional in slurm_sprint_job_info(). * slurmrestd/v0.0.37 - Add update_time field to nodes query to allow clients to only get nodes list based on change timestamp. * common/parse_config - catch and propagate return codes when handling a match on a key-value pattern. This implies error codes detected in the handlers are now not ignored and users of _handle_keyvalue_match() can fatal(). * common/hostlist - fix hostlist_delete_nth() xassert() upper bound check. * API change: Removed slurm_kill_job_msg and modified the function signature for slurm_kill_job2. slurm_kill_job2 should be used instead of slurm_kill_job_msg. * Fix non-zero exit code for scontrol ping when all controllers are down. * Enforce a valid configuration for AccountingStorageEnforce in slurm.conf. If the configuration is invalid, then an error message will be printed and the command or daemon (including slurmctld) will not run. * slurmrestd/v0.0.37 - Add update_time field to partitions/reservations query to allow clients to only get the entities list when something changed. * slurmdbd.service - add "After" relationship to all common names for MariaDB to reduce startup delays. * slurmrestd/v0.0.37 - Correct displaying node states that are UNKNOWN. * slurmrestd/v0.0.37 - Add flags to node states. * Fix first job on fresh cluster not being assigned JobId=1 (or FirstJobId). * squeue - make it so --nodelist is sensitive to --clusters. * squeue - do --nodelist node validation in the same order as listing. * Removed AccountingStoreJobComment option. Please update your config to use AccountingStoreFlags=job_comment instead. * AccountingStoreFlags=job_script allows you to store the job's batch script. * AccountingStoreFlags=job_env allows you to store the job's env vars. * Add sacct -o SubmitLine to get the submit line of a job/step. * Removed DefaultStorage{Host,Loc,Pass,Port,Type,User} options. * Fix NtasksPerTRES delimiter from : to = in scontrol show job output. * Removed CacheGroups, CheckpointType, JobCheckpointDir, MemLimitEnforce, SchedulerPort, SchedulerRootFilter options. * Make job accounting queries use consistent timeframes with and w/o jobs. * --cpus-per-task and --threads-per-core now imply --exact. This fixes issues where steps would be allocated the wrong number of CPUs. * configure: the --with option handling has been made consistent across the various optional libraries. Specifying *-with-foo=/path/to/foo will only check that directory for the applicable library (rather than, in some cases, falling back to the default directories), and will always error the build if the library is not found (instead of a mix of error messages and non- fatal warning messages). * configure: replace --with-rmsi_dir option with proper handling for * -with-rsmi=dir. * Pass additional job environment variables to MailProg. * Add SLURM_JOB_WORK_DIR to Prolog, Epilog. * Removed sched/hold plugin. * Fix srun overwriting SLURM_SUBMIT_DIR and SLURM_SUBMIT_HOST when within an existing allocation. * step_ctx code has been removed from the api. * cli_filter/lua, jobcomp/lua, job_submit/lua now load their scripts from the same directory as the slurm.conf file (and thus now will respect changes to the SLURM_CONF environment variable). * SPANK - call slurm_spank_init if defined without slurm_spank_slurmd_exit in slurmd context. * job_container/tmpfs - Remove need for .active file to allow salloc without an interactive step to work. * slurmd - Delay background node registration on every failure up to 128s on startup. * slurmctld - Always notify slurmd that node registration was accepted to avoid slurmd needless attempting to re-register if there is configuration issue. * Put node into "INVAL" state upon registering with an invalid node configuration. Node must register with a valid configuration to continue. * Make --cpu-bind=threads default for --threads-per-core -- cli and env can override. * jobcomp/elasticsearch - Use data_t to serialize data. The plugin now has the JSON-C library as a prerequisite. * scrontab - create the temporary file under the TMPDIR environment variable (if set), otherwise continue to use TmpFS as configured in slurm.conf. * Add LastBusyTime to "scontrol show nodes" and slurmrestd nodes output, which represents the time the node last had jobs on it. * slurmd - allow multiple comma-separated controllers to be specified in configless mode with *-conf-server * sacctmgr - changed column headings to "ParentID" and "ParentName" instead of "Par ID" and "Par Name" respectively. * Perl API - make sure man pages are installed under the --prefix given to configure. * Manually powering down of nodes with scontrol now ignores SuspendExc. * SALLOC_THREADS_PER_CORE and SBATCH_THREADS_PER_CORE have been added as input environment variables for salloc and sbatch, respectively. They do the same thing as *-threads-per-core. * Distinguish queued reboot requests (REBOOT) from issued reboots (REBOOT^). * Set the maximum number of open files per process to 4096 to avoid performance issues when closing the entire range with closeall(). * auth/jwt - add support for RS256 tokens. * Relax reservation purge due to any invalid uid after creation time. * Reject srun that requests both --exclusive and --overlap. * service files - change dependency to network-online rather than just network to ensure DNS and other services are available. * RSMI: Fix incorrect PCI BDF bits. * plugins/cli_filter - Convert to using data_t to serialize JSON. * Fix testing array job after regaining locks in backfill. * Don't display node's comment with "scontrol show nodes" unless set. * Add "Extra" field to node to store extra information other than a comment. * scrontab - Use /tmp instead of TmpFS if TMPDIR is not set. * Add ResumeTimeout, SuspendTimeout and SuspendTime to Partitions. * sreport - change to sorting TopUsage by the --tres option. * slurmrestd - do not run allow operation as SlurmUser/root by default. * Allow map_cpu and mask_cpu for non-whole node allocation. * TaskPluginParam=verbose is now treated as a default. Previously it would be applied regardless of the job specifying a *-cpu-bind. * Add "node_reg_mem_percent" SlurmctldParameter to define percentage of memory nodes are allowed to register with. * Show correct number of SocketsPerBoard in slurmd -C with hwloc2. * Alter sreport's cluster utilization report column name from 'Reserved' to 'Planned' to match the nomenclature of the 'Planned' node. * Add StateComplete format option to sinfo to show base_state+flags. * "scontrol show node" now shows State as base_state+flags instead of shortened state with flags appended. eg. IDLE# *> IDLE+POWERING_UP. Also "POWER" state flag string is "POWERED_DOWN". * slurmd/req - add missing job_env_t's het_job_id initialization off the request in _rpc_{abort,terminate}_job(). This caused problems for Native Cray builds when joining a CNCU job_container plugin with Epilog configured. * Fix joining a CNCU job_container on a Native Cray build before executing the UnkillableStepProgram for a HetJob step. * slurmrestd/v0.0.35 - Plugin has been tagged as deprecated. * srun - Job steps requiring more cores than available to be rejected unless '--overlap' is specificed. * Add bf_node_space_size to SchedulerParameters. * Add scontrol update node state=POWER_DOWN_FORCE and POWER_DOWN_ASAP as new ways to power off and reset especially CLOUD nodes. * Define and separate node power state transitions. Previously a powering down node was in both states, POWERING_OFF and POWERED_OFF. These are now separated. * Create a new process called slurmscriptd which runs PrologSlurmctld and EpilogSlurmctld. This avoids fork() calls from slurmctld, and can avoid performance issues if the slurmctld has a large memory footprint. * Added new Script option to DebugFlags for debugging slurmscriptd. * scrontab - add ability to update crontab from a file or standard input. * scrontab - add ability to set and expand variables. * Pass JSON of job to node mappings to ResumeProgram. * If running steps in an allocation with CR_PACK_NODE or -mpack the srun will only attempt to allocate as much as needed from the allocation instead of always trying to allocate every node in the allocation. * Jobs that request the whole node now check to see if any gres are allocated. * Rename SbcastParameters to BcastParameters. * Make srun sensitive to BcastParameters. * RSMI: Add gres_links_create_empty() and preserve RSMI enumeration order. * GPUs: Use index instead of dev_num for CUDA_VISIBLE_DEVICES * Don't run epilog on nodes if job never launched. * QOS accrue limits only apply to the job QOS, not partition QOS. * Add --gpu-bind=per_task: option, --gpus-per-task will now set this option by default. * Treat any return code from SPANK plugin that is not SLURM_SUCCESS to be an error or rejection. * Print the statistics for extern step adopted processes in sstat. * Fix SLURM_NODE_ALIASES to work for ipv6 node addrs. * Add support for automatically detecting and broadcasting executable shared object dependencies for sbcast and srun *-bcast.- Updated to 20.11.8: * slurmctld - fix erroneous "StepId=CORRUPT" messages in error logs. * Correct the error given when auth plugin fails to pack a credential. * Fix unused-variable compiler warning on FreeBSD in fd_resolve_path(). * acct_gather_filesystem/lustre - only emit collection error once per step. * Add GRES environment variables (e.g., CUDA_VISIBLE_DEVICES) into the interactive step, the same as is done for the batch step. * Fix various potential deadlocks when altering objects in the database dealing with every cluster in the database. * slurmrestd: - handle slurmdbd connection failures without segfaulting. - fix segfault for searches in slurmdb/v0.0.36/jobs. - remove (non-functioning) users query parameter for slurmdb/v0.0.36/jobs from openapi.json - fix segfault in slurmrestd db/jobs with numeric queries - add argv handling for job/submit endpoint. - add description for slurmdb/job endpoint. * slurmrestd/dbv0.0.36: - Fix values dumped in job state/current and job step state. - Correct description for previous state property. * srun: - fix broken node step allocation in a heterogeneous allocation. - leave SLURM_DIST_UNKNOWN as default for --interactive. * Fail step creation if -n is not multiple of --ntasks-per-gpu. * job_container/tmpfs - Fix slowdown on teardown. * Fix problem with SlurmctldProlog where requeued jobs would never launch. * job_container/tmpfs - Fix issue when restarting slurmd where the namespace mount points could disappear. * sacct: - avoid truncating JobId at 34 characters. - fix segfault when printing StepId (or when using --long). * scancel - fix segfault when --wckey filtering option is used. * select/cons_tres - Fix memory leak. * Prevent file descriptor leak in job_container/tmpfs on slurmd restart. * perlapi/libslurmdb - expose tres_req_str to job hash. * scrontab - close and reopen temporary crontab file to deal with editors that do not change the original file, but instead write out then rename a new file. * sstat - fix linking so that it will work when --without-shared-libslurm was used to build Slurm. * Clear allocated cpus for running steps in a job before handling requested nodes on new step. * Don't reject a step if not enough nodes are available. Instead, defer the step until enough nodes are available to satisfy the request. * Don't reject a step if it requests at least one specific node that is already allocated to another step. Instead, defer the step until the requested node(s) become available. * Better handling of --mem=0. * Ignore DefCpuPerGpu when --cpus-per-task given.- Updated to 20.11.7 which fixes CVE-2021-31215 (bsc#1186024) - New features in 20.11.7: * slurmd - handle configless failures gracefully instead of hanging indefinitely. * select/cons_tres - fix Dragonfly topology not selecting nodes in the same leaf switch when it should as well as requests with *-switches option. * Fix issue where certain step requests wouldn't run if the first node in the job allocation was full and there were idle resources on other nodes in the job allocation. * Fix deadlock issue with Slurmctld. * torque/qstat - fix printf error message in output. * When adding associations or wckeys avoid checking multiple times a user or cluster name. * Fix wrong jobacctgather information on a step on multiple nodes due to timeouts sending its the information gathered on its node. * Fix missing xstrdup which could result in slurmctld segfault on array jobs. * Fix security issue in PrologSlurmctld and EpilogSlurmctld by always prepending SPANK_ to all user-set environment variables. CVE-2021-31215. - New features in 20.11.6: * Fix sacct assert with the --qos option. * Use pkg-config --atleast-version instead of --modversion for systemd. * common/fd - fix getsockopt() call in fd_get_socket_error(). * Properly handle the return from fd_get_socket_error() in _conn_readable(). * cons_res - Fix issue where running jobs were not taken into consideration when creating a reservation. * Avoid a deadlock between job_list for_each and assoc QOS_LOCK. * Fix TRESRunMins usage for partition qos on restart/reconfig. * Fix printing of number of tasks on a completed job that didn't request tasks. * Fix updating GrpTRESRunMins when decrementing job time is bigger than it. * Make it so we handle multithreaded allocations correctly when doing - -exclusive or --core-spec allocations. * Fix incorrect round-up division in _pick_step_cores * Use appropriate math to adjust cpu counts when --ntasks-per-core=1. * cons_tres - Fix consideration of power downed nodes. * cons_tres - Fix DefCpuPerGPU, increase cpus-per-task to match with gpus-per-task * cpus-per-gpu. * Fix under-cpu memory auto-adjustment when MaxMemPerCPU is set. * Make it possible to override CR_CORE_DEFAULT_DIST_BLOCK. * Perl API - fix retrieving/storing of slurm_step_id_t in job_step_info_t. * Recover state of burst buffers when slurmctld is restarted to avoid skipping burst buffer stages. * Fix race condition in burst buffer plugin which caused a burst buffer in stage-in to not get state saved if slurmctld stopped. * auth/jwt - print an error if jwt_file= has not been set in slurmdbd. * Fix RESV_DEL_HOLD not being a valid state when using squeue --states. * Add missing squeue selectable states in valid states error message. * Fix scheduling last array task multiple times on error, causing segfault. * Fix issue where a step could be allocated more memory than the job when dealing with --mem-per-cpu and --threads-per-core. * Fix removing qos from assoc with -= can lead to assoc with no qos * auth/jwt - fix segfault on invalid credential in slurmdbd due to missing validate_slurm_user() function in context. * Fix single Port= not being applied to range of nodes in slurm.conf * Fix Jobs not requesting a tres are not starting because of that tres limit. * acct_gather_energy/rapl - fix AveWatts calculation. * job_container/tmpfs - Fix issues with cleanup and slurmd restarting on running jobs.- Ship REST API version and auth plugins with slurmrestd. - Add YAML support for REST API to build (bsc#1185603).- Udpate to 20.11.5: - New features: * New job_container/tmpfs plugin developed by NERSC that can be used to create per-job filesystem namespaces. Documentaiion and configuration can be found in the respecting man page. - Bug fixes: * Fix main scheduler bug where bf_hetjob_prio truncates SchedulerParameters. * Fix sacct not displaying UserCPU, SystemCPU and TotalCPU for large times. * scrontab - fix to return the correct index for a bad #SCRON option. * scrontab - fix memory leak when invalid option found in #SCRON line. * Add errno for when a user requests multiple partitions and they are using partition based associations. * Fix issue where a job could run in a wrong partition when using EnforcePartLimits=any and partition based associations. * Remove possible deadlock when adding associations/wckeys in multiple threads. * When using PrologFlags=alloc make sure the correct Slurm version is set in the credential. * When sending a job a warning signal make sure we always send SIGCONT beforehand. * Fix issue where a batch job would continue running if a prolog failed on a node that wasn't the batch host and requeuing was disabled. * Fix issue where sometimes salloc/srun wouldn't get a message about a prolog failure in the job's stdout. * Requeue or kill job on a prolog failure when PrologFlags is not set. * Fix race condition causing node reboots to get requeued before ResumeTimeout expires. * Preserve node boot_req_time on reconfigure. * Preserve node power_save_req_time on reconfigure. * Fix node reboots being queued and issued multiple times and preventing the reboot to time out. * Fix run_command to exit correctly if track_script kills the calling thread. * Only requeue a job when the PrologSlurmctld returns nonzero. * When a job is signaled with SIGKILL make sure we flush all prologs/setup scripts. * Handle burst buffer scripts if the job is canceled while stage_in is happening. * When shutting down the slurmctld make note to ignore error message when we have to kill a prolog/setup script we are tracking. * scrontab - add support for the --open-mode option. * acct_gather_profile/influxdb - avoid segfault on plugin shutdown if setup has not completed successfully. * Reduce delay in starting salloc allocations when running with prologs. * Alter AllocNodes check to work if the allocating node's domain doesn't match the slurmctld's. This restores the pre*20.11 behavior. * Fix slurmctld segfault if jobs from a prior version had the now-removed INVALID_DEPEND state flag set and were allowed to run in 20.11. * Add job_container/tmpfs plugin to give a method to provide a private /tmp per job. * Set the correct core affinity when using AutoDetect. * slurmrestd - mark "environment" as required for job submissions in schema.- Udpate to 20.11.04 * Fix node selection for advanced reservations with features. * mpi/pmix: Handle pipe failure better when using ucx. * mpi/pmix: include PMIX_NODEID for each process entry. * Fix job getting rejected after being requeued on same node that died. * job_submit/lua - add "network" field. * Fix situations when a reoccuring reservation could erroneously skip a period. * Ensure that a reservations [pro|epi]log are ran on reoccuring reservations. * Fix threads-per-core memory allocation issue when using CR_CPU_MEMORY. * Fix scheduling issue with --gpus. * Fix gpu allocations that request --cpus-per-task. * mpi/pmix: fixed print messages for all PMIXP_* macros * Add mapping for XCPU to --signal option. * Fix regression in 20.11 that prevented a full pass of the main scheduler from ever executing. * Work around a glibc bug in which "0" is incorrectly printed as "nan" which will result in corrupted association state on restart. * Fix regression in 20.11 which made slurmd incorrectly attempt to find the parent slurmd address when not applicable and send incorrect reverse*tree info to the slurmstepd. * Fix cgroup ns detection when using containers (e.g. LXC or Docker). * scrontab - change temporary file handling to work with emacs. - Removed check-for-lipmix.so.MAJOR.patch - Added: load-pmix-major-version.patch- Update to 20.11.03 - This release includes a major functional change to how job step launch is handled compared to the previous 20.11 releases. This affects srun as well as MPI stacks - such as Open MPI - which may use srun internally as part of the process launch. One of the changes made in the Slurm 20.11 release was to the semantics for job steps launched through the 'srun' command. This also inadvertently impacts many MPI releases that use srun underneath their own mpiexec/mpirun command. For 20.11.{0,1,2} releases, the default behavior for srun was changed such that each step was allocated exactly what was requested by the options given to srun, and did not have access to all resources assigned to the job on the node by default. This change was equivalent to Slurm setting the --exclusive option by default on all job steps. Job steps desiring all resources on the node needed to explicitly request them through the new '--whole' option. In the 20.11.3 release, we have reverted to the 20.02 and older behavior of assigning all resources on a node to the job step by default. This reversion is a major behavioral change which we would not generally do on a maintenance release, but is being done in the interest of restoring compatibility with the large number of existing Open MPI (and other MPI flavors) and job scripts that exist in production, and to remove what has proven to be a significant hurdle in moving to the new release. Please note that one change to step launch remains - by default, in 20.11 steps are no longer permitted to overlap on the resources they have been assigned. If that behavior is desired, all steps must explicitly opt-in through the newly added '--overlap' option. Further details and a full explanation of the issue can be found at: https://bugs.schedmd.com/show_bug.cgi?id=10383#c63 - Other changes from 20.11.03 * Fix segfault when parsing bad "#SBATCH hetjob" directive. * Allow countless gpu:srun, sbatch->srun sequence. * Reject job credential if non-superuser sets the LAUNCH_NO_ALLOC flag. * Make it so srun --no-allocate works again. * jobacct_gather/linux - Don't count memory on tasks that have already finished. * Fix 19.05/20.02 batch steps talking with a 20.11 slurmctld. * jobacct_gather/common - Do not process jobacct's with same taskid when calling prec_extra. * Cleanup all tracked jobacct tasks when extern step child process finishes. * slurmrestd/dbv0.0.36 - Correct structure of dbv0.0.36_tres_list. * Fix regression causing task/affinity and task/cgroup to be out of sync when configured ThreadsPerCore is different than the physical threads per core. * Fix situation when --gpus is given but not max nodes (-N1-1) in a job allocation. * Interactive step - ignore cpu bind and mem bind options, and do not set the associated environment variables which lead to unexpected behavior from srun commands launched within the interactive step. * Handle exit code from pipe when using UCX with PMIx.- Fix fallout introduced by: "Replace '%service_del_postun -n' with '%service_del_postun_without_restart'" for older Leap/SLE versions.- Fix Provides:/Conflicts: for libnss_slurm (bsc#1180700).- Add support for configuration files from external plugins. While built-in plugins have their configuration added in slurm.conf, external SPANK plugins add their configuration to plugstack.conf To allow packaging easily spank plugins, their configuration files should be added independently at /etc/spack/plugstack.conf.d and plugstack.conf should be left with an oneliner including all the files under /etc/spack/plugstack.conf.d- Update to 20.11.02 * Fix older versions of sacct not working with 20.11. * Fix slurmctld crash when using a pre-20.11 srun in a job allocation. * Correct logic problem in _validate_user_access. * Fix libpmi to initialize Slurm configuration correctly. - Update to 20.11.01 * Fix spelling of "overcomited" to "overcomitted" in sreport's cluster utilization report. * Silence debug message about shutting down backup controllers if none are configured. * Don't create interactive srun until PrologSlurmctld is done. * Fix fd symlink path resolution. * Fix slurmctld segfault on subnode reservation restore after node configuration change. * Fix resource allocation response message environment allocation size. * Ensure that details->env_sup is NULL terminated. * select/cray_aries - Correctly remove jobs/steps from blades using NPC. * cons_tres - Avoid max_node_gres when entire node is allocated with - -ntasks-per-gpu. * Allow NULL arg to data_get_type(). * In sreport have usage for a reservation contain all jobs that ran in the reservation instead of just the ones that ran in the time specified. This matches the report for the reservation is not truncated for a time period. * Fix issue with sending wrong batch step id to a < 20.11 slurmd. * Add a job's alloc_node to lua for job modification and completion. * Fix regression getting a slurmdbd connection through the perl API. * Stop the extern step terminate monitor right after proctrack_g_wait(). * Fix removing the normalized priority of assocs. * slurmrestd/v0.0.36 - Use correct name for partition field: "min nodes per job" -"min_nodes_per_job". * slurmrestd/v0.0.36 - Add node comment field. * Fix regression marking cloud nodes as "unexpectedly rebooted" after multiple boots. * Fix slurmctld segfault in _slurm_rpc_job_step_create(). * slurmrestd/v0.0.36 - Filter node states against NODE_STATE_BASE to avoid the extended states all being reported as "invalid". * Fix race that can prevent the prolog for a requeued job from running. * cli_filter - add "type" to readily distinguish between the CLI command in use. * smail - reduce sleep before seff to 5 seconds. * Ensure SPANK prolog and epilog run without an explicit PlugStackConfig. * Disable MySQL automatic reconnection. * Fix allowing "b" after memory unit suffixes. * Fix slurmctld segfault with reservations without licenses. * Due to internal restructuring ahead of the 20.11 release, applications calling libslurm MUST call slurm_init(NULL) before any API calls. Otherwise the API call is likely to fail due to libslurm's internal configuration not being available. * slurm.spec - allow custom paths for PMIx and UCX install locations. * Use rpath if enabled when testing for Mellanox's UCX libraries. * slurmrestd/dbv0.0.36 - Change user query for associations to optional. * slurmrestd/dbv0.0.36 - Change account query for associations to optional. * mpi/pmix - change the error handler error message to be more useful. * Add missing connection in acct_storage_p_{clear_stats, reconfig, shutdown}. * Perl API - fix issue when running in configless mode. * nss_slurm - avoid deadlock when stray sockets are found. * Display correct value for ScronParameters in 'scontrol show config'- Update to version 20.11.0 Slurm 20.11 includes a number of new features including: * Overhaul of the job step management and launch code, alongside improved GPU task placement support. * A new "Interactive Step" mode of operation for salloc. * A new "scrontab" command that can be used to submit and manage periodically repeating jobs. * IPv6 support. * Changes to the reservation logic, with new options allowing users to delete reservations, allowing admins to skip the next occurance of a repeated reservation, and allowing for a job to be submitted and eligible to run within multiple reservations. * Dynamic Future Nodes - automatically associate a dynamically provisioned (or "cloud") node against a NodeName definition with matching hardware. * An experimental new RPC queuing mode for slurmctld to reduce thread contention on heavily loaded clusters. * SlurmDBD integration with the Slurm REST API. Also check https://github.com/SchedMD/slurm/blob/slurm-20-11-0-1/RELEASE_NOTES- Updated to 20.02.6, addresses two security fixes: * PMIx - fix potential buffer overflows from use of unpackmem(). CVE-2020-27745 (bsc#1178890) * X11 forwarding - fix potential leak of the magic cookie when sent as an argument to the xauth command. CVE-2020-27746 (bsc#1178891) - And many other bugfixes, full log and details available at: * https://lists.schedmd.com/pipermail/slurm-announce/2020/000045.html- Replace '%service_del_postun -n' with '%service_del_postun_without_restart' '-n' is deprecated and will be removed in the future.- Updated to 20.02.5, changes: * Fix leak of TRESRunMins when job time is changed with --time-min * pam_slurm - explicitly initialize slurm config to support configless mode. * scontrol - Fix exit code when creating/updating reservations with wrong Flags. * When a GRES has a no_consume flag, report 0 for allocated. * Fix cgroup cleanup by jobacct_gather/cgroup. * When creating reservations/jobs don't allow counts on a feature unless using an XOR. * Improve number of boards discovery * Fix updating a reservation NodeCnt on a zero-count reservation. * slurmrestd - provide an explicit error messages when PSK auth fails. * cons_tres - fix job requesting single gres per-node getting two or more nodes with less CPUs than requested per-task. * cons_tres - fix calculation of cores when using gres and cpus-per-task. * cons_tres - fix job not getting access to socket without GPU or with less than --gpus-per-socket when not enough cpus available on required socket and not using --gres-flags=enforce binding. * Fix HDF5 type version build error. * Fix creation of CoreCnt only reservations when the first node isn't available. * Fix wrong DBD Agent queue size in sdiag when using accounting_storage/none. * Improve job constraints XOR option logic. * Fix preemption of hetjobs when needed nodes not in leader component. * Fix wrong bit_or() messing potential preemptor jobs node bitmap, causing bad node deallocations and even allocation of nodes from other partitions. * Fix double-deallocation of preempted non-leader hetjob components. * slurmdbd - prevent truncation of the step nodelists over 4095. * Fix nodes remaining in drain state state after rebooting with ASAP option. - changes from 20.02.4: * srun - suppress job step creation warning message when waiting on PrologSlurmctld. * slurmrestd - fix incorrect return values in data_list_for_each() functions. * mpi/pmix - fix issue where HetJobs could fail to launch. * slurmrestd - set content-type header in responses. * Fix cons_res GRES overallocation for --gres-flags=disable-binding. * Fix cons_res incorrectly filtering cores with respect to GRES locality for - -gres-flags=disable-binding requests. * Fix regression where a dependency on multiple jobs in a single array using underscores would only add the first job. * slurmrestd - fix corrupted output due to incorrect use of memcpy(). * slurmrestd - address a number of minor Coverity warnings. * Handle retry failure when slurmstepd is communicating with srun correctly. * Fix jobacct_gather possibly duplicate stats when _is_a_lwp error shows up. * Fix tasks binding to GRES which are closest to the allocated CPUs. * Fix AMD GPU ROCM 3.5 support. * Fix handling of job arrays in sacct when querying specific steps. * slurmrestd - avoid fallback to local socket authentication if JWT authentication is ill-formed. * slurmrestd - restrict ability of requests to use different authentication plugins. * slurmrestd - unlink named unix sockets before closing. * slurmrestd - fix invalid formatting in openapi.json. * Fix batch jobs stuck in CF state on FrontEnd mode. * Add a separate explicit error message when rejecting changes to active node features. * cons_common/job_test - fix slurmctld SIGABRT due to double-free. * Fix updating reservations to set the duration correctly if updating the start time. * Fix update reservation to promiscuous mode. * Fix override of job tasks count to max when ntasks-per-node present. * Fix min CPUs per node not being at least CPUs per task requested. * Fix CPUs allocated to match CPUs requested when requesting GRES and threads per core equal to one. * Fix NodeName config parsing with Boards and without CPUs. * Ensure SLURM_JOB_USER and SLURM_JOB_UID are set in SrunProlog/Epilog. * Fix error messages for certain invalid salloc/sbatch/srun options. * pmi2 - clean up sockets at step termination. * Fix 'scontrol hold' to work with 'JobName'. * sbatch - handle --uid/--gid in #SBATCH directives properly. * Fix race condition in job termination on slurmd. * Print specific error messages if trying to run use certain priority/multifactor factors that cannot work without SlurmDBD. * Avoid partial GRES allocation when --gpus-per-job is not satisfied. * Cray - Avoid referencing a variable outside of it's correct scope when dealing with creating steps within a het job. * slurmrestd - correctly handle larger addresses from accept(). * Avoid freeing wrong pointer with SlurmctldParameters=max_dbd_msg_action with another option after that. * Restore MCS label when suspended job is resumed. * Fix insufficient lock levels. * slurmrestd - use errno from job submission. * Fix "user" filter for sacctmgr show transactions. * Fix preemption logic. * Fix no_consume GRES for exclusive (whole node) requests. * Fix regression in 20.02 that caused an infinite loop in slurmctld when requesting --distribution=plane for the job. * Fix parsing of the --distribution option. * Add CONF READ_LOCK to _handle_fed_send_job_sync. * prep/script - always call slurmctld PrEp callback in _run_script(). * Fix node estimation for jobs that use GPUs or --cpus-per-task. * Fix jobcomp, job_submit and cli_filter Lua implementation plugins causing slurmctld and/or job submission CLI tools segfaults due to bad return handling when the respective Lua script failed to load. * Fix propagation of gpu options through hetjob components. * Add SLURM_CLUSTERS environment variable to scancel. * Fix packing/unpacking of "unlinked" jobs. * Connect slurmstepd's stderr to srun for steps launched with --pty. * Handle MPS correctly when doing exclusive allocations. * slurmrestd - fix compiling against libhttpparser in a non-default path. * slurmrestd - avoid compilation issues with libhttpparser < 2.6. * Fix compile issues when compiling slurmrestd without --enable-debug. * Reset idle time on a reservation that is getting purged. * Fix reoccurring reservations that have Purge_comp= to keep correct duration if they are purged. * scontrol - changed the "PROMISCUOUS" flag to "MAGNETIC" * Early return from epilog_set_env in case of no_consume. * Fix cons_common/job_test start time discovery logic to prevent skewed results between "will run test" executions. * Ensure TRESRunMins limits are maintained during "scontrol reconfigure". * Improve error message when host lookup fails. - Refresh patch: pam_slurm-Initialize-arrays-and-pass-sizes.patch- Add support for openPMIx also for Leap/SLE 15.0/1 (bsc#1173805). - Do not run %check on SLE-12-SP2: Some incompatibility in tcl makes this fail. - Remove unneeded build dependency to postgresql-devel. - Disable build on s390 (requires 64bit).- Bring QA to the package build: add %%check stage. - Remove cruft that isn't needed any longer. - Add 'ghosted' run-file. - Add rpmlint filter to handle issues with library packages for Leap and enterprise upgrade versions.- Updated to 20.02.3 which fixes CVE-2020-12693 (bsc#1172004). - Other changes are: * Factor in ntasks-per-core=1 with cons_tres. * Fix formatting in error message in cons_tres. * Fix calling stat on a NULL variable. * Fix minor memory leak when using reservations with flags=first_cores. * Fix gpu bind issue when CPUs=Cores and ThreadsPerCore > 1 on a node. * Fix --mem-per-gpu for heterogenous --gres requests. * Fix slurmctld load order in load_all_part_state(). * Fix race condition not finding jobacct gather task cgroup entry. * Suppress error message when selecting nodes on disjoint topologies. * Improve performance of _pack_default_job_details() with large number of job * arguments. * Fix archive loading previous to 17.11 jobs per-node req_mem. * Fix regresion validating that --gpus-per-socket requires --sockets-per-node * for steps. Should only validate allocation requests. * error() instead of fatal() when parsing an invalid hostlist. * nss_slurm - fix potential deadlock in slurmstepd on overloaded systems. * cons_tres - fix --gres-flags=enforce-binding and related --cpus-per-gres. * cons_tres - Allocate lowest numbered cores when filtering cores with gres. * Fix getting system counts for named GRES/TRES. * MySQL - Fix for handing typed GRES for association rollups. * Fix step allocations when tasks_per_core > 1. * Fix allocating more GRES than requested when asking for multiple GRES types.- Treat libnss_slurm like any other package: add version string to upgrade package.- Updated to 20.02.1 with following changes" * Improve job state reason for jobs hitting partition_job_depth. * Speed up testing of singleton dependencies. * Fix negative loop bound in cons_tres. * srun - capture the MPI plugin return code from mpi_hook_client_fini() and use as final return code for step failure. * Fix segfault in cli_filter/lua. * Fix --gpu-bind=map_gpu reusability if tasks > elements. * Make sure config_flags on a gres are sent to the slurmctld on node registration. * Prolog/Epilog - Fix missing GPU information. * Fix segfault when using config parser for expanded lines. * Fix bit overlap test function. * Don't accrue time if job begin time is in the future. * Remove accrue time when updating a job start/eligible time to the future. * Fix regression in 20.02.0 that broke --depend=expand. * Reset begin time on job release if it's not in the future. * Fix for recovering burst buffers when using high-availability. * Fix invalid read due to freeing an incorrectly allocated env array. * Update slurmctld -i message to warn about losing data. * Fix scontrol cancel_reboot so it clears the DRAIN flag and node reason for a pending ASAP reboot.- Remove legacy_cray: with 20.02 the special treatment for cray-specific plugins on SLE version prior to 15SP2 is no longer required.- slurm-plugins will now also require pmix not only libpmix (bsc#1164326)- Removed autopatch as it doesn't work for the SLE-11-SP4 build.- Disable %arm builds as this is no longer supported.- pmix searches now also for libpmix.so.2 so that there is no dependency for devel package (bsc#1164386) * added patch file check-for-lipmix.so.MAJOR.patch * reworded patch file Remove-rpath-from-build.patch to use %autopatch- Update to version 20.02.0 (jsc#SLE-8491) * Fix minor memory leak in slurmd on reconfig. * Fix invalid ptr reference when rolling up data in the database. * Change shtml2html.py to require python3 for RHEL8 support, and match man2html.py. * slurm.spec - override "hardening" linker flags to ensure RHEL8 builds in a usable manner. * Fix type mismatches in the perl API. * Prevent use of uninitialized slurmctld_diag_stats. * Fixed various Coverity issues. * Only show warning about root-less topology in daemons. * Fix accounting of jobs in IGNORE_JOBS reservations. * Fix issue with batch steps state not loading correctly when upgrading from 19.05. * Deprecate max_depend_depth in SchedulerParameters and move it to DependencyParameters. * Silence erroneous error on slurmctld upgrade when loading federation state. * Break infinite loop in cons_tres dealing with incorrect tasks per tres request resulting in slurmctld hang. * Improve handling of --gpus-per-task to make sure appropriate number of GPUs is assigned to job. * Fix seg fault on cons_res when requesting --spread-job. - Move to python3 for everything but SLE-11-SP4 * For SLE-11-SP4 add a workaround to handle a python3 script (python2.7 compliant).- Add explicit version dependency to libpmix as well. 'slurm-devel' has a tight version dependency on libpmix - allowing multiple libpmix versions in one package repository is therefore essential.- Update to version 20.02.0-rc1 * sbatch - fix segfault when no newline at the end of a burst buffer file. * Change scancel to only check job's base state when matching -t options. * Save job dependency list in state files. * cons_tres - allow jobs to be run on systems with root-less topologies. * Restore pre-20.02pre1 PrologSlurmctld synchonization behavior to avoid various race conditions, and ensure proper batch job launch. * Add new slurmrestd command/daemon which implements the Slurm REST API.- Update to version 20.02.0-0pre1, highlights are Highlights: * Exclusive behavior of a node includes all GRES on a node as well as the cpus. * Use python3 instead of python for internal build/test scripts. The slurm.spec file has been updated to depend on python3 as well. * Added new NodeSet configuration option to help simplify partition configuration sections for heterogeneous / condo*style clusters. * Added slurm.conf option MaxDBDMsgs to control how many messages will be stored in the slurmctld before throwing them away when the slurmdbd is down. * The checkpoint plugin interface and all associated API calls have been removed. * slurm_init_job_desc_msg() initializes mail_type as uint16_t. This allows mail_type to be set to NONE with scontrol. * Add new slurm_spank_log() function to print messages back to the user from within a SPANK plugin without prepending "error: " from slurm_error(). * Enforce having partition name and nodelist=ALL when creating reservations with flags=PART_NODES. * SPANK - removed never-implemented slurm_spank_slurmd_init() interface. This hook has always been accessible through slurm_spank_init() in the S_CTX_SLURMD context instead. * sbcast - add new BcastAddr option to NodeName lines to allow sbcast traffic to flow over an alternate network path. * Added auth/jwt plugin, and 'scontrol token' subcommand. PMIx - improve * performance of proc map generation. Deprecate kill_invalid_depend in * SchedulerParameters and move it to a new option called DependencyParameters. * Enable job dependencies for any job on any cluster in the same federation. * Allow clusters to be added automatically to db at startup of ctld. Add * AccountingStorageExternalHost slurm.conf parameter. The * "ConditionPathExists" condition in slurmd.service has been disabled by default to permit simpler installation of a "configless" Slurm cluster. * In SchedulerParameters remove deprecated max_job_bf and replace with bf_max_job_test. * Disable sbatch, salloc, srun --reboot for non-admins. SPANK - added support * for S_JOB_GID in the job script context with spank_get_item(). * Prolog/Epilog - add SLURM_JOB_GID environment variable. configuration file changes: * The mpi/openmpi plugin has been removed as it does nothing. MpiDefault=openmpi will be translated to the functionally-equivalent MpiDefault=none. command changes (see man pages for details) * Display StepId=.batch instead of StepId=.4294967294 in output of "scontrol show step". (slurm_sprint_job_step_info()) * MPMD in srun will now defer PATH resolution for the commands to launch to slurmstepd. Previously it would handle resolution client*side, but with a non*standard approach that walked PATH in reverse. * squeue - added "--me" option, equivalent to --user=$USER. * The LicensesUsed line has been removed from 'scontrol show config'. Please see the 'scontrol show licenses' command as an alternative. * sbatch - adjusted backoff times for "--wait" option to reduce load on slurmctld. This results in a steady*state delay of 32s between queries, instead of the prior 10s delay. - Removed following deprecated patches: * removed patch slurmctld-rerun-agent_init-when-backup-controller-takes-over.patch * removed patch split-xdaemon-in-xdaemon_init-and-xdaemon_finish-for.patch * removed patch slurmctld-uses-xdaemon_-for-systemd.patch * removed patch slurmd-uses-xdaemon_-for-systemd.patch * removed patch slurmdbd-uses-xdaemon_-for-systemd.patch * removed patch slurmsmwd-uses-xdaemon_-for-systemd.patch * removed patch removed-deprecated-xdaemon.patch- standard slurm.conf uses now also SlurmctldHost on all build targets (bsc#1162377)- Fix a missed systemd_requires -> systemd_ordering conversion.- Remove special OHPC compatibility macro: these settings should be applied univerally. - Add a Recommends for mariadb to slurm-slurmdbd: it is recommened to run the database on the same machine as the daemon.- BuildRequire pkgconfig(systemd) instead of systemd: allow OBS to shortcut through the -mini flavors. - Use systemd_ordering instead of systemd_requires: systemd is never a strict requirement; but in case the system is scheduled for installation together with systemd, we want systemd to be installed prior to slurm.- start slurmdbd after mariadb (bsc#1161716)- Fix base_ver for SLE 15 SP2.- Update to version 19.05.5 (jsc#SLE-8491) * Check %docdir/NEWS for details. * Includes security fixes CVE-2019-19727, CVE-2019-19728, CVE-2019-12838. * Disable i586 builds as this is no longer supported. * Create libnss_slurm package to support user and group resolution thru slurmstepd. * slurm-2.4.4-rpath.patch -> Remove-rpath-from-build.patch Obsoleted: - pam_slurm_adopt-avoid-running-outside-of-the-sshd-PA.patch - pam_slurm_adopt-send_user_msg-don-t-copy-undefined-d.patch - pam_slurm_adopt-use-uid-to-determine-whether-root-is.patch- Deprecate "ControlMachine" only for SLURM version upgrades and products newer than 1501. This ensures that the original setting is retained for the SLURM version shipped origianlly with SLE-15-SP1 or Leap 15.1.- Update to v18.08.9 for fixing CVE-2019-19728 (bsc#1159692). * Wrap END_TIMER{,2,3} macro definition in "do {} while (0)" block. * Make sview work with glib2 v2.62. * Make Slurm compile on linux after sys/sysctl.h was deprecated. * Install slurmdbd.conf.example with 0600 permissions to encourage secure use. CVE-2019-19727. * srun - do not continue with job launch if --uid fails. CVE-2019-19728.- added pmix support jsc#SLE-10800- Use --with-shared-libslurm to build slurm binaries using libslurm. - Make libslurm depend on slurm-config.- Fix ownership of /var/spool/slurm on new installations and upgrade (boo#1158696).- Fix permissions of slurmdbd.conf (bsc#1155784, CVE-2019-19727). - Fix %posttrans macro _res_update to cope with added newline (bsc#1153259).- Add package slurm-webdoc which sets up a web server to provide the documentation for the version shipped.- Move srun from 'slurm' to 'slurm-node': srun is required on the nodes as well so sbatch will work. 'slurm-node' is a requirement when 'slurm' is installed (bsc#1153095).- Set %base_ver for SLE-15-SP2 to 18.08 (for now).- Edit sample configuration to deprecate "ControlMachine", "ControlAddr", "BackupController" and "BackupAddr" in favor "SlurmctldHost".- Fix logic of slurm-munge recommends: slurm-munge requires munge already, so if we have munge installed we recommend slurm-munge as the authentication when installing slurm or slurm-node.- Fix build for SLE-11-SP4 and older.- added cray depend libraries to seperate package, as they are now built, since json is enabled- Updated to 18.08.8 for fixing (CVE-2019-12838, bsc#1140709, jsc#SLE-7341, jsc#SLE-7342) * Update "xauth list" to use the same 10000ms timeout as the other xauth commands. * Fix issue in gres code to handle a gres cnt of 0. * Don't purge jobs if backfill is running. * Verify job is pending add/removing accrual time. * Don't abort when the job doesn't have an association that was removed before the job was able to make it to the database. * Set state_reason if select_nodes() fails job for QOS or Account. * Avoid seg_fault on referencing association without a valid_qos bitmap. * If Association/QOS is removed on a pending job set that job as ineligible. * When changing a jobs account/qos always make sure you remove the old limits. * Don't reset a FAIL_QOS or FAIL_ACCOUNT job reason until the qos or account changed. * Restore "sreport -T ALL" functionality. * Correctly typecast signals being sent through the api. * Properly initialize structures throughout Slurm. * Sync "numtask" squeue format option for jobs and steps to "numtasks". * Fix sacct -PD to avoid CA before start jobs. * Fix potential deadlock with backup slurmctld. * Fixed issue with jobs not appearing in sacct after dependency satisfied. * Fix showing non-eligible jobs when asking with -j and not -s. * Fix issue with backfill scheduler scheduling tasks of an array when not the head job. * accounting_storage/mysql - fix SIGABRT in the archive load logic. * accounting_storage/mysql - fix memory leak in the archive load logic. * Limit records per single SQL statement when loading archived data. * Fix unnecessary reloading of job submit plugins. * Allow job submit plugins to be turned on/off with a reconfigure. * Fix segfault when loading/unloading Lua job submit plugin multiple times. * Fix printing duplicate error messages of jobs rejected by job submit plugin. * Fix printing of job submit plugin messages of het jobs without pack id. * Fix memory leak in group_cache.c * Fix jobs stuck from FedJobLock when requeueing in a federation * Fix requeueing job in a federation of clusters with differing associations * sacctmgr - free memory before exiting in 'sacctmgr show runaway'. * Fix seff showing memory overflow when steps tres mem usage is 0. * Upon archive file name collision, create new archive file instead of overwriting the old one to prevent lost records. * Limit archive files to 50000 records per file so that archiving large databases will succeed. * Remove stray newlines in SPANK plugin error messages. * Fix archive loading events. * In select/cons_res: Only allocate 1 CPU per node with the --overcommit and - -nodelist options. * Fix main scheduler from potentially not running through whole queue. * cons_res/job_test - prevent a job from overallocating a node memory. * cons_res/job_test - fix to consider a node's current allocated memory when testing a job's memory request. * Fix issue where multi-node job steps on cloud nodes wouldn't finish cleaning up until the end of the job (rather than the end of the step). * Fix issue with a 17.11 sbcast call to a 18.08 daemon. * Add new job bit_flags of JOB_DEPENDENT. * Make it so dependent jobs reset the AccrueTime and do not count against any AccrueTime limits. * Fix sacctmgr --parsable2 output for reservations and tres. * Prevent slurmctld from potential segfault after job_start_data() called for completing job. * Fix jobs getting on nodes with "scontrol reboot asap". * Record node reboot events to database. * Fix node reboot failure message getting to event table. * Don't write "(null)" to event table when no event reason exists. * Fix minor memory leak when clearing runaway jobs. * Avoid flooding slurmctld and logging when prolog complete RPC errors occur. * Fix GCC 9 compiler warnings. * Fix seff human readable memory string for values below a megabyte. * Fix dump/load of rejected heterogeneous jobs. * For heterogeneous jobs, do not count the each component against the QOS or association job limit multiple times. * slurmdbd - avoid reservation flag column corruption with the use of newer flags, instead preserve the older flag fields that we can still fit in the smallint field, and discard the rest. * Fix security issue in accounting_storage/mysql plugin on archive file loads by always escaping strings within the slurmdbd. CVE-2019-12838.- Fix build dependency issue around libibmad-devel introduced in SLE-12-SP4.- Add BuildRequires to address warnings during build: * for libcurl-devel, libssh2-devel and rrdtool-devel * for libjson-c-devel and liblz4-devel where available, disable these with --without-json and --without-lz4 where not. * disable DataWarp (--without-datawarp).- Update SLURM to 18.08.7: * Set debug statement to debug2 to avoid benign error messages. * Add SchedulerParameters option of bf_hetjob_immediate to attempt to start a heterogeneous job as soon as all of its components are determined able to do so. * Fix underflow causing decay thread to exit. * Fix main scheduler not considering hetjobs when building the job queue. * Fix regression for sacct to display old jobs without a start time. * Fix setting correct number of gres topology bits. * Update hetjobs pending state reason when appropriate. * Fix accounting_storage/filetxt's understanding of TRES. * Set Accrue time when not enforcing limits. * Fix srun segfault when requesting a hetjob with test_exec or bcast options. * Hide multipart priorities log message behind Priority debug flag. * sched/backfill - Make hetjobs sensitive to bf_max_job_start. * Fix slurmctld segfault due to job's partition pointer NULL dereference. * Fix issue with OR'ed job dependencies. * Add new job's bit_flags of INVALID_DEPEND to prevent rebuilding a job's dependency string when it has at least one invalid and purged dependency. * Promote federation unsynced siblings log message from debug to info. * burst_buffer/cray - fix slurmctld SIGABRT due to illegal read/writes. * burst_buffer/cray - fix memory leak due to unfreed job script content. * node_features/knl_cray - fix script_argv use-after-free. * burst_buffer/cray - fix script_argv use-after-free. * Fix invalid reads of size 1 due to non null-terminated string reads. * Add extra debug2 logs to identify why BadConstraints reason is set.- Do not build hdf5 support where not available.- Add support for version updates on SLE: Update packages to a later version than the version supported originally on SLE will receive a version string in their package name.- added the hdf5 job data gathering plugin- Add backward compatibility with SLE-11 SP4- Update to version 18.08.05-2: This version obsoletes: Fix-contrib-perlapi-to-build-with-the-fix-for-CVE-2019-6438-750cc23ed.patch - Fix spec file for older SUSE versions.- Update to version 18.08.05: * Add mitigation for a potential heap overflow on 32-bit systems in xmalloc. (CVE-2019-6438, bsc#1123304). * Other fixes: + Backfill - If a job has a time_limit guess the end time of a job better if OverTimeLimit is Unlimited. + Fix "sacctmgr show events event=cluster" + Fix sacctmgr show runawayjobs from sibling cluster + Avoid bit offset of -1 in call to bit_nclear(). + Insure that "hbm" is a configured GresType on knl systems. + Fix NodeFeaturesPlugins=node_features/knl_generic to allow other gres other than knl. + cons_res: Prevent overflow on multiply. + Better debug for bad values in gres.conf. + Fix double accounting of energy at end of job. + Read gres.conf for cloud nodes on slurmctld. + Don't assume the first node of a job is the batch host when purging jobs from a node. + Better debugging when a job doesn't have a job_resrcs ptr. + Store ave watts in energy plugins. + Add XCC plugin for reading Lenovo Power. + Fix minor memory leak when scheduling rebootable nodes. + Fix debug2 prefix for sched log. + Fix printing correct SLURM_JOB_ACCOUNT_PACK_GROUP_* in env for a Het Job. + sbatch - search current working directory first for job script. + Make it so held jobs reset the AccrueTime and do not count against any AccrueTime limits. + Add SchedulerParameters option of bf_hetjob_prio=[min|avg|max] to alter the job sorting algorithm for scheduling heterogeneous jobs. + Fix initialization of assoc_mgr_locks and slurmctld_locks lock structures. + Fix segfault with job arrays using X11 forwarding. + Revert regression caused by e0ee1c7054 which caused negative values and values starting with a decimal to be invalid for PriorityWeightTRES and TRESBillingWeight. + Fix possibility to update a job's reservation to none. + Suppress connection errors to primary slurmdbd when backup dbd is active. + Suppress connection errors to primary db when backup db kicks in + Add missing fields for sacct --completion when using jobcomp/filetxt. + Fix incorrect values set for UserCPU, SystemCPU, and TotalCPU sacct fields when JobAcctGatherType=jobacct_gather/cgroup. + Fixed srun from double printing invalid option msg twice. + Remove unused -b flag from getopt call in sbatch. + Disable reporting of node TRES in sreport. + Re-enabling features combined by OR within parenthesis for non-knl setups. + Prevent sending duplicate requests to reboot a node before ResumeTimeout. + Down nodes that don't reboot by ResumeTimeout. + Update seff to reflect API change from rss_max to tres_usage_in_max. + Add missing TRES constants from perl API. + Fix issue where sacct would return incorrect array tasks when querying specific tasks. + Add missing variables to slurmdb_stats_t in the perlapi. + Fix nodes not getting reboot RPC when job requires reboot of nodes. + Fix failing update the partition list of a job. + Use slurm.conf gres ids instead of gres.conf names to get a gres type name. * Disable slurmctld-rerun-agent_init-when-backup-controller-takes-over.patch: Believed to be fixed by commit c1a537dbbe6 See: https://bugs.schedmd.com/show_bug.cgi?id=5511 * Add Fix-contrib-perlapi-to-build-with-the-fix-for-CVE-2019-6438-750cc23ed.patch: Fix fallout from 750cc23ed for CVE-2019-6438.- Update to 18.08.04, with following highlights * Fix message sent to user to display preempted instead of time limit when a job is preempted. * Fix memory leak when a failure happens processing a nodes gres config. * Improve error message when failures happen processing a nodes gres config. * Don't skip jobs in scontrol hold. * Allow --cpu-bind=verbose to be used with SLURM_HINT environment variable. * Enhanced handling for runaway jobs * cons_res: Delay exiting cr_job_test until after cores/cpus are calculated and distributed. * Don't check existence of srun --prolog or --epilog executables when set to "none" and SLURM_TEST_EXEC is used. * Add "P" suffix support to job and step tres specifications. * Fix jobacct_gather/cgroup to work correctly when more than one task is started on a node. * salloc - set SLURM_NTASKS_PER_CORE and SLURM_NTASKS_PER_SOCKET in the environment if the corresponding command line options are used. * slurmd - fix handling of the -f flag to specify alternate config file locations. * Add SchedulerParameters option of bf_ignore_newly_avail_nodes to avoid scheduling lower priority jobs on resources that become available during the backfill scheduling cycle when bf_continue is enabled. * job_submit/lua: Add several slurmctld return codes and add user/group info * salloc/sbatch/srun - print warning if mutually exclusive options of --mem and --mem-per-cpu are both set. - Refreshed: * pam_slurm_adopt-avoid-running-outside-of-the-sshd-PA.patch- restarting services on update only when activated - added rotation of logs - Added backported patches which harden the pam module pam_slurm_adopt (BOO#1116758) which will be in slurm 19.05.x * added pam_slurm_adopt-avoid-running-outside-of-the-sshd-PA.patch [PATCH 1/3] pam_slurm_adopt: avoid running outside of the sshd PAM * added pam_slurm_adopt-send_user_msg-don-t-copy-undefined-d.patch [PATCH 2/3] pam_slurm_adopt: send_user_msg: don't copy undefined data * added pam_slurm_adopt-use-uid-to-determine-whether-root-is.patch [PATCH 3/3] pam_slurm_adopt: use uid to determine whether root is logging on - package slurm-pam_slurm now depends on slurm-node and not on slurm- fixed code in %pretrans section to be compatible with lua 5.1- Added missing perl-base dependency.- Moved HTML docs to doc package.- Moved config man pages to a separate package: This way, they won't get installed on compute nodes.- Update to 18.08.3 * Add new burst buffer state of "teardown-fail" to indicate the burst buffer teardown operation is failing on specific buffers. * Multiple backup slurmctld daemons can be configured * Enable jobs with zero node count for creation and/or deletion of persistent burst buffers. * Add "scontrol show dwstat" command to display Cray burst buffer status. * Add "GetSysStatus" option to burst_buffer.conf file. * Add node and partition configuration options of "CpuBind" to control default task binding. * Add "NumaCpuBind" option to knl.conf * Add sbatch "--batch" option to identify features required on batch node. * Add "BatchFeatures" field to output of "scontrol show job". * Add support for "--bb" option to sbatch command. * Add new SystemComment field to job data structure and database. * Expand reservation "flags" field from 32 to 64 bits. * Add job state flag of "SIGNALING" to avoid race condition. * Properly handle srun --will-run option when there are jobs in COMPLETING state. * Properly report who is signaling a step. * Don't combine updated reservation records in sreport's reservation report. * node_features plugin - Add suport for XOR & XAND of job constraints (node feature specifications). * Improvements to how srun searches for the executible when using cwd. * Now programs can be checked before execution if test_exec is set. * Report NodeFeatures plugin configuration with scontrol and sview commands. * Add acct_gather_profile/influxdb plugin. * Add new job state of SO/STAGE_OUT * Correct SLURM_NTASKS and SLURM_NPROCS environment variable for heterogeneous job step. * Expand advanced reservation feature specification to support parenthesis and counts of nodes with specified features. * Defer job signaling until prolog is completed * Have the primary slurmctld wait until the backup has completely shutdown before taking control. * Fix issue where unpacking job state after TRES count changed could lead to invalid reads. * Heterogeneous job steps allocations supported with Open MPI. * Remove redundant function arguments from task plugins. * Add Slurm configuration file check logic using "slurmctld -t" command. * Add the use of a xml file to help performance when using hwloc. * Remove support for "ChosLoc" configuration parameter. * Configuration parameters "ControlMachine", "ControlAddr", "BackupController" and "BackupAddr" replaced by an ordered list of "SlurmctldHost" records. * Remove --immediate option from sbatch. * Add infrastructure for per-job and per-step TRES parameters. * Add DefCpuPerGpu and DefMemPerGpu to global and per-partition configuration parameters. * Add ValidateMode configuration parameter to knl_cray.conf. * Disable local PTY output processing when using 'srun --unbuffered'. * Change the column name for the %U (User ID) field in squeue to 'UID'. * CRAY - Add CheckGhalQuiesce to the CommunicationParameters. * When a process is core dumping, avoid terminating other processes in that task group. * CPU frequency management enhancements: If scaling_available_frequencies file is not available, then derive values from scaling_min_freq and scaling_max_freq values. * Add pending jobs count to sdiag output. * Add configuration paramerers SlurmctldPrimaryOnProg and SlurmctldPrimaryOffProg, which define programs to execute when a slurmctld daemon changes state. * Add configuration paramerers SlurmctldAddr for use with virtual IP to manage backup slurmctld daemons. * Explicitly shutdown the slurmd process when instructed to reboot. * Add ability to create/update partition with TRESBillingWeights through scontrol. * Calcuate TRES billing values at submission. * Add node_features plugin function "node_features_p_reboot_weight()". * Add NodeRebootWeight parameter to knl.conf configuration file. * Completely remove "gres" field from step record. Use "tres_per_node", "tres_per_socket", etc. * Add "Links" parameter to gres.conf configuration file. * Force slurm_mktime() to set tm_isdst to -1. * burst_buffer.conf - Add SetExecHost flag to enable burst buffer access from the login node for interactive jobs. * Append ", with requeued tasks" to job array "end" emails if any tasks in the array were requeued. * Add ResumeFailProgram slurm.conf option to specify a program that is called when a node fails to respond by ResumeTimeout. * Add new job pending reason of "ReqNodeNotAvail, reserved for maintenance". * Remove AdminComment += syntax from 'scontrol update job'. * sched/backfill: Reset job time limit if needed for deadline scheduling. * For heterogeneous job component with required nodes, explicitly exclude those nodes from all other job components. * Add name of partition used to output of srun --test-only output. * sdiag output now reports outgoing slurmctld message queue contents. * Improve escaping special characters on user commands when specifying paths. * Add salloc/sbatch/srun option of --gres-flags=disable-binding to disable filtering of CPUs with respect to generic resource locality. * SlurmDBD - Print warning if MySQL/MariaDB internal tuning is not at least half of the recommended values. * Add ability to specify a node reason when rebooting nodes with "scontrol reboot". * Add nextstate option to "scontrol reboot". * Consider "resuming" (nextstate=resume) nodes as available in backfill future scheduling. * Add TimelimitRaw sacct output field to display timelimit numbers. * Add support for sacct --whole-hetjob=[yes|no] option. * Make salloc handle node requests the same as sbatch. * Add shutdown_on_reboot SlurmdParameter to control whether the Slurmd will shutdown itself down or not when a reboot request is received. * Add cancel_reboot scontrol option to cancel pending reboot of nodes. * Make Users case insensitive in the database based on Parameters=PreserveCaseUser in the slurmdbd.conf. * Improve scheduling when dealing with node_features that could have a boot delay. * Changed the default AuthType for slurmdbd to auth/munge. * Added 'remote-fs.target' to After directive of slurmd.service file. * Remove drain on node when reboot nextstate used. * Speed up pack of job's qos. * Add sacctmgr options to prevent/manage job queue stuffing: - GrpJobsAccrue= - MaxJobsAccrue= * MinPrioThreshold Minimum priority required to reserve resources when scheduling. * Add control_inx value to trigger_info_msg_t to permit future work in the trigger management code to distinguish which of multiple backup controllers has changed state. * NOTES: PreemptType=preempt/job_prio has been removed - use PreemptType=preempt/qos instead. * Bluegene support was deprecated has now been removed * cgroup_allowed_devices_file.conf was removed. It was never used by default, as ConstrainDevices was not set. If needed, refer to the cgroups.conf man page on how to create one. * slurm.epilog.clean: Removed. User should use pam_slurm_adopt instead. - Refreshed: * removed-deprecated-xdaemon.patch * slurmctld-uses-xdaemon_-for-systemd.patch * slurmd-uses-xdaemon_-for-systemd.patch * slurmdbd-uses-xdaemon_-for-systemd.patch * slurmsmwd-uses-xdaemon_-for-systemd.patch * slurmctld-rerun-agent_init-when-backup-controller-takes-over.patch- Move config man-pages to config package.- added correct link flags for perl bindings (bsc#1108671) * added correct linker search path in slurm-2.4.4-rpath.patch * perl:Switch is required by slurm torque wrappers- Fix Requires(pre) and Requires(post) for slurm-config and slurm-node. This fixes issues with failing slurm user creation when installed during initial system installation (bsc#1109373).- Update to 17.11.9 * Fix segfault in slurmctld when a job's node bitmap is NULL during a scheduling cycle. Primarily caused by EnforcePartLimits=ALL. * Remove erroneous unlock in acct_gather_energy/ipmi. * Enable support for hwloc version 2.0.1. * Fix 'srun -q' (--qos) option handling. * Fix socket communication issue that can lead to lost task completition messages, which will cause a permanently stuck srun process. * Handle creation of TMPDIR if environment variable is set or changed in a task prolog script. * Avoid node layout fragmentation if running with a fixed CPU count but without Sockets and CoresPerSocket defined. * burst_buffer/cray - Fix datawarp swap default pool overriding jobdw. * Fix incorrect job priority assignment for multi-partition job with different PriorityTier settings on the partitions. * Fix sinfo to print correct node state.- When using a remote shared StateSaveLocation, slurmctld needs to be started after remote filesystems have become available. Add 'remote-fs.target' to the 'After=' directive in slurmctld.service (boo#1103561).- Update to 17.11.8 * Fix incomplete RESPONSE_[RESOURCE|JOB_PACK]_ALLOCATION building path. * Do not allocate nodes that were marked down due to the node not responding by ResumeTimeout. * task/cray plugin - search for "mems" cgroup information in the file "cpuset.mems" then fall back to the file "mems". * Fix ipmi profile debug uninitialized variable. * PMIx: fixed the direct connect inline msg sending. * MYSQL: Fix issue not handling all fields when loading an archive dump. * Allow a job_submit plugin to change the admin_comment field during job_submit_plugin_modify(). * job_submit/lua - fix access into reservation table. * MySQL - Prevent deadlock caused by archive logic locking reads. * Don't enforce MaxQueryTimeRange when requesting specific jobs. * Modify --test-only logic to properly support jobs submitted to more than one partition. * Prevent slurmctld from abort when attempting to set non-existing qos as def_qos_id. * Add new job dependency type of "afterburstbuffer". The pending job will be delayed until the first job completes execution and it's burst buffer stage-out is completed. * Reorder proctrack/task plugin load in the slurmstepd to match that of slurmd and avoid race condition calling task before proctrack can introduce. * Prevent reboot of a busy KNL node when requesting inactive features. * Revert to previous behavior when requesting memory per cpu/node introduced in 17.11.7. * Fix to reinitialize previously adjusted job members to their original value when validating the job memory in multi-partition requests. * Fix _step_signal() from always returning SLURM_SUCCESS. * Combine active and available node feature change logs on one line rather than one line per node for performance reasons. * Prevent occasionally leaking freezer cgroups. * Fix potential segfault when closing the mpi/pmi2 plugin. * Fix issues with --exclusive=[user|mcs] to work correctly with preemption or when job requests a specific list of hosts. * Make code compile with hdf5 1.10.2+ * mpi/pmix: Fixed the collectives canceling. * SlurmDBD: improve error message handling on archive load failure. * Fix incorrect locking when deleting reservations. * Fix incorrect locking when setting up the power save module. * Fix setting format output length for squeue when showing array jobs. * Add xstrstr function. * Fix printing out of --hint options in sbatch, salloc --help. * Prevent possible divide by zero in _validate_time_limit(). * Add Delegate=yes to the slurmd.service file to prevent systemd from interfering with the jobs' cgroup hierarchies. * Change the backlog argument to the listen() syscall within srun to 4096 to match elsewhere in the code, and avoid communication problems at scale.- slurmctld-rerun-agent_init-when-backup-controller-takes-over.patch: Fix race in the slurmctld backup controller which prevents it to clean up allocations on nodes properly after failing over (bsc#1084917). - Handled %license in a backward compatible manner.- Add a 'Recommends: slurm-munge' to slurm-slurmdbd.- Shield comments between script snippets with a %{!?nil:...} to avoid them being interpreted as scripts - in which case the update level is passed as argument (see chapter 'Shared libraries' in: https://en.opensuse.org/openSUSE:Packaging_scriptlet_snippets) (bsc#1100850).- Update from 17.11.5 to 17.11.7 - Fix security issue in handling of username and gid fields CVE-2018-10995 and bsc#1095508 what implied an update from 17.11.5 to 17.11.7 Highlights of 17.11.6: * CRAY - Add slurmsmwd to the contribs/cray dir * PMIX - Added the direct connect authentication. * Prevent the backup slurmctld from losing the active/available node features list on takeover. * Be able to force power_down of cloud node even if in power_save state. * Allow cloud nodes to be recognized in Slurm when booted out of band. * Numerous fixes - check 'NEWS' file. Highlights of 17.11.7: * Notify srun and ctld when unkillable stepd exits. * Numerous fixes - check 'NEWS' file. - Add: slurmsmwd-uses-xdaemon_-for-systemd.patch * Fixes daemoniziation in newly introduced slurmsmwd daemon. - Rename: split-xdaemon-in-xdaemon_init-and-xdaemon_finish-for-systemd-compatibilty.patch to split-xdaemon-in-xdaemon_init-and-xdaemon_finish-for.patch * remain in sync with commit messages which introduced that file- Avoid running pretrans scripts when running in an instsys: there may be not much installed, yet. pretrans code should be done in lua, this way, it will be executed by the rpm-internal lua interpreter and not be passed to a shell which may not be around at the time this scriptlet is run (bsc#1090292).- Add requires for slurm-sql to the slurmdbd package.- Package READMEs for pam and pam_slurm_adopt. - Use the new %%license directive for COPYING file.- Add: * split-xdaemon-in-xdaemon_init-and-xdaemon_finish-for-systemd-compatibilty.patch * slurmctld-uses-xdaemon_-for-systemd.patch * slurmd-uses-xdaemon_-for-systemd.patch * slurmdbd-uses-xdaemon_-for-systemd.patch * removed-deprecated-xdaemon.patch Fix interaction with systemd: systemd expects that a daemonizing process doesn't go away until the PID file with it PID of the daemon has bee written (bsc#1084125).- Make sure systemd services get restarted only when all packages are in a consistent state, not in the middle of an 'update' transaction (bsc#1088693). Since the %postun scripts that run on update are from the old package they cannot be changed - thus we work around the restart breakage.- fixed wrong log file location in slurmdbd.conf and fixed pid location for slurmdbd and made slurm-slurmdbd depend on slurm config which provides the dir /var/run/slurm (bsc#1086859).- added comment for (bsc#1085606)- Fix security issue in accounting_storage/mysql plugin by always escaping strings within the slurmdbd. CVE-2018-7033 http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2018-7033 (bsc#1085240). - Update slurm to v17.11.5 (FATE#325451) Highlights of 17.11: * Support for federated clusters to manage a single work-flow across a set of clusters. * Support for heterogeneous job allocations (various processor types, memory sizes, etc. by job component). Support for heterogeneous job steps within a single MPI_COMM_WORLD is not yet supported for most configurations. * X11 support is now fully integrated with the main Slurm code. Remove any X11 plugin configured in your plugstack.conf file to avoid errors being logged about conflicting options. * Added new advanced reservation flag of "flex", which permits jobs requesting the reservation to begin prior to the reservation's start time and use resources inside or outside of the reservation. A typical use case is to prevent jobs not explicitly requesting the reservation from using those reserved resources rather than forcing jobs requesting the reservation to use those resources in the time frame reserved. * The sprio command has been modified to report a job's priority information for every partition the job has been submitted to. * Group ID lookup performed at job submit time to avoid lookup on all compute nodes. Enable with PrologFlags=SendGIDs configuration parameter. * Slurm commands and daemons dynamically link to libslurmfull.so instead of statically linking. This dramatically reduces the footprint of Slurm. * In switch plugin, added plugin_id symbol to plugins and wrapped switch_jobinfo_t with dynamic_plugin_data_t in interface calls in order to pass switch information between clusters with different switch types. * Changed default ProctrackType to cgroup. * Changed default sched_min_interval from 0 to 2 microseconds. * Added new 'scontrol write batch_script ' command to fetch a job's batch script. Removed the ability to see the script as part of the 'scontrol -dd show job' command. * Add new "billing" TRES which allows jobs to be limited based on the job's billable TRES calculated by the job's partition's TRESBillingWeights. * Regular user use of "scontrol top" command is now disabled. Use the configuration parameter "SchedulerParameters=enable_user_top" to enable that functionality. The configuration parameter "SchedulerParameters=disable_user_top" will be silently ignored. * Change default to let pending jobs run outside of reservation after reservation is gone to put jobs in held state. Added NO_HOLD_JOBS_AFTER_END reservation flag to use old default. Support for PMIx v2.0 as well as UCX support. * Remove plugins for obsolete MPI stacks: - lam - mpich1_p4 - mpich1_shmem - mvapich * Numerous fixes - check 'NEWS' file. - slurmd-Fix-slurmd-for-new-API-in-hwloc-2.0.patch plugins-cgroup-Fix-slurmd-for-new-API-in-hwloc-2.0.patch: Removed. Code upstream. - slurmctld-service-var-run-path.patch: Replaced by sed script. - Fix some rpmlint warnings.- moved config files to slurm-config package (FATE#324574).- Moved slurmstepd and man page into slurm-node due to slurmd dependency - Moved config files into slurm-node - Moved slurmd rc scripts into slurm-node - Made slurm-munge require slurm-plugins instead of slurm itself - slurm-node suggested slurm-munge, causing the whole slurm to be installed. The slurm-plugins seems to be a more base class (FATE#324574).- split up light wight slurm-node package for deployment on nodes (FATE#324574).- added /var/spool/ directory and removed duplicated entries from slurm.conf- Package so-versioned libs separately. libslurm is expected to change more frequently and thus is packaged separately from libpmi.- Updated to 17.02.9 to fix CVE-2017-15566 (bsc#1065697). Changes in 17.0.9 * When resuming powered down nodes, mark DOWN nodes right after ResumeTimeout has been reached (previous logic would wait about one minute longer). * Fix sreport not showing full column name for TRES Count. * Fix slurmdb_reservations_get() giving wrong usage data when job's spanned reservation that was modified. * Fix sreport reservation utilization report showing bad data. * Show all TRES' on a reservation in sreport reservation utilization report by default. * Fix sacctmgr show reservation handling "end" parameter. * Work around issue with sysmacros.h and gcc7 / glibc 2.25. * Fix layouts code to only allow setting a boolean. * Fix sbatch --wait to keep waiting even if a message timeout occurs. * CRAY - If configured with NodeFeatures=knl_cray and there are non-KNL nodes which include no features the slurmctld will abort without this patch when attemping strtok_r(NULL). * Fix regression in 17.02.7 which would run the spank_task_privileged as part of the slurmstepd instead of it's child process. * Fix security issue in Prolog and Epilog by always prepending SPANK_ to all user-set environment variables. CVE-2017-15566. Changes in 17.0.8: * Add 'slurmdbd:' to the accounting plugin to notify message is from dbd instead of local. * mpi/mvapich - Buffer being only partially cleared. No failures observed. * Fix for job --switch option on dragonfly network. * In salloc with --uid option, drop supplementary groups before changing UID. * jobcomp/elasticsearch - strip any trailing slashes from JobCompLoc. * jobcomp/elasticsearch - fix memory leak when transferring generated buffer. * Prevent slurmstepd ABRT when parsing gres.conf CPUs. * Fix sbatch --signal to signal all MPI ranks in a step instead of just those on node 0. * Check multiple partition limits when scheduling a job that were previously only checked on submit. * Cray: Avoid running application/step Node Health Check on the external job step. * Optimization enhancements for partition based job preemption. * Address some build warnings from GCC 7.1, and one possible memory leak if /proc is inaccessible. * If creating/altering a core based reservation with scontrol/sview on a remote cluster correctly determine the select type. * Fix autoconf test for libcurl when clang is used. * Fix default location for cgroup_allowed_devices_file.conf to use correct default path. * Document NewName option to sacctmgr. * Reject a second PMI2_Init call within a single step to prevent slurmstepd from hanging. * Handle old 32bit values stored in the database for requested memory correctly in sacct. * Fix memory leaks in the task/cgroup plugin when constraining devices. * Make extremely verbose info messages debug2 messages in the task/cgroup plugin when constraining devices. * Fix issue that would deny the stepd access to /dev/null where GRES has a 'type' but no file defined. * Fix issue where the slurmstepd would fatal on job launch if you have no gres listed in your slurm.conf but some in gres.conf. * Fix validating time spec to correctly validate various time formats. * Make scontrol work correctly with job update timelimit [+|-]=. * Reduce the visibily of a number of warnings in _part_access_check. * Prevent segfault in sacctmgr if no association name is specified for an update command. * burst_buffer/cray plugin modified to work with changes in Cray UP05 software release. * Fix job reasons for jobs that are violating assoc MaxTRESPerNode limits. * Fix segfault when unpacking a 16.05 slurm_cred in a 17.02 daemon. * Fix setting TRES limits with case insensitive TRES names. * Add alias for xstrncmp() -- slurm_xstrncmp(). * Fix sorting of case insensitive strings when using xstrcasecmp(). * Gracefully handle race condition when reading /proc as process exits. * Avoid error on Cray duplicate setup of core specialization. * Skip over undefined (hidden in Slurm) nodes in pbsnodes. * Add empty hashes in perl api's slurm_load_node() for hidden nodes. * CRAY - Add rpath logic to work for the alpscomm libs. * Fixes for administrator extended TimeLimit (job reason & time limit reset). * Fix gres selection on systems running select/linear. * sview: Added window decorator for maximize,minimize,close buttons for all systems. * squeue: interpret negative length format specifiers as a request to delimit values with spaces. * Fix the torque pbsnodes wrapper script to parse a gres field with a type set correctly. - Fixed ABI version of libslurm.- Trim redundant wording in descriptions.- Updated to slurm 17-02-7-1 * Added python as BuildRequires * Removed sched-wiki package * Removed slurmdb-direct package * Obsoleted sched-wiki and slurmdb-direct packages * Removing Cray-specific files * Added /etc/slurm/layout.d files (new for this version) * Remove /etc/slurm/cgroup files from package * Added lib/slurm/mcs_account.so * Removed lib/slurm/jobacct_gather_aix.so * Removed lib/slurm/job_submit_cnode.so - Created slurm-sql package - Moved files from slurm-plugins to slurm-torque package - Moved creation of /usr/lib/tmpfiles.d/slurm.conf into slurm.spec * Removed tmpfiles.d-slurm.conf - Changed /var/run path for slurm daemons to /var/run/slurm * Added slurmctld-service-var-run-path.patch (FATE#324026).- Made tmpfiles_create post-install macro SLE12 SP2 or greater - Directly calling systemd-tmpfiles --create for before SLE12 SP2- Allows OpenSUSE Factory build as well - Removes unused .service files from project - Adds /var/run/slurm to /usr/lib/tmpfiles.d for boottime creation * Patches upstream .service files to allow for /var/run/slurm path * Modifies slurm.conf to allow for /var/run/slurm path- Move wrapper script mpiexec provided by slrum-torque to mpiexec.slurm to avoid conflicts. This file is normally provided by the MPI implementation (boo#1041706).- Replace remaining ${RPM_BUILD_ROOT}s. - Improve description. - Fix up changelog.- Spec file: Replace "Requires : slurm-perlapi" by "Requires: perl-slurm = %{version}" (boo#1031872).- Trim redundant parts of description. Fixup RPM groups. - Replace unnecessary %__ macro indirections; replace historic $RPM_* variables by macros.- slurmd-Fix-for-newer-API-versions.patch: Stale patch removed.- Use %slurm_u and %slurm_g macros defined at the beginning of the spec file when adding the slurm user/group for consistency. - Define these macros to daemon,root for non-systemd. - For anything newer than Leap 42.1 or SLE-12-SP1 build OpenHPC compatible.- Updated to 16.05.8.1 * Remove StoragePass from being printed out in the slurmdbd log at debug2 level. * Defer PATH search for task program until launch in slurmstepd. * Modify regression test1.89 to avoid leaving vestigial job. Also reduce logging to reduce likelyhood of Expect buffer overflow. * Do not PATH search for mult-prog launches if LaunchParamters=test_exec is enabled. * Fix for possible infinite loop in select/cons_res plugin when trying to satisfy a job's ntasks_per_core or socket specification. * If job is held for bad constraints make it so once updated the job doesn't go into JobAdminHeld. * sched/backfill - Fix logic to reserve resources for jobs that require a node reboot (i.e. to change KNL mode) in order to start. * When unpacking a node or front_end record from state and the protocol version is lower than the min version, set it to the min. * Remove redundant lookup for part_ptr when updating a reservation's nodes. * Fix memory and file descriptor leaks in slurmd daemon's sbcast logic. * Do not allocate specialized cores to jobs using the --exclusive option. * Cancel interactive job if Prolog failure with "PrologFlags=contain" or "PrologFlags=alloc" configured. Send new error prolog failure message to the salloc or srun command as needed. * Prevent possible out-of-bounds read in slurmstepd on an invalid #! line. * Fix check for PluginDir within slurmctld to work with multiple directories. * Cancel interactive jobs automatically on communication error to launching srun/salloc process. * Fix security issue caused by insecure file path handling triggered by the failure of a Prolog script. To exploit this a user needs to anticipate or cause the Prolog to fail for their job. CVE-2016-10030 (bsc#1018371). - Replace group/user add macros with function calls. - Fix array initialzation and ensure strings are always NULL terminated in - pam_slurm.c (bsc#1007053). - Disable building with netloc support: the netloc API is part of the devel branch of hwloc. Since this devel branch was included accidentally and has been reversed since, we need to disable this for the time being. - Conditionalized architecture specific pieces to support non-x86 architectures better.- Remove: unneeded 'BuildRequires: python' - Add: BuildRequires: freeipmi-devel BuildRequires: libibmad-devel BuildRequires: libibumad-devel so they are picked up by the slurm build. - Enable modifications from openHPC Project. - Enable lua API package build. - Add a recommends for slurm-munge to the slurm package: This is way, the munge auth method is available and slurm works out of the box. - Create /var/lib/slurm as StateSaveLocation directory. /tmp is dangerous.- Create slurm user/group in preinstall script.- Keep %{_libdir}/libpmi* and %{_libdir}/mpi_pmi2* on SUSE.- Fix build with and without OHCP_BUILD define. - Fix build for systemd and non-systemd.- Updated to 16-05-5 - equvalent to OpenHPC 1.2. * Fix issue with resizing jobs and limits not be kept track of correctly. * BGQ - Remove redeclaration of job_read_lock. * BGQ - Tighter locks around structures when nodes/cables change state. * Make it possible to change CPUsPerTask with scontrol. * Make it so scontrol update part qos= will take away a partition QOS from a partition. * Backfill scheduling properly synchronized with Cray Node Health Check. Prior logic could result in highest priority job getting improperly postponed. * Make it so daemons also support TopologyParam=NoInAddrAny. * If scancel is operating on large number of jobs and RPC responses from slurmctld daemon are slow then introduce a delay in sending the cancel job requests from scancel in order to reduce load on slurmctld. * Remove redundant logic when updating a job's task count. * MySQL - Fix querying jobs with reservations when the id's have rolled. * Perl - Fix use of uninitialized variable in slurm_job_step_get_pids. * Launch batch job requsting --reboot after the boot completes. * Do not attempt to power down a node which has never responded if the slurmctld daemon restarts without state. * Fix for possible slurmstepd segfault on invalid user ID. * MySQL - Fix for possible race condition when archiving multiple clusters at the same time. * Add logic so that slurmstepd can be launched under valgrind. * Increase buffer size to read /proc/*/stat files. * Remove the SchedulerParameters option of "assoc_limit_continue", making it the default value. Add option of "assoc_limit_stop". If "assoc_limit_stop" is set and a job cannot start due to association limits, then do not attempt to initiate any lower priority jobs in that partition. Setting this can decrease system throughput and utlization, but avoid potentially starving larger jobs by preventing them from launching indefinitely. * Update a node's socket and cores per socket counts as needed after a node boot to reflect configuration changes which can occur on KNL processors. Note that the node's total core count must not change, only the distribution of cores across varying socket counts (KNL NUMA nodes treated as sockets by Slurm). * Rename partition configuration from "Shared" to "OverSubscribe". Rename salloc, sbatch, srun option from "--shared" to "--oversubscribe". The old options will continue to function. Output field names also changed in scontrol, sinfo, squeue and sview. * Add SLURM_UMASK environment variable to user job. * knl_conf: Added new configuration parameter of CapmcPollFreq. * Cleanup two minor Coverity warnings. * Make it so the tres units in a job's formatted string are converted like they are in a step. * Correct partition's MaxCPUsPerNode enforcement when nodes are shared by multiple partitions. * node_feature/knl_cray - Prevent slurmctld GRES errors for "hbm" references. * Display thread name instead of thread id and remove process name in stderr logging for "thread_id" LogTimeFormat. * Log IP address of bad incomming message to slurmctld. * If a user requests tasks, nodes and ntasks-per-node and tasks-per-node/nodes != tasks print warning and ignore ntasks-per-node. * Release CPU "owner" file locks. * Update seff to fix warnings with ncpus, and list slurm-perlapi dependency in spec file. * Allow QOS timelimit to override partition timelimit when EnforcePartLimits is set to all/any. * Make it so qsub will do a "basename" on a wrapped command for the output and error files. * Add logic so that slurmstepd can be launched under valgrind. * Increase buffer size to read /proc/*/stat files. * Prevent job stuck in configuring state if slurmctld daemon restarted while PrologSlurmctld is running. Also re-issue burst_buffer/pre-load operation as needed. * Move test for job wait reason value of BurstBufferResources and BurstBufferStageIn later in the scheduling logic. * Document which srun options apply to only job, only step, or job and step allocations. * Use more compatible function to get thread name (>= 2.6.11). * Make it so the extern step uses a reverse tree when cleaning up. * If extern step doesn't get added into the proctrack plugin make sure the sleep is killed. * Add web links to Slurm Diamond Collectors (from Harvard University) and collectd (from EDF). * Add job_submit plugin for the "reboot" field. * Make some more Slurm constants (INFINITE, NO_VAL64, etc.) available to job_submit/lua plugins. * Send in a -1 for a taskid into spank_task_post_fork for the extern_step. * MYSQL - Sightly better logic if a job completion comes in with an end time of 0. * task/cgroup plugin is configured with ConstrainRAMSpace=yes, then set soft memory limit to allocated memory limit (previously no soft limit was set). * Streamline when schedule() is called when running with message aggregation on batch script completes. * Fix incorrect casting when [un]packing derived_ec on slurmdb_job_rec_t. * Document that persistent burst buffers can not be created or destroyed using the salloc or srun --bb options. * Add support for setting the SLURM_JOB_ACCOUNT, SLURM_JOB_QOS and SLURM_JOB_RESERVAION environment variables are set for the salloc command. Document the same environment variables for the salloc, sbatch and srun commands in their man pages. * Fix issue where sacctmgr load cluster.cfg wouldn't load associations that had a partition in them. * Don't return the extern step from sstat by default. * In sstat print 'extern' instead of 4294967295 for the extern step. * Make advanced reservations work properly with core specialization. * slurmstepd modified to pre-load all relevant plugins at startup to avoid the possibility of modified plugins later resulting in inconsistent API or data structures and a failure of slurmstepd. * Export functions from parse_time.c in libslurm.so. * Export unit convert functions from slurm_protocol_api.c in libslurm.so. * Fix scancel to allow multiple steps from a job to be cancelled at once. * Update and expand upgrade guide (in Quick Start Administrator web page). * burst_buffer/cray: Requeue, but do not hold a job which fails the pre_run operation. * Insure reported expected job start time is not in the past for pending jobs. * Add support for PMIx v2. Required for FATE#316379.- Setting 'download_files' service to mode='localonly' and adding source tarball. (Required for Factory).- version 15.08.7.1 * Remove the 1024-character limit on lines in batch scripts. task/affinity: Disable core-level task binding if more CPUs required than available cores. * Preemption/gang scheduling: If a job is suspended at slurmctld restart or reconfiguration time, then leave it suspended rather than resume+suspend. * Don't use lower weight nodes for job allocation when topology/tree used. * Don't allow user specified reservation names to disrupt the normal reservation sequeuece numbering scheme. * Avoid hard-link/copy of script/environment files for job arrays. Use the master job record file for all tasks of the job array. NOTE: Job arrays submitted to Slurm version 15.08.6 or later will fail if the slurmctld daemon is downgraded to an earlier version of Slurm. * In slurmctld log file, log duplicate job ID found by slurmd. Previously was being logged as prolog/epilog failure. * If a job is requeued while in the process of being launch, remove it's job ID from slurmd's record of active jobs in order to avoid generating a duplicate job ID error when launched for the second time (which would drain the node). * Cleanup messages when handling job script and environment variables in older directory structure formats. * Prevent triggering gang scheduling within a partition if configured with PreemptType=partition_prio and PreemptMode=suspend,gang. * Decrease parallelism in job cancel request to prevent denial of service when cancelling huge numbers of jobs. * If all ephemeral ports are in use, try using other port numbers. * Prevent "scontrol update job" from updating jobs that have already finished. * Show requested TRES in "squeue -O tres" when job is pending. * Backfill scheduler: Test association and QOS node limits before reserving resources for pending job. * Many bug fixes. - Use source services to download package. - Fix code for new API of hwloc-2.0. - package netloc_to_topology where avialable. - Package documentation.- version 15.08.3 * Many new features and bug fixes. See NEWS file - update files list accordingly - fix wrong end of line in some files- version 14.11.8 * Many bug fixes. See NEWS file - update files list accordingly- add missing systemd requirements - add missing rclink- version 14.03.9 * Many bug fixes. See NEWS file - add systemd support/sbin/ldconfig/sbin/ldconfigs390zl38 170663131522.05.1122.05.11-150300.7.9.122.05.11-150300.7.9.1libslurm.so.38libslurm.so.38.0.0/usr/lib64/-fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -gobs://build.suse.de/SUSE:Maintenance:32297/SUSE_SLE-15-SP3_Update/7fd5e89bcc53d38ed4c5fbb0acf60be8-slurm_22_05.SUSE_SLE-15-SP3_Updatedrpmxz5s390x-suse-linuxELF 64-bit MSB shared object, IBM S/390, version 1 (SYSV), dynamically linked, BuildID[sha1]=ea99288328bbfa791dd38c98f129a299fcc61b41, strippedPRRRRRRRR RRR R RR RRR RRRR 0coutf-849e4445f67ecd1a7d5adb875183d43ff64be3c536504100bcecc564d3b20b9e6?7zXZ !t/ ]"k%{;~=GuV<ܖO"WTb_M} TNB8\K]u!g&]HU 9Y/=f5Cd:Zr`#$$ecs Eb AfW89Շv&ցh7sRT|87 Z{n5\&.8q88g,RM@b @x6eH4*}h7xb83e~ٛ[U[Kq kLs}eYc< &q4(d|ªg( Qfdj!j~UgkKp J`e{xFsB> QFZLH<;n+CCBw:y Mh hysM uF 4/wn`;/7BCr g=\ K;҄-@AL۷B? m>R3Yq[꾑J.Ų~ֵ 7$$Qg:]/6C zF!|ET# y .D uR<' .5(#fb~(TU*X z@9:v[u|]PFhJ;ebwY}L>[gGuI_|0'[qF! +I6we 8ѫ$;G-I(>lCn,IC; Q)ur?y:KqL~9+ WӔJH#z,`5=HMcrڿ 35 굲tQ,dcfc4 SԉJaw!AǪj(2zSFMWUQ* p@4:l&dv!DOA;tٸط^AB9\ύ)#&|l5 ikԠUmF7dOs7g甁 Ѥp {8G%V?==b9Pp/ )PHCv V3$_K5P!y%N7z]sA,We-Pamkuho17^ݛl^eb,/d{F bAa9;x rzn;YlI ;aM4wHϥkjo<]㟥D %6Q޸C :&@Ƞ -%[*F`;qp x o%YܭMfWg [gIh(HGrT1JQtĘy|<(ΡWaI0NwN >ZĮH< neq9ĎPn_NX!^:wtR5>U"8 f_30l|iL2pΣ`-/6'ea|Y Em%Lwq1QË٨h*{]U>4rR+!kua vL$E Wgʴ=-u~ B(#Q =z ^jr=vG2)cP^[!cu ]|; b n z KHɰq?x Eh0A,E'mٕlAaύ2/-${N.NA!Cnl+[[Uk4kQi6}l/B'?%fFpFޗ38,EOs_i`/}7}E0bLWf: S{*I5W4h5O;:Ԫ(C:H"׵(`OjJGA{$z}'XF{Aw0h]2PYEuJU=:}\–lS 񱀫Z ܤd#=ַܪ[)SL 2j ?//ATFyHeRp6%8j1ky(q`z(E^qۃ| `?T7Xk`40Dڜ}p{TVrh/hI&8Rށ1Lxsp) Œt>2K[БF ': n|WkődHj 6s mUd!EBaZ6AՁaM爾D }| H~erzZ6!@e)y7[2pb\M,<Ԙ !|$2j\j֌5ISx  [B{=ضt]d5D(O/ u"VSfsFn;:\ JQbϵ,h&rpJzp\-mY@qmPc4PbM`_ 6~2hdqY4IQH$Lx g%B htЭ>>˯$iP 4bJR~Ōޡ=cibf7Ae=k%S)']9V#V#.[Lp x6YQaf$ymA Ealc~B ޭCYL馊4^I;"@JK\,+Vnel0fӊ)& //r5eu Ha /bّxQW _"L+Q ,`_^j.3֞d*핀j\ls!?:;~ Wm{Y<3ZKF+I[D" ?k r2x(Sl8q]hRLw@>/b>K{z2qF`l#Oʆ2ёʐ;zĘ(:xďYS˖L DްFaXI eCYcIOKQwznN`|B %*+3 5{I{SFB$|6?G~4adeR +qz dՑ&PrM#,~x["Q]bfYn#X}/ɢ<1 "{,b!Y͚ i;옏Ԑ&L|, [q'xW\i. 5w.nr*Ρ4i$Mc5ٵ.S*;+ Qϫ A_d2t9bQ̷oMRP⥾pG6SkFh tj3Hd[TVQ>3y =И2;~?ca].P `u}0;) )+Ջv6Q *t;So"/݆qtpwwc\nǁ@5>]3-vFP凊^!YWv'~cHijᠨI6͓t׼>d7qKHMyf _[dG . g$.仩x|CCjTƏPwwϳuBg1o~4$#.)C?PhB !cw+1ƨ5o~\޺TqL(8!o=C.X5_#߯sr/)Jf5&dXL&LqnGzs q+ pjE +3wkYmh5dg2| '`=$w.nv .7Dl8!5uًꮈP .O~l x! ܟ'4"09A8cRNOXӄ7.+ŝac2fIw"ޞHlY{M)uO=X@`QpSP#Ym"6;I< {Z~tUp,+6}-ބ@!cVFiWӽʡ1M14TIsr97^²W ~y ׽C*}@knF?Zk aw %*m&᷻`b5o8c7,6mN[W yЊ3ğ x[: <*R o11A`=M~e`f/7GP`Ak7Y%E50 `JK3zLz秦'Gf%hb6`&%[f  uKoMd"$m=&ߐxl!gILMu.4Thy>DҶ*뺕7葒? !w hNB7S;xۅdm'38OI[!?I#0lW HU1f' \KpFNXa+KIӭ qMW#7Lzͮ342[)~]Opxue@~\ A93s*e ,Ʂ8c)M1 ƷfJPaȺBA|<8D@.MM"@E R5i%)n@ CE˾,f cxRD Ơ K\+kŶ8lhr)ڢĤ>łg,u0bsTF7*᾿ P)\;0GZn@1Ce,APZV&=SI ̈yup#+ Y?IWC ";:+7_PCIOd$rqriU LhY>r,pc nS}}+ 4ⱱS|T+ LW3_@(a3?Bq?&ewV}0|ɞњQxBxցX%wX1io-(mTt-Ȓ@zLS[*ts; WM3WL 6qq9L؛uP2(Gg9@PǁWtS>ѼNØ%8LRa.gX?kڍ6Vskٕ-)"742R(ʮAe3˰ISaN{!ΘbE"JA~k G kUefYSD# D~!^ x,";60P~.4$ ?ݲRO.Ӻt:j-  Q3`D4f1/%(=F`ComvjbTMe`oXB Vv wqmxQ5\z]mL=0<~Q),Q;ƚi7Pf0ժԍk1WKVpy o9؇&`P+;cOk(PKӽ5EY<~]'Cq; “CuxqWHdfnxDZ/'dpΓP7\-K :'9bY\f␥SYh=emI4oZfGHo @hjVS E7G("׶,]y,Ʃ] eHS7`ScŠJzғ5TՅM`69p :ľ:TqLvZsf;P]TfyUemmԪڿquj3/ <]??:W*)}cG (9 kͱҒ/AEř0(23P/}Pk}na>["=z36]).N0:uj8hOF P\jBES9;+ f 󋂞d fLs'ZN-E6FD2vgkI5GfnNk-ʉ?(=Av%WpG'4v1 M;gE胟"sȈV(a߈aR2"33f%?@ #lM> )Ά~hV9O468ɳ z ;"Ԩj߱;ڝ?Dg}@ AJXL}O7ѫ(׾\LƉ &M }3R]e1q %*(½W%=a&e3ljsIUq\ASAyQς|vo Q7g7 [: aڄ"ր4N+ s0)VOM/=(&H%D%Fsxz{jXr{PBk 2Nujrdlw2[J]xm{ {5v@wA֠X8ةObo 6+uG9#`:q'4'tI|'{AN !4>ʏO>BKU)Fd޽-[ef~L40J>qLӂgc[)EPrq/Mُp\be.Xm}K5RI/r+Ǡz3I|Q{3MBErA^RW3Y; c@IZ!PLx 60.5,&zˉm6e紣Β.1I%${LA_ w|\4-iYyJs R1Ӄ8u$7FRKߠ~Oy3"lׂ,=bV ]^u 0Jx64O`1Fm!yyGWuO2Nq\R;^ft;h: [ [N&~.3r=# T'T[S39l L"7?M'ҩ zYOA3tjcO|R!} 9=d B/9#NuPFuK?{W۸M;_N'>nΐ<#JǟB t~ XE,ᆯfbF$]z_TbMroѠs]{R+LzkŚrѲ)بJyRe1.їc#mZ |IG _mJwLd@.-mR"_bLu #7៞ic P>Z~P'Z_dE)\ij'PSi +-[ wjM'k>ǎZ ]Wg"Uӵ+F-~]侮ԭp^אG_\9e{>8a@kf9,6+lR尒QT<_]TOe 3na!&g|Љ@yQɎ@T) E6T1"NHiE9GwsUhe\}+~7BJ=Hn{`U Z3{ hhY/N&L~u x.ѹ0p{R +HzFhЁzڅ}d\q@n"OF|ؕxM4|pqd 4+׹{l) t*7nN7q9C!@\jxTnc~DƜ*:}yNEjz"bcNߪn6wxu(Li@;U x5Z ZcU&t4tk HJ7n>w/\+:leꉣlCD% hAo.DL\"㊻(HGϣi~usrd/DzWw\A'H\숚^]5{KE?Hs}6e`NOCe=LG`vhw'9l[Rv0LJU .6ӻYHEwL-Y?b砯3_Yri^dԶN'={A1wNp( Tc;:Ҙe1  f7$P3K"wҤﾼk3>}|K[F)͕2[ >w K{`9EV˹C^DjA2$%!nlڠj&8Mh~cr(q۞&1] L#B?5E)]3>z;*)m#-S_u32_q͹x;okFn&H#R%?A*F>#Ӵ*Os]K#)>*Pn~-drrZ_cĂ[u@ݥY棨sK/̏|q7#se9]H9pXNMpK0&h (_Yw' i6^NĨ&lU?A]+=] ;mma]_ɪ.Fv?6\ $T"zj]d_xX]j\g ޥNeA$ej CN./S4plKyp'!9 &jߵ>DD*{zү7EtKKvGlZWomZ˺ab8l}G'Qk6MPdCs3𣃺t=upk¹SUpJ>{9hH&@{Էsw9u vVUX҄Oj?Ql/rMjf}Tcש5&.➣Oׯ_EfF_0DF] Υ_?@b߄NIG]1pEaڠUt Gez/+hTVUOk^ljD!])s2,@^*B +MAYd~4 (x,w.uA%$0>uBozU)Bk>Wϝab^N^SpCWZ8GHkφp~;!m)kpR u4^@h9{qb WS`-i7gWh#@[YKcIclMBc;(pN+:م64w; wS!,)dLF,A(g6^8~]DTD/ի?mPw&zEN܌l$8Gގ$9(hm(h([jS3bLw~5ll3+O7U*~`!c4 ;9DJeUbQ!G۷ [ךgn JMl J-78f,;(_KI= rMrāܙ"Kc_0j Px aC:c E8נCfkt핖OX67,FDElƁ&@-LR [~5!IƧu|/nyWXR\zl* ͫ wJi[ RKϥ];A f2 )saB{}_T]jm3`Z S]aҞ>P3+ ^s<wi,>:߂bg> "5TqĪ}b:hq2Y7nUDlhK{+ۘYj8J5?5+`7)3!F0ycmU|O]g_E2:$z6\q]z'})ye%DRopXv)'J~F$6YOՁX2n=:?sUa.>EoTV;M5[̀8#*>͚Ku~ǪKXc$A߿IRK.ڄ쁀&Yh/%v.Nv z6Vg>Ü[4xʪ6DlBā8T@WR"%eKMqNڨ r|3%zFgnϒwIaJd^SG\,*6Bʓ#Cz8}u[hvjҡY5ܶ~ۘ\[W Vqc%Nav:6tOՊUa^HD 9ئI ΦT&!T=w4aZ&Kؓ-9?S="wm!^x/8E3}tXAx(// (Ae n"slT$uqU3JAnn[bgs˓>ZaZ!..hqj:e7.懠@lmd@С5[[/H'^#gK>>݋߹ޚ͘Qdrd\wqm >Dc$GREC=>a!㛥a$ߕWY3e&K9 0u+ⱄt=ys|q^K4O4؊ م0a4umlK'p&P$r΃n<9C\({ZEy twz x!,>hrUuEwcCNa,S̉(C*uSݤ6wS̥xPldMl΀E4 b6khVDrǵ\ Ȩ>0DTU !vzJ:j񹂃m01- fH"/=3_vTcPsi4sRZ_QhAr4ICKƒV >7d˸KJS~3iGRqP -aF̄zA<&/lTT}vb 1C2>'Wui+RDL0Kw힤SF`1> 7s&jb۲⓴X=Owu%S恦YW<˜"bd"Wk!ķAO~(\N9ԇD.j`ڑC>r?v9r$Ϥa@fᠲሣBȜozzs|%_̘L4]L|a*‚}y30Ln $'T7z]ZNQṡ37){<HP-a7-LWݸ삛{8SVTCVw=3b\1ͧDD?Fcvz>$G Q˞( zkm=sف(L5XƷ(eٵ!qGAFԩ}?K#xH-g12 \F[~mHcg壚 ^I||`|.J̵˰QC硍DVr V e LIcԑțf>YJ E$:V@g{HWo8!D[8OFZ^̳/H gYcݯh!E]7rN#@ktw5 ZFB;b 鸦%BXDm?יla:q`LZsJa?d! dvVHd0tQg3^2,~u-:+$)/uCRG3ۉ{пa׿$ /==P[9Wt$j,|  ;=LCEk{$v%UAçQJbϜF9 pL93\CJ0k!r5n/NgB @r^l.$8T/`!տwQTd-/YG@E/,HJ^Ơo5"w}w2?̌T\HQ / uL!3$ Eu IJXSvq/qĊ)PJdֵ䂣Eš;X7EvT% cWnFdB]f? qNފkMp S o솈9a18AE|F0n7ZST>'hOCՖ JJLeIt^,OD[?fNI?@n.ͿCJ_?@ka Þ#Dgb-v_ob>BN2eirןaYU^̩3Q%[toQEy/ʰNyOH-JJ'_#}\o[^9GfA^ھcݥyLhs 5ƸKƆ*RHԽEm+~y&v2$%d 2$(/e4FfbR$#BQ$3#h#61{C.&J.+ u<Ad@oxTAi&wD>Eug4F0f0q ?F9z? 5UΚ|h< %sRcs&7 ]9AzCZ dOjጽE+A\t?ȈzH鞁a!dNN Ldo9SqOjQ5oלOH(p9JmT'c+$Dye 7Zfߚ x֕ ;kygX:VlMgxO\GF# $_O%p#'ַ{;ƨ`rRHV%`;(LcDH %aj2S79lTD$F`V]z&/R3G-<#@NRҾrwPQn0*[`5(3F%?q<"*,zyxD(A 2/Q{GٕπT:JgB;)Y-#ǵrRWJ7ƛnbVg^oz;ݬy]t&綶krǎFEЃ3m]㈿6p]ӋdHC~7S備s8X!_Xh/|MdRU|qf0$ n;mg hLidgzBDJO7yI2gv[lzxH:~,[r0W r] t9|{[_ow ^ B(gW"Qn09l, YJLS({ID,B FH F]vOT3Âڿs 8l֣HCPG595> ђWE!:y@lV/]S%%T-֮jʳ_F {+)h*.UzPf{tFm?R8?dzh|SOͱe !/x}_Cofς n:O܈6I)P?nfhG3C@h@$P Wev!/h0Bq6%,`X@ě㽅=s3IK4V$mKnݖªf@ȈƺcQsF#F* LNiGgܩi|Zu91f!,hׁAY]X@J}!"w \䢽/ E5~a|D8$tuF/.02kpiRT*O"zޑkUJdR҃lH(^л5ngW?-yz B^HoŒ*+Sa.iziP9^zٲP hak^F|q}Be !֝ 1q./ &UNS N-;T4i6\X]|_7nN+uʇuxnZgw/3_.WFk%)ۼ[[8S/bX,jCZl e?[vlX#(SoٮĂw,Al_1憪mj"6):r2)_T&9*w\DXi#mbAB_,*8ِex̆.\؊|32OebOO@"rӡ.a}ҧW"[Zɒ2?~"Hڑ6GgZʶ!W8f/\1߸?2`Vк62"?W1ݶo6xA!QN{$:]^h{ -}$CJNٜJ%`o/ET_ KԱK`o&q0\'R0n{VDD:uG'zHg2lK@FxF@T64xYYU7b[EL/`>M(#8S.s{liWV8MfJ8xwEJ',9G'VpREi4,ͪ4]pwA .]فTnMLBν+hzgO*JLJW|l|3ȾfV%6_6w#l{N,]+ G1"Yzήf~T"(yTa7xN5|Moɋ6 sfA;g²,xJrK5 EYr)còB9+}Xfsv[P|uZf; 4⺨㐡%/d6~4 ax.cy"=7&~YugL{8yvi /ST~6~aѳ'|\(>oU;hBx{ ąQ۷D3Jt* Zn =) ;tIVҔZjƾ3lOcF,cI{3^Qʼ˞64 ^jwN&rJ!H+5Fi5_26dudlB}iEj&Ϣ}8D7ҫ< R\`!p-r!螙Ϊ&mpˀV'族$a\^ݩ!q X݀Ʒ09ֵ$S97}f,yߧia]G}? +y11>(FWO*I[Z4z`@.n= +;I$=R@%mQL#7-򅍝-' <5ГQ ;Ƹ-mM ۘ/va[쁟?)˒2UWэBf[/qEr! 2;k#P}2Y[$xxJGTy, 9'Z]:|v^~}%@ExF u~,Aϖ@) [) F +{SgWOH0{S'U7e⇔l?+;|{n7,L^%-[HX$fnR9 NkW<V?R.X^L^({i0{wU'xmY =oZRtОGq@0]D#CNWfD n:.\ubPjaoKnotMf}X\#۠+}Am&=j.tUڌf enaýK^\;n\q<ρAVJQǚ:n˳y1e{ 1zȊF>ysbAV P 𠉓);c"`@.b+V_ˊ~~z3DAxpRݭ>3X1؆Ov]V+YĒ{~}|kM(XXWGv&p4-O_:|ߗBs| ^!CLȬw)av-~cpG2 k:bUE@I]~6SK7^ 9rP] *#-R ]Bås[c tF1R+:9͡O=`ݡN{?[iYA`pDԯk/OӯKW׼w+P_a^㋹X2z ^E6k/ ͞=pS=u$=߯eqyެWh0Ud``!Haal^Hf#.4&F<2$dZJu.\H'։nh64e7yap #Loj aM Gbn QB(G_Ԧq"(^J0M?0ƶ n&#D >D#Raœ/RFr k^E۪9d=%q)wѢx3_eԁ:;\YPŏT)/ x/MC]g5Foffp]/qUBP[Y"-o;6.p$6Ir';6L%g$p 0LfֶA1!ZbրK'8왕ĵvGm@#|ck3rag- XȲ` є7} 2)(zaC*egQ!Rh7dϙ "Tl{\v ȗkbd0Wܣ-Y+c=려MMoDU ='@Q'@QU6!UFg DDcͺ>\T2ˡ-)nx(2(L[L lm!!IfS&@/n6GoQS6W~]n;Xe|;F?gs [!;՘!W՘9el6" _CS@Y2 pw;SVEv[_)@6&Sh8oH4(E8aRf94N.mu,S,Jgݴ'ȓUoj1\6emi^;[Xpd!FIW\fDyBB5Y:.V[ཊ6k`oyЖ@w/U? X-Y#5%P"7e/)ݶBz+~qDul܊__]҆ Qf5X'%Î \]L/?N )98@ G8a.Kv&^ZK[3N1%|w\:P\=u+7Ro>:dUD43$ ߨ O4ھ69`qjŭ#ıQo[^ᅗ N|wb~"VP NBG;4]AɑiˈiZ6Q̠|n )Ruku_?y4ޱ/>UXGhlO# Dx~J[⍵@CâN6ɀ7 ($/aZ[ BԄ^oDc# Vqrڏd;dow;`g`Wz2ZAXwP>%jon :,5ĈN y H7y(n8k7=}8SW=@&"Z>7ªf]'fs;q. Kpl/5=|0@!Sô%UX8h]n7v)m%(C+EeGAkaO'Z[n͞2ƒaCv{8`ET*7u4>\I[|?P27ٽ@[r+Ǎ2p_'Ω}Ic>:{Y^^NצVnT^#Qm0MfmoR%jrLnDKa O3櫴wP\S8$3˚u/ 7yAI`qkQ!(5_LEdG/0;Maq-x԰<6S]hhgRcwO "9bN3ʜ{Bxcj`;1G9}Б qΆ1QXWtd6Y_Du5|/g3o}>GF7zD'd&(ȱ^Ysf*q՟ikCVZz|> n{׬ |GĆ&1F Ӑ䭼a pCV1n }vFALvX1@)3֜7_.EKc p?!~LenS<5š9 H8Xi^u_@! fj gxy#SH w`HR|U.upBĻta1@<pۦ8DV&ߪs(r0ޚs%7 -j_նq4 Qǝpr?3Ke,9D<_Dz,>D "['?jʉ/~Esi'6ڿ] ERqPOv ,S`l ,FbR ubОx:<=g2]p#Oڇ 4@vߦڀnۃNJre9wx[3.Y06ZH6睳&p@JDiFY>LpFrOe^?e 2|7ZBb!X؇ѓE*Znj.~ݲXrl1o̿ޖ_iwCB.j5o]ZyMTg ^O쏁:ޞq tٳ8ϪɷB" PfZ՗7XH{sҴpxn_ ٝ-u˪w-91VsXm=(\DNl+aB hqN˯3.mT, *,.;N4ʘ32M^mPM&3ܧ2V=r+ؼD0%t5%n4_+wWO%ިC%~I[]iOtk4: i0N}-L#3fP#$+mVm%X~U ~q׷/л{ >ʢLK[Z rg̗ĊmI)Ɠ֐uA$6^S]R)_xCLWfUFjgsetr?㘅"OM֔!Ӱ4cu,}BHi Q((4_hZM^=wOsK+Se`({k(T` Q2]OXg (7$61<||[D{ryw 4=$j{MzϾM"%.Bn:\R 'P1Fh%g .qւ߷#̶`6K#{,M?S -x s1˰7b`\U_ںk&m/f j'2EHʕ:72kC. uPɱ=xh!6aZ}>(8b!QfĊEJW Hrzv?q+@Ԑ#{h` c=ڿYxĆ+Q(A3K%QyQӼ.WwV 9oء%(E:j9ALHkxOr&xQ 80:@I՜oEOubO; r`U_[ |Pݎ`'wpHAjN7c­‹ib>U&FGCa*Q#`m6 t-$3EX;r>^[jgTƤ)ьLrQk I%w~n"/Z=k8-CD f_'AHA ՗+p.dg&[g}A'Ow 8#XڣHwۨfT{4V6J=e #ುO<ь|Ȓk$)H,[(Q$^tvZ*QjyayY/^!s@A]Qu-!;31U MXo_Lև瓖$?Ct}n'+y[.R-D]pIy=]nQ$6Qx'#mLMMkdM{h/-bE`G{l2ޑ[J\{!()@)*o&s_r+]ˌlsoN9ÒdTZ0?1[5$~3-= 3[XzUmzJqblxc>W^>߭aG0G)/Cd;~F9k'j٬>qߴy ӇtŇ.4C3ZU2,zn}iQ@Wg%dt06ZE8( T6z5cBJl ~O|WUr٪UU2{!83=մCg4'y*墵,#h&DI_B_j%{Bv97Zb1T/nG)g=Bzf9)%>Կ4!noK &Ih _lzqbDx -/b9:U\bkTsnnYhBZRoe ZZ;gIf{CT{7.C#OΆ;PuMfLLZ߾O BAw7x1k ω`5(PԚL1D|lihՑ}W oX2錡b CFRʧ8 nІ{1!:TDZ}= 7=n]X|J!I) JlkPbם?#>-B[WPTZPuE|[I!g 7،B[Yڤ!:`g;/vh11q?;d$L aEtKe|TňTu1Mn#b?`5zQ,6g?Eۭ@ ȶwirias\2,DWem>+0yiyTz̓($Zly!HQgzVf`Ѵ|/F.,ea-,A+y55QʼnI# #w'\Tc aZ1͕l' z;z՟V 7dZ* ?Lr_t :/iټ̖A^ndJ ,< 7 (IǞv<烋P(fcEOgLU/2_0:<_oůNSQ[9QYijs뿁aC )!@ |a=2'i@%58T/ÚyA VW< z^őuHjϼS}?4._>Fm[ϓlΐ0n̉I;R_ozAT? '¯!e4Qa\)sd(Yuz!W1|I$ĜVb1P{ u5rY]ݲry Sk^YhOË q_ew]gy] wJNT4M+b_mQ8V "t] -!+'`\j_}=IkAv!\iݮ03hZ(, yG#&լ?8~̅Os^ҕ8Z6@֍؛dE!z f8:{EfVH ƟG"yQ[RsB5jingŻz[<|w+6ޣ hTڨitG *Bth(=mb7\? fz.g|l7z@>+g(+x=mK ]uh~ XUӇYX<53 EONdlQX6;37VVMrcK()ǵ<ź\2WIˈʫbq<&cGZx+3Mu[MU;̺A,&VsF^RRXTRJ9)w6 6z+rfS6"C6Pdj\P'"h*o=1z I |2[<1@*M.53Q |84{\ԛ"f>_?S'w;VJѾ"g1$(h.`!B Km#3Sv5 ݤkz`9ɗ-6F{ІI06QΫle )+g37 P'GBҖ<.QMa`Ԡ/$׵/duֆm?Ǵs \G~1&blQܔd!4?|"-wvǠGU!oy0 ,T>Pv_ Cnm zn^{񂭥%k)+RG쫱UEwq d6@mG= ϲb +HƘ sq6|¯jDr'Mʔہ܌=(-Bna~~>J-;VLwo'#^ LmtC.64wcn^l|]Nַ3~o%Qx]퀖K:@(H " *\m,$FC,(mO8E O?/:YxSoK%U`D3{#Pd14JZe :G2̙'WG4j0ܥȹıd{w@MxY./P &Hr})$[Ja–J9vTFS(qѣcՃp"^kJGC%M ;eReG[))K}Zcٻ5aBM0)(a`b>;]# h lh1z(U8G%bz)_fWnթ59l~ \&]ڮ9^A:㶒H Xpw+L@垥O{U܀`@xַgo4BN d/ m&gԐ!*BC.}F~YC;^(:(-mⰺw P`PdW""L.`= h>]PBCa'e`ɦ"45-Ն Ý't|a2Mo,y'ifDE+Cp3G66LQ;쀃 Q9͡S>A 5DC`&,y_7mITa6;xAi>ii*ɿ}Td_InpqGr 9'7g6 I'PEX9c|u;Fp}S+h{X-&N q9 KPpQ\sρroqbX*:)n c…*g!<~uNj(',z]7^="wp\]`VtǙLjr=iɳ"b7\1.;x'7;_)|u? u#jO _%\])XCH\41./~t}[:Eoƻ} ɼNL|Jܳ\IvU|`O14^ӵ^Uґu +_) ~[q>%4duHm.l&uvc9N@N_DSd=b(0*e4 T XJVrwpl!+p4V*um W9@D3lQU 5H L{~;/hD Q^EsF O&/eD6^' `~.%|o?QY y֟wb<*2SSI'} v՚r0o2錨9)cwSHe3K„,RL ,~HSXo%?)mvY-oY[4@a[ Iq+9i6絏fr(}:4{.BLa 1$Zh2I[7\pO~`~%4tFR`;,+=wprH*,m[69-жa-zO ҄we~ o Rϒ3Мȫ)ZW&`ih :AU6}vUfX4ksz.[^" -+k~l|6=oEiy.#Weufr] ;;6Ca gc@ƆF! bF7`j,t?W~!)u iL5.kt?`1r LWLbg:ڮ7@f 2^w( P 7{#p0t9l8sgɷ.5!#]@K{t ux)8zk7 z;.lQ9=X3eΐ{@|8WÅ/kh JՋ oZp0]@;nֽ::ћ_I< hSdlErDt M~W-+7MԅL[˥m `b1$!/lӖR,'ٯrQzKy[Z`W}i~ڭ'FG2i7R!՘lhj]űxʚ NrǤQ;XDw'` F gkPM|ɥD-FI/;!. iP"M^r2 $@GJ`|Rs .HS:g:Fc:owN/CwJAPSCX+"R~RyޑV/|_H)dZj+3C+t .ѽDLSDh=wYq@3A)AVޒ#ZelrK/\UH")^J%B |xlgpU 'x-ziXL I9afT\4Ah0`bEjI)\<ڀ%(q\''xM0*RY~f5*r>O]q[K#OF`/|t&%N'eccG͓\Q?H[ ߵ4xJ4:]Ɖ}t'ylrFtMfYg/m8զ M[Na`El9l j*I1lT{YƝC'CuG1ٸ)*WaD遨 R716m-L'_}<2IGť]XJQDDVxKXQQhp2%3̗+;q!֝-ܛ }O 6Z%FvrSl0n cAT?kL`0j^~Q>c}Fi:'vQfZ΀9r04e#Ԭ-ةH*)~kK9@Ŗ߹XCu,= Tv T[:3+0X?s{Ԯ俻6=TNt=~r ԮKV4r0{Ήn W:wH]+^%yrZqIB{EEH-R:ZnrJn紆QNƆ=w=!SY&=SJN+1opaUJj'q[kب`5MgO]״bq˳$Y>^ %7^ijJ(џxcy[C ߴ=TI e8B,azD* 誋O `N '%q*e+yE,ݬQORE|f.(I3>{^.](*SfccH6F:tI ӢX1֗rPB vo*9⧶d˼Njc!ڻ=,$<$焂vz}l_h [^^&IutZ:IRLx -cJ 94J ?rHyLJRgU%Y1b%ت~"\A/E2FfE;Q;ʢ U1hG!=={n8/6ȥJddtBKYr#Hm(F\àFH9Ea@s2u/`Qe7rZW-rBUETB;76DzYRS;} rؼ|ι*EU=m(KYӐmGx{\ids/msgI(x!_ (0!;xDX-c 2%:`ڏTtv`__zeoDZ >Ag!b=\#); j-2M$KiQRe\nفH`>HInounOhD| k& ~M`ԣ n- rAӧ.{u` }QY*E;M~cs~-t"%dpeWЩYWeX37·P씨p49VN1Jlvgk@$Pdڼ%Jդ%Thd&@2#)<J"-ʘk +#gDjMWLqyūߐ处aUS#/⨄`0}&f0Ś&1헝x?+X;J2Q#},x!_کt\&-%`!ȷMs]Q7Pjp+A89! =V i>*>E때Rr]i"kczIzkKgĵy:5{+!ksF@~,sN4Qq~$\# %|e/jSJgkxLoOFhePp/ak/|MáWn6$т"W5j3: vnCjBW$)s 9~G?Ҏ (A'gG1E_:z[68( uQtgf@Q)X wl틚vA? q]8FYO݂f+APfre18:- IźA"yxp^85Y[h4j˞C&7uR]j8}1mH z[!@]5=Ї2NˑdС-L#uy`(VhZ?i &%J%\z۞&Q5I*.%]CFbT}}nIZ+qg;̍zt2V)O6hUjK51m +(PKy_:~vRSVrSY8d Qp9B"?`Y`%-a#X**ܺo&qxO0Q,T/ `h3/U1oVP)wD k:|R.jWX#HPuIJۣC9i.U«WJQ(嚦W1O[R$>1 Z2/WT:5A;uԨx/ڀh)>Z,# B1mC_ִ|=yAީp)y P c~^߁G"ic X>;!-e396t}ZL.F#%gU yE15sj5  + a\MVji2j.UE6@ QٱUOqq̈oduϞws"䪻l_.6«&] ݘ}_Z)Tr,è:u' h8LY"#| ׉W}LluUr“9]@rϢQB,E~Z4.^3 "#`wY[QǕJBթ K}P&%^X?-TӧJfq7[lB38Q5T=sEcJ`p˷(儩XI {pRّD4Υ;:I 7ib[).p5@t`ܭzM=ʻRc@Nnhh!%@F Yj:p2N{t2XcJD (uPW+ J{O-^/7)ah1G\]D̝ خ-٧IحH&^ 4;[#:c0ũ\Dj&ny*q'K6ޡRqS,m^2;pK.(_ڷ3Iv@abi`c6`K"Ew-_f@{K-!{m,(kXc2 $N1r!K6UT7+߮YO W^{Ǎ5#a 1oP0LKEk3ZON) 4Fھ{ r^"UT7_AV9;!K`|zZdfvօy^Dzyál6E%S4|twpe\hJ"-`Ρ)Cx㊎jӯ0`2mHNŊG0HC!Kwoq$Ի°g@=XhHxd?,WlkN4%s]+cJ.<;rgq=ʫՏ9pxŃo;ɠpث%#%# Yz~S8 -?{g Jy  W5ѣ:~i_`g\!k?ؕ4nDWI DDc9 Wս6E(<ҠH)cM ?n(ذ0?z=ڽhǙ\^H&^|?V8'tdN%p5v͑'OT7* HCZ^iծwp5]+PBJ MaV):"+̿(Wm9)cE\9M {H@Uv`H`E=B+ۏ oW }j;?x%$7ܣDY #6~KںH&I$"y&`TOҠ}#ӊ:yPkT,}bADWݼ2ܺ܎S>@Cw^"ŶE!u6yz uM1쓅:&yDF:gVneսiֵYȆ%Ghu^vXJ^Ei_=e31[9sA db ~k* j_iԟ?ftP ^'9;6N#JQo]6vv$U86ۭ9F',;Ek9': QD#>AwJȭC?؊^,\of'SЧ3||Խ$BoPb 2#E BS8~n!p$j2@cj}d'J*ҮQ?p,KpxwnκkCXrxs#Lp!/Cu+G, cD|M nZ݀mL7wS{ԹCB#Q1 7kF,?RT ]`uDG6[ MNiq&4 '\K";粩 Ț_Ȋ\dH4r+Ŏ:j8;(D>v^@vE蓃R""E vPW*vf'w5ظ$;o)LJÃB1*)*IJv1 Ŋ,j;gY}(A7aヵ ~uLS2zyRyL>'[H57<;Ɔ+8S&}<ː?<슣aHo׶ z^>F$]b'|[Iϐs'|.&FlCIBݾBq4RW>kSt$Նل^wU><"M߯vEh&RuuCU Z~gTx/#j;ϻX}qT@+::Y/ R LLYuuкVNnd{m4H&gAee_ Y=QvI5P@2!Uk+H*^FaywCUͦs0Ȑ{ai&&QD qS%VrC l7pիE@*dӏ%ߝ]ڥ. jZQ*CQ5acȾNhQyQ'񒽌ϹLK$cm~O 5=G&s:xGG|OT$E="oa߳D91*wOt'yb}c&[\lC qXQڤI:XR̹]d )d0|*@l;|ֆƱHp&Ѵ]NipF$FweAb.ΉCgv.9yf ަjn' DU4ZrȂG8˟uB_BW?N48]V]c㍓W n'+̳G<f X$avp8}.vlw+@/̴&\bE>&!`3j) 5曃ȋ04(%b;W`x}}VE{a?ayT_EnHl4+Ab21$ hաCJIhN@\M[ҩ;,,Y6s+}V7Rm"s)ٽl{PjV[m&9/F\>@2"Ek׋?fc+E;Eށ&*#ML)-y"p9VBҰ"x33LFfW$m̸E}wƕ+HHgS<3gXgB5)^C DER0u|67I=:A^>iz&fֺrKKrB*{{e *4.m"yr[\y;ρ"',-cq83ޫ#I" 3J̱To~X4uZʱ@v{*p!`CzbXH:V1FyRADx.v1ByWeR!j$5rŤd.+{ګKK<6QGBEpR-vJ4T3>5ufqy] u|?_6>^d=ktSݼ2NS}jrz*Qt̽\z) F3GVvf]FMr&w~^>mHjTnG'|MdGōh~AKSZP]k.' HVg'ǾPGZ:s"CBmjQӁ6Ϩޜ1а5{@Qf%l۶)[0ɼojlC|pӸшِ^8FJ-"v\hTcFfq6ۉ=@iFD1^wO/H@-#&R2-PK]MLlIu'!o5"KN;z=,vFޗ.9^ Ə}}D XofP@ !9Βkx+M(dO*H9fy77vݗf?$("McqWhOE>U%`Efoʾ`/rWYLWۓ\ {㕗rP簭5pJX_m$GSĠިE9n%%<4CѢMr҅Jr*Z?XOTAv<_D\qK8{NآoA]>]"_zq Dt.Yx8]@-} [^~ɁA$6ۺ$#c O8(/&3`Lض36+$6qjF{odYO:k_s(:wXU~ӇOj7g5ꢑ<0N#tLa<3 bX%(CRj`؁ȮJ^mSwӓZ q)SZ}BN[eJBbBdԈx Kxd?+8D32]Ïq9N><1_m\ܨxur ab}=w,:Ӝθ%M_)EO&u/KRP!JB D YZr%-Wp|2łOHVOdU S}kEPn2Ues.T_!3[80*q_"ǥPQahcTyĦuE̠ "%1al V+"ŹVdɧmDE#, X4-wh37>~ BlTa[TBIƫHn,,tpg"!.72 F<;^"#(T.xMc ,׋KLWXȐL5I] %(Ђc2Ļ`g8iCD_HIVW]1fYz ՑOhg-Z>{G+sEa$mad"ߤ(q>箽aDƐ]GO$N_)\ӰTviY,pɡ:"!ʼ=:=ߕue a }Π bE@9J&"HI'P?-*$::*%~9KCB#HPɣbрRa^!:4!x/칏)S;8@<پ«>yP+g spup!`Ƚoc mQSȪՆ!V ^PVsl$A+f‚̯Q J܎hFTXb֮*15{¤'"9UUd[98^A<~G]YmV4C}(ߨ}G'ҊeBBGi_o @Tw`&o3gOX@?b{kwṆ<xt.kAP56+ĦB[uCU -h{;*4!l C]s\Nyc4@Np.h;,[fgB:(9O\9-N̍7%fVhMU,07GL:k\jѭ)RH׉T$=%< GV. X%0]2i ?b>6ΊK"@@I?2_1{ģKRb[ C?ff8Il0(f:!o9Bg#}Q93OJ Vh) GZ:Xu37И,~7@s%o&45{,7 K˻,; G]Ö~o0b].aeP3kjE 'oyNcK+'mw0!Vͺv0mv1jraYiKYk׃WVf;LIP֋2٨ToZ:r3½k >  ;5B7ZsG+QoQ'#GkUלMPRbI4u۫O㜂$SHwp+RLEZiCη̿sQj'8}(8pgM\PCъ/#E-CGO]Z:*pƖ'Bʪ*}R+-`+=T]ZF_Y1g/u);L;AіUC3z;xdR7RIh'Y79#/B@);6u4!ى ēn/7@isxi ’B ,dMaj-JګG̍:@3W#8(xJRgKn+E,B 3*U X#o򃰋):u,Ki.2*#°8:XBXV8L|cɸEW4r F5fno(GRFz-)H0ЌOj b*CN e p6@$6l@?A뚵^" e[{$o%nK쳘m&޾G _k\6,o{M4sTsI7OtDjZeXIt.'!Yqł1Cs2igiG•2#jEk,4UlĊ6( z=(Қ5Ju4;}@ 28懺jojcRPg y! RJtݮq@?(5EOR;HG~pc=4(X"R቟jf0#%>bO8L|%]BQ3kWmxX$,m ia7#HtYS(3gѮ- `d:= }pjοd+e*ȌX9xNzMQp٫e}UBц~PhkƒdMet"Գ`밵tU_F$-B8]|-H=~82gTfa_ d[u)mWmc7At(g 6L`Xc(uoL=[VNJ,dV$7Ow{0ҡ+]LڦTy1r)"gn;I,I|bC]j?[meaZ?KU&xA+S7,c2mpCƨ9 z_hIns2̨Ϋp*cԎo|R ?6rSЕQ_ӀDdK'HuA(u+mJVL߬7A xaJNjR@_留w[)>RmɈQ9\am)97O {Յ~ZNq  M;F ]틹2Rl9Qy@Jv'ʮq'L>eyӔ 쨺,/9NB=o`ρ E_ƀs ĝ=+K=q$8FtTs2;{^n[u/MZbƽV۳9KC`G[džҚ~hέ;4VKπd[5νcKQCҝ}Dz8H$-B$ߛIyKc؆|@f-- QגfkdϢP!#GV :W v$bxsؒz&= N9sxص3R'PFy!L$/@mƢD0?lUzǿu)6Q?LQ'ޮ_#^U~:8(,eOGџgP*uNP\=-`[}y;:@:<.)V![G3 oTL2pRz0Nɪ-y&v0 צGN9 ~TBlt.F^ᑘH $vqV^ӬȜ.9Qi 9_(؀tCWvԲf@FhMGe)wp.L.Jk[C2jDVvJƣ}_LQ(m ][ǾC 7M/NĴ:d=w[DYr骲R3GB+4`3AzeD7 6q֘ۿ9Pa7Ù;0epz>;4Rhأb!Ʒ=V6+7@(NG%UTh4,[%܉?u֞qzO-lnٍ# fsd6WIx  O1M~1 2&v `WvIH=L߈ %u1:=`֑JwF%mƹ&H|yWNY;[=3 r;.c>T8坵mnٛPr7VN8zxK=SL4mN<"a?eV}@t3/_ AO-P9RutŠK 4o4_ o+#L 1Z˟% ulOvy#3WD68ٙk?0t /d5 PZ A/1{ScQ"U+ )굹P5vn!p%\0:82TA13qVԊ@Sw]/ 7>?7H(!cXѫ \^wJ"j6j^I[c _W%,m7DnM.%G]ZnИ(7sCG%rOP>˕`:s(LQV4 h.-83LMຊasΉGӚ#ȋcK۳5eT 67/&O5.i^.ǫ0Qɘ6,[㳗y\yɑq&'A#r 74pMk4u[C֣fRGP\ ߸qv{[3Y{PHE&#'>"^ymYIKED/;"8~I(DN=Tnu^(!O77"JVL & r~.4yYUq̣ifkU$98<{#ѳ'-E  ly+=h (+_YYm/0$U#_Sg8D'{3Xavܿ+iF }J01k/Fn ˼tv Ͳؽ0UΤK0I^?$zfԬcasF2ncY"Y>ԃFZv,1}:d퇋S K>+ bt]1j{`C4gwG&|g^'UzPc17!8;4*vxȐQi @dӷW n(Ow khpv Xũn4?0rǓ$O: pS*Xt+afE`?4>FD XhP!"a|g3 k:JM˩GBџjχ,>uq͏%*TDȢ:t6e,05Sےg*\ l\1gx H"0MC¤8[FvKGtg嵑 .f5UbհNЏz_o$TXt`Ѯs?ɷ"Ge&s8IN}8e;DclmգXdjSWT09ү ɐ5uPKזiڋΫgOi<@'yrVԀU#Xj`U<8%c._;PhhW׹G"p}WAD9Po%f,VYlG+oC0iMN% Q{l}p3 bEMIB{5|t(SaiP ɮWUC,YL)˩W_ 6݆l3HuP&?B61IuuGWNZX] 6 =:͇S$>nP2kXjq"ޗ c>/K5{ͮFEK&064?<ǥPZNb'<~6683Ҁ4SٔMZ &ORO֍) ެ<0с3-D,nrna:bp7(d1prpPit\%(l\i0AC o:YJп}090lz(Ckd}Xٓy LE0~:_l]_ ~A\l%/1HUk\Yμ<ů.%QHpHXnW"5bq+MYjnpIgZO!Bc5Z"3Up59+H?ʑ[В:Ij-/\8e+/d/0t \*m-cbR36ROdqdQM3U c힌æ%ҕ(whK Y<4\C\.g|E@%.C3E5[R͝~\^K{ߴC%1R-[nNh:NJ_Q$1 NgWd؝m҂4SȠ|rcۄKMDG?U [~.;Wl8Bؾt<_LVOY>Ť[9-тCҭ'hWO6[LhIqo] +a-(uJ'RL}̠c U@Dv/FbE'!L/swl!2D7ȦAmqQyY+^Cӵ&aC~YJ&W?Ն;*ĭ aK4l )"GI#tXkZrq8ʇYFRl!fnfWU11kVBri\IJ;Svq7K&U~we{ "pc XȭmPhGIbr͸2Zb ,8/i-S9hYcye[\CdMlK?kwnt\YI#βVw8ku24eEV쾑M$mrk2 /25[0 ˄&L!StvRQP|d]:'CKԗO^0 è7I>ndǼ7@#',saSQ$/zXwJtQþ&4h,KfXPQePc&KN$nnUcX>87 Q&Hms;(X/CrXPݸ>T l=xy$5xϜa;Un n|/W | l͗ Jm>J<2W.pWMMk-J" }R\6dTVy3SΎ?qWm|G,'mk!7#-*ˀ砪rGY%}PQ %x.Ox*(}*㶠7.w>6PkO ILͶޮGaf߃De_Xc,!ZYXMN0$veٹd%m)hVkų0RTBayMJ5 ʹXR zEEYL=fb"'`l@90W=ݕO(<1< "(IPnuyIXFqCL&A6-⹾7F{^-{#1 !!! WB *5i_[S#t=J}6&}zpk}گw<\3^q~" N`LrV OnW9&f $/@iZG( kp (݇Y=|,LʮrE#ࢗIU(Կ5hSGiF3%plB~&p9Vj\JmAcQD*mdyO(<{ƴzZ_uTP @oD-ib+14eq,mPw} pM]ceQڒH_HI@6;*]6 $FD"KI֪{ƢDOq9oVc3j^iU-Hܨ2$ ['q=QlӗښѯJ"I>dV)gɶ{L,Vc_<öaͨy | sZ0%3;rQ<RҰ&O,$D610 M"mp>U⿮LI'A[_:%_*NCU6t\9͸i&;z^.H K|1%<I sx~)&5ym,?$^Ry ?;Q [z P~۩:[fD+t2@Q 6xWu<$acn&aPoeN,N [ެ& rQf}7kWtwx`8+~ GXY#zv*4bk  AF D ]%XK5IZ6xiZԇGiujR7ڒ/B:TtMnf(y떒MHdANk|g'aw|ucO1*T& |/m=8G)A.$եmVA Jv+^e ^t';qU 4fߗ# c p-+k6ix0 V8Ħ?ȿW UG^̦̀ )E@6Iua0KS^"VOfE\ B0x!1G& !2˔m*ri =S,:O䯈uhrYUn$ೋUj fa¹=B+yĚ:bD?]5i ^-%CKm,(s sqC 24i(NaPQ$+8 \]lG/=^r >siAWh46.^q:U|X &5-K"0۝;̩$ɒquj~uTi(/!9XR\YV5);nׄGc_.wl1RxV@B HהycCj4Eq$: 5 eTc YwgkJEUwE+~&Q:h9(CE D3>cbx ToiW`jvo H,݆c2W1~ #H~ERe|dG$}9^/k wTҷ4 ѱj̺9JR| %,֧gdzI*KPb!Q UOo0`r=Da~`(CQkzhJ͆XJ> Eny?߶mx' aӵ.)[|Ӗ r's.d=]zvDռVdmUTC2-vUWj[D(?sF")[*|+tϮ(m1hUfъ2*u>*hdֶZlKzZ&y+P&e;Im鋫escQd#[WCW<K-wb0l;m\JxjZ<E:NVm̩ڦ+r݄8ᱡd(y5=Vy ;yr PHN[*頯27*5/Pwqi*h=K_TOj4?R>(á8vdH g|2{?i"xoD}ܳX'QI <>"1NfxF/r^d>&Yo"m&LNnʍ[V_D MvoE \3C :%wr}^,MnȽd%zWp,iXD=J*?h@4h;1V <8։dXyȜrGHuҋqPz4m! me%=} CKp:!V#B+2F79;}CR97n 4+\лzqHJ:uy&v='yOJLE 'iJ*-,clj|χX5UxEo{8C)2A !=p}t4ϾfdI_W79k8M\U@,p7rI>VHzةi7 `8U,X i E{hV`;؎5՗غ4,:X`βjQDXmf% ~Cӎ˰kfKƸ\w)FK@X|qeqTi-!}VF\H(Ep~y=f̿!X1V<~Ę#e rozrzdKq/?|TZ=`"fVSaeY/L;kd* 48A|t'R N>p Seben'I␹ݔ.>a_x̨m cd"jQ{րd|]`GlE OQ9u^sbO'L1g?u]!%X?H1]fe8%2_nζڜ3@K)˺g~e7Cxb CȑR~$8\uk6zkUh5hdlhτ 45B8rLOF@>EUe |V>ΰe0OLNl~^ZɏV,n{l c,shu30IwY p&)TڬX:l"X4 y?Ov mO4> j|jB<oj5~AmI|Jp^/ՙNj5?hAptܻ .241>tߍXJkt\`luXa2wg+D,3< ΧL\uaZ#Jxu|iFs[nHg52+fIp=t"Uo]&Jvc g'y_/P("j7͝^ aw:`Jxr_c*i4\cb#< ͢|D{ K4@Yitzex:Xxxm *Ҋc17FlJKLr !ۊ7Xė˴Иս!.ӅB, چP5d?eexB-}ѻ3R3m_k;s M=)&W\k>z~v) [($& +U "34~fzM :4B|2闱EGnVM I3#3+wǿ˾Z#04ǻu$;3P/Urß7[RL"' |[7z)2XRujps)~'vK٦2}Z?[bP1>ψ4%@J L &⿫u0?5c>22X (x4em/|M&lAy)90E!YixG4wYmլƊ?S/ 7vV $ᠰ͡o]|ٵob$QEա㽼XU1;i$gmkd֓6p駓58$1:{$=W(z.CORg1lfQ?G=TF"O2+0d04,De+,?׈>:U#جuέOse‚ߣ2z)9E.jTӺWvk@8n^TZGmsXTkʉXiV!QdN7VH2YH1:g]}_j p=*^ Wotq K{FFѫNUF1H6daq{ovA/v61(X{A͍$ȁ r(PS<=uq*WC+TlWu4eiHg&Xէ3=J㥼zEç^Ilh`˶3@{?&",[c\{05/6~_n׌q<_5o\ [˂Xa@rt:G:N9bCqr@& - b~8db 솱gCrZ RTjTv7kf".&U>سPhh T=#[ s@h,':(Mj"* H)n*' v#ńIl%j;5~Q zaQĶZ-ZtǿP K]8+7^/7rW آ 4REbԥiTS^OQ5LhnRyW{Lw"kǥ[j3˳jY'3`UenDoMVIri;G 5m'$Poё@ '#I0y<&|c;bqi-W5 94$ Cj<t93jZd}D,nz ͽ"'G\Tk.ʮxqvM{ B4HANʀr_~` ɮԞe>(m>}\R>)FAff"e(j4)!φSZ3h~G_Ef Α ? r}b=/v]T|ܳY=+/s[y^WtU1TU=/DW-XjR&r{Z?4G 0Cvиe5:R:ßImm:7o0n~ r&oƪ-?UE7oUHpEثg'I^ÔXGX#幘==Sh3;!4ybTnEyDby9N:liݾj F~WWxOGg5܇mk^F=DpE;{cy1|S_n.݊*i|P_CxYXUE0֚f:}ޒq8X2[-Y=.{{c}_6aN,P0ffBzrM!ò@X|=fY`VEX"o7T, aAf8ۺ0|l 856l$,x,G=>%%0lSĜ=A5L(n6.lb_DS.jnMH[-d.'^Cި}-g<[+ }՞G-eZσwypZ'4ě~F.;b9|k*qpdn'$zUSX#`צ%gkF{{c1m}xec<, Qd$"JfEL:}^ Om;FZ?;$TPt >W~KǃE~k~̲ؽ ڟ~.?+QTc,g|eDy׊N<G((E&_c:Dǰڱ81:\Fn53%I{m ˸2777oxL 2#*5_-7vzVP8٫m, ~05bVҝ|ro⦟K0d±3at vm e_J)ȱzm?ĥzO^D=k ѧ_odgg7Be|WBT}2;|p_?ӛm}4OqxJ-{Z!0] 6.#g6Zs-rQSzwzxP݆ǿ2KG}g)jEv#3^&8&TLDa|鼬|{#:C0%|J{@p+]P/N7ӗLwR*vxyڬmy6v_fVˣS'իBu~![XeZݹޕ&!lȇ7>&/yoh0JkXSY5$]}:tpgh3TǠ[ΔAtp9/~o1C.<×CybcH?9Om3[~ 2tl&q2>}S0j˞𧢆/|%w_Taj׫ӛ CYkUDD$8p:\#hOr] P /܈[BϣDmg2WH~t=~7p_f\gL`1xY2q8l<6 NfQjjoc@R *'ri7VD^^ D4vSQ2l_ !b8Pei$fZ]C $#Wrt>_$s{A7SZrAyl#V)gILS0/U_!N0K2qsVFN"h>9ܡ믖 n*WH", MyNq% !vzAI?4N20RE լPUə``E=92a8Xq?$З~'- U_yKoO-}Ãр<YiNDXR#'bF΄w˨1 `66{q4x \W ! a kұajjbvdТZ<6;LeF1N%sT d+uD+EF:^hQ6D8cq-nvҞffNXt" X6(DQ#uLD'/@23˜|PG֝R)BA5Hsrk-9RϤ~FW^Tm£c,3U(:,}zn`aN6%!V !QO;b쯗c/!dQ~On@.zýY=&86'"ήwi sN3d|y^u8DU3P4f˧v_):RIzCXwqCZL8.3d |Bp#i.6>*±Iq~5Vmfb &$6 R۬"̤᰹~%9Ӟt,=EV`nd!D,"1Pc""C5D>M"uا2/5e仐 ,˱&y4N~?;rݐц/#A9'広45CkOqd!6sqlmtg#!Q {Y(j:BI#%V~{ &9aKШa jQ C X,f(3L&D \ r| AwF4fPa|eq69NȤ]wC!R_\s]EY!fOS}~3,BkkF5.٦@3'<;:LoZD.ݞgp[>sd9SʧlTF[j>ꐮUL L"zO_;"!ӥ 1 ɯ-&C~+DY"=,OǺə=.J3uU]t ,ޡ &A / aaJsp(>o˲fP F+-0١S# ЬwMp%4V}5)} (y=<\[H S9BP~:K3J`Ҩ3KR!)\:E9M.:xǛ p>^>i9 &fe<#T rfY 9!/3h.ixnG?\]h7g ~n?TXSGsY/l@o E EE" 8ITW{'1\uU ldYkfBa$Ij ,"dĆ-R@T"m0D  cW{mN&#a1M^;O*՞2{F21GCf+˨؀%%@M]Iu6TTjT>]ts #vi_ ?D5&ŝ:s"HD˺IZΞ^#"/ʧTcAmn0>/3R>Ӏ<AoEeX;_r\gH>_3"?>pğI ViJ0S7G՞$169]lI_Q*@ڇR}Bi yMU:ڗa6 G'qDA#p7Ǝs\gGOeQQ) Y\]0&碛~;}>XjsY1-lZde-Fd ?|?uC[1>6-TNArx7QWkR("  @U%bcX(b$PDDDH-J eb(ETZ~&M7@ڝMsK\wOfP0)M naUMʸ___ˋD*Q0oVj+!qN rЉa&2oIAhwnuF ꫇}I5n8wF8p HL* 6| #8DqKaDWu尒-]DA%FFAqɍ"$ U J>4yWBr B|G897T=/daљ02fE~tTLqJ<)TDL<36x1YOEJtoϑ޳@ԅ#Q8$"G)˟f[Wxs ) *帍 $S41I[uӉ6QK: YQBB.QoBj"ulZOVWȴ4¸j`L_T2BI#Sxv<ū}f4޽5y5v$0tYϭ"2ݓR#¼OSl> -wJpg!=u)bEf>&4L]T+WզegOĕͩ6tu׍N~ܲ`qhye1Vaի#gEnDΦ?g߻_,]eH5vg{Hp*BxdwK(P0 0qV#k׺77ĽM;ND8o3h(wn\Z!{8 : r8ٴ}dgbx`oO3Ƚ=4ѷ(xDr`I&8ӃM!Щ"SjJ17\ !_cKUKX]i]Mҭ*S${#emIjD{rC?/;ZA\ɴbGJq8D<~6$#?eSiH$>2.}kawٴwޯ1N۷d$ɨ|虢 Sa(W<SՊep05iyjT#֣d8QTa!عq1yW*,LlJýn(t%InYo>\rdw{Fr0h ϙ۽YO=ij{_q1b0 lϖEO[Cj(4=]į1$ QMO 0G9ui n(|L4X0J#{ҁ'5kJfd0r.b'&uuֻDq}MđN> 55>_,1$o|PT6uϲV( (:= ؂ו3\ϼ܎N& J7}% ;91I&hX_16t_;9uCwga}pf ϐp/qc.|;uZ Q}>=5Fd= Y}!`W֓>b<ٸ=l <9JnCͷ@S"G 1b DZR40աy0V2.,F;J6 mfĹ `p,"JeL[˰e:YR.PmEXpaU Ŧ9$o7M^T-&v,8a YжgY˔QC,9eL昚dD G?Y>Â(]ӯTSPVr(ؠQhŅTFa#%bL(ѤQb(,+QX,*jWLR\R q0R[hBaD` QjEECV&-A-,1Ю,UH|*@\%&8P:mn^*}u}>|'+m -ְbu(7;}ǥ.!;r@[ 0}ΒSSP+,(Ktr#fGvL9P2(eK r#uĉ!Gg0*Cژ͓0d {<3Uw[z" @v{=_Pe=w: Ur\ݢݻ LZK˻2I1qRy:i CKTXYwJ VTˀR4.Z ȭL!fud~ qɮs!('YLQ&[dtmd2IvѮ"q%bvsAn,YaZPhJ =){"ebȥ*(*"aI^LDZ2 [bŃGeE8ȉCՄR"`TdXJ#*b**mBTY(> &Pf MTf42f*UdT%CSy d2,^~Bi">G,@y̚&}x7A$}B='z-T7Co!7"Wxͣ< "fXTEPT1C8K ؽ`!B<|VAXC# KOib,K4U!A ^׿GL,7RS֏%Sm!>IW o᜛M]/ Ct>cS1=בwͤ|bWGE#Cf`Tέ^r>o %l6g$Z=C0mg n&*n@2RB&a'gwvzF9ţ' 5Iۘ&qV4 p 8K Ъ)ȾXp:FǚcFhmJXB)=sJrЮ6F;~~}Mjqi zI'77rf+a'3aw"S]*&oӺ]d2Kڃw (@v~D0P Ď.U(\d9py츷ex2q=3YOu X&7{ , 匜AfFX:_ Q>Ϧ\Ư'R6i7B\(310Nby5|Fߡ~ÀTC]BiWoجd]34@>݇;V}H\,5Tl8mıK