M&CS High Performance Computing - GPU Partition

Introduction

M&CS also has two GPU nodes available for GPU-based compute needs. It is available as mcs.gpu.q and was purchased at the same time as the CPU nodes.

A part of the TU/e umbrella cluster can be used by the Department of Mathematics and Computer Science. It is made available for all research staff for small- to medium-scale job-based compute needs. The facility originates from a collectively expressed need by M&CS research in 2019 for low-barrier (close to home but serviced) compute resources that could facilitate most of the computations. The main goal was to enable an experimental environment to test jobs where one would not have to worry about unnecessarily burning rented compute hours from external resources.

Documentation

For documentation on the umbrella cluster, you can use the navigation on the left of this page.

Technical specifications

All specifications of the machines in the cluster can be found here. Users are given their own 'home' folder under /home/[username] and shared scratch space is available under /scratch-shared/[username]. The M&CS GPU nodes have an additional 200 GB and 500 GB of node-specific scratch space available under $TMPDIR. Please review this page to see what storage is appropriate for different types of data (and for how long data will be stored). The mentioned file systems are not backed up.

Community

See also the community section of the general M&CS HPC page.

Policies & guidelines

The GPU nodes are shared by all researchers of M&CS. Policy states that all hardware must remain available for everybody at all times.

• Please consider at any time the impact of the job you’re about to run (given (expected) wall time, allocation of resources, etc.). Be especially considerate of running jobs that need less resources than what has been allocated (e.g. if you allocate 16 cores, but your script only starts 4 threads); this prevents others from using the unused resources.
• Do not compile code or run compute tasks on the log-in node.

If you have any questions about whether the M&CS HPC cluster is the best fit for your work, please review the other solutions or contact your Research IT consultant.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

import tensorflow_io as tfio
import tensorflow as tf

widar_hdf5 = tfio.IOTensor.from_hdf5(filename=<<PATH TO HDF5 FILE>>, spec={
  '/inputs': tf.TensorSpec(shape=(121, 2000, 6), dtype=tf.float32),
  '/task_labels': tf.TensorSpec(shape=(6,), dtype=tf.int8),
  '/domain_labels': tf.TensorSpec(shape=(150,), dtype=tf.int8)
})
widar_inputs = widar_hdf5('/inputs')
widar_task_labels = widar_hdf5('/task_labels')
widar_domain_labels = widar hdf5('/domain_labels')

def get_sample(instance):
  return (
    widar_inputs[instance], (
      widar_task_labels[instance],
      widar_domain_labels[
      ...
      ]
    )
  )

train_instances = [1, 2, 4, 8, 16, 17, 18, 19, 24, 46, 61, 63, 128, ...]

train_set = tf.data.Dataset.from_tensor_slices(train_instances)

train_set = train_set.shuffle(
  buffer_size=len(train_instances),
  seed=42,
  reshuffle_each_iteration=True
).repeat()\
.map(
  map_func=get_sample,
  num_parallel_calls=4,
  deterministic=False
).batch(
  batch_size=12,
  drop_remainder=True
)
train_set = train_set.prefetch(20) 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

import tensorflow as tf

train_set = tf.data.TextLineDataset(
  [
    "filel.txt",
    "file2.txt",
    ...
  ]
)

train_set = train_set.shuffle(
  buffer_size=100,
  seed=42,
  reshuffle_each_iteration=True
).repeat()\
.map(
  map_func=lambda x: x + 1,
  num_parallel_calls=4,
  deterministic=False
).batch(
  batch_size=12,
  drop_remainder=True
)

train_set = train_set.prefetch(20) 

Rules (partition policies; technically enforced)

• Slurm's Fairshare is used¹
• Wall-time limit (7 days for M&CS)
• Required to specify the number of cores/RAM explicitly for GPU queue
• At most 2 GPUs are allocated simultaneously to a single user
• At most 8 cores and 128GB RAM can be used on node A
  At most 7 cores and 32GB RAM can be used on node B

¹ For each researcher, usage in the last 5 minutes is computed as a fraction of available resources. Jobs from researchers with a lower usage fraction are started first. This allows for a fair distribution of jobs during popular times.