Commit 7158eede authored by Yuxin Wu's avatar Yuxin Wu

avoid divide-by-zero in GPUUtilizationTracker (#999)

parent e51323c1
...@@ -61,3 +61,13 @@ As long as the type of data is supported, the data will be dispatched to and log ...@@ -61,3 +61,13 @@ As long as the type of data is supported, the data will be dispatched to and log
As a result, tensorboard will show not only summaries in the graph, but also your custom data. As a result, tensorboard will show not only summaries in the graph, but also your custom data.
For example, a precise validation error often needs to be computed manually, outside the TensorFlow graph. For example, a precise validation error often needs to be computed manually, outside the TensorFlow graph.
With a uniform monitor backend, this number will show up in tensorboard as well. With a uniform monitor backend, this number will show up in tensorboard as well.
### Textual Logs
```python
from tensorpack.utils import logger
# logger has the methods of Python's logging.Logger
logger.info("Hello World!")
```
See [APIs of utils.logger](../modules/utils.html#module-tensorpack.utils.logger)
...@@ -105,9 +105,10 @@ class GPUUtilizationTracker(Callback): ...@@ -105,9 +105,10 @@ class GPUUtilizationTracker(Callback):
if stop_evt.is_set(): # or on exit if stop_evt.is_set(): # or on exit
return return
evt.clear() evt.clear()
# Ignore the last datapoint. Usually is zero, makes us underestimate the util. if cnt > 1:
stats -= data # Ignore the last datapoint. Usually is zero, makes us underestimate the util.
cnt -= 1 stats -= data
cnt -= 1
rst_queue.put(stats / cnt) rst_queue.put(stats / cnt)
break break
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment