ioztat is a storage load analysis tool for OpenZFS

Related tags

Miscellaneousioztat
Overview

ioztat

ioztat is a storage load analysis tool for OpenZFS. It provides iostat-like statistics at an individual dataset/zvol level.

The statistics offered are read and write operations per second, read and write throughput per second, and the average size (in kilobytes) of read and write operations issued in the current reporting interval. Viewing these statistics at the individual dataset level allows system administrators to identify storage "hot spots" in larger multi-tenant systems--particularly those with many VMs or containers operating essentially independent workloads.

This sample output shows activity which has taken place in the most recent second, on a the ssd zpool of a ZFS virtualization host:

[email protected]:~# ioztat -y -c1 ssd
dataset                                         w/s      wMB/s        r/s      rMB/s   wareq-sz   rareq-sz
ssd                                            0.00       0.00       0.00       0.00       0.00       0.00
   images                                      0.00       0.00       0.00       0.00       0.00       0.00
      DC1                                     17.96       0.10       3.99       0.05       5.66      12.29
      DC2                                     21.95       0.14       0.00       0.00       6.59       0.00
      QB                                       2.00       0.01       0.00       0.00       7.17       0.00
      SAP-TC                                   2.99       0.02       0.00       0.00       6.83       0.00
      SAP4-WIN2019                             2.99       0.05       0.00       0.00      17.07       0.00
      nagios                                   0.00       0.00       0.00       0.00       0.00       0.00
      qemu                                     0.00       0.00       0.00       0.00       0.00       0.00
         autostart                             0.00       0.00       0.00       0.00       0.00       0.00
   iso                                         0.00       0.00       0.00       0.00       0.00       0.00
   unsnapped                                   0.00       0.00       0.00       0.00       0.00       0.00
      rp9                                      0.00       0.00       0.00       0.00       0.00       0.00

For the most part, ioztat behaves the same way that the system standard iostat tool does, with similar arguments.

usage: ioztat [-h] [-s SORT] [-i INTERVAL] [-c COUNT] [-y] [-b]
              pool [pool ...]

iostat for ZFS datasets

positional arguments:
  pool         ZFS pool

optional arguments:
  -h, --help   show this help message and exit
  -s SORT      Sort by: name / wps / wMBps / rps / rMBps
  -i INTERVAL  Time between each report
  -c COUNT     Number of reports generated
  -y           Skip the initial "summary" report
  -b           Use binary (power-of-two) prefixes

The only required argument is the name of at least one zpool to monitor. Without any other arguments, ioztat first prints a summary record showing activity per dataset since the most recent system boot, then prints a new record showing the most recent activity once per second. The -i argument can be used to change the report interval, and the -c argument can be used to limit ioztat to a certain number of intervals before exiting.

For those who wish a continually-updated, easy to read summary of pool activity, watch -n1 ioztat poolname -c1 -y will suit nicely.

Comments
  • Rework formatting

    Rework formatting

    This makes ioztat output more closely resemble zpool iostat:

                                  operations      bandwidth         opsize
    dataset                       read   write    read   write    read   write
    --------------------------  ------  ------  ------  ------  ------  ------
    rpool
       ROOT
          default                   23       0   62.6K       0    2.7K     759
       cvs                           0       0    1.6K     892   24.2K  108.4K
       home                          0       0       0       0       0       0
          freaky                     3       2   29.7K    8.3K   10.2K    3.9K
    

    This also adds -e for exact values (-p in zpool iostat, which we're currently using elsewhere) and -H for scripted mode that drops the header and separates fields by tabs.

    The intent is for this to eventually fix #23, probably by adjusting the space for dataset.

    opened by Freaky 12
  • Painting the bikeshed

    Painting the bikeshed

    Should bandwidth be changed to throughput? This is more correct, but deviates from zpool-iostat(8).

    Should we default to binary mode, since that's all zpool-iostat(8) supports?

    Should we default to a single iteration? This is what iostat(1) and zpool-iostat(8) both do. They also accept interval and count as positional arguments.

    Is + as an ellipsis for truncated dataset names reasonable, or should it be ... despite the additional space it uses?

    opened by Freaky 8
  • Overwrite old reports in terminal by default

    Overwrite old reports in terminal by default

    I thought that this script put a little too much into the terminal. This PR makes it so that only the most recent report is visible, which can be disabled with the new -n argument.

    opened by Aluminite 4
  • No warning if zfs version too low (was: doesn't find datasets on Debian Buster, zfs 0.7.12-2)

    No warning if zfs version too low (was: doesn't find datasets on Debian Buster, zfs 0.7.12-2)

    ioztat -Hey as suggested by the man page doesn't return anything.

    ioztat -SIn rpool, also suggested by the man page, gives "dataset not found: 'rpool'" (rpool of course exists).

    Is there a way to further debug this? Am I missing a dependency?

    opened by steffen-AEI 3
  • Make numeric precision variable

    Make numeric precision variable

    Matching ZFS number formatting behaviour, use between 0 and 2 decimal places according to available space, and further reduce the column size to again match zpool-iostat and make more room for dataset names.

    This includes the careful rounding behaviour of the previous implementation, which I believe ZFS lacks.

    opened by Freaky 3
  • Fix -y when not combined with -c

    Fix -y when not combined with -c

    Fix an error when args.count is None. While I'm here also fix a count of zero that was ignored due to zero being falsey.

    Simplify the logic slightly by changing skipsummary from a bool to a generic skip count.

    opened by Freaky 3
  • Output is insensitive to terminal width

    Output is insensitive to terminal width

    The current output width is 106 columns, which by happy coincidence is precisely the width of my usual terminals.

    This coincidence seems unlikely to be universal, and some sensitivity to this would seem to be in order.

    opened by Freaky 3
  • Fix and simplify overwrite mode

    Fix and simplify overwrite mode

    This fixes overwrite mode by replacing the line counting and iterated cursor repositioning with a clear screen and placing the cursor at 1,1 - exactly what GNU watch does.

    I've also done some tidying, hoisting some iteration and filtering logic into dedicated generator functions which helps to further simplify the main loop.

    opened by Freaky 3
  • python3 shebang--does this work in FreeBSD?

    python3 shebang--does this work in FreeBSD?

    @Freaky, I discovered after merging your patches today that ioztat does not work on the Python2.7 that my Ubuntu workstation at the home office provides with "env python". I had to update the shebang to "env python3" to get it to work again.

    This should be fine for Ubuntu, but does it work in FreeBSD-land as well? I don't know if there's a python3 executable in your environment, or whether there is only python and you get what you get.

    opened by jimsalterjrs 3
  • Format name with all intermediates for disambiguation

    Format name with all intermediates for disambiguation

    Fixes #5

    This replaces the "just-the-rightmost-dataset-name-component" logic by "components-of-the-name-unique-after-the-previous-one". Output with default options looks like

    dataset                                         w/s      wMB/s        r/s      rMB/s   wareq-sz   rareq-sz
    zroot                                          0.00       0.00       0.00       0.00       0.00       0.00
       ROOT/default                                0.00       0.00       3.88       0.00       0.00       0.16
       tmp                                         0.00       0.00       0.00       0.00       0.00       0.00
    

    (e.g. here instead of default being indented two, and ROOT being invisible, we get the dataset-name-components that distinguish it from zroot, the previously-printed name)

    opened by adriaandegroot 3
  • Unmounted datasets lead to confusing gaps in output

    Unmounted datasets lead to confusing gaps in output

    Stats for unmounted datasets aren't exported, so you can end up with incomplete hierarchies in ioztat output. For example here rpool and rpool/ROOT aren't mounted, leaving rpool/ROOT/default without the path leading up to it:

    dataset                                         w/s      wMB/s        r/s      rMB/s   wareq-sz   rareq-sz
          default                                  0.00       0.00       0.97       0.00       0.00       0.05
    

    Similarly, rpool/usr/local/poudriere/ports/default misses the mountpoint-less poudriere/ports segments:

       usr                                         0.00       0.00       0.97       0.00       0.00       0.01
          local                                    0.00       0.00       0.00       0.00       0.00       0.00
                   default                         0.00       0.00       0.00       0.00       0.00       0.00
    
    opened by Freaky 3
  • using dataclasses

    using dataclasses

    Hi,

    I started of my own version of this, but found yours basically the last moment … My name of choice was objset, but the poll is finished I guess.

    I used dataclasses - ported here. And updated the fields to match current OpenZFS git.

    opened by commonism 7
  • no activity reported for zfs replication

    no activity reported for zfs replication

    is it expected that ioztat does not seem to report activity for zfs replication ?

    if this is not a bug but by design, maybe it's worth mentioning in the

    opened by devZer0 0
  • Packages

    Packages

    Produce .deb/.rpm/etc packages. sanoid is already doing this. I'm currently using sanoid package in latest ubuntu release.

    Getting these included will take some time but is worthwhile I think.

    enhancement help wanted 
    opened by morganchristiansson 2
Releases(v2.0.1)
  • v2.0.1(Mar 4, 2022)

  • v2.0.0(Mar 3, 2022)

    Added

    • An iostat(8) man page
    • CHANGELOG.md, this Changelog
    • -e and -H flags for use in scripting [#26]
    • -I flag for totals since last update instead of per-second
    • -o flag to overwrite prior reports [#9] [#20]
    • -P and -p flags to override dataset name display [#19]
    • -S flag for including child dataset statistics in parents [#32]
    • -T d|u flag for adding a timestamp to each report [#25]
    • -V, --version flags [#13] [#14]
    • -x flag for extended statistics, including unlink counts under -xx [#33]
    • interval and count positional arguments [#31]

    Changed

    • Shebang line is now /usr/bin/env python3 [#18]
    • Formatting has changed from resembling GNU iostat(8) to ZFS zpool-iostat(8) [#26]
    • Header is now printed periodically on a tty unless -N is specified [#26]
    • Display is clamped to the terminal width by truncating dataset names if necessary [#26]
    • Sort field names have been changed, with fallbacks for compatibility [#26]
    • Sort options are now case-insensitive [#26]
    • count now defaults to 1 unless an interval is specified [#31]
    • dataset is now an optional argument [#20]
    • Binary (1024-based) formatting is now default, with new -D flag for decimal
    • Average I/O sizes are now hidden beyind -x flag by default to reduce clutter [#33]
    • Exit with an error if a requested dataset does not exist or is not mounted

    Removed

    • -b flag. Binary mode is now the default to match other iostat tools.

    Fixed

    • count and interval must now be positive [#31]
    • WIFSIGNALED() status is now propagated properly to caller
    • Sleep interval now adjusts to compensate for runtime
    • Unhandled BrokenPipeError exception on SIGPIPE
    • Unhandled FileNotFoundError exception on Linux if a dataset is destroyed while enumerating files in /proc
    • Unhandled CalledProcessError exception on FreeBSD if a requested pool does not exist when using sysctl(8) fallback
    Source code(tar.gz)
    Source code(zip)
  • v1.1.0(Jan 20, 2022)

    Thanks to the efforts of @Freaky and @adriaandegroot, ioztat has new features, improved output readability, and now works on FreeBSD as well as Linux!

    New features include initial selection at the dataset level rather than pool level, optional suppression of zero-activity datasets with -z, and optional suppression of recursion into child datasets with -n.

    Source code(tar.gz)
    Source code(zip)
  • v1.0.0(Jan 12, 2022)

    The first release of ioztat builds on efforts from the Reddit r/zfs community, including:

    u/55rzs (initial creation) u/d1722825 (substantial refactoring and cleanup) u/mercenary_sysadmin (addition of the -y flag to allow for easy use with the GNU watch command)

    Enjoy!

    Source code(tar.gz)
    Source code(zip)
Owner
Jim Salter
Jim Salter
Machine Learning powered app to decide whether a photo is food or not.

Food Not Food dot app ( 🍔 🚫 🍔 ) Code for building a machine Learning powered app to decide whether a photo is of food or not. See it working live a

Daniel Bourke 48 Dec 28, 2022
Beatsaber for Python

beatsaber Beatsaber for Python It was automatically generated with mkpylib. If you're reading this message, it m

Shawn Presser 3 Jul 30, 2021
A middle-to-high level algorithm book designed with coding interview at heart!

Hands-on Algorithmic Problem Solving A one-stop coding interview prep book! About this book In short, this is a middle-to-high level algorithm book de

Li Yin 1.8k Jan 02, 2023
Gitlab py scripts

Gitlab py scripts The code can be used to gather the list of GitHub groups/projects and the permissions of the users in those groups/projects. group/p

Roghuchi 1 Aug 29, 2022
Pokemon catch events project to demonstrate data pipeline on AWS

Pokemon Catches Data Pipeline This is a sample project to practice end-to-end data project; Terraform is used to deploy infrastructure; Kafka is the t

Vitor Carra 4 Sep 03, 2021
Groupe du projet Python en 2TL2-4

Présentation Projet EpheCom Ce logiciel a été développé dans le cadre scolaire. EpheCom est un logiciel de communications - vocale et écrite - en temp

1 Dec 26, 2021
Experimental proxy for dumping the unencrypted packet data from Brawl Stars (WIP)

Brawl Stars Proxy Experimental proxy for version 39.99 of Brawl Stars. It allows you to capture the packets being sent between the Brawl Stars client

4 Oct 29, 2021
Plock : A stack based programming language

Plock : A stack based programming language

1 Oct 25, 2021
Oppia is an online learning tool that enables anyone to easily create and share interactive activities

Oppia is an online learning tool that enables anyone to easily create and share interactive activities (called 'explorations'). These activities simulate a one-on-one conversation with a tutor, makin

Oppia 4.7k Dec 29, 2022
Interactive class notebooks for ECE4076 Computer Vision, weeks 1 - 6

ECE4076 Interactive class notebooks for ECE4076 Computer Vision, weeks 1 - 6. ECE4076 is a computer vision unit at Monash University, covering both cl

Michael Burke 9 Jun 16, 2022
Compiler Final Project - Lisp Interpreter

Compiler Final Project - Lisp Interpreter

2 Jan 23, 2022
Ant Colony Optimization for Traveling Salesman Problem

tsp-aco Ant Colony Optimization for Traveling Salesman Problem Dependencies Python 3.8 tqdm numpy matplotlib To run the solver run main.py from the p

Baha Eren YALDIZ 4 Feb 03, 2022
A Python module for decorators, wrappers and monkey patching.

wrapt The aim of the wrapt module is to provide a transparent object proxy for Python, which can be used as the basis for the construction of function

Graham Dumpleton 1.8k Jan 06, 2023
🇮🇳 A Indian Flag Animation Project Made With Python

🇮🇳 A Indian Flag Animation Project Made With Python

MuFaz-TG 2 Oct 21, 2022
Adjust the white point, gamma or make your XDR display darker without losing HDR peak luminance or the ability to adjust display brightness

XDR Tuner Adjust the white point, gamma or make your XDR display darker without losing HDR peak luminance or the ability to adjust display brightness

François Simond 16 Dec 28, 2022
Some out-of-the-box hooks for pre-commit

pre-commit-hooks Some out-of-the-box hooks for pre-commit. See also: https://github.com/pre-commit/pre-commit Using pre-commit-hooks with pre-commit A

pre-commit 3.6k Dec 29, 2022
OnTime is a small python that you set a time and on that time, app will send you notification and also play an alarm.

OnTime Always be OnTime! What is OnTime? OnTime is a small python that you set a time and on that time, app will send you notification and also play a

AmirHossein Mohammadi 11 Jan 16, 2022
Telegram bot for Urban Dictionary.

Urban Dictionary Bot @TheUrbanDictBot A star ⭐ from you means a lot to us! Telegram bot for Urban Dictionary. Usage Deploy to Heroku Tap on above butt

Stark Bots 17 Nov 24, 2022
Stop ask your soraka to ult you, just ult yourself

Lollo's auto-ultimate script Are you tired of your low elo friend who can't ult you with soraka when you ask for it? Use Useless Support and just ult

9 Oct 20, 2022
Aevsploit İçin Destekde Bulun Papara: 1427113016

Aevsploit İçin Destekde Bulun Papara: 1427113016 Toolu Geliştirmek İçin Fikirlerinizi Bekliyorum Telegram

9 Jun 07, 2022