This issue is intended to start a broad discussion about improving CHAP usability, possibly generating specific issues or sub-projects as the discussion goes on. I am hoping to elicit feedback from both developers and scientists, related either to ideas and/or thoughts on implementation or to relevant anecdotes related to peoples' use of CHAP.
Disclaimer: Specific ideas and / or concerns described below reflect my own limited use and imperfect knowledge of everything CHAP already does or can do, as well as personal opinions, and so may differ widely from other peoples’ experience or priorities. Nonetheless I have had enough conversations among our developers and scientists to believe that this is a timely and important discussion.
I would start by separating this issue into three stages of interaction with CHAP:
- Configuring CHAP: I find configuring a CHAP pipeline outside the context of pipelines that are automatically generated and executed from scans highly cumbersome and difficult to remember and to generalize. pipeline files are complex, and the existence of templates, while useful, does not greatly alleviate that difficulty. Automation is an answer to this problem that has already proven successful. But at-scan-time automation is not sufficient, since one of the main virtues and objectives of CHAP was to permit re-executing the same or modified pipelines on data with conditions or in ways that differ from what is done during automation.
- Interacting with & running CHAP: We routinely experience the need to inspect and/or modify parts of a pipeline dynamically – including in ways not originally anticipated. In idealized view, all or virtually all functionality in CHAP should be (easily, naturally) accessible from within a python script that imports CHAP. This comment parallels at least 2 ongoing efforts already underway (for EDD and for tomography). So it may already be possible. If so, I am interested in using this space (or a separate issue) to pose additional questions about how to do so.
- Interacting with CHAP outputs, e.g. nexus files: I think CHAP’s current use of nexus files for outputs has been a big success. They can house more or less arbitrary metadata and data, they point towards the use of standards, and they can be read by both standard h5 readers as well as more specific nexus file readers like nexpy. However in my view, practical use of those files is difficult. I envision a “batteries included” version of or sister package to CHAP would hold a suite of functions or class extensions that would ease the burden I am discussion.
Item 3 – further detail: Personally, interacting with a nexus file requires a cumbersome step of browsing its structure with either “h5dump” command or by using nexpy, then transcribing trees into code or scripts that load the one or more datasets I am after.
In other words, determining what data a nexus files contains is an ad-hoc process of browsing and discovery that must necessarily be repeated for every nexus file I encounter.
Another way of describing this situation is that a nexus file merely replaces a directory structure (e.g. for raw data on DAQ) with a tree structure (group names), wherein the group names encode critical information about what’s there. An extension to CHAP I imagine would contain a suite of functions or class extensions that eliminate or all but eliminate this burden.
For instance, I imagine a function such as chap_help(‘output.nxs’) that would yield either plain-english or structured contents of the file, including CHAP-specific conventions such as the the raw data it is built from (such as specfiles and scans), the type of data if contains (edd, saxswaxs, etc), what kind of data it contains, etc.
I envision such a function (or nexus file obect extension?) as having two purposes: one for interactive use and another for scripting. For instance,
datasets = chap_help(‘output.nxs’, type=’edd’)
that would yield a list of the available edd datasets in the file.
Thoughts/Feedback/Additional notes welcoe.
This issue is intended to start a broad discussion about improving CHAP usability, possibly generating specific issues or sub-projects as the discussion goes on. I am hoping to elicit feedback from both developers and scientists, related either to ideas and/or thoughts on implementation or to relevant anecdotes related to peoples' use of CHAP.
Disclaimer: Specific ideas and / or concerns described below reflect my own limited use and imperfect knowledge of everything CHAP already does or can do, as well as personal opinions, and so may differ widely from other peoples’ experience or priorities. Nonetheless I have had enough conversations among our developers and scientists to believe that this is a timely and important discussion.
I would start by separating this issue into three stages of interaction with CHAP:
Item 3 – further detail: Personally, interacting with a nexus file requires a cumbersome step of browsing its structure with either “h5dump” command or by using nexpy, then transcribing trees into code or scripts that load the one or more datasets I am after.
In other words, determining what data a nexus files contains is an ad-hoc process of browsing and discovery that must necessarily be repeated for every nexus file I encounter.
Another way of describing this situation is that a nexus file merely replaces a directory structure (e.g. for raw data on DAQ) with a tree structure (group names), wherein the group names encode critical information about what’s there. An extension to CHAP I imagine would contain a suite of functions or class extensions that eliminate or all but eliminate this burden.
For instance, I imagine a function such as chap_help(‘output.nxs’) that would yield either plain-english or structured contents of the file, including CHAP-specific conventions such as the the raw data it is built from (such as specfiles and scans), the type of data if contains (edd, saxswaxs, etc), what kind of data it contains, etc.
I envision such a function (or nexus file obect extension?) as having two purposes: one for interactive use and another for scripting. For instance,
datasets = chap_help(‘output.nxs’, type=’edd’)that would yield a list of the available edd datasets in the file.
Thoughts/Feedback/Additional notes welcoe.