Report the offset of an input stream error

[gennaroprota]

archive_exception said only that the input failed, without giving a clue as to where the error occurred. It now names the offset at which the input went wrong, for text, binary and XML input archives alike.

[robertramey]

Note that the archive interface doesn't presume that an archive is a type of io stream. This opens the way to certain special purpose archives which might do something like implement a deep copy, calculate a CRC, etc, etc, There might be an example of such a custom archive in the documentation. Consider this when adding to stream behavior.

[gennaroprota]

Hmm, thanks for the heads-up. However, the archive interface is untouched: basic_iarchive, common_iarchive, interface_iarchive and archive_exception are all unchanged. The offset is produced in three places that already hold a stream or a stream buffer of their own:

• basic_binary_iprimitive has std::basic_streambuf<Elem, Tr> & m_sb as a member.
• basic_text_iprimitive<IStream> already needs a real stream for ios_flags_saver and basic_istream_locale_saver.
• basic_xml_iarchive already needs get_is(), and the grammar calls get(), fail() and eof() on it. The two implementations return std::istream & and std::wistream &.

demo_trivial_archive.cpp, the example archive_reference.html points at, has a save_binary and no stream at all, and includes none of these headers. A CRC or deep copy archive would never reach this code, and if it threw archive_exception it would get the message it always got.

But one point is worth mentioning: an archive deriving from basic_xml_iarchive whose get_is() returned something stream-like but not a std::basic_istream would now fail to compile, since the offset comes from get_is().rdbuf(). Both in-tree implementations return real streams but, in general, this might break some code. What do you think?

[robertramey]

OK, I was just concerned that the fact that the archive interface doesn't presume a stream might be overlooked. I would be easy to do. If one had nothing else to do, it might be a good idea to add some more archives examples like calculating a CRC or a deep copy, or something for which a stream might not be useful. We're splitting hairs here - you're really doing a great job and understanding a lot of the very subtle details. Few who have looked into the library realize these issues - hence lots of suggestions are not totally thought out.

[correaa]

Yes, this is an example of a Boost-based archive that it is not backed by a std::stream:

https://github.com/llnl/b-mpi3/blob/master/include/mpi3/package_archive.hpp

I don't know of other examples.

[robertramey]

I skimmed through your package archive code and related b-mpi2 project. A couple of observations:

a) This is quite the body of work!!!! I'm impressed with the volume, scope, effort, apparent quality of implementation and the documentation in the *.md file. On the face of it - seems quite boost worthy. Of course if/when it gets reviewed, we always find things to address, but still ...

b) It's not clear to me what the purpose of the package_archive is and what it's for. If it's an implementation detail not interesting to the user, so it doesn't belong in the "official" documentation, you could include this information as comments in the headers. FWIW it's my custom to do this: user documentation in the documentation, implementation documentation as part of the headers.

c) The directory "serialization hack" which seems to just include code from the serialization library got my attention. I suspect it's there so that one need not link to the serialization library itself. If I'm correct about this it seems short sighted. Seems that improvements/bug fixes in the serialization library won't be propagated to this library. Not that I don't appreciate that linking with a boost library either statically or dynamically is kind of pain.

It's not clear what your ultimate goal here is? Given the amount of effort you've invested, It seems that getting accepted by boost would be the ultimate goal. But of course I can' read your mind. In any case, congratulations on a professional piece of work. When I wrote the library, I had the hope that others would make their own special purpose archives. Seems that hasn't occurred as often I had hoped/expected. In any case, thank you for your efforts and your contributions to making the serialization better!

[correaa]

Hi Robert,

First of all, thank you for initiating the great Boost.Serialization library.
I am convinced that Serialization shouldn't be an afterthought, I think it is a fundamental piece of generic programming.

I skimmed through your package archive code and related b-mpi2 project. A couple of observations:

a) This is quite the body of work!!!! I'm impressed with the volume, scope, effort, apparent quality of implementation and the documentation in the *.md file. On the face of it - seems quite boost worthy. Of course if/when it gets reviewed, we always find things to address, but still ...

Thank you Robert, yes, Joaquin Lopez Muñoz suggested the same. For context, B-MPI3 is no more than a modernization (from scratch) of Boost.MPI to C++1X plus whatever was added from MPI v1.0 to the current MPI v5.0 (Remote-Memory-Access and Shared NUMA Memory)

b) It's not clear to me what the purpose of the package_archive is and what it's for. If it's an implementation detail not interesting to the user, so it doesn't belong in the "official" documentation, you could include this information as comments in the headers. FWIW it's my custom to do this: user documentation in the documentation, implementation documentation as part of the headers.

You are right, next time I touch that code I will document what it does.
For the moment it is really an implementation detail, not exposed directly at the moment.

It is an implementation detail that provides the magic to communicate any serializable value type through MPI.
At the end it is not very difference from an archive on streams, MPI Packages can receive certain "built-in" types (including "runtime" types, also called MPI derived types). The archive serves as an intermediary between the "user" (the B-MPI3 library internals) and the MPI Package handle (C-interface, https://www.mpich.org/static/docs/v4.1/www3/MPI_Pack.html plus the MPI_Datatype infrastructure)

c) The directory "serialization hack" which seems to just include code from the serialization library got my attention. I suspect it's there so that one need not link to the serialization library itself. If I'm correct about this it seems short sighted. Seems that improvements/bug fixes in the serialization library won't be propagated to this library. Not that I don't appreciate that linking with a boost library either statically or dynamically is kind of pain.

Yes, this is an optional hack that allows to include Boost.Serialization without linking.
It is only active if the _MAKE_BOOST_SERIALIZATION_HEADER_ONLY is defined.

I have an early user that wanted to add B-MPI3 but, according to their own assessment a long time ago, they didn't want to add binary library dependencies, so I had to implement this.

Yes, as any hack it is short sighted.

Having said that, I think the core of Boost.Serialization could in fact be header only.
In fact I think it is the "Archives" that should handle the complexity that sometimes need library compilation.

It's not clear what your ultimate goal here is? Given the amount of effort you've invested, It seems that getting accepted by boost would be the ultimate goal. But of course I can' read your mind.

The ultimate goal is to incorporate MPI to moderm HPC C++ programs, and generalize the communication of complex types.
Two codes use it now, https://qmcpack.org and https://gitlab.com/npneq/INQ

Yes, the original idea was to include it in Boost, the B in B-MPI3 is for Boost.
At the moment I am very busy reviewing Boost.Multi, which unsurprisingly is made to be compatible with what I call generic serialization (including Boost.Serialization).
Another possible outcome is that this interface will be end up, with changes, being proposed an official C++ interface of MPI https://link.springer.com/chapter/10.1007/978-3-032-07194-1_10

The feature requests I made to Boost.Serialization in issues and conversations in Slack, are inspired in my experience with both pieces work (Multi and B-MPI3), specifically a) make Serialization header-only b) accept generic named-value-pairs c) generate names automatically for named-value pairs

Talking about hacks, I have this piece of glue code to work with Boost.Serialization without depending on it: https://github.com/correaa/boost-multi/blob/develop/include/boost/multi/detail/serialization.hpp

In any case, congratulations on a professional piece of work. When I wrote the library, I had the hope that others would make their own special purpose archives. Seems that hasn't occurred as often I had hoped/expected. In any case, thank you for your efforts and your contributions to making the serialization better!

I have to admit that it took me a lot of time to create custom archives that worked with Boost.Serialization.
If remember correctly the hardest part was to decide at what level implement the actual code given the multiple layers of inheritance.

As an early exercise I started with this, another archive type for YML files:
https://gitlab.com/correaa/boost-archive-yml

Also I think the serialization of indirect object like pointers is something was very difficult to get right and I had to ignore. It is possible that they do not work with my archive types.

[robertramey]

"Having said that, I think the core of Boost.Serialization could in fact be header only.
In fact I think it is the "Archives" that should handle the complexity that sometimes need library compilation."

I think this goal is largely realized by using "polymorphic_archives". I believe that it should be possible to compile an app against this header without including any archive code and without linking to the serialization library. Note that the serialization library compiled code is about archives. I tried to make this distinction clear by creating two separate namespaces: serialization and archive. This decision (like all my decisions - lol) received its share of criticism but not enough to make me change it. Given all this, I don't think the files in serialization_hack should be necessary. If they are, perhaps this might be eliminated by some careful refactoring. Or perhaps not.

Some small refactoring might make serialization of pointers to objects optional - you might like this idea.

Including serialization a YML and JSON would be a great addition to the library - be it as examples, or as pre-made archives added to the package.

You guys are doing a great job! I feel like I'm going to live forever!

[correaa]

"Having said that, I think the core of Boost.Serialization could in fact be header only. In fact I think it is the "Archives" that should handle the complexity that sometimes need library compilation."

I think this goal is largely realized by using "polymorphic_archives".

Yes, polymorphic can serve this purpose because they effectively behave like pointer-to-implementation.
But it mixes up with other polymorphic semantics.

(Also has other problems as we discussed before, some of these problems could be mitigated with a more general/flexible handling of named-value-pair)

I believe that it should be possible to compile an app against this header without including any archive code and without linking to the serialization library.

I have to take a look again, my understanding (perhaps outdated) is that you can write a header-only archive type, but as soon as this archive requires basic_archive OR the application uses serialization then linking to Boost.Serialization becomes mandatory, that is why I added the hack.
But I am not 100% sure, maybe I am mistaken.

Note that the serialization library compiled code is about archives. I tried to make this distinction clear by creating two separate namespaces: serialization and archive. This decision (like all my decisions - lol) received its share of criticism but not enough to make me change it.

I don't think it was a bad choice, but I understand that it creates confusion. For example, the code for the archives needs to be linked to boost_serialization. there is no boost_archives, although if it works like you are saying it would have made more sense to have libboost_archive object.

Given all this, I don't think the files in serialization_hack should be necessary. If they are, perhaps this might be eliminated by some careful refactoring. Or perhaps not.

It would be nice if that is the case, I will explore it again when I look at this.

Some small refactoring might make serialization of pointers to objects optional - you might like this idea.

That is genuinely a good idea.
The way I see it is that "generic" serialization shouldn't bother with serializing objects that are not themselves "values" in the generic sense (or proxies references to values). That includes pointers and even smart pointers.
The efforts to serialize structures with internal pointers is admirable but IMO it is outside the scope of serialization, and too general of a problem to handle.

Including serialization a YML and JSON would be a great addition to the library - be it as examples, or as pre-made archives added to the package.

Yes, I proposed the guys in Boost.JSON to add Boost Json archives since they have all the technology to do it.
The conceptual problem I see is that Serialization and Parsing are really two different problems that are related but they are conflated far too often.

You guys are doing a great job! I feel like I'm going to live forever!

I hope I had more time to dedicate to actually contribute code for Boost.Serialization or at least contribute with full fledged ideas, but Gennaro is doing a great job indeed.