String handling routines

[ivan-pi]

Let's start a discussion on routines for string handling and manipulation. The thread over at j3-fortran already collected some ideas:

• split - given a separator, splits the string into some form of array
• upper/lower - convert a character string to all upper/lower case

The discussion also mentioned the proposed iso_varying_string module, which was supposed to include some string routines. I found three distinct implementations of this module:

• ISO_VARYING_STRING Module by Rich Townsend
• iso_varying_string implementation by Brad Richardson (@everythingfunctional)
• ISO_VARYING_STRING due to J.L.Schonfelder (author of the iso_varying_string proposal; the module dates back to 1998)

I also found the following Fortran libraries targeting string handling:

• Strings For Fortran by Brad Richardson @everythingfunctional
• Fortran Character String Utilities by George Benthien
• M_STRINGS from Urban Jost @urbanjost (part of General-Purpose Fortran tools)
• String_Functions by David Frank
• StringiFor by Stefano Szaghi @szaghi
• fortranString from @bceverly
• fortran-string-utility-module by @tomedunn
• fortran-string from @dongli
• strings by @jchristopherson
• fortran_libstring from @koiking213
• fortran-strings by @eengl
• flibs by @arjenmarkus contains several modules for handling strings
• functional-fortran by @milancurcic implements several functions on strings
• ZstdFortranLib by @zbeekman has some conversion to/from other intrinsic kinds, sub, gsub, split, join, and conversion on concatenation. WIP though
• LibString, F77 string library by Giulio Vistoli & Alex Pedretti

It is likely that several of the tools in the list of popular Fortran projects also contain some tools for working with strings. Given the numerous implementations it seems like this is one of the things where the absence of the standard "... led to everybody re-inventing the wheel and to an unnecessary diversity in the most fundamental classes" to borrow the quote of B. Stroustrup in a retrospective of the C++ language.

For comparison here are some links to descriptions of string handling functions in other programming languages:

• Python: String Methods, String constants, custom string formatting, template strings
• Ruby: String class & methods
• D: std.string, std.utf, std.path, std.regex, std.ascii (see related issue Proposal for ascii #11), std.encoding, std.windows.charset, std.conv
• C: C string handling on wikipedia
• C++: std::string class, <string> header, C++ string handling on Wikipedia, Boost libraries for string and text processing
• Julia: Strings, Common operations
• MATLAB: Characters and Strings
• Rust: Strings - Rust By Example, Primitive Type str, Struct std::string::String

Obviously, for now we should not aim to cover the full set of features available in other languages. Since the scope is quite big, it might be useful to break this issue into smaller issues for distinct operations (numeric converions, comparisons, finding the occurence of string in a larger string, joining and splitting, regular expressions).

My suggestion would be to start with some of the easy functions like capitalize, count, endswith, startswith, upper, lower, and the conversion routines from numeric types to strings and vice-versa.

[jacobwilliams]

This is a great summary! I feel like most of what we need has already been done in these projects (and others), so mainly we need to just gather it all together. Some important things to decide:

• what do we call the string class? (I vote string).
• what do we call the various individual methods?
• Are they going to be OO (s.lower()) or functional (lower(s))?

[everythingfunctional]

Should we base it on the ISO_VARYING_STRING module? If so, the class is VARYING_STRING and the procedures are functional (lower(s)).

Should we utilize intrinsic function names? like real(string, [kind, [status]]) and have it stop if the conversion fails and no status variable is provided?

[milancurcic]

functional-fortran implements several functions on strings:

Further, these functions (and their corresponding operators) are compatible with character strings: complement, empty, head, init, intersection, insert, last, reverse, set, sort, split, tail, and union.

(Caution: split in functional-fortran is not quite what's been discussed at j3-fortran repo. It merely splits the string in two and returns the first or second part)

[certik]

Thanks for this initiative and listing the current landscape. I think we definitely want stdlib to have good string support.

(For conversion from real/integer numbers to strings, I implemented a function str to be used like this: https://github.com/certik/fortran-utils/blob/b43bd24cd421509a5bc6d3b9c3eeae8ce856ed88/tests/strings/test_str.f90, implemented here and here, so one can do things like "Number i = " // str(i) // ".".)

@ivan-pi do you want to go ahead and create a table of the basic subroutines and let's brainstorm how they should be named, to be consistent with other languages and/or the above various string implementations if possible. And also if they should be functions or subroutines and what arguments to accept.

[certik]

@jacobwilliams is right about raising the question how to represent the string. We should start with that.

I would recommend (as usual) to have a lowest level API that operates on the standard Fortran (allocatable where appropriate) character. Then, have a higher level API that operates on a string type, and simply calls the lower level API. Regarding a name, see #26, it seems most people agree that the convention to name derived type is to append _t, so it would be string_t.

That way people can use these low level API routines right away. For example in my codes I do not need to modify any data structures and can start using it. The higher level string_t API can then be used by codes that choose to refactor them, or in new codes. If the syntax is not as nice, some people might opt for the lower level API anyway.

[everythingfunctional]

I would vote that the low level API be based on functions, pure and elemental where possible and appropriate. I would stick with the Fortran convention of optional status parameters where there is the possibility of things going wrong, and if one is not provided and something goes wrong it crashes. I have tended to use that convention in any routines that go from a string to some intrinsic like:

if (present(status)) then
    read(string, *) result
else
    read(string, *, iostat=status) result
end if

Honestly, I thought the ISO_VARYING_STRING standard did a great job of covering all of the intrinsic functions available for character(len=*) variables, and extending IO to work with that type (put, get, put_line). Aside from the strange interface for split I think it's a great starting point.

[ivan-pi]

My plan was to go through the libraries above and create a table of the most commonly available routines in the next days.

I agree we should consider both low-level routines which work directly on strings of type character(len=*) and a high-level string_t type.

The book Fortran Tools for VAX/VMS and MS-DOS by Jones & Crabtree contains a description of a Fortran string-handling library. Interestingly, they decided to use null-terminated strings like in C, meaning they needed to build a separate set of functions from the intrinsic ones (concatenation operator // and length function). They later used these tools to develop a compiler for a subset of the Fortran language itself! Their conclusion about strings was:

Fortran is often maligned for its lack of facilities for character-oriented processing. ... The apparent deficiency of Fortran for string manipulation is primarily because of the methods traditionally used rather than because of a shortcoming of the language itself. The main shortcoming of Fortran for string handling is the lack of a standard library of routines for often-needed functions. As Fortran programmers we are faced with a choice: we either invest the up-front effort required to create our own standard library or we live with the continuing effort of hacking together a solution each time we are presented with similar problems.

[certik]

Where is the latest ISO_VARYING_STRING implementation? Most links are dead by now. The only version I was able to find so far is this one: http://fortrangis.sourceforge.net/doc/iso__varying__string_8F90_source.html.

[ivan-pi]

Where is the latest ISO_VARYING_STRING implementation? Most links are dead by now. The only version I was able to find so far is this one: http://fortrangis.sourceforge.net/doc/iso__varying__string_8F90_source.html.

I have linked three distinct implementations in the top post. The links from the gfortran compiler pages are dead as well as the link in Modern Fortran Explained by MCR.

Edit: An informal description of the iso_varying_string module for Varying Length Character Strings in Fortran can be found at: http://numat.net/fortran/is1539-2-99.html

[certik]

@ivan-pi thanks. I like your plan. It looks like the iso_varying_string is in the "high-level" API category, as it operates on a VARYING_STRING derived type. Our low-level API would be similar, but operating directly on character(len=*).

[jacobwilliams]

Building the low-level API on character(len=*) variables will be problematic for some operations, since they can't be resized. The high-level API will need to call routines that operate on character(len=:),allocatable variables. So you may end up with two slightly different routines in some cases. So are there really three APIs?

• character(len=*)
• character(len=:),allocatable
• string_t or whatever we call it

That seems complicated to me... but it would cover all the bases...

[milancurcic]

I think there are two possible APIs here: intrinsic and derived-type one.

For the intrinsic API, character(len=*) works well for input strings. If the function will return a string of known size, you return a charecter(len=something) string. If unknown, you return an allocated character(len=:), allocatable string. User doesn't need to know which one it is.

I also see the intrinsic one as the starting point. Higher-level (derived type) implementation is likely to use the intrinsic API internally.

[everythingfunctional]

My understanding, and somebody correct me if I don't have this quite right, is that the ISO_VARYING_STRING standard was created before character(len=:), allocatable (around 2001 I think?), but then when character(len=:), allocatable was added to the standard, it was supposed to function like variable length strings, and so the former was mostly abandoned. However, I have found most compilers to be buggy with their implementation. Memory leaks when used as the return from a function, failure to properly reallocate on assignment, false-positive warnings about accessing uninitialized memory, etc.

If allocatable character actually worked we wouldn't a new derived type for strings. You would just use the intrinsic type and move on. But I think as written in the standard, it probably will never truly work properly in all cases (especially as in read statements, since other allocatable arrays don't and aren't supposed to).

If there is a new type for strings, I don't think a lower level library or API should be exposed, and it should probably not be based on allocatable characters.

[certik]

As I mentioned above, you can use this trick to return character(len=N) strings from functions. The downside is that the string operation gets executed twice --- once to compute the length in the pure procedure, and second time to actually return it. So we probably don't want to do it that way. What I was thinking is to do what @milancurcic suggested: use character(len=*) as well as character(len=N) where we can, and use character(len=:),allocatable to avoid doing the operation twice as I described above. And that's the low level API. Below I provide two examples: #69 (comment) and #69 (comment), to show one one would decide whether to expose character(len=*) or character(len=:), allocatable.

As @everythingfunctional mentioned, for example GFortran used to have huge problems with allocatable strings and leaked memory. The latest version has improved a lot. Given that this is standard Fortran, and stdlib is a standard library, I think it is ok if we depend on the standard, and if there are compiler bugs, we'll try to workaround them and ensure they are reported. Regarding read statements, see #14 that would handle that. I think we should at least try to create a consistent low level API, not give up without even trying. If it truly cannot be done, only then we'll have to do what you propose, and only expose the string_t type and report the bugs to compilers (and keep the list somewhere) and propose improvements to the language itself, so that it can be done in the future.

[everythingfunctional]

I thought you could only use intrinsic procedures in variable declaration statements. Learned something new. That's a neat trick, but like you said, not particularly efficient.