From cc8a122b57a46dcf4ec66072f7a62932e9e0f7e5 Mon Sep 17 00:00:00 2001 From: Nathan McRae Date: Wed, 14 Feb 2024 19:02:46 -0800 Subject: [PATCH] Flesh out readme --- SaneTsv/readme.md | 36 +++++++++++++++++++++++++++++++++--- 1 file changed, 33 insertions(+), 3 deletions(-) diff --git a/SaneTsv/readme.md b/SaneTsv/readme.md index 5a94db7..df15f83 100644 --- a/SaneTsv/readme.md +++ b/SaneTsv/readme.md @@ -20,7 +20,7 @@ Implementations of the format do not need to handle file reading and writing dir # Typed TSV -Typed TSV allows for typing of columns. All column names in a typed TSV must end with ':' (0x3A) and then one of the following types: +Typed TSV builds on Sane TSV to allow for typing of columns. All column names in a typed TSV must end with ':' (0x3A) and then one of the following types: - 'string' - 'boolean' @@ -49,10 +49,40 @@ Aside from the 'binary' column type, all fields must be UTF-8 encoded text. Each - 'uint32' and 'uint64' are unsigned 32 and 64 bit integers respectively. They should be formatted like this regex: `[1-9][0-9]*` - 'int32' and 'int64' are signed 32 and 64 bit integers respectively. They should be formatted like this regex: `-?[1-9][0-9]*` (except that '-0' is not allowed) +Typed TSV files should have the .ytsv extension (.ttsv is already used). + # Commented TSV -Commented lines start with a '#' character at the beginning of the line. Unescaped '#' characters are not allowed on a line that does not start with a '#'. Any '#' characters in fields must be escaped. +Commented TSV builds on Typed TSV and allows for more flexibility in the format by including line comments. They are kept distinct so that some applications of it can take advantage of the extra flexibility, while others can stick with the more restricted Typed TSV format. + +Commented lines start with a '#' character at the beginning of the line. Unescaped '#' characters are not allowed on a line that does not start with a '#'. Any '#' characters in fields must be escaped. Any unescaped '#' after the start of a line are errors. Comments must be UTF-8 encoded text. -Comments after the last record are an error. \ No newline at end of file +Comments are associated with the record beneath them. If a comment appears at the top of the file, it is associated with the file as a whole. + +Comments after the last record are an error. + +Commented TSV files should have the .ctsv extension. + +## Extending the Commented TSV Format + +Because it can include comments, this format lends itself well to extension. For example, if we wanted to extend the type system to include physical units, we could do so like this: + +``` +# UnitsTSV V1.0.0 +id:uint32\tdatetime:string\tmeasurement1:m:float64\tmeasurement2:v:float64\tmeasurement3:1/s:float64 +``` + +Note that extended formats must remain parseable by baseline parsers, hence we must include the base types after the new types. + +Extending formats may also have restrictions. For example, they could disallow record comments and only allow the file comment above the header. + +Extended formats may still use the .ctsv extension, though they could use a dedicated one as well. + +## Ideas for Extension + +- Physical units +- Multiformats +- ISO 8601 +- https://github.com/multiformats/unsigned-varint \ No newline at end of file