Map DateTime to google.protobuf.Timestamp and document temporal string formats

`DateTime` fields now use the well-known `google.protobuf.Timestamp` type
(with `google/protobuf/timestamp.proto` imported automatically) instead of
`string`. A Timestamp cannot be malformed the way a string can, and proto
consumers get language-native timestamp types. Note that a Timestamp is a
UTC instant, so a publisher's original UTC offset is not preserved;
`t.protobuf type: "string"` remains available to override.

To support this, `t.protobuf` gains two options usable by any scalar:

- `import:` maps a scalar to an externally defined proto type, emitting the
  needed `import` statement in `schema.proto`.
- `comment:` documents the expected format on each generated field, used by
  the built-in `string`-typed temporal scalars (`Date`, `LocalTime`,
  `TimeZone`) whose proto type is wider than the ElasticGraph type (e.g.
  `// ISO 8601 date`). Values are still validated at ingestion time, just
  as with JSON ingestion.

Author: jwils

elasticgraph-proto_ingestion/README.md

@@ -95,6 +95,22 @@ ElasticGraph.define_schema do |schema|
 end
 ```
 
+A custom scalar can also map to an externally defined proto type by passing `import:` with the
+proto file that defines it, and `comment:` documents the expected format on each generated field
+(useful when the proto type is wider than the ElasticGraph type):
+
+```ruby
+# in config/schema/phone_number.rb
+
+ElasticGraph.define_schema do |schema|
+  schema.scalar_type "PhoneNumber" do |t|
+    t.mapping type: "keyword"
+    t.json_schema type: "string"
+    t.protobuf type: "string", comment: "E.164 phone number"
+  end
+end
+```
+
 ### Stable Field Numbers
 
 `schema_artifacts:dump` automatically reads and writes `proto_field_numbers.yaml`
@@ -138,23 +154,30 @@ enums:
 
 The generated `schema.proto` uses these built-in scalar mappings:
 
-| ElasticGraph Type | Protobuf Type |
-|-------------------|------------|
-| `Boolean`         | `bool`     |
-| `Cursor`          | `string`   |
-| `Date`            | `string`   |
-| `DateTime`        | `string`   |
-| `Float`           | `double`   |
-| `ID`              | `string`   |
-| `Int`             | `int32`    |
-| `JsonSafeLong`    | `int64`    |
-| `LocalTime`       | `string`   |
-| `LongString`      | `int64`    |
-| `String`          | `string`   |
-| `TimeZone`        | `string`   |
-| `Untyped`         | `string`   |
+| ElasticGraph Type | Protobuf Type               |
+|-------------------|-----------------------------|
+| `Boolean`         | `bool`                      |
+| `Cursor`          | `string`                    |
+| `Date`            | `string`                    |
+| `DateTime`        | `google.protobuf.Timestamp` |
+| `Float`           | `double`                    |
+| `ID`              | `string`                    |
+| `Int`             | `int32`                     |
+| `JsonSafeLong`    | `int64`                     |
+| `LocalTime`       | `string`                    |
+| `LongString`      | `int64`                     |
+| `String`          | `string`                    |
+| `TimeZone`        | `string`                    |
+| `Untyped`         | `string`                    |
 
 Additionally:
+- `DateTime` uses the [well-known `Timestamp` type](https://protobuf.dev/reference/protobuf/google.protobuf/#timestamp);
+  `schema.proto` imports `google/protobuf/timestamp.proto` automatically. Note that a `Timestamp`
+  is a UTC instant, so a publisher's original UTC offset is not preserved.
+- `string`-typed temporal scalars (`Date`, `LocalTime`, `TimeZone`) are wider than the
+  ElasticGraph types they carry, so generated fields of these types document the expected format
+  in a comment (e.g. `// ISO 8601 date, e.g. "2024-11-25"`). Values are validated when events
+  are ingested, just as with JSON ingestion.
 - List types become `repeated` fields.
 - Lists of lists (e.g. `[[Float!]!]!`) are not supported because Protocol Buffers cannot represent
   them directly. Schema artifact generation raises an error identifying the unsupported field.

elasticgraph-proto_ingestion/lib/elastic_graph/proto_ingestion/schema_definition/api_extension.rb

@@ -21,21 +21,21 @@ module SchemaDefinition
       # Module designed to be extended onto an {ElasticGraph::SchemaDefinition::API} instance
       # to enable protobuf schema artifact generation.
       module APIExtension
-        # Maps built-in ElasticGraph scalar types to proto field types.
-        PROTO_TYPES_BY_BUILT_IN_SCALAR_TYPE = {
-          "Boolean" => "bool",
-          "Cursor" => "string",
-          "Date" => "string",
-          "DateTime" => "string",
-          "Float" => "double",
-          "ID" => "string",
-          "Int" => "int32",
-          "JsonSafeLong" => "int64",
-          "LocalTime" => "string",
-          "LongString" => "int64",
-          "String" => "string",
-          "TimeZone" => "string",
-          "Untyped" => "string"
+        # Maps built-in ElasticGraph scalar types to their `protobuf` configuration options.
+        PROTO_OPTIONS_BY_BUILT_IN_SCALAR_TYPE = {
+          "Boolean" => {type: "bool"},
+          "Cursor" => {type: "string"},
+          "Date" => {type: "string", comment: %(ISO 8601 date, e.g. "2024-11-25")},
+          "DateTime" => {type: "google.protobuf.Timestamp", import: "google/protobuf/timestamp.proto"},
+          "Float" => {type: "double"},
+          "ID" => {type: "string"},
+          "Int" => {type: "int32"},
+          "JsonSafeLong" => {type: "int64"},
+          "LocalTime" => {type: "string", comment: %(ISO 8601 local time, e.g. "14:23:12")},
+          "LongString" => {type: "int64"},
+          "String" => {type: "string"},
+          "TimeZone" => {type: "string", comment: %(IANA time zone identifier, e.g. "America/Los_Angeles")},
+          "Untyped" => {type: "string"}
         }.freeze
 
         # Wires up the protobuf extensions when this module is extended onto an API instance.
@@ -51,7 +51,8 @@ def self.extended(api)
             if type.is_a?(SchemaElements::ScalarTypeExtension)
               # Use the reverted (original) type name so that built-in scalars renamed via
               # `type_name_overrides` still resolve to their proto types.
-              type.protobuf type: PROTO_TYPES_BY_BUILT_IN_SCALAR_TYPE.fetch(type.type_ref.with_reverted_override.name)
+              options = PROTO_OPTIONS_BY_BUILT_IN_SCALAR_TYPE.fetch(type.type_ref.with_reverted_override.name)
+              type.protobuf(type: options.fetch(:type), import: options[:import], comment: options[:comment])
             end
           end
         end

elasticgraph-proto_ingestion/lib/elastic_graph/proto_ingestion/schema_definition/schema.rb

@@ -82,6 +82,7 @@ def initialize(
           @proto_enum_value_numbers_by_enum = normalize_proto_enum_value_number_mappings(proto_field_number_mappings)
           @used_field_numbers_by_message = {}
           @used_enum_value_numbers_by_enum = {}
+          @imports = ::Set.new
           @message_definitions_by_name = {}
           @enum_definitions_by_name = {}
           @type_name_by_message_name = {}
@@ -100,6 +101,7 @@ def to_proto
           sections = [
             %(syntax = "proto3";),
             "package #{@package_name};",
+            *render_imports,
             render_definitions
           ]
 
@@ -154,8 +156,12 @@ def register_type(type)
           when SchemaElements::ObjectInterfaceAndUnionExtension
             register_message(type)
           when SchemaElements::ScalarTypeExtension
-            # Scalars don't get their own proto definition. Resolving their proto field type below
+            # Scalars don't get their own proto definition, but scalars mapped to an externally
+            # defined proto type require its import. Resolving their proto field type below also
             # surfaces a clear error during registration when a custom scalar is not configured.
+            if (import = type.protobuf_import)
+              @imports << import
+            end
           else
             raise Errors::SchemaError, "Type `#{type.name}` cannot be converted to proto."
           end
@@ -177,7 +183,7 @@ def register_message(type)
             .filter_map(&:to_indexing_field)
             .map do |field|
               field_name = Identifier.field_name(field.name)
-              repeated, field_type = proto_field_type_for(
+              repeated, field_type, type_comment = proto_field_type_for(
                 field.type,
                 context_type_name: type.name,
                 context_field_name: field.name
@@ -189,19 +195,16 @@ def register_message(type)
                 name_in_index: field.name_in_index
               )
 
-              comment =
-                if field_name == field.name
-                  nil
-                else
-                  "source name: #{field.name}"
-                end
+              comment_parts = [] # : ::Array[::String]
+              comment_parts << "source name: #{field.name}" unless field_name == field.name
+              comment_parts << type_comment if type_comment
 
               FieldDefinition.new(
                 name: field_name,
                 type: field_type,
                 field_number: field_number,
                 repeated: repeated,
-                comment: comment
+                comment: comment_parts.empty? ? nil : comment_parts.join("; ")
               )
             end
 
@@ -352,7 +355,9 @@ def proto_field_type_for(type_ref, context_type_name:, context_field_name:)
             raise Errors::SchemaError, "Type `#{base_type_ref.unwrapped_name}` cannot be resolved for proto generation."
           end
 
-          [list_depth == 1, register_type(resolved)]
+          base_type_name = register_type(resolved)
+          base_type_comment = (SchemaElements::ScalarTypeExtension === resolved) ? resolved.protobuf_comment : nil
+          [list_depth == 1, base_type_name, base_type_comment]
         end
 
         def list_depth_and_base_type(type_ref)
@@ -367,6 +372,12 @@ def list_depth_and_base_type(type_ref)
           [list_depth, current]
         end
 
+        # Renders `import` statements for referenced external proto files as a single contiguous
+        # section. Returns `[]` when no external types are referenced.
+        def render_imports
+          return [] if @imports.empty?
+          [@imports.sort.map { |import| %(import "#{import}";) }.join("\n")]
+        end
         def render_definitions
           rendered_enums = @enum_definitions_by_name.values.sort_by(&:name).map { |definition| render_enum(definition) }
           rendered_messages = @message_definitions_by_name.values.sort_by(&:name).map { |definition| render_message(definition) }

elasticgraph-proto_ingestion/lib/elastic_graph/proto_ingestion/schema_definition/schema_elements/scalar_type_extension.rb

@@ -18,12 +18,28 @@ module ScalarTypeExtension
           # @dynamic protobuf_type
           attr_reader :protobuf_type
 
-          # Configures the protobuf type for this scalar type.
+          # Proto file to import for the configured protobuf type, if it is an externally defined type.
+          # @dynamic protobuf_import
+          attr_reader :protobuf_import
+
+          # Comment rendered on generated proto fields of this scalar type (e.g. to document a string format).
+          # @dynamic protobuf_comment
+          attr_reader :protobuf_comment
+
+          # Configures the protobuf type for this scalar type. In addition to proto scalar types,
+          # an externally defined message type can be used by passing `import:` with the proto file
+          # that defines it (e.g. a well-known type such as `google.protobuf.Timestamp`). When the
+          # proto type is wider than the ElasticGraph type (such as a `string`-typed `Date`),
+          # `comment:` documents the expected format on each generated field.
           #
-          # @param type [String] protobuf scalar type name
+          # @param type [String] protobuf type name
+          # @param import [String, nil] proto file to import for the type, when it is not a proto scalar
+          # @param comment [String, nil] comment rendered on generated fields of this type
           # @return [void]
-          def protobuf(type:)
+          def protobuf(type:, import: nil, comment: nil)
             @protobuf_type = type
+            @protobuf_import = import
+            @protobuf_comment = comment
           end
 
           # Returns this scalar's proto field type.

elasticgraph-proto_ingestion/sig/elastic_graph/proto_ingestion/schema_definition/api_extension.rbs

@@ -2,7 +2,7 @@ module ElasticGraph
   module ProtoIngestion
     module SchemaDefinition
       module APIExtension: ::ElasticGraph::SchemaDefinition::API
-        PROTO_TYPES_BY_BUILT_IN_SCALAR_TYPE: ::Hash[::String, ::String]
+        PROTO_OPTIONS_BY_BUILT_IN_SCALAR_TYPE: ::Hash[::String, ::Hash[::Symbol, ::String]]
 
         def self.extended: (::ElasticGraph::SchemaDefinition::API & APIExtension) -> void
         def proto_schema_artifacts: (?package_name: ::String) -> void

elasticgraph-proto_ingestion/sig/elastic_graph/proto_ingestion/schema_definition/schema.rbs

@@ -63,6 +63,7 @@ module ElasticGraph
         @proto_enum_value_numbers_by_enum: ::Hash[::String, ::Hash[::String, ::Integer]]
         @used_field_numbers_by_message: ::Hash[::String, ::Set[::Integer]]
         @used_enum_value_numbers_by_enum: ::Hash[::String, ::Set[::Integer]]
+        @imports: ::Set[::String]
         @message_definitions_by_name: ::Hash[::String, MessageDefinition]
         @enum_definitions_by_name: ::Hash[::String, EnumDefinition]
         @type_name_by_message_name: ::Hash[::String, ::String]
@@ -103,10 +104,11 @@ module ElasticGraph
           ::ElasticGraph::SchemaDefinition::SchemaElements::TypeReference,
           context_type_name: ::String,
           context_field_name: ::String
-        ) -> [bool, ::String]
+        ) -> [bool, ::String, ::String?]
         def list_depth_and_base_type: (
           ::ElasticGraph::SchemaDefinition::SchemaElements::TypeReference
         ) -> [::Integer, ::ElasticGraph::SchemaDefinition::SchemaElements::TypeReference]
+        def render_imports: () -> ::Array[::String]
         def render_definitions: () -> ::String
         def proto_enum_value_name: (::String, ::String) -> ::String
         def proto_zero_enum_value_name: (::String) -> ::String

elasticgraph-proto_ingestion/sig/elastic_graph/proto_ingestion/schema_definition/schema_elements/scalar_type_extension.rbs

@@ -4,8 +4,10 @@ module ElasticGraph
       module SchemaElements
         module ScalarTypeExtension: ::ElasticGraph::SchemaDefinition::SchemaElements::ScalarType
           attr_reader protobuf_type: ::String?
+          attr_reader protobuf_import: ::String?
+          attr_reader protobuf_comment: ::String?
 
-          def protobuf: (type: ::String) -> void
+          def protobuf: (type: ::String, ?import: ::String?, ?comment: ::String?) -> void
           def to_proto_field_type: () -> ::String
         end
       end

elasticgraph-proto_ingestion/spec/unit/elastic_graph/proto_ingestion/schema_definition/api_extension_spec.rb

@@ -29,11 +29,12 @@ module SchemaDefinition
 
           expect(proto_schema_from(results)).to include(
             "package sales.v1;",
+            'import "google/protobuf/timestamp.proto";',
             "string id = 1;",
             "int32 count = 2;",
             "double cost = 3;",
             "bool active = 4;",
-            "string created_at = 5;",
+            "google.protobuf.Timestamp created_at = 5;",
             "int64 size_bytes = 6;"
           )
         end

elasticgraph-proto_ingestion/spec/unit/elastic_graph/proto_ingestion/schema_definition/schema_spec.rb

@@ -135,6 +135,78 @@ module SchemaDefinition
           expect(proto_schema_from(results)).to include("fixed64 occurred_at = 2;")
         end
 
+        it "maps `DateTime` fields to `google.protobuf.Timestamp`, importing its proto file once" do
+          results = define_proto_schema do |s|
+            s.object_type "Event" do |t|
+              t.field "id", "ID"
+              t.field "created_at", "DateTime"
+              t.field "updated_at", "DateTime"
+              t.index "events"
+            end
+          end
+
+          expect(proto_schema_from(results)).to eq(<<~PROTO)
+            syntax = "proto3";
+
+            package elasticgraph;
+
+            import "google/protobuf/timestamp.proto";
+
+            message Event {
+              string id = 1;
+              google.protobuf.Timestamp created_at = 2;
+              google.protobuf.Timestamp updated_at = 3;
+            }
+          PROTO
+        end
+
+        it "renders format comments on fields whose scalar type documents one" do
+          results = define_proto_schema do |s|
+            s.object_type "Person" do |t|
+              t.field "id", "ID"
+              t.field "birth_date", "Date"
+              t.index "people"
+            end
+          end
+
+          expect(proto_schema_from(results)).to include(
+            %(string birth_date = 2; // ISO 8601 date, e.g. "2024-11-25")
+          )
+        end
+
+        it "combines source name and format comments on a single field" do
+          results = define_proto_schema do |s|
+            s.object_type "Person" do |t|
+              t.field "id", "ID"
+              t.field "option", "Date"
+              t.index "people"
+            end
+          end
+
+          expect(proto_schema_from(results)).to include(
+            %(string option_ = 2; // source name: option; ISO 8601 date, e.g. "2024-11-25")
+          )
+        end
+
+        it "supports `import:` and `comment:` on custom scalar types" do
+          results = define_proto_schema do |s|
+            s.scalar_type "Money" do |t|
+              t.mapping type: "keyword"
+              t.protobuf type: "myapp.types.Money", import: "myapp/types/money.proto", comment: "amount + currency"
+            end
+
+            s.object_type "Order" do |t|
+              t.field "id", "ID"
+              t.field "total", "Money"
+              t.index "orders"
+            end
+          end
+
+          generated = proto_schema_from(results)
+          expect(generated).to include('import "myapp/types/money.proto";')
+          expect(generated).to include("myapp.types.Money total = 2; // amount + currency")
+        end
+
         it "can assign field numbers from configured mappings" do
           results = define_proto_schema do |s|
             s.configure_proto_field_number_mappings(