Support Monitor Layout types + PostgreSQL Example by lukelowry · Pull Request #616 · ORNL/GridKit

lukelowry · 2026-10-11T13:18:02Z

Description

Adds an optional layout setting for monitor output. With "layout": "long", each row holds one variable at one sample instead of one column per variable, so results load straight into SQL tables no matter how many variables are monitored. This works for CSV, JSON, and YAML.

Closes #597.

Wide output is unchanged, and printing is faster:

Format	`develop`	This PR
CSV	0.199 s	0.156 s
JSON	0.377 s	0.179 s
YAML	0.310 s	0.181 s

Print time only, 2000 monitors × 4 variables over 500 samples.

Proposed changes

Add VariableMonitorLayout (WIDE, LONG) to SinkSpec, set by a layout key in the case's monitors list. It defaults to WIDE.
Long rows are sample, t, class, id, variable, value rather than the t, id, value sketched in the issue:
- class and id are the device's class and id from the case file, since an id alone can repeat across classes.
- variable gets its own column, so queries can filter or group by variable without parsing names.
- sample is needed for a unique key, since the samples just before and after an event share the same t.
Monitors hand their variables to the controller once (appendEntries), and the controller writes every format and layout from that list. This replaces the per-format virtuals in VariableMonitorBase and the formatting code in VariableMonitor.
Layouts are dispatch tags (Wide, Long) like the format tags, so each sink is a Sink<FormatT, LayoutT> resolved at compile time.
Remove the unused VariableMonitorController::addVariable.
Add a DynamicSimulation/ACTIVSg2000 example that runs a bus fault and loads the long output into PostgreSQL with a single psql script.

Checklist

All tests pass.
Code compiles cleanly with flags -Wall -Wpedantic -Wconversion -Wextra.
The new code follows GridKit™ style guidelines.
There are unit tests for the new code.
The new code is documented.
The feature branch is rebased with respect to the target branch.
The CHANGELOG.md has been updated to reflect the changes.

Further comments

@pelesh Feedback on PR length/scope

@tsybinae Thank you for the suggestion and guidance.

tsybinae · 2026-10-15T19:27:06Z

Hi. I am reviewing the code. It builds, runs, and produces the csv. The timestamp looks a bit odd, for instance, row 569215 looks like this

1704,7.0916666666666668e+00,Genrou,5281_1_genrou,omega,2.2408284627934353e-05

This effectively means that t = 7.09 something. Is this thing going to merge on t if I need to left join?

Thank you.

lukelowry · 2026-10-16T02:17:45Z

@tsybinae Thanks! Join on sample, since t repeats at events (before and after). Also, the time just has a rounding issue due to floating point precision, so joining on t is not recommended.

Let me know if you have any othr issues!

tsybinae · 2026-10-16T12:58:18Z

This functionality is not specific to the csv task, so the merge can go ahead. There may be more precision related questions within a few month.

tsybinae

Builds and runs cleanly.

superwhiskers

core idea seems like the right approach. just some comments on implementation of formatting that should be addressed before this goes further.

superwhiskers · 2026-10-16T14:54:03Z

+      /// Type used for dispatch
+      struct Wide
+      {
+      };
+
+      /// Type used for dispatch
+      struct Long
+      {
+      };
+


instead of using the type for dispatch, i'd recommend using a template parameter instead. you can take in the VariableMonitorLayout as a template parameter and remove the need for this.

superwhiskers · 2026-10-16T15:06:27Z

-      void append(std::string& out, Json json) const override
+      void append(std::string& out, Json json) const
      {
-        std::ostringstream os;
-        os.precision(std::numeric_limits<RealT>::digits10 + 1);
-        os << std::scientific;
-
-        std::string indent = "  ";
        if (json.after_first)
        {
-          os << indent << ",\n";
+          out += "  ,\n";
        }
-        os << indent << "{\n";
-        indent.append(2, ' ');
+        out += "  {\n    \"t\": ";
+        VariableMonitorDetail::formatReal(out, *time_);

-        os << indent << std::quoted("t") << ": " << *time_ << ",\n";
-        for (auto&& var : variables_)
+        const Entry* previous = nullptr;
+        for (auto&& entry : entries_)
        {
-          os << indent << std::quoted(var.label) << ": " << *var.value << ",\n";
+          if (previous && previous->device_class == entry.device_class && previous->id == entry.id)
+          {
+            out += ",\n      \"";
+          }
+          else
+          {
+            if (previous)
+            {
+              out += "\n    }";
+            }
+            out += ",\n    \"";
+            out += entry.device_class;
+            out += '_';
+            out += entry.id;
+            out += "\": {\n      \"";
+          }
+          out += entry.variable;
+          out += "\": ";
+          VariableMonitorDetail::formatReal(out, entry.value());
+
+          previous = &entry;
+        }
+        if (previous)
+        {
+          out += "\n    }";
        }
+        out += "\n  }";
+      }


since we are already pulling in nlohmann/json, i would recommend using its json serialization capabilities here. that would make this easier to read

superwhiskers · 2026-10-16T15:08:02Z

+        out += "  - t: ";
+        VariableMonitorDetail::formatReal(out, *time_);
+        out += '\n';

-        auto after_first = false;
-        for (auto* mon : monitors_)
+        const Entry* previous = nullptr;
+        for (auto&& entry : entries_)
        {
-          if (after_first)
+          if (!previous || previous->device_class != entry.device_class || previous->id != entry.id)
          {
-            os << ",\n";
+            out += "    ";
+            out += entry.device_class;
+            out += '_';
+            out += entry.id;
+            out += ":\n";
          }
-          std::string monitor_out;
-          mon->append(monitor_out, Json());
-          os << monitor_out;
-          after_first = true;
-        }
-
-        indent.erase(indent.size() - 2);
-        os << '\n'
-           << indent << "}";
+          out += "      ";
+          out += entry.variable;
+          out += ": ";
+          VariableMonitorDetail::formatReal(out, entry.value());
+          out += '\n';

-        out += os.str();
+          previous = &entry;
+        }


we don't have a yaml library so i understand doing this manually, but it could be improved by using <format> to avoid repeatedly concatenating new strings to the output

superwhiskers · 2026-10-16T15:11:37Z

      /// Type used for dispatch
      struct Csv
      {
        /// Delimiter for CSV line output
        std::string delim{","};
      };

      /// Type used for dispatch
      struct Json
      {
        /// Implementation detail used to prevent a comma before the first block
        mutable bool after_first{false};
      };

      /// Type used for dispatch
      struct Yaml
      {
      };


while these weren't added here either, i think we should also replace these with a template parameter to dispatch instead. this would be more consistent.

nkoukpaizan · 2026-10-16T14:57:53Z

+  "monitors": [
+    {
+      "file_name": "ACTIVSg2000.long.csv",
+      "format": "csv",
+      "layout": "long"
+    }
+  ],


I would add an output_layout option to solver.json instead of bringing back monitor sinks in case.json.

nkoukpaizan · 2026-10-16T14:59:59Z

@@ -0,0 +1,24 @@
+\set ON_ERROR_STOP on


I haven't looked at SQL in a while. I recommend adding some comments in this file.

nkoukpaizan · 2026-10-16T15:07:03Z

+This example runs the reusable
+[ACTIVSg2000 case](../../../../cases/PhasorDynamics/ACTIVSg2000/README.md)
+through a bus fault and loads its long monitor output into PostgreSQL with
+[psql](https://www.postgresql.org/docs/current/app-psql.html).
+
+```bash
+DynamicSimulation ACTIVSg2000.solver.json
+psql -f ACTIVSg2000.sql
+```


This doesn't currently run the example. The user has to run it manually and figure out how to get psql on their own.

nkoukpaizan · 2026-10-16T15:09:22Z

 | --- | --- |
 | [ThreeBusBasic](Toy/ThreeBusBasic/README.md) | A three-bus line opened and reclosed. |
 | [ThreeBusConstantSource](Toy/ThreeBusConstantSource/README.md) | A three-bus constant signal source example. |
+| [ACTIVSg2000](ACTIVSg2000/README.md) | A bus fault with long monitor output loaded into PostgreSQL. |


Perhaps we should include SQL in the folder name to clarify the purpose. As is, I expected similar setups to the other examples.

nkoukpaizan · 2026-10-16T15:39:12Z

-       */
-      virtual void appendHeader(std::string&, Csv) const = 0;
+      /// Returns the current value of a monitored variable
+      using GetterT = std::function<double()>;


Everything is now converted to double before printing, including float, int, bool, etc.

lukelowry requested review from pelesh, superwhiskers and tsybinae October 11, 2026 13:18

lukelowry self-assigned this Oct 11, 2026

lukelowry added the enhancement New feature or request label Oct 11, 2026

lukelowry force-pushed the lukel/monitor-long-dev branch from 54bd651 to bba5a78 Compare October 13, 2026 22:04

Support Layout types WIDE and LONG

757dcfe

lukelowry force-pushed the lukel/monitor-long-dev branch from bba5a78 to 757dcfe Compare October 14, 2026 22:24

improve output

bed6f96

tsybinae approved these changes Oct 16, 2026

View reviewed changes

superwhiskers requested changes Oct 16, 2026

View reviewed changes

nkoukpaizan reviewed Oct 16, 2026

View reviewed changes

Conversation

lukelowry commented Oct 11, 2026

Description

Proposed changes

Checklist

Further comments

Uh oh!

tsybinae commented Oct 15, 2026

Uh oh!

lukelowry commented Oct 16, 2026

Uh oh!

tsybinae commented Oct 16, 2026

Uh oh!

tsybinae left a comment

Choose a reason for hiding this comment

Uh oh!

superwhiskers left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants