Repository navigation
OpenTelemetry tracing and metrics #1196
Description
Activity
Yes. It is on my radar.
I'm able to collect EF Core traces using EntityFrameworkCore Instrumentation for OpenTelemetry .NET.
2024-10-19T08:59:05.6990000 2024-10-19T01:59:05.6986664 info: Microsoft.EntityFrameworkCore.Database.Command[20101] 2024-10-19T08:59:05.6990000 Executed DbCommand (24ms) [Parameters=[], CommandType='Text', CommandTimeout='0'] 2024-10-19T08:59:05.7000000 SELECT "c"."Id", "c"."Brand" 2024-10-19T08:59:05.7000000 FROM "CatalogBrands" AS "c" 2024-10-19T08:59:05.7010000 Activity.TraceId: 3bc7ce1e7495b4f44f636917ef5fb63e 2024-10-19T08:59:05.7010000 Activity.SpanId: 273cf93473216c11 2024-10-19T08:59:05.7010000 Activity.TraceFlags: Recorded 2024-10-19T08:59:05.7010000 Activity.ParentSpanId: 1e296c6cd31eb0f1 2024-10-19T08:59:05.7020000 Activity.ActivitySourceName: OpenTelemetry.Instrumentation.EntityFrameworkCore 2024-10-19T08:59:05.7020000 Activity.ActivitySourceVersion: 1.0.0-beta.12 2024-10-19T08:59:05.7030000 Activity.DisplayName: /var/lib/firebird/data/firebirdDb 2024-10-19T08:59:05.7030000 Activity.Kind: Client 2024-10-19T08:59:05.7040000 Activity.StartTime: 2024-10-19T08:59:05.6726693Z 2024-10-19T08:59:05.7040000 Activity.Duration: 00:00:00.0280476 2024-10-19T08:59:05.7040000 Activity.Tags: 2024-10-19T08:59:05.7040000 db.system: firebird 2024-10-19T08:59:05.7050000 db.name: /var/lib/firebird/data/firebirdDb 2024-10-19T08:59:05.7050000 peer.service: localhost 2024-10-19T08:59:05.7060000 Resource associated with Activity: 2024-10-19T08:59:05.7060000 service.name: apiservice 2024-10-19T08:59:05.7060000 service.instance.id: k4n16oeao8 2024-10-19T08:59:05.7080000 telemetry.sdk.name: opentelemetry 2024-10-19T08:59:05.7080000 telemetry.sdk.language: dotnet 2024-10-19T08:59:05.7080000 telemetry.sdk.version: 1.9.0Reacted by NorpaNetHere is an example of the EF Core trace in the Aspire dashboard.
And the console log.
Has any work been done in this area already?
I'm interested in this feature and would be happy to contribute with some PRs.
A proposed plan of action could be:
- Replace all static classes from
FirebirdSql.Data.Loggingwith corresponding components from the .NET ILogger API; - Create an
ActivitySource(static) forFirebirdSql.Data; - Integrate tracing instrumentation by adding
Activitiesthroughout the code; - (Optional) Add Metrics?
This would only utilize components from .NET runtime, without adding any extra dependencies.
Reacted by Brandon Williams- Replace all static classes from
Has any work been done in this area already?
Just some research looking into what exists now, and what possible improvements could be made to enhance the observability of the system.
@fdcastel - I just recognized your username and thank you very much for the docker images!
Reacted by F.D.CastelHas any work been done in this area already?
Not really. I played with replacing the current implementation
ILogger, but nothing committed.Metrics is definitely something worth discussing and adding. Spans/tracing makes probably sense for command execution. Both metrics and spans need to have sensible attributes.
Has any work been done in this area already?
Not really. I played with replacing the current implementation
ILogger, but nothing committed.- If you wish, I can continue what you started in a separate fork;
- Or start from zero, if you are not satisfied with the current state of what you have today;
- Or we can just sit and wait for you to do everything 😉
Metrics is definitely something worth discussing and adding. Spans/tracing makes probably sense for command execution. Both metrics and spans need to have sensible attributes.
Agreed. About this, Microsoft.Data.SqlClient is probably a good benchmark.
You can start with logging from scratch.
Reacted by F.D.CastelQuick question:
Fork from here or https://github.com/cincuranet/FirebirdSql.Data.FirebirdClient ?
Here.
3 remaining items
- Reacted by F.D.Castel
I can foresee extension method APIs for
MeterProviderBuilderandTraceProviderBuilderthat could be created in order to assist in subscribing to FirebirdSql.Data OpenTelemetry tracing and metrics./// <summary> /// Extension method for setting up FirebirdSql.Data OpenTelemetry tracing. /// </summary> public static class TracerProviderBuilderExtensions { /// <summary> /// Subscribes to FirebirdSql.Data activity source to enable OpenTelemetry tracing. /// </summary> public static TracerProviderBuilder AddFirebird( this TracerProviderBuilder builder) => builder.AddSource("FirebirdSql.Data"); } /// <summary> /// Extension method for setting up FirebirdSql.Data OpenTelemetry metrics. /// </summary> public static class MeterProviderBuilderExtensions { /// <summary> /// Subscribes to FirebirdSql.Data meter to enable OpenTelemetry metrics. /// </summary> public static MeterProviderBuilder AddFirebird( this MeterProviderBuilder builder) => builder.AddMeter("FirebirdSql.Data"); }
Regarding the extension methods, I believe the best course of action would be to create a new package
OpenTelemetry.Instrumentation.FirebirdSql, in line with OpenTelemetry guidelines.This would allow the
FirebirdSql.Data.FirebirdClientproject to remain free of any OpenTelemetry packages dependencies.However, I've deferred this discussion once the instrumentation code is more refined.
So, please give it a try in real scenarios and bring us your feedback. So far, I’ve only tested it within the Aspire Dashboard (and OpenObserve) on my end with some basic use cases.
Reacted by Brandon WilliamsWill do. Initially I was motivated because I saw a gap with Aspire integrations, but now I see this could be more useful than that. Where I work we are responsible for a large and growing number of Firebird databases running in retail stores across the United States. There, I support a small team of database administrators who I know would appreciate additional metrics, logs, and traces to help them analyze our software's performance and behavior. I will try and get their feedback.
Reacted by F.D.CastelPlease note that Aspire Dashboard never intended to be a production system. Is more a "developer helper"
(or, in Microsoft words, "purpose-built to enhance the development experience").If you want something battle tested, Seq is probably a good option. Though, of course, there are several other professional options available.
Reacted by Brandon WilliamsAnd, on a completely unrelated note, it’s great to hear that Firebird is being utilized in large-scale operations in the U.S.! 🚀
Reacted by Brandon WilliamsFirst off, thank you for putting together this OpenTelemetry instrumentation proposal! I’ve done some initial testing in real scenarios, and here are a few thoughts and observations:
-
Tracing
- Having
Activityspans around each command is great—it fits well with how other ADO.NET providers handle telemetry. The use of attributes likedb.systemanddb.query.text(ordb.statement) is aligned with OpenTelemetry conventions. - I especially appreciate that exceptions are caught and recorded properly in the span. It’s very helpful for diagnosing issues.
- Having
-
Metrics
- The new histograms (
db.client.operation.duration, etc.) and counters (db.client.connection.count,db.client.connection.max) give valuable insights into query performance and connection pool usage. It’s easy to see at a glance when we’re approaching pool limits.
- The new histograms (
-
Connection Instrumentation
- Measuring
Open()andOpenAsync()times is really useful. If you ever plan to capture “time waiting for a connection from the pool,” that might be a nice next step.
- Measuring
-
Performance & Logging
- Overall, overhead seems minimal when no listeners are attached. Using
Microsoft.Extensions.Loggingis also a nice modernization, letting us plug in our existing log pipeline. - Parameter logging toggles are a great touch—some environments can’t store full param data, so having a quick way to disable that is appreciated.
- Overall, overhead seems minimal when no listeners are attached. Using
-
Potential Small Tweaks
- You may want to rename a couple of attributes (e.g.,
db.statementinstead ofdb.query.text) to exactly match OTel specs, but that’s more of a nitpick than a requirement. - Sanitizing or truncating large SQL statements or sensitive parameter values might be worth considering in certain scenarios.
- You may want to rename a couple of attributes (e.g.,
Overall, this is a fantastic addition and makes the Firebird ADO.NET experience much more observability-friendly. Thanks again for taking the time to implement these changes—I’m looking forward to continuing to test them out and will share more feedback as I gather more data in production environments.
Reacted by F.D.Castel-
Thank you @willibrandon for the valuable feedback. 👍🏻
I will be away for the next ~10 days but I will surely address your suggestions upon my return.
Meanwhile, if you can, please provide more detailed points you would like to see addressed.
Whenever possible, citing sources. E.g. Where did you get that
db.statementis the recommended attribute name? I gotdb.query.textfrom here.@cincuranet What about the creation of a new package as discussed here?
@cincuranet Additionally, if possible, could you offer guidance on how to measure the 'time spent waiting for a connection from the pool,' as referenced by @willibrandon in the third point?
@fdcastel Thanks for the update, and I really appreciate you taking time to review this in detail! I realize now that my mention of
db.statementcame from an older version of the OpenTelemetry specs, where that attribute was commonly used. The latest Database Client Semantic Conventions indeed recommenddb.query.textfor the full SQL text and potentiallydb.query.summaryfor a lower-cardinality representation.Sorry for any confusion that caused—your reference is spot on. I look forward to refining the implementation with the new guidance once you’re back. In the meantime, if there are any other details you’d like me to clarify or test, just let me know!
I realize now that my mention of
db.statementcame from an older version of the OpenTelemetry specsYou're not alone! 😅 When I was coding, I ended up using an old version of the document for a good chunk of it.
Eventually, I caught my mistake and updated the attributes to match the latest version.
In the meantime, if there are any other details you’d like me to clarify or test, just let me know!
If possible, please share any other additional metrics/attributes you think are relevant for your real-world scenario.
As I said earlier I tried to stick to the specs exactly, but some of them seemed pretty useless. So, I left those out -- at least until someone can show me a solid reason to include them.
What about the creation of a new package as discussed #1196 (comment)?
We don't need separate package. .NET has all the pieces in the box (
Activity,ILogger, ...) and OTEL plugs into (or rather on top) it.
Hello, thanks for the great work.
I am working on .NET Aspire integrations for Firebird and wonder if the .NET data provider is instrumented with OpenTelemetry tracing and metrics?