Is 12 Factor App Right About Logging

Is the 12FactorApp right about Logging? Phil Wilkins OCI October 2022

Speaker Phil Wilkins Cloud DeveloperEvangelist Copyright © 2022, Oracle and/or its affiliates Philip.Wilkins@Oracle.com https://bit.ly/devrel-slack-emea @Phil Wilkins mp3monster.org / cloud-native.info / oracle-integration.cloud linkedin.com/in/philwilkins github.com/mp3monster @mp3monster

3 Copyright ©2022, Oracle and/or its affiliates https://www.manning.com/books/unified-logging-with-fluentd https://bit.ly/FluentdBook Philip.Wilkins@Oracle.com https://bit.ly/devrel-slack-emea @Phil Wilkins mp3monster.org / cloud-native.info / oracle-integration.cloud linkedin.com/in/philwilkins github.com/mp3monster @mp3monster

4 Join our publicOracle DevRel Workspace oracledevrel.slack.com Join the dedicated Slack channel to be part of the conversation and raise your questions to our Experts: Step 1: Access the Slack OracleDevRel Workspace following this link: https://bit.ly/devrel-slack-emea Oracle Cloud Free Tier + Special Promo Try Always Free. No Time Limits. Step 2: Search for Phil Wilkins philip.wilkins@oracle.com Free credits you can use for additional OCI services 300$ 500$ in Oracle Cloud Credits – 1 month

12 Factor App saysabout logging …. 6 Copyright © 2022, Oracle and/or its affiliates https://12factor.net/logs

12 Factor App saysabout logging …. 7 Copyright © 2022, Oracle and/or its affiliates Logs provide visibility into the behavior of a running app. In server-based environments they are commonly written to a file on disk (a “logfile”); but this is only an output format. https://12factor.net/logs

12 Factor App saysabout logging …. 8 Copyright © 2022, Oracle and/or its affiliates Logs are the stream of aggregated, time- ordered events collected from the output streams of all running processes … Logs in their raw form are typically a text format with one event per line (though backtraces from exceptions may span multiple lines). Logs have no fixed beginning or end, but flow continuously as long as the app is operating. https://12factor.net/logs

12 Factor App saysabout logging …. 9 Copyright © 2022, Oracle and/or its affiliates A twelve-factor app never concerns itself with routing or storage of its output stream … It should not attempt to write to or manage logfiles. each running process writes its event stream, unbuffered, to stdout. During local development, the developer will view this stream in the foreground of their terminal to observe the app’s behavior. https://12factor.net/logs

12 Factor App saysabout logging …. 10 Copyright © 2022, Oracle and/or its affiliates each process’ stream will be captured by the execution environment, collated together with all other streams from the app, routed to one or more final destinations for viewing and long-term archival. These archival destinations are not visible to or configurable by the app Open-source log routers (such as Logplex and Fluentd) are available for this purpose. https://12factor.net/logs

12 Factor App saysabout logging …. https://12factor.net/logs 11 Copyright © 2022, Oracle and/or its affiliates The event stream for an app can be routed to a file, or watched via real-time tail in a terminal log indexing and analysis system such as Splunk, or a general-purpose data warehousing system such as Hadoop/Hive introspecting an app’s behavior over time, including – event search, trend analysis, alerting

What is Fluentd? 16 Copyright © 2022, Oracle and/or its affiliates

Highly Pluggable Framework Input •TCP/UDP • Unix Sockets • HTTP • Many file formats • SNMP traps • OS (Linux/UNIX) • Log4J, SLF4J and other related frameworks for .Net, JavaScript Output • ALM solutions e.g. Splunk, cloud native solutions, loggly, logzio etc. • HTTP • Prometheus • Grafana • Many file formats • DB (SQL/NoSQL) • Event Streams e.g. Kafka, Kenesis, MQTT • Social notifications e.g. Jabber, Slack, emai, twilio l etc • Support mgmt tools like Pager Duty Buffer / Cache • Custom in memory cache • Redis & Coherence Formatter • XML • JSON • CSV/TSV etc. • Compressed formats Storage • S3 buckets • DB (No) SQL • File Filters • Value based conditions • REGEX expressions Parser • Multline text to single event • Event info extraction e.g. date & time Custom components Amusing Ruby Gems it is possible to build any custom components using the framework provided

Data to ActionableInformation 20 Copyright © 2022, Oracle and/or its affiliates Information Source Capture • Infra structure such as CPU, memory use • JVM use • App Log Files • SNMP Traps Structure & Route • Get the raw data to the appropriate tooling in a format that can be processed Aggregate & Analyze •Data from multiple sources •Merge in time series Visualize Data • Search for log events • Present trends e.g. memory consumption, • Rate of storage consumption Notify & Alert •Push events into JIRA Svc Desk, Slack, etc •Rules on severity dictate behaviours Fluentd – Optimal Fluentd – Leverage Other Tools

Data Egress costs$€£¥ 35 Copyright © 2022, Oracle and/or its affiliates OCI $0.02 - $0.16 per GB $0.08 per GB $0.09 per GB $0.01 - $0.02 per GB $0.0085 - $0.05 per GB $0.01 - $0.15 per GB $0.085 per GB $0.043 per GB Traffic costs vary by Geographic region – South America & parts of Asia are most expensive Europe & North America cheapest

Balancing $€£¥ withvisibility 36 Copyright © 2022, Oracle and/or its affiliates OCI - Consider adapting a centre of mass for critical E2E visibility - Centre of mass = where most log traffic is generated - Do need to trade off technology need, capacity & bandwidth cost - But have regional richer logs in the event of needing more detail

Possible scaling &deployment approaches 37 Copyright © 2022, Oracle and/or its affiliates Single instance - Often how people tend to think of log unification tools (multiples when app bundles solution - Presents lots of issues for multi- cloud / hybrid Resilient Pairing - Monolith approach of hot standby approach (Presents lots of issues for multi-cloud / hybrid) - OR 1 Node per cloud/DC Distributed Instances with Central Focus - More resilient, - Ability to control data flow, local control of connection failure - Supports highly distributed Fluentd has a very small footprint - Lots of deployments – becomes an issue of patching/config change - Even smaller footprint possible with Fluent Bit - From deployment on IoT to containers, servers and Mainframes

Possible scaling &deployment approaches 38 Copyright © 2022, Oracle and/or its affiliates Single instance - Often how people tend to think of log unification tools (multiples when app bundles solution - Presents lots of issues for multi- cloud / hybrid Resilient Pairing - Monolith approach of hot standby approach - Presents lots of issues for multi- cloud / hybrid Distributed Instances with Central Focus - More resilient, - Ability to control data flow, local control of connection failure - Supports highly distributed Fluentd has a very small footprint - Lots of deployments – becomes an issue of patching/config change - Even smaller footprint possible with Fluent Bit - From deployment on IoT to containers, servers and Mainframes

Fluentd – Scaling& Aggregation 39 Copyright © 2022, Oracle and/or its affiliates App A (Front End) Svr App B VM Svc Shared Persistence / Analytics Platform Ops Alerting Service Pod App A (Mid Tier) Svr App A (Mid Tier) VM App C VM Svc Pod

Fluentd – Scaling& Aggregation – Cloud / DC 40 Copyright © 2022, Oracle and/or its affiliates App A (Front End) Svr App B Container Svc Shared Persistence / Analytics Platform Ops Alerting Service Pod App A (Mid Tier) Svr App A (Mid Tier) VM App C VM Svc Pod Cloud Region Own DC

Fluentd – Scaling& Aggregation - Kubernetes / Container 41 Copyright © 2022, Oracle and/or its affiliates App A (Front End) Pod App B Pod Svc Shared Persistence / Analytics Platform Ops Alerting Service Pod App A (Mid Tier) Pod App A (Mid Tier) Pod App C Pod Svc Pod Worker Node Worker Node

Fluentd – Scaling& Aggregation - Kubernetes / Container – using Side Cars 42 Copyright © 2022, Oracle and/or its affiliates App A (Front End) Pod App B Pod Pod Shared Persistence / Analytics Platform Ops Alerting Service Pod App A (Mid Tier) Pod App A (Mid Tier) Pod App C Pod Pod Pod Worker Node Worker Node Side Car Side Car Side Car

46 Copyright ©2022, Oracle and/or its affiliates Central Node (Node 2) (Single Instance) Node 1 (Instance n) common • filter Slack Stdout / Monitoring Warboard Op Analytics & AI Ops filters • Transsform (event  message) • Match + copy: • Out file • relabel labelPipeline basic-file.txt basic-file2.txt label-pipeline-file-output.* forwarder We need to control this flow to avoid a notification storm! Log Sim Log Sim

OCI Managed Service 47Copyright © 2022, Oracle and/or its affiliates Central Node (Node 2) (Single Instance) Node 1 (Instance n) common • filter Slack Stdout / Monitoring Warboard • Transsform (event  message) • Match + copy: • Out file • relabel labelPipeline basic-file2.txt basic-file.txt label-pipeline-file-output.* forwarder We need to control this flow to avoid a notification storm! Multi Node – Demo OpenSearch Log Analytics Log Sim Log Sim

Addressing Real-world Challenges Inthe realworld, we have more significant challenges … • Highly distributed solutions that need to have logging and monitoring consolidated • Tracing for reactive and solutions – context switching rather than threads in the execution • Often different teams want to use different tools – security want Splunk, DBAs want OEM, infrastructure teams want Nagios – making the setup of environments more complex than needs be • Some operational events are more critical than others – need to filter those out • Make legacy solutions easier to operate, isolate log events and tag them with operational code references – so process & care are embedded without impacting the app 50 Copyright © 2022, Oracle and/or its affiliates

Is the 12Factor app right about Logging ? 51 Copyright © 2022, Oracle and/or its affiliates … Largely, BUT DON’T interpret it literally! 1 Treating log events as streams offers lots of opportunities … • Just archiving logs misses the benefit of treating logs like a stream • Tools for stream processing have developed massively since this was written 2 Writing to stdout DOESN’T mean … • Don’t use logging frameworks – the spirit is don’t tie log handling to the application. • Frameworks ensure consistency and a minimum level of log content • If you use logging frameworks – why create more work in reparsing stdout to extract semantic meaning • Use a logging framework that passes the log event in a language agnostic manner • Be mindful of the control over stdout such as K8s if you do go that way 3 As solutions are decomposed and independently deploy and scale … • Aggregating logs is key to better understanding what is happening end to end • Like networking considerations – don’t bind aggregation to the application logic

Questions / Thankyou Copyright © 2022, Oracle and/or its affiliates Phil Wilkins Cloud Developer Evangelist Philip.Wilkins@Oracle.com https://bit.ly/devrel-slack-emea @Phil Wilkins mp3monster.org / cloud-native.info / oracle-integration.cloud linkedin.com/in/philwilkins github.com/mp3monster @mp3monster

OCI Architecture Center-- Free Content & More URLS are https://oracle.com/goto/... Copyright © 2022, Oracle and/or its affiliates 53 Reference Architectures GitHub - DevRel /ref-archs Playbooks /playbooks /gh-devrel /deployed Built & Deployed Live Labs /labs Tutorials /tutorial Blogs Developer Open Source Learning Videos Apex PaaS Community GitHub - Oracle /gh-oracle Cloud Customer Connect /connect /open /dev /paas /apex /blog /youtube Oracle Community /community GitHub - Samples /gh-samples URLS are https://oracle.com/goto/...

55 oracledevrel.slack.co m Join the dedicatedSlack channel to be part of the conversation and raise your questions to our Experts: Step 1: Access the Slack OracleDevRel Workspace following this link: https://bit.ly/devrel-slack-emea Step 2: Search for Phil Wilkins philip.wilkins@oracle.com Join our public Oracle DevRel Workspace