Green Screens Server Monitoring

by Green Screens Ltd.

System APM (Application Performance Monitoring) is one of the crucial elements for any service and system admins. This is especially important to large organizations as its advanced monitoring might provide excellent information not only about service memory, threads and web requests, but also user behavior.

One of the big advantages web based terminals versus classical desktop clients have is APM as it bring much more than you might ever think.

Here is the answer why Green Screens Terminal Services for IBM i is not "just another terminal"...

In our case - Green Screens Terminal Services for IBM i, it is very interesting to companies to know the metrics of how their production system is used. For an example, which RPG/COBOL programs are used the most, in which period of the day. When operators are the most active or which spools files are printed at which period of the day, how often, and so many more. This is only to scratch the surface.

What will be possible ....?

  • User behavior - to monitor when users are active in which period of the day. This data can be interconnected with program usage
  • Program usage - to monitor which terminal screen is used, when, how often, in which period of  the day. As screens are linked to RPG or COBOL programs, they are monitored automatically. Can be very helpful in detecting resource usage and optimize programs.
  • Printing - to monitor which reports are used in which period of the day, amount of pages printed, or data transferred. This might help to optimize the amount of printing - paper usage.  
  • Detecting program bottleneck - often, spool files and some RPG/COBOL programs are using complex queries, which makes additional pressure on the system resources. In big systems, it is not easy to detect performance bottleneck. Being able to detect such issues can save a lot of money to the companies.
  • AI business process optimization - Exporting such time series data can be extremely helpful to data analysts and data scientist to use the data in AI tools, which helps in process optimization and better time distribution for employees. It can help to detect and optimize employee time management, which has big impact on company performance.
  • Detecting infrastructure issues - Except performance usage, advanced technical usage is also possible, allowing to monitor and detect for an example network issues. It is not the first time we encountered response time issues, most often caused by overprotected infrastructure where every service, firewall, VPN, proxy and AV software brings additional layer and additional breaking point which is extremely hard to detect.

Monitoring examples

Here is a small list of monitoring use examples to think about and what you can do with the data.

  • telnet requests (avg. / time / period of the day)
  • telnet sessions (counter, peak / period of the day)
  • printer sessions  (counter, peak / period of the day)
  • printer spools released  (counter, peak / period of the day)
  • strpccmd command calls  (counter, peak / period of the day)
  • screen calls - (which screen is used the most - RPG/COBOL)
  • session durations (longevity, counter, connection times)
  • security features monitor (how many OTP, WebAuth, login types, etc..)
  • printer release stats - when printing is the most active
  • telnet response times - (IBM i <-> GS response times)
  • network response times (web <-> GS server)
  • average spool sizes (print sizes through time, average, max, min, pages and bytes)

The solution

Today, many solutions exist being commercial or open-source, however we will focus on free open-source tools and integration packs allowing to monitor our product from a single place.

As Green Screens Terminal Services is written in Java, it allows excellent and easy integration with different products out of the box without touching the code.

Here are the base products that can be used for a powerful product monitoring for free.

Prometheus

Prometheus is a data collector server which periodically collects data from running services. Installation is simple, based on a single program distributed in a single zip file. Simply, extract and start.

Configuration boils down to add remote services, to which Prometheus will periodically connect to collect data. Add Green Screens Server to prometheus.yml file, restart Prometheus and after a few seconds it will be able to show collected data through its basic web console.

Prometheus also allows an alternative approach, where remote service connects to a Prometheus server and push data up. It is up to a system admin and security to determine which way to go, as which options are easier to implement.

Exporting Prometheus data and using it with AI tools can be very revealing in optimizing company processes.

Grafana

Grafana is a web based visualization tool to present Prometheus data in graphical form as charts, graphs and gauges. Installation is easy, based on a single program. Configuration is also easy, requires adding a data source - in this case Prometheus server. After Prometheus server is added, it is possible to create various dashboards to visualize data.

In our case, we are interested to monitor Green Screens Server JVM memory, threads, peak usage, request counts, etc. and all mentioned above in a beautiful UI interface with detail graphs and charts.

What is interesting about Grafana is dashboard export, allowing to exchange dashboard monitoring settings. This feature allows us to easily distribute updated versions with a new monitoring features. Examples created by the community can be found here.

If you want to feel Grafana dashboards live, check this playground demo.

InspectIT Ocelot

InspectIT Ocelot is a JavaAgent which starts along with Java program (Green Screens Server) and use class load interception mechanism to inject its own monitoring logic. Also starts a service on localhost:8888 where Prometheus connects to get the data.

OpenTelemetry

OpenTelemetry is another JavaAgent similar to InspectIT. Main difference is in possibility to use extensions libs. When properly configured, can be used with Prometheus and Grafana also. OpenTelemetry is community driven compared to InspectIT Ocelot and most likely a new emerging standard. Some modules are currently in alpha state and will take a few months to become production ready.

Java Integration

The base operation is ingress mode - Prometheus connects to get data, we need to add some module to Java application (Green Screens Server in this case). This can be done in three ways...

  1. Monitoring Client code integration into the product
    This option require from us to integrate monitoring code in various parts of our product which brings more fine-tuned monitoring, but also create dependency.
    As OpenTelementry is on a good path to become a standard, it is the most likely the best candidate for future integration.
  2. Java Agents - generic code injection on class bootstrap
    JavaAgent is a special class loaded at JVM startup which intercepts every Java class load, and injects its own program logic into targeted methods. This saves us from changing the code and adding monitoring logic into any module by code injection technique. It is the most powerful approach as it has access to absolutely every method call and its arguments without the need to change existing  code.
  3. Exposing JMX data
    Green Screens Server will have a new web module "service.apm" in the near future which is just a servlet registering Prometheus JMX exporter to open URL "/service.apm/metrics" used by Prometheus server to retrieve extra metrics data.

    Idea behind this approach is to use JMX to store statistics data instead of using 3rd party monitoring library. Having JMX exporter available through URL allows services such as Prometheus to connect and extract statistics data.

    JMX is a generic framework available natively in Java JVM without requirements for any additional modules. Combined with JMX yml based filter, we can create powerful generic monitoring JavaAgent and separate web module for Prometheus monitoring. JMX data can be also monitored with desktop Eclipse based application JMC - Java Monitoring Console.

The final workflow of all services integration looks like on the image below.

Conclusion

We hope that this blog post manage to convince you that Green Screens Terminal Services for IBM i is much more than just a telnet and print service. Even it solves basic requirements for daily usage for anyone working on IBM i, we also hope that we manage to give some insight how it differs from classic desktop client which belongs to category of "just another terminal".

Except bringing ease of management, significantly better security, IBM i server isolation and many more, it also brings additional value - resource usage monitoring, user behavior monitoring and network bottleneck detection helping to optimize business processes within the company and thus potentially reduce operational cost.

Final word

If one wants to learn more about monitoring Green Screens Server, here is a link to installation and setup manual for your Green Screens Server.

Please note, that in current early  Alpha state, some features as advanced terminal and spool behavior are not available as first release for Green Screens Monitoring service is planned for the end of 2021. Some dashboard elements will not show the data as they are linked to values exposed with service.apm module (not available yet).