Category Archives: Dyalog ’24

Posts relating to the 2024 user meeting

Dyalog ’24 Videos: Week 6 – Remembering the “Old”, Welcoming the “New”

Posted on by
Reply

The main theme this week is remembering how we got here – and welcoming a new generation of APL users. There are three talks and a panel discussion looking back on careers spent working on or with APL, and a final panel discussion with members of the new generation, which will hopefully wipe away some of the nostalgia.

John Daintree’s presentation Array Notation: A Journey of Discovery is the exception from the historical theme – the release was delayed until this week due to technical difficulties. However, John does point out that it is nearly a full decade since Phil Last’s initial APL Array Notation presentation at Dyalog ’15 in Sicily, and that Adám Brudzewsky has subsequently given several overviews of the evolution and experimental use of the notation (Literal Notation for Arrays and Namespaces at Dyalog ’17, Array Notation Mk III at Dyalog ’18, and Array Notation RC1 at Dyalog ’20). As Chief Architect at Dyalog Ltd, John has been working on the implementation of array notation within the Dyalog interpreter and IDE, and offers a sneak peek at array notation features that will be in version 20.0 – and some that are probably too adventurous to make it this time.

Veli-Matti Jantunen inherited reasponsibility for PxEdit when the original author decided to join FinnAir as a pilot. (Kimmo Linna is now an Airbus 350 Captain but still finds time to contribute tools to the Dyalog community – for example, RSConnect). In PxEdit to GitHub – Refurbishing a Seasoned APL Application to Open Source, Veli-Matti explains that PxEdit was originally a temporary hack for handling larger-than-usual tables at Statistics Finland, but it grew in popularity as Veli-Matti added functionality that users needed (like Unicode support). To cater to users in developing countries, Veli-Matti avoided using system features that would require the newest versions of operating systems, doing a lot of work in “raw APL” instead. 25 years later, PxEdit is used by 115 organisations in 45 different countries. Veli-Matti’s next task is to make it open source on GitHub, in the hope that a new person will pick it up so that he can retire!

The next retrospective presentation is Charles Brenner’s Some APL Pioneers that I Knew. If you are interested in the history of programming languages and the personalities behind languages like APL, this highly entertaining talk is a treat. On his way, Charles met, or worked with, Larry Breed, Kenneth Iverson, Eugene McDonnell, Adin Falkoff, Garth Foster, Curtis Jones, and Paul Berry – to name a few. Charles’ sense of humour is extra dry, so listen carefully to pick up several jokes that the audience did not get quickly enough to laugh during the presentation…

Geoff Streeter worked on Dyalog APL from the day it was conceived in 1981 until he retired 18 months ago. Key Technical Decisions During the Development of Dyalog APL is a walk through the history of the implementation and critical decisions made, starting with: Should we use C or Assembly Language? Target UNIX? Build a 1st generation (flat) or 2nd generation (nested) interpreter? Should it include productivity tools like ⎕FMT, and if so from which pre-existing APL interpreters? In his own words, expect some nostalgia, some repentance, some triumphalism, and some serendipity. Thankfully, many of those early decisions turned out to be excellent. None of us would have been in Glasgow in September 2024, more than four decades after the release of Dyalog version 1.0, if it had not been for Geoff!

Finally, Stephen Taylor wraps up the historical theme as the host of two panel discussions. In Let’s Put the Future Behind Us, we hear from the team who nursed the company and the product from a very shaky birth, where the company was reconstructed more than once – Pete Donnelly explains how he mortgaged his own house because he believed in the future of the product! Geoff Streeter worked without pay at times, but explains that he was not concerned as he abdicated worrying to management. Andy Shiers remembers how the sales of IBM hardware to British Airways helped pay for continued work on APL. Just past the halfway point in Dyalog Ltd’s current history (this is hard for me to fathom), Pete handed the baton to Gitte Christensen, who tells the story of how she fell in love with APL and “came home” in 2005 to take over the helm of a language vendor after starting her career at one of the original APL vendors, I. P. Sharp Associates (IPSA).

However, we cannot end a Dyalog user meeting looking back! In The New Breed Plugs In, Stephen asks Martina Crippa, Josh David, Sandra Persson, and Gilgamesh Athoraya how they found APL (or how APL found them, as some of them put it). What attracted them to APL? Beauty, having fun, and creativity seem to be important components of many explanations, but I should not say more – listen to the stories!

——————————————

This week’s videos:

Materials for all presentations can be downloaded from the Dyalog ’24 webpage.

Email, RSS Follow

Dyalog ’24 Videos: Week 5 – Best Practices

Posted on by
Reply

Most of the recordings that we are releasing in week 5 concern “best practices” for the use of APL in large or complex systems. Many of these are mentioned in the invited keynote Dyalog APL in the Biggest Data Centres at the Heart of the Investment Management Industry by Oliver Lanz from the product development team at SimCorp, one of the biggest users of Dyalog.

In his talk, Oliver gives an overview of the architecture of Simcorp Dimension, which is the backbone of more than 50% of the top investment management firms worldwide; it manages over €35 trillion combined. Oliver explains the position that Dyalog has within their architecture, and how shared code files, the .NET bridge, multi-threading, and Simcorp’s homemade type checking system, all contribute to making the system secure and efficient. He introduces many of the topics discussed in the other talks released this week, and explains why it is important to Simcorp that Dyalog Ltd and the APL community develop and use such tools.

One of the points raised by Oliver Lanz in his talk is that developers are increasingly required to use automated tools to measure code quality, and document this to auditors. As applications are exposed to the outside world over the internet, an important goal of automated analysis is to detect and prevent vulnerabilities that would allow attackers to access or modify data without permission. In his talk Static Analysis of APL in APL, Brandon Wilson describes the work that he is doing with Aaron Hsu to teach the co-dfns parser about traditional functions so that it can parse existing APL application code and enable static analysis of APL code. Aaron follows up his compiler and parser status in Co-dfns Roadmap and Updates.

As APL applications become part of elaborate architectures where APL code provides and/or makes use of other services, another requirement that is emerging is the provision of telemetry to support the monitoring. This can be used to track performance over time and – when things go wrong – troubleshoot system failures that involve calls between multiple components, potentially implemented using different technologies. In their presentation on Telemetry and Protobuf, Gilgamesh Athoraya presents how OpenTelemetry can be used from Dyalog and demonstrates a sample application that emits telemetry data. Telemetry data can, optionally, be emitted using Protobuf (Protocol Buffers), a mechanism for serialising structured data. Sandra Persson joined him to present an APL implementation of Protobuf and a plugin that generates APL code from schema files.

Kai Jaeger is the most prolific tool builder in the Dyalog community. He is the author of many useful tools, both for use at runtime and during development. He has contributed the vast majority of the packages available on the Tatin package manager (which is also his work). In Developing in Dyalog with Modern Tools, Kai takes us through “a day in the life of an APL developer”, demonstrating the most important tools that he uses on a daily basis, to make development both easier and safer. Amongst other things, Kai demonstrates Git, GitHub, Cider, Tatin, Snippets, CommTools, FiRe, APLGit2, Testing (with code cverage), and CodeBrowser for code review.

I must admit that Brandon Wilson’s Data Parallel Proof Verification was not something I thought that I would fully appreciate, as my understanding of proofs and how to use them is extremely limited. However, I was still intrigued by the insight (obvious once Brandon has pointed it out) that proofs are tree structures, with a top-level proof being constructed from several smaller proofs. Brandon has been experimenting with using co-dfns/Hsu-style tree wrangling to validate proofs in parallel, and the results are very promising. As mathematicians move in the direction of software systems to make proofs more reliable, it matters a lot whether your new proof can be verified in a split second or it takes hours or even days. Although there are few current APLers who need a proof verification system, this could help make APL more visible to communities who are currently unaware of it.

——————————————

This week’s videos:

Materials for all presentations can be downloaded from the Dyalog ’24 webpage.

Email, RSS Follow

Dyalog ’24 Videos: Week 4 – Migrating APL Code, and New GUI Frameworks

Posted on by
Reply

The presentations released this week are about APL applications on the move. Some applications are moving from one APL implementation to another, for example from APL2 or APL+Win to Dyalog APL. Others are moving from one platform to another, for example from a mainframe or the Microsoft Windows desktop to Linux in docker containers in the cloud. Sometimes both moves are being considered – the need to switch platforms often triggers a review of which APL system to use on the new platform.

Migrating to Dyalog APL

OLE or ActiveX is an old technology which is being replaced by web services using tools like Jarvis. However, quite a few existing applications use it to make APL business logic available from tools like Microsoft Excel. Dyalog allows you to turn any namespace into an OLE Server with an API that is specific to your application – this requires the design of an object-oriented API. APL+Win comes with a simpler (evil tongues might say less secure) but in some ways easier-to-use framework in which a single OLE Server called WSEngine allows you to load any workspace and execute code within it. Jürgen Wiedemann from Dittrich und Partner Consulting (DPC) shows how the Dyalog framework can be used to implement something very similar to the APl+Win tool. Not only does this simplify some migrations, it also provides a simpler OLE Server framework for users of Dyalog who don’t need or want an application-specific API.

We are in the process of migrating the METSIM® application from APL+Win to Dyalog. As part of this project, we are developing general-purpose tools for moving APL source code from other APL systems to Git, performing automated translation and/or emulation of language features not directly supported by Dyalog. Karl Holt and I present the current state of these tools, including the current prototype of a function named ∆WI, which emulates ⎕WI from APL+Win or MicroAPL APLX, and will help move GUI applications to Dyalog with minimal application code changes. All these tools will be made available free as public repositories on GitHub.

Markos Mitsos of ERGO insurance has taken a completely different approach to his migration from APL+Win to Dyalog. Rather than use tools like those described in the previous two presentations, which aim to minimise the changes required to application code, Markos has carefully studied the Dyalog APL language and tools, and decided to refactor his old code to take full advantage of the new platform. This obviously takes much longer than using semi-automated migration tools. In his presentation on workspace transformations, Markos explains why he feels the extra work was worth it.

New GUI Frameworks

First, a bit of fun: RayLib is a simple and easy-to-use library for video game programming (although it can be used for more serious purposes). Brian Ellingsgaard is the creator of raylib-bqn, an interface from BQN, which is an array language with many similarities to Dyalog APL. He spent this summer with us as an intern, creating a similar interface from Dyalog – the result is raylib-apl. In his presentation, he gives examples of 2D and 3D game programming using RayLib…including the mandatory 3D duck rendering!

The original goal of our Everywhere WC project (EWC) was to allow existing applications using the Microsoft Windows GUI based on the ⎕WC system function to be moved to other platforms, including Linux in the cloud. EWC provides cover-functions that emulate the ⎕WC family of functions using a JavaScript (React) application to recreate equivalent GUIs. This allows applications to be migrated to Linux and macOS (or Microsoft Windows) desktops using the HTMLRenderer, or to any platform with a web browser to render the user interfaces – with minimal changes. However, the implementation also supports changing the look and feel using CSS styling, Flex layout for responsive UI, and several new controls based on JavaScript libraries like the ApexCharts graphics package. In this presentation, I also suggest that it has potential for completely new applications.

In the 1980s, Pete Donnelly and Geoff Streeter oversaw the design and development of ⎕SM, a system function for creating text-based (“80×25”) user interfaces. When Graphical User Interfaces came with Microsoft Windows, Pete worked with John Daintree to design ⎕WC, a system function for creating GUIs under Windows. In both cases, these features were quickly mastered by APL users, and subsequently used to build many of the most successful applications written in Dyalog. Now that applications increasingly need to be cross-platform and/or remotely controlled, Pete has been Noodling*, searching for intuitive ways for APL users to define cross-platform, HTML-based UI, based on popular widget libraries. Given his track record, it is worth paying attention to Pete’s Noodles!

* John Daintree refers to experiments as “Noodles”

——————————————

This week’s videos:

Materials for all presentations can be downloaded from the Dyalog ’24 webpage.

Email, RSS Follow

Dyalog ’24 Videos: Week 3 – The APL Forge, Education, and Data Science

Posted on by
Reply

The recordings released this week focus on activities related to attracting new users to APL.

For many years, the APL Problem Solving Competition was a cornerstone of our efforts to promote APL to a new generation of potential users. This year, we introduced the APL Forge, where prizes are awarded for original work done in APL, rather than solutions to “classical” computer science problems. Our Managing Director (CEO) Stine Kromberg introduced the new format and presented the Grand Prize to this year’s winner, Holder Hoover. Holden, who is an aviation geek and still in High School, then explained how he built his own system for receiving and processing the ADS-B data that is transmitted by nearly all aircraft so that he can track aircraft within range of receivers operated by himself and a friend in the Azores. In three days (apparently between two exams), he managed to decode data using APL that he had spent three months failing to handle in JavaScript.

We were very pleased to learn that Sergey Ichtchenko and his two collaborators, Dani Adham and Hex, have been working on a MOOC (Massive Open Online Course) for the University of Helsinki, and we invited Sergey to Dyalog ’24 to tell us more about this project. When completed, the APL MOOC will be an online APL course for university students, worth 5 ECTS credits (or 3% of a Bachelor’s Degree) to Finnish and other European students. You can review the work in progress, and send comments or words of encouragement to the team, at aplmooc.fi.

Data Science is a term that – along with artificial intelligence – is used to describe a very wide range of data processing techniques, from cross-tabulation and standard statistical techniques over regression, to large and small “language models”. The other three talks released this week look at the use of APL as a tool for Data Science from different perspectives.

In his presentation Dyalog for Data Science, Jesús Galán López compares APL to popular Python libraries, solving some well-known introductory problems in what used to be a statistics curriculum but is now called Data Science. As he concludes: “If you are an APLer and you have been using APL to extract useful information from data: Congratulations, you are a Data Scientist! And if you are a Data Scientist – Try APL!”.

Stephen Mansour is the creator of TamStat, an application written in Dyalog that he uses to teach statistics to University students. A core component of the design of TamStat is the use of defined operators to create a domain-specific notation that is not only flexible and powerful but makes the fundamentals of statistics easier to grasp – hence the name “Taming Statistics”. His presentation on Taming Regression using APL illustrates how, by designing user functions and operators to be similar to APL primitives, you can produce a notation that is a user-friendly Tool of Thought, helping you to understand statistics in ways that are close to impossible in other languages. In addition to being available as a free application, the source code of TamStat is free and open source, and available for use in your own applications.

Not all data science applies to rectangular data that can be easily grouped and cross-tabulated. In 2015 I did a presentation in the Talks at Google series. Towards the end of my presentation, I was asked about Iverson’s original intention of having APL natively apply to arrays and trees…”What happened to the latter?”. Arthur Whitney was in the room, and I asked out loud “What happened to the trees, Arthur?”, to which he responded “I don’t know” and can be seen shaking his head. I then waffle on a bit about nested arrays and how they lead to poor performance. Today, thanks to Aaron Hsu, I have learned that regular flat APL arrays allow for extremely efficient representations of trees (his thesis on the co-dfns compiler shows that this can be used to achieve orders-of-magnitude improvements in performance in compilers). In his presentation Climbing Trees and Catching Bugs, our Senior Intern (spending his second consecutive summer with us!) Asher Harvey-Smith introduces a tutorial that he is working on, with the goal of making Aaron’s techniques easily approachable to newcomers. He also talks about some of the other fun he had this summer.

——————————————

This week’s videos:

Materials for all presentations can be downloaded from the Dyalog ’24 webpage.

Email, RSS Follow

Dyalog ’24 Videos: Week 2 – New Functionality

Posted on by
Reply

The recordings released this week focus on active development projects at Dyalog Ltd.

The first three presentations describe features that are expected to be in the next version of Dyalog. In Setting and Getting Variable Values Mk II, Adám Brudzewsky describes a set of new system functions for setting and getting variables’ values using arrays. Until now, this required the use of the primitive function execute (⍎), in ugly and inefficient expressions. Also, execute is increasingly seen as a potential security risk, as careless use allows code injection attacks – so these new Dyalog v20.0 features will not only increase the readability and performance of application code, they will also make applications more secure.

John Daintree returns with WC Plugins. This is not a talk about tools for unclogging drains, but a general mechanism that will allow both the Dyalog development team and users of Dyalog to create self-describing libraries in C, which the ⎕WC (and ⎕NEW) system functions will be able to create instances of, in the same way that they can currently be used to create GUI objects and OLE or OCX components. This will allow us to open-source parts of the Dyalog interpreter, such as our HTMLRenderer interface to the Chromium Embedded Framework, the Conga interface to TCP and UDP, and an upcoming cryptographic library. It will also make it easier for users to contribute to the Dyalog ecosystem.

Peter Mikkelsen is one of the youngest members of the Dyalog development team. In New Function for Shell Calls, he describes the work that he has done to create a modern version of one of the oldest system functions in Dyalog APL – the ⎕SH function for making shell calls (known as ⎕CMD under Microsoft Windows). Among other things, this new function can be interrupted and can run the shell command on a separate operating system thread. It allows you to separate and redirect output streams, and specify what encodings are used. It even allows you to stream output to and from the external code asynchronously, using tokens and callbacks. It is slightly surprising that it cannot make toast!

The other two recordings this week are about features that are already available, but more people need to know about!

Adám Brudzewsky shows how text files containing APL code or data are all that you need to define and run an application in Initialising and Starting from Text Files. You can also extend your APL development environment with a few text files in the right place!

Finally, in New Tatin Packages, Brian Becker reintroduces the Tatin package manager, and describes three new Tatin packages that Dyalog Ltd have recently made available – a package for loading and using .NET packages hosted in the NuGet system (dyalog-NuGet), a package for managing APL processes in the same way on all platforms supported by Dyalog (dyalog-APLProcess), and a somewhat experimental package for tapping into the OpenAI platform and generate images, have conversations with large language models (LLMs), and access other “Artificial Intelligence” features (dyalog-OpenAI).

——————————————

This week’s videos:

Materials for all presentations can be downloaded from the Dyalog ’24 webpage.

Email, RSS Follow

Dyalog ’24 Videos: Week 1 – Welcome to the Dyalog ’24 Presentations!

Posted on by
Reply

We’ve finished processing the first recordings of the presentations given at Dyalog’24 in Glasgow. The first four are now available to watch, and you can look forward to five or six new recordings each week until the end of November.

As usual, the Dyalog user meeting opened with presentations by the CEO and CTO, who provided statements of direction for our technical and marketing activities. As mentioned in my blog post introducing the videos after last year’s user meeting, we have a new CEO since 1 January – Stine Kromberg is now at the helm, and opened the meeting. I once again followed up the CEO’s introduction with a whirlwind tour of what I consider to be the most important stops on our technical road map – also introducing many of the talks that will follow later.

The two other talks we are releasing this week are from the other end of the meeting, appearing first due to technical issues that have delayed a couple of other recordings. First up is Aarush Bhat, who is making his first presentation at a Dyalog user meeting. Aarush is based in Bangalore and has been working for Dyalog for just over a year now. So far, most of his efforts have been concentrated on building a new framework for very thoroughly testing primitive elements of the Dyalog APL language; in this presentation, he explains what he has been able to do so far.

At the other end of the spectrum, we have lost count of how many presentations our Chief Architect John Daintree has made! In the third of his three presentations this year, he talks about limits in the Dyalog APL interpreter. These limits include things like the maximum rank of an array, the number of different data types we can support, the maximum number of constants and names in an expression, and so on. Many of the structures that impose these limits have remained unchanged since Dyalog version 1.0, but some of the limits are now coming under pressure by migrants from APL systems that have different limits, and some of the new features that we are adding to Dyalog, like array notation – and new primitives.

Any change to such structures has potential impacts on performance. Work will need to be done to continue to allow the sharing of arrays between different versions of Dyalog through workspaces, component files, and TCP sockets. If we do make changes, we’d like to get them all over and done with in one go, and ideally not change them again for another 40 years.

I hope you enjoy watching the first set of recordings from Dyalog ’24!

——————————————

This week’s videos:

Materials for all presentations can be downloaded from the Dyalog ’24 webpage.

Email, RSS Follow