Although run-time performance is rarely the most important reason for selecting APL, good performance often becomes important during the lifetime of an application (especially if it is successful and, therefore, has to deal with growing data volumes and numbers of users). Array-oriented programming naturally encourages Subject Matter Experts to use dense and pointer-free structures, which allow APL-based solutions to do things like balancing thousands of portfolios in a fraction of the time that more traditional solutions need.

The “set functions” – membership (∊), index of (⍳), intersection (∩), union (∪), and without (~) – are already some of the most highly-tuned primitives in the history of APL because they are critical to the performance of very many APL applications. New parallel instruction sets keep appearing in modern processors, and the balance between processor and memory performance is in a constant state of flux. Also, computer scientists continually improve algorithms for searching. Karta Kooner’s talk on the performance of Set Functions describes the approach that he is taking as we embark on another round of optimisations.

Sometimes, the best way to improve application performance or reliability is to split the application into multiple processes that can run independently and be scaled up by adding more processes as required. Apache Kafka is a widely-used tool for connecting such processes and reliably forwarding streams of messages between them. Stefan Kruger presents the benefits of Kafka and his initial experiments on what a Dyalog interface to Kafka might look like.

Application performance increasingly depends on how much memory you use, and how efficiently you move data around. If you want to help ensure that your APL algorithms have the best possible mechanical sympathy with Dyalog APL, Richard Smith’s “Introduction to the Workspace” will help you understand how the interpreter manages the memory that holds your arrays in the workspace.

An APL compiler promises to help APL users take advantage of highly parallel hardware like General-Purpose Graphics Processing Units (GPGPUs). If the compiler is self-hosted, it also makes it practical to quickly port APL to virtual environments, or base an APL implementation on other programming languages such as Python or JavaScript. Aaron Hsu presents an update on recent enhancements to the Co-dfns compiler, and plans for the near future.

Although I have been an APL developer for more than four decades, it is only recently that I understood how APL can be used to efficiently and elegantly handle tree structures using simple arrays. Brandon Wilson has been studying techniques developed by Aaron Hsu that make it possible to parallelise compilation of APL by the Co-dfns compiler. YAML parsers are notoriously difficult to write accurately, and Brandon hopes to find an effective description of YAML through APL that can help the community better understand its edge cases.

Next week the final videos from Dyalog ’23 will be published (along with my final blog post on the subject) – a great way to end the year!

——————————————

This week’s videos:

• D11: Dyalog + Kafka = True? (Stefan Kruger)
• D07: Future Performance Improvements in Set Functions (Karta Kooner)
• D06: An Introduction to the Workspace (Richard Smith)
• D15: Co-dfns Update 2023 (Aaron Hsu)
• U11: A YAML Parser in APL – Leveraging a Linear Dataflow Model to Unleash APL’s Expressiveness (Brandon Wilson)

Materials for all presentations can be downloaded from the Dyalog ’23 webpage.

Follow

An increasing number of APL systems serve business logic as services, in addition to providing a user interface. Some recent APL applications have no user interface at all, and are only available as services.

For this reason, Dyalog’s web service framework, Jarvis, features prominently in Brian Becker’s overview of the current state of tool development at Dyalog Ltd. Brian is the APL Tools Architect, and the author of both Jarvis and HttpCommand, our tool for making web requests (Jarvis provides server functionality, while HttpCommand provides client functionality).

Jarvis is the foundation of many new APL projects, two of which are featured this week. First, Finn Flug from Dittrich & Partner Consulting GmbH (DPC) tells the story of an application that started its life as a COM service implemented in APL+Win, was converted to a web service using Jarvis, and was then deployed as a docker container. Docker containers are also the execution platform for Claus Madsen’s talk. For decades, Claus has been writing software in APL for pricing financial instruments; he is one of Denmark’s leading experts in this field. Claus is not a software engineer, so when a new client wanted a pricing application as a web service, he needed a little help from his friends at Dyalog Ltd. He wanted to replace the binary workspaces that he had been using to hold his source code with docker containers, and for these docker containers to be rebuilt and published on Amazon Web Services every time he committed a change to one of his APL source files (managed on GitHub). To his surprise, since making this change he has found that he feels more in control of his APL development workflow than ever before – without having lost any flexibility.

The only talk that doesn’t mention Jarvis is Stig Nielsen’s talk on “Worker Bees”, where he demonstrates a mechanism which is similar to Dyalog APL’s isolates. The difference is that where the isolate workspace launches separate processes to host isolates, Stig makes use of a .NET class that allows a single process to launch multiple Dyalog interpreters that all run within the same process. This mechanism has only been made available to selected clients because some features of Dyalog (like file holds and locks or the use of DLLs) are restricted in this mode – but it has the advantage that data can be shared in-process, between multiple APL interpreters and components written in C#.

——————————————

This week’s videos:

• D09: Dyalog Tools Update (Brian Becker)
• U05: Converting a COM Server to a Jarvis-based Web Service (Finn Flug)
• U01: APL Worker Bees – Another Way to Run APL Tasks in Parallel (Stig Nielsen)
• U06: Dyalog, AWS, Jarvis, Docker…What’s Not to Like? (Claus Madsen)

Materials for all presentations can be downloaded from the Dyalog ’23 webpage.

Follow

This week, the focus is on the use of tools and interfaces in applications. Mark Wolfson from BIG integrates data from 1,000 retailers to provide business intelligence to both the retailers and manufacturers. Over the last couple of years, Mark has migrated his application from IBM APL2 to Dyalog APL. He tells the story of how Dyalog’s tools and features (including SQAPL, Jarvis, HttpCommand, .NET integration, ⎕JSON, and ⎕CSV) have enabled BIG to respond to customer needs faster and more flexibly than ever before.

Kai Jaeger has a long history of developing tools and utilities for Dyalog APL. He is also the main developer of the new Tatin Package Manager. In addition to developing the package manager itself, he has been working on moving his tools and small “APL-cations” to Tatin and GitHub, where Kai is now responsible for 90% of the existing packages. His presentation also briefly introduces several new packages and recent enhancements to existing tools.

Following immediately after Kai, I attempted to demonstrate how much easier packages can make your life by creating a small APL application within the new Cider project management system, using one Tatin package and one NuGet package. The Tatin package is Dyalog’s HTTP Client utility, HttpCommand (a rough equivalent of cURL), which is used to retrieve data from the web, and the NuGet package is MailKit, which reads e-mail using the POP3 protocol.

The new .NET eco-system is a potential source of many very useful packages that will work in the same way under Microsoft Windows, Linux, and macOS. Modern .NET APIs make widespread use of generic classes and methods, which means that they work the same way with different data types. Although this is how we have always worked in APL, Dyalog’s .NET bridge assumes that .NET expects specific data types. In his “Part 2” talk, John presents his thoughts on how Dyalog can embrace .NET generics.

The final two talks this week are about potential future packages. First, Josh David takes a look at open-source statistical libraries available to the APL developer, including KokoStats and TamStat. Rich Park follows with a demonstration of a simple interface for producing graphs from APL data using Vega-lite, a declarative language for describing visual presentations using JSON. Charts are rendered using a JavaScript library and, thanks to the HTMLRenderer, data can be provided directly from APL to produce graphics ready for publication on the web.

It’s great to see that the APL ecosystem is not only alive and well, but expanding!

——————————————

This week’s videos:

• U02: Transforming and Streamlining a Complex Development Process using Dyalog (Mark Wolfson)
• U03: The Evolution of APLTree and APL-cation (Kai Jaeger)
• D08: Using Packages (Morten Kromberg)
• D12: Dyalog Version 20.0 – Part 2 (John Daintree)
• D16: Statistical Libraries for Dyalog (Josh David)
• D18: Vega Charts with Dyalog (Rich Park)

Materials for all presentations can be downloaded from the Dyalog ’23 webpage.

Follow

I am always thrilled to hear from people who have been able to use APL to gain insight into difficult subjects, or use APL to pass knowledge on to others. A truly fascinating and surprising example of this at Dyalog ’23 was the talk titled “quAPL – A Quantum Computing Library in APL”, in which Marcos Frenkel presented work that he has done together with Santiago Núñez-Corrales and Bruno Abreu at the National Center for SuperComputing Applications, University of Illinois at Urbana-Champaign in the USA.

I say “surprising” because I had imagined that quantum computers were a bit like the analog computers of old, where you would initialise some “circuits” and then let the thing settle into some kind of minimal energy state – and, therefore, about as far from APL as you could get. However, when you take a closer look at quantum computing, qubit states are “matrix multiplications all the way down”. To quote Marcos: APL naturally expresses quantum concepts, leading to higher level programming constructs.

Jesús Galán López is a metallurgist at Ghent University in Belgium. In two presentations at Dyalog ’23, he explains how he has been exploring the use of APL for scientific and engineering work. He takes us on a deep dive into the use of APL to help students write better models of processes like grain growth in metals than they would be able to using traditional programming languages.

Dyalog had two interns this summer: last week’s video collection featured work done on an Array Notation implementation by Kamila Szewczyk from Saarland University. This week, Asher Harvey-Smith talks about his experience as an intern, some of the teaching materials that he worked on over the summer, and his adventures teaching algebra to his mates using APL after he returned to the University of Warwick.

Not all education happens at an institution: sometimes you have to learn APL on your own, with the help of online friends. If you are really keen, you can attend conferences and hunt down Aaron Hsu. The final talk this week is by Brandon Wilson. In a talk titled “Introspections of an Intermediate APLer – The Good, Bad, and Ugly of Learning APL“, he (in his own words) offers the perspective of a firmly middle-intermediate APLer trying to make the most of the glittering promises of “APL as a tool of thought” while the pains and joys are salient.

——————————————

This week’s videos:

• U08: quAPL – A Quantum Computing Library in APL (Marcos Frenkel)
• U09: APL and Metallurgy (Jesús Galán López)
• U10: Teaching Linear Algebra with APL (Asher Harvey-Smith)
• U15: Introspections of an Intermediate APLer – The Good, Bad, and Ugly of Learning APL (Brandon Wilson)
• U12: Grain Growth and Array Programming (Jesús Galán López)

Materials for all presentations can be downloaded from the Dyalog ’23 webpage.

Follow

This week’s collection consists of presentations by members of the development team, and concentrates on language features and development tools that the developers are currently designing or already implementing. Many of these features are likely to be in version 20.0 of Dyalog, which we plan to release in early 2025, about a year after version 19.0 becomes available. In other words, there is still time to comment on these ideas and propose changes or additions…please take a look and let us know your thoughts!

John Daintree presents an update to the design (and prototype implementation) of so-called Token-by-Token (TBT) debugging, which will make it possible to step through code one primitive at a time rather than one line at a time. This promises to be a wonderful teaching tool, and will also help experienced users debug complex expressions – especially trains and other tacit constructs.

In the presentation on “Setting and Getting Variable Values”, Adám Brudzewsky proposes extensions that will simplify (and speed up) a number of frequently-occurring patterns, where application code needs to reference or update variables based on arrays containing variable names. At the moment, most solutions require complicated, inefficient, and uncomfortable uses of the execute primitive (which some code scanners might identify as potentially unsafe).

Adám’s second presentation also proposes ways to simplify frequently-occurring use cases, this time related to a relatively recent addition to the Dyalog language, the key operator (⌸). Similar to an SQL “group by” clause, key performs operations on groups of data that are partitioned using key values – and it suffers from similar issues if you want the results to be in a particular order, or include entries for key values that do not occur in the data which is at hand.

Peter Mikkelsen proposes a new mechanism for making shell commands from Dyalog. Rather than extending the existing mechanism (⎕SH and its synonym ⎕CMD), which is widely used by existing code bases, Peter proposes a new system function that will manage both binary and encoded input and output, and include features such as whether to error on a non-zero exit code and ways to redirect input and output

The final talk in this block is by Kamila Szewczyk, a student from Saarland University, who was an intern at Dyalog Ltd this summer. In this whirlwind talk, she covers her implementation of a prototype of an APL Array Notation deserialiser, which will be available as an I-Beam in version 19.0, and probably used by Link 4.0. When she needed a break from that project, she created internal experimental builds of the interpreter to try out a large number of interesting ideas that are present in other array language implementations, such as automatic numerical differentiation, the obverse operator, and a monadic inner product where, for example, -.× computes the determinant.

——————————————

This week’s videos:

• D03: Dyalog Version 20.0 – Part 1 (John Daintree)
• D04: Setting and Getting Variable Values (Adám Brudzewsky)
• D05: Revisiting ⎕SH and ⎕CMD (Peter Mikkelsen)
• D13: Giving Key a Vocabulary (Adám Brudzewsky)
• U16: An Implementation of APL Array Notation (Kamila Szewczyk)

Materials for all presentations can be downloaded from the Dyalog ’23 webpage.

Follow