The APL Quest Series

It seemed like a normal Friday until mid-afternoon. But on 4 February 2022, I embarked on a journey that, at the time, seemed to stretch impossibly far into the future — a future that wasn’t entirely known yet. In the APL Orchard chat room on Stack Exchange, a dozen APLers, some experts and some newbies, held the very first APL Quest chat event.

In this first session, we explored the oldest APL Problem Solving Competition phase 1 problem – question number 1 from 2013 – presented our solutions, and discussed them for about half an hour. The following week, I recorded a video where I went through some of these solutions, with the code posted on GitHub. This pattern continued each week; chat events on Fridays, and a follow-up video released (usually) the week after, each session looking at a different phase 1 problem.

APL Quest chat in progress

Inspiration

Originally, the idea came from Richard Savenije, inspired by LeetCode. Both he and Stefan Kruger, who later became my colleague, were frustrated with LeetCode’s assumptions about programming languages – assumptions that didn’t really hold for APL. Aside from that, their test framework didn’t permit APL submissions anyway.

So, Richard suggested that we should make our own problems website, and Stefan pointed out that my colleague Rich Park had already set up a website that offered automatic checking of solutions to simple problems. With the help of a summer intern, Rich had populated the site with all APL Problem Solving Competition phase 1 problems from 2013 until 2021 (the 2022 round was scheduled to launch two months later). Richard suggested using this site for weekly puzzles, one every Friday afternoon, and we found a time that suited the people present.

Next, we had to decide on a format. Should it be a Zoom meeting or a chat event? Earlier, there had been a couple of chat events series in the APL Orchard; the APL Cultivation series ran weekly from October 2017 until May 2018 and semi-weekly from 28 November 2019 until August 2020. Stefan Kruger was in the process of converting these to a since-completed book, APL Cultivations. After some discussions, we decided to have the sessions in chat, and I came up with the idea of recording a short screen cast after each event.

I got the arithmetic wrong, and claimed we’d have 100 weeks worth of problems, as the 2022 problems would be available by the time we had explored the other problems, giving us almost two years of material. Actually, the 2023 problems were ready when we got to them! But either way, the end seemed very far away…

Technical Details

And yet, here we are, after 110 weeks, 110 live chat events, and 110 published videos – a total of over 22 hours of video contents! We never missed a week, even on various holidays, though Rich Park did have to substitute for me a few times when I was prevented from hosting, and sometimes I had to push off recording and/or publishing a video until two or three weeks after the chat event. Sometimes, I’d host the chat event while travelling in a car or a train, and sometimes I’d record videos in hotel rooms or in other people’s homes. By always using a plain white (or off-white) background, I was able to get a consistent look, irrespective of where I was, though I sometimes had to move furniture to sit in front of a bare wall, and once I had to drape my bedspread over the hotel room’s television…

Recording an APL Quest session

Speaking of looks, I’ve received praise for the technical quality of my videos; for their smooth integration of presentation and live coding and for their nice design. So, for those who are interested in how I did it…the introductory screen and the problem statement are simply PowerPoint slides with Fade transitions. After the problem description slide, I faded to a blank slide, and from there, I switched application to RIDE (on Microsoft Windows, you have to install it separately), which was running in full-screen mode with the Language Bar and Status Bar hidden. When I started, the newest released RIDE was version 4.3, but I was running pre-releases of v4.4 and Dyalog v18.2 that added the Nord theme which I had fallen in love with in 2021. By matching my slide colours to the RIDE theme, I achieved a seamless transition without having to do any video editing.

I’m somewhat of a typeface enthusiast. Previously, I had searched for what I considered good sans-serif fonts, and for this project, I choose the humanist Go font. For APL code, I went with my own SAX2 font which I had created by extracting letterforms from the old SAX (SHARP APL for UNIX) manual, and then extended to cover the characters necessary today. However, it bothered me that the SAX font looked so thin next to Go, due to being digitised from the golf ball of a IBM Selectric without accounting for the visual weight normally added by the typewriter’s ink ribbon.

I had to hack font selection into RIDE, which I was able to do because RIDE is built using normal web technology, in particular CSS. After some experimentation, I found the way to do it. After I switched to RIDE v4.5, I was able to set the fontface in an official manner, but even this wasn’t enough. I had relied on PowerPoint’s reasonable auto-bolding of the SAX font, which otherwise didn’t include bold, but RIDE v4.5 wouldn’t let me style the APL font further, so I had to hack RIDE again!

“Hacking” RIDE v4.5

This time, I didn’t need to modify source files, but rather found that RIDE was “vulnerable” (though not in a dangerous way) to CSS injection through its font input field. Making the font bold didn’t look right, as that would only smear the letters horizontally, but adding a “text stroke” had the desired effect. If you want your RIDE to resemble what you can see in the videos, set the APL font to SAX2')}.monaco-editor *{-webkit-text-stroke:.67px currentcolor}. With the visuals set, I recorded using OBS Studio and, on the rare occasion that I needed to edit something, I used DaVinci Resolve.

Concluding the Series

When we finished that last session, it felt rather anti-climactic, but I can look back on a very enjoyable time. And of course, the efforts that all participants put into this are not forgotten; we’ve got an amazing chat and video series that future APLers can enjoy. Thank you to everyone who contributed; chat participants, video commenters, colleagues (especially Brian Becker who authored all the problems), and most of all my wife, who often had to encourage me to record the next video(s) and also gave me the time and space to do so, even if it meant single-handedly keeping our children quiet. If you’re up for a marathon, you can watch the entire 22-hour series on YouTube, and APL Quest on the APL Wiki includes links to all the problems, chat sessions, code, and videos.

Dyalog ’23 Videos: Week 8 – Celebrate Solstice with the Last of Our Dyalog ’23 Videos

Whether you are celebrating a winter holiday, looking forward to the days getting longer from tomorrow, or enjoying summer south of the equator, we hope you have time to enjoy the final collection of light-hearted presentations from Dyalog ’23.

Andy Shiers has overall responsibility for making sure that Dyalog gets correctly built and tested before it reaches the users – and that it is supported once it has been released. In his “Fireside Chat”, Andy covers a wide range of topics that he regularly encounters but feels might have not been sufficiently emphasised in other talks or in our documentation – and that you might need to know about. Andy is back by popular demand from user meeting delegates, after a gap of a few years.

The development team at SimCorp Italiana have been regular contributors to Dyalog user meetings, with insightful and amusing anecdotes about the relationship between humans who work with technology. This year, in “Once Upon A File”, the stories are mostly about the trials and tribulations of importing data.

Ray Cannon has been working on synthesising music on various computing devices. From a humble start where he tapped notes out himself using “A Pointy Stick”, he learned how to generate chords, add harmonics, attack, decay, and reverberate – and store the result in .WAV files. The result is a rendering of J.S. Bach’s Tocatta and Fugue in D Minor, BMV565 – complete with animations – on multiple organ pipes, all done in APL.

We hope that you enjoyed the presentations from Dyalog ’23, that you have a Happy New Year ahead, and that we will see many of you at Dyalog ’24 in Glasgow (15-19 September 2024).

——————————————

This week’s videos:

Materials for all presentations can be downloaded from the Dyalog ’23 webpage.

Dyalog ’23 Videos: Week 7 – Performance and Scaling

Although run-time performance is rarely the most important reason for selecting APL, good performance often becomes important during the lifetime of an application (especially if it is successful and, therefore, has to deal with growing data volumes and numbers of users). Array-oriented programming naturally encourages Subject Matter Experts to use dense and pointer-free structures, which allow APL-based solutions to do things like balancing thousands of portfolios in a fraction of the time that more traditional solutions need.

The “set functions” – membership (), index of (), intersection (), union (), and without (~) – are already some of the most highly-tuned primitives in the history of APL because they are critical to the performance of very many APL applications. New parallel instruction sets keep appearing in modern processors, and the balance between processor and memory performance is in a constant state of flux. Also, computer scientists continually improve algorithms for searching. Karta Kooner’s talk on the performance of Set Functions describes the approach that he is taking as we embark on another round of optimisations.

Sometimes, the best way to improve application performance or reliability is to split the application into multiple processes that can run independently and be scaled up by adding more processes as required. Apache Kafka is a widely-used tool for connecting such processes and reliably forwarding streams of messages between them. Stefan Kruger presents the benefits of Kafka and his initial experiments on what a Dyalog interface to Kafka might look like.

Application performance increasingly depends on how much memory you use, and how efficiently you move data around. If you want to help ensure that your APL algorithms have the best possible mechanical sympathy with Dyalog APL, Richard Smith’s “Introduction to the Workspace” will help you understand how the interpreter manages the memory that holds your arrays in the workspace.

An APL compiler promises to help APL users take advantage of highly parallel hardware like General-Purpose Graphics Processing Units (GPGPUs). If the compiler is self-hosted, it also makes it practical to quickly port APL to virtual environments, or base an APL implementation on other programming languages such as Python or JavaScript. Aaron Hsu presents an update on recent enhancements to the Co-dfns compiler, and plans for the near future.

Although I have been an APL developer for more than four decades, it is only recently that I understood how APL can be used to efficiently and elegantly handle tree structures using simple arrays. Brandon Wilson has been studying techniques developed by Aaron Hsu that make it possible to parallelise compilation of APL by the Co-dfns compiler. YAML parsers are notoriously difficult to write accurately, and Brandon hopes to find an effective description of YAML through APL that can help the community better understand its edge cases.

Next week the final videos from Dyalog ’23 will be published (along with my final blog post on the subject) – a great way to end the year!

——————————————

This week’s videos:

Materials for all presentations can be downloaded from the Dyalog ’23 webpage.

Dyalog ’23 Videos: Week 6 – Tools and Services

An increasing number of APL systems serve business logic as services, in addition to providing a user interface. Some recent APL applications have no user interface at all, and are only available as services.

For this reason, Dyalog’s web service framework, Jarvis, features prominently in Brian Becker’s overview of the current state of tool development at Dyalog Ltd. Brian is the APL Tools Architect, and the author of both Jarvis and HttpCommand, our tool for making web requests (Jarvis provides server functionality, while HttpCommand provides client functionality).

Jarvis is the foundation of many new APL projects, two of which are featured this week. First, Finn Flug from Dittrich & Partner Consulting GmbH (DPC) tells the story of an application that started its life as a COM service implemented in APL+Win, was converted to a web service using Jarvis, and was then deployed as a docker container. Docker containers are also the execution platform for Claus Madsen’s talk. For decades, Claus has been writing software in APL for pricing financial instruments; he is one of Denmark’s leading experts in this field. Claus is not a software engineer, so when a new client wanted a pricing application as a web service, he needed a little help from his friends at Dyalog Ltd. He wanted to replace the binary workspaces that he had been using to hold his source code with docker containers, and for these docker containers to be rebuilt and published on Amazon Web Services every time he committed a change to one of his APL source files (managed on GitHub). To his surprise, since making this change he has found that he feels more in control of his APL development workflow than ever before – without having lost any flexibility.

The only talk that doesn’t mention Jarvis is Stig Nielsen’s talk on “Worker Bees”, where he demonstrates a mechanism which is similar to Dyalog APL’s isolates. The difference is that where the isolate workspace launches separate processes to host isolates, Stig makes use of a .NET class that allows a single process to launch multiple Dyalog interpreters that all run within the same process. This mechanism has only been made available to selected clients because some features of Dyalog (like file holds and locks or the use of DLLs) are restricted in this mode – but it has the advantage that data can be shared in-process, between multiple APL interpreters and components written in C#.

——————————————

This week’s videos:

Materials for all presentations can be downloaded from the Dyalog ’23 webpage.

Dyalog ’23 Videos: Week 5 – Tools and Interfaces

This week, the focus is on the use of tools and interfaces in applications. Mark Wolfson from BIG integrates data from 1,000 retailers to provide business intelligence to both the retailers and manufacturers. Over the last couple of years, Mark has migrated his application from IBM APL2 to Dyalog APL. He tells the story of how Dyalog’s tools and features (including SQAPL, Jarvis, HttpCommand, .NET integration, ⎕JSON, and ⎕CSV) have enabled BIG to respond to customer needs faster and more flexibly than ever before.

Kai Jaeger has a long history of developing tools and utilities for Dyalog APL. He is also the main developer of the new Tatin Package Manager. In addition to developing the package manager itself, he has been working on moving his tools and small “APL-cations” to Tatin and GitHub, where Kai is now responsible for 90% of the existing packages. His presentation also briefly introduces several new packages and recent enhancements to existing tools.

Following immediately after Kai, I attempted to demonstrate how much easier packages can make your life by creating a small APL application within the new Cider project management system, using one Tatin package and one NuGet package. The Tatin package is Dyalog’s HTTP Client utility, HttpCommand (a rough equivalent of cURL), which is used to retrieve data from the web, and the NuGet package is MailKit, which reads e-mail using the POP3 protocol.

The new .NET eco-system is a potential source of many very useful packages that will work in the same way under Microsoft Windows, Linux, and macOS. Modern .NET APIs make widespread use of generic classes and methods, which means that they work the same way with different data types. Although this is how we have always worked in APL, Dyalog’s .NET bridge assumes that .NET expects specific data types. In his “Part 2” talk, John presents his thoughts on how Dyalog can embrace .NET generics.

The final two talks this week are about potential future packages. First, Josh David takes a look at open-source statistical libraries available to the APL developer, including KokoStats and TamStat. Rich Park follows with a demonstration of a simple interface for producing graphs from APL data using Vega-lite, a declarative language for describing visual presentations using JSON. Charts are rendered using a JavaScript library and, thanks to the HTMLRenderer, data can be provided directly from APL to produce graphics ready for publication on the web.

It’s great to see that the APL ecosystem is not only alive and well, but expanding!

——————————————

This week’s videos:

Materials for all presentations can be downloaded from the Dyalog ’23 webpage.

Dyalog ’23 Videos: Week 4 – Research and Education

I am always thrilled to hear from people who have been able to use APL to gain insight into difficult subjects, or use APL to pass knowledge on to others. A truly fascinating and surprising example of this at Dyalog ’23 was the talk titled “quAPL – A Quantum Computing Library in APL”, in which Marcos Frenkel presented work that he has done together with Santiago Núñez-Corrales and Bruno Abreu at the National Center for SuperComputing Applications, University of Illinois at Urbana-Champaign in the USA.

I say “surprising” because I had imagined that quantum computers were a bit like the analog computers of old, where you would initialise some “circuits” and then let the thing settle into some kind of minimal energy state – and, therefore, about as far from APL as you could get. However, when you take a closer look at quantum computing, qubit states are “matrix multiplications all the way down”. To quote Marcos: APL naturally expresses quantum concepts, leading to higher level programming constructs.

Jesús Galán López is a metallurgist at Ghent University in Belgium. In two presentations at Dyalog ’23, he explains how he has been exploring the use of APL for scientific and engineering work. He takes us on a deep dive into the use of APL to help students write better models of processes like grain growth in metals than they would be able to using traditional programming languages.

Dyalog had two interns this summer: last week’s video collection featured work done on an Array Notation implementation by Kamila Szewczyk from Saarland University. This week, Asher Harvey-Smith talks about his experience as an intern, some of the teaching materials that he worked on over the summer, and his adventures teaching algebra to his mates using APL after he returned to the University of Warwick.

Not all education happens at an institution: sometimes you have to learn APL on your own, with the help of online friends. If you are really keen, you can attend conferences and hunt down Aaron Hsu. The final talk this week is by Brandon Wilson. In a talk titled “Introspections of an Intermediate APLer – The Good, Bad, and Ugly of Learning APL“, he (in his own words) offers the perspective of a firmly middle-intermediate APLer trying to make the most of the glittering promises of “APL as a tool of thought” while the pains and joys are salient.

——————————————

This week’s videos:

Materials for all presentations can be downloaded from the Dyalog ’23 webpage.