Case Conversion: Mapping and Folding

Dyalog v18.0 introduced ⎕C, which converts the case of characters in an array by mapping to lower case, mapping to upper case, or folding. This superseded an earlier experimental I-beam (819⌶) that could map to lower or upper case but not fold – this I-beam is currently deprecated and will be removed in Dyalog v20.0.

There’s often confusion about the difference between mapping and folding. Mapping is used when you want characters in an array to be in a particular case, whereas folding is used when you want to eliminate case to perform case-insensitive comparisons. The confusion arises because case insensitive comparisons can usually be done well by mapping to lower case rather than folding – indeed, on first inspection, mapping to lower case and folding appear to be the same thing.

The difference is best illustrated through some examples. Used monadically, the I-beam maps to lower case and ⎕C folds:

      819⌶'Hello'
hello
      ⎕C'Hello'
hello

A case-insensitive match function that uses 819⌶ to map both arguments to lower case before checking if they match might look like this:

      cc←≡⍥(819⌶)

In many cases it will appear to work without issues:

      'hello' cc 'HELLO'
1
      'hello' cc 'GOODBYE'
0

However, it doesn’t always work. Greek, for example, has two different lower-case sigma characters (σ and ς) but only one upper case (Σ). ίσως and ΊΣΩΣ are case-insensitively equal, but the function does not work:

      'ίσως' cc 'ΊΣΩΣ'
0

This is because when these two arrays are mapped to lower case they become ίσως and ίσωσ respectively, which do not match. If we use folding instead:

      cc←≡⍥⎕C

the comparisons work as expected:

      'hello' cc 'HELLO'
1
      'hello' cc 'GOODBYE'
0
      'ίσως' cc 'ΊΣΩΣ'
1

This works because folding converts the Greek words to ίσωσ and ίσωσ respectively – every different sigma character, even the lower-case ones, have been changed to σ, and now the two arrays match.

The main use of case conversion is to perform caseless comparisons, so you might wonder why mapping to upper and lower case is supported at all. There are still occasions where you might need that – most notably, when formatting text for display.

If you are still using 819⌶ to perform caseless comparison you should change to using ⎕C to get correct behaviour. And do not forget that 819⌶ will not be supported beyond Dyalog v19.0.

Want to learn more? Adám Brudzewsky explains mapping and folding in this webinar, beginning at 00:06:33.

Welcome Karl Holt

Karl had always imagined that he would spend his career in academia, as had many who know him. However, after becoming disenchanted with the whole university environment halfway through his Master’s degree, he sought a role outside academia and joined Dyalog Ltd. Although he’s only been here for a few months, he feels that he has already settled in well.

Karl was one of the few computer science students who actually enjoyed the courses on project management; he was also part of an even smaller minority who enjoyed writing mathematical proofs! He not only brings his mathematics and coding abilities with him, but also a fresh energy to the team.

Karl is a perfectionist, and often spends countless hours contemplating the right way to pick up a cup and otherwise trivial daily activities. This fits perfectly with how he has been mentored by our CTO, Morten, as every time he has shown Morten what he found to be a specifically beautiful APL expression, Morten has shown him five different ways to do the same thing, each of which is even prettier and has its own strength!

Even though the majority of Karl’s education has been in the natural sciences, he also enjoys many facets of the “softer” ones. He has a broad range of interests, and always enjoys hearing about what other people are working on – everything from quantum physics to anthropological studies contemplating the queerness of algae. His favorite read in 2023 was “Invisible Women“, a book disclosing the big data gaps within political minorities, especially women’s experiences in a world designed for men.

Karl is just as happy being an interested listener to a conversation as he is being the main speaker. He keeps a long list of projects he would love to work on if he someday had the time; for now, he spends his spare time playing board games, solving fun puzzles, staring at bodies of water, and hanging out with his friends and many siblings.

Employee Spotlight: Stefan

About to ski down to the fjord in Norway’s Sunnmøre Alps

It’s now one year since Stefan joined Dyalog Ltd, and we asked him what he made of the experience so far. “I can’t believe how time flies. It feels like it was only yesterday I arrived at the Bramley office for the first time, to pick up my laptop and meet everyone.”

Prior to joining us, Stefan worked for IBM, where he discovered APL more or less by accident. “Coming from a huge, anonymous mega-corp, it’s a welcome change being able to actually get to know the whole company,” Stefan continues. “Here, of course people have their particular core areas, but everyone gets stuck in with a little bit of everything.”

Stefan takes a particular interest in the Mac platform, as he is a life-long, unapologetic Apple user. Everywhere else he’s worked (especially at IBM, inventor of the PC), developers demand Apple hardware, so he was surprised to find that at Dyalog Ltd, that’s more unusual.

Epic powder day in Italy’s Courmayeur

Outside work, Stefan likes the adventurous side of the great outdoors. He spent many years rock climbing home and abroad, until chronic elbow tendinitis forced a stop to that. Back country skiing is another hobby, and he tends to take all his vacation days in winter. “Living aboard a boat in Svalbard, skiing with a rifle to ward against polar bears was a bucket list experience”, he says. “Sampling the famous japow and live volcanos on Hokkaido, and ticking couloirs in Canada’s Purcells range is up there, too.” Stefan rides bikes to keep fit, and his garage has accumulated far too many bikes over the years.

“Dyalog is my dream gig,” he says. “I always wanted to work for a company where the people really care for the product they make.”

Top of the Grandpa Peabody boulder in The Buttermilks, Bishop, California

Employee Spotlight: Jada

This week marks one year since Jada joined Dyalog Ltd, and we asked her about her experiences thus far.

“It’s truly been a whirlwind of a year. I don’t know where the time has gone! It’s been intense, but in the best way possible. I’ve been able to get a feel for everything that being a part of Team Dyalog involves in a very short amount of time, from my first user meeting (Dyalog ’23) to my first product release (Dyalog v19.0). It’s been a steep, but fun learning curve.”

Jada joined us when it became apparent that the administrative needs of the company were growing. She was tasked with taking on some of the jobs that were being done by the Karen and Stine (then the Financial Officer). However, Jada was familiar with Team Dyalog even before she joined, as she had previously attended Dyalog ’19, Dyalog ’22, and various Dyalog internal events as a guest of her partner, Rich Park. Through Rich, Jada was introduced to Dyalog and the world of array programming. “I’ll be honest, a lot of the technical stuff does not yet make sense to me. But that just means there’s so much still to learn, which keeps my work life very interesting. Seeing others try to understand APL code reminds me of Derrida’s views on language, intention, and the need for deconstruction.”

Jada’s daily routine has begun to take shape. “Initially, I had so much to learn, it was rather daunting. From our products/services, our customers, my colleagues, the history of the company, to my actual job…I could go on! But all that initial familiarisation work was worth it, and I had plenty of support along the way.”

So, what does Jada actually do? “Well, think of all the things you need to do to run a business. That’s what I do. I help wherever I can. I deal with office maintenance, renewals for licences, billing, sales of physical products (books, APL keyboards), contract reviews, and all things compliance. I’m also an Apprentice Cat Herder – luckily, I’m learning from the best! Karen has really been my rock this year. She’s one of the most thorough and competent professionals I’ve ever met. I’m hoping to continue to learn a lot from her and Stine.”

Jada also enjoys her life outside work. “I like to read (fiction and Japanese manga), love going for long walks and the occasional run in the woods, and canoeing the Thames with Rich. I always enjoy a good night out as well – gigs, bars, and dancefloors call my name! My latest hobby has been learning to play bass guitar. I’ve got no immediate plans to sell out Wembley, but it’s fulfilling to set my sights on a piece, learn it bar by bar, and gradually improve and gain muscle memory. It can be frustrating at times, but practising is so much fun. My rhythm has improved a lot too.”

Overall, Jada has really enjoyed her first year with us. “I’m so grateful to be here. You see a lot of corporate speak about ‘family’ to distract from poor business practices, but not at Dyalog Ltd. Although this is a company, the heart and soul are the people behind it and the community around us. We’re real, we care, we listen. There’s no apathy or shilling. The passion for Dyalog is infectious. My colleagues’ hard work, meticulousness, and honesty motivate me to apply those qualities to my own work. When I work hard and produce quality work, it’s recognised and that makes me feel valued. I couldn’t ask for a better job with better people.”

Numeric Case Conversion

Dyalog APL version 18.0, released in June 2020, introduced the Case Convert system function ⎕C. It was a replacement for the long-lived (since version 15.0, from June 2016) I-beam 819⌶, which was then deprecated. (By the way, did you know that the digits 819 were chosen to be reminiscent of the letters BIg as in big — uppercase — letters?) It is expected that 819⌶ will be disabled in the next major version of Dyalog APL.

⎕C already has several advantages over 819⌶, for example the ability to case-fold rather than lowercase. (Did you know that Cherokee syllabary case-folds to uppercase?) With today’s release of Dyalog version 19.4.1, we’re happy to announce a further extension of ⎕C, covering scaled format (also known as scientific or E notation) and complex numbers.

By default, APL uses the letter E to separate mantissa and exponent in very large and very small numbers. Similarly, the letter J is used to separate real and imaginary parts of complex numbers:

      2 20 * 64 ¯24
1.844674407E19 5.960464478E¯32
      ¯1 ¯5*0.5
0J1 0J2.236067977

For input, however, e and j are accepted in addition to E and J:

      1E4 1J4 ≡ 1e4 1j4
1

You can now conveniently mitigate this asymmetry using ⎕C:

      ⎕C 2 20 * 64 ¯24
1.844674407e19 5.960464478e¯32
      ⎕C ¯1 ¯5 * 0.5
0j1 0j2.236067977

We hope that this added functionality will be exploited to its fullest by all our users. Please contact us if you experience any stability issues with the new feature.

The APL Quest Series

It seemed like a normal Friday until mid-afternoon. But on 4 February 2022, I embarked on a journey that, at the time, seemed to stretch impossibly far into the future — a future that wasn’t entirely known yet. In the APL Orchard chat room on Stack Exchange, a dozen APLers, some experts and some newbies, held the very first APL Quest chat event.

In this first session, we explored the oldest APL Problem Solving Competition phase 1 problem – question number 1 from 2013 – presented our solutions, and discussed them for about half an hour. The following week, I recorded a video where I went through some of these solutions, with the code posted on GitHub. This pattern continued each week; chat events on Fridays, and a follow-up video released (usually) the week after, each session looking at a different phase 1 problem.

APL Quest chat in progress

Inspiration

Originally, the idea came from Richard Savenije, inspired by LeetCode. Both he and Stefan Kruger, who later became my colleague, were frustrated with LeetCode’s assumptions about programming languages – assumptions that didn’t really hold for APL. Aside from that, their test framework didn’t permit APL submissions anyway.

So, Richard suggested that we should make our own problems website, and Stefan pointed out that my colleague Rich Park had already set up a website that offered automatic checking of solutions to simple problems. With the help of a summer intern, Rich had populated the site with all APL Problem Solving Competition phase 1 problems from 2013 until 2021 (the 2022 round was scheduled to launch two months later). Richard suggested using this site for weekly puzzles, one every Friday afternoon, and we found a time that suited the people present.

Next, we had to decide on a format. Should it be a Zoom meeting or a chat event? Earlier, there had been a couple of chat events series in the APL Orchard; the APL Cultivation series ran weekly from October 2017 until May 2018 and semi-weekly from 28 November 2019 until August 2020. Stefan Kruger was in the process of converting these to a since-completed book, APL Cultivations. After some discussions, we decided to have the sessions in chat, and I came up with the idea of recording a short screen cast after each event.

I got the arithmetic wrong, and claimed we’d have 100 weeks worth of problems, as the 2022 problems would be available by the time we had explored the other problems, giving us almost two years of material. Actually, the 2023 problems were ready when we got to them! But either way, the end seemed very far away…

Technical Details

And yet, here we are, after 110 weeks, 110 live chat events, and 110 published videos – a total of over 22 hours of video contents! We never missed a week, even on various holidays, though Rich Park did have to substitute for me a few times when I was prevented from hosting, and sometimes I had to push off recording and/or publishing a video until two or three weeks after the chat event. Sometimes, I’d host the chat event while travelling in a car or a train, and sometimes I’d record videos in hotel rooms or in other people’s homes. By always using a plain white (or off-white) background, I was able to get a consistent look, irrespective of where I was, though I sometimes had to move furniture to sit in front of a bare wall, and once I had to drape my bedspread over the hotel room’s television…

Recording an APL Quest session

Speaking of looks, I’ve received praise for the technical quality of my videos; for their smooth integration of presentation and live coding and for their nice design. So, for those who are interested in how I did it…the introductory screen and the problem statement are simply PowerPoint slides with Fade transitions. After the problem description slide, I faded to a blank slide, and from there, I switched application to RIDE (on Microsoft Windows, you have to install it separately), which was running in full-screen mode with the Language Bar and Status Bar hidden. When I started, the newest released RIDE was version 4.3, but I was running pre-releases of v4.4 and Dyalog v18.2 that added the Nord theme which I had fallen in love with in 2021. By matching my slide colours to the RIDE theme, I achieved a seamless transition without having to do any video editing.

I’m somewhat of a typeface enthusiast. Previously, I had searched for what I considered good sans-serif fonts, and for this project, I choose the humanist Go font. For APL code, I went with my own SAX2 font which I had created by extracting letterforms from the old SAX (SHARP APL for UNIX) manual, and then extended to cover the characters necessary today. However, it bothered me that the SAX font looked so thin next to Go, due to being digitised from the golf ball of a IBM Selectric without accounting for the visual weight normally added by the typewriter’s ink ribbon.

I had to hack font selection into RIDE, which I was able to do because RIDE is built using normal web technology, in particular CSS. After some experimentation, I found the way to do it. After I switched to RIDE v4.5, I was able to set the fontface in an official manner, but even this wasn’t enough. I had relied on PowerPoint’s reasonable auto-bolding of the SAX font, which otherwise didn’t include bold, but RIDE v4.5 wouldn’t let me style the APL font further, so I had to hack RIDE again!

“Hacking” RIDE v4.5

This time, I didn’t need to modify source files, but rather found that RIDE was “vulnerable” (though not in a dangerous way) to CSS injection through its font input field. Making the font bold didn’t look right, as that would only smear the letters horizontally, but adding a “text stroke” had the desired effect. If you want your RIDE to resemble what you can see in the videos, set the APL font to SAX2')}.monaco-editor *{-webkit-text-stroke:.67px currentcolor}. With the visuals set, I recorded using OBS Studio and, on the rare occasion that I needed to edit something, I used DaVinci Resolve.

Concluding the Series

When we finished that last session, it felt rather anti-climactic, but I can look back on a very enjoyable time. And of course, the efforts that all participants put into this are not forgotten; we’ve got an amazing chat and video series that future APLers can enjoy. Thank you to everyone who contributed; chat participants, video commenters, colleagues (especially Brian Becker who authored all the problems), and most of all my wife, who often had to encourage me to record the next video(s) and also gave me the time and space to do so, even if it meant single-handedly keeping our children quiet. If you’re up for a marathon, you can watch the entire 22-hour series on YouTube, and APL Quest on the APL Wiki includes links to all the problems, chat sessions, code, and videos.