Welcome Rodrigo Girão Serrão

The story of how Rodrigo got his first internship at Dyalog is, in his opinion, a textbook example of serendipity. As 2020 started, Rodrigo began actively participating in an online code golf community, where people try to solve programming challenges in as few bytes of code as possible. Whilst his golfing skills were possibly lacking, the challenges he posted were usually well accepted. Posting many challenges meant Rodrigo got exposed to answers in all sorts of programming languages, from C, Java, Python and JavaScript, to Jelly, 05AB1E, Husk…and APL. Because of the context and the aspect of it, Rodrigo first thought APL was one of those “esolangs” and not a serious programming language.

Rodrigo’s fascination with APL led him to start frequenting The APL Orchard chatroom, where a small number of brilliant people convened to discuss all things APL. Here he met Adám Brudzewsky, who was keen on teaching APL to newcomers, and so began Rodrigo’s journey to learn APL.

His interest in APL kept growing, and he found it to be a simple and expressive language that also incorporated his affinity with mathematics. One day, while lurking in The APL Orchard, Adám asked Rodrigo if he would be interested in taking an intern position at Dyalog…a few emails later it was established that Rodrigo would work as a part-time intern at Dyalog during the Summer of 2020. This enabled Dyalog to make the most of Rodrigo’s skills in teaching and technical writing, and meant Rodrigo could indulge his passion for sharing knowledge about mathematics and programming while still finishing his MSc in Applied Mathematics. After his internship, Rodrigo took some time to complete his MSc thesis before returning to Dyalog to finish what he had started and hopefully to take part in many other interesting projects. When he is not working for Dyalog, Rodrigo may be found leading a Portuguese APL meetup, writing a blog post for his website (mathspp.com), or maybe leading a workshop or course. Other than working, Rodrigo likes to spend time with his loved ones, read fantasy books, eat chocolate, and watch silly comedy movies.

Welcome Shuhao Yang

Image

Shuhao joined Dyalog straight after he completed his Master’s degree in quantum computing, which happened to be during the second COVID-related lockdown in the UK. This has given him a quite unusual experience of starting a career as he has yet to meet a single member of the Dyalog team face to face! Shuhao obtained his Bachelor’s degree in mathematics – although he was interested in computer science, he studied mathematics as a way of looking for the root of CS and computing. He has broad interests across different software including Matlab, Python and LaTeX and has developed a solid knowledge on C++.

Shuhao enjoys the romantic theories in computer science and always wanted to work in one of the summit areas of CS – compilers, graphics and operating systems. He’s very happy that he now has the opportunity to work on the Dyalog APL interpreter.

The APL Orchard

If you have not already been there, I highly recommend visiting the APL Orchard (apl.chat). Adám Brudzewsky originally started this chat room on Stack Exchange to teach and answer questions, including many successful introductory sessions to people who wanted to learn APL. Since then, it has become an extremely active discussion forum, with a very wide range of interesting conversations, from regularly helping more newcomers get started with APL to theoretical discussions about the design of future array languages that will improve APL and perhaps compete against it some day.

Interested in APL?

One of Adám’s infamous invitations to a personalised APL introduction

Several of the most active participants have written their own APL implementations and some are working on array languages that are quite different from APL. It is a lively crowd, generating lots of thought-provoking discussions that feel a lot like the big arguments that I remember witnessing between Iverson, Benkard, Bernecky, Falkoff, Trenchard More and others at APL conferences back in the days when the APL language was still being born.

The participants include a mix of current, past and future employees of Dyalog Ltd, and it is a source of much useful inspiration for future work on our product and healthy challenges to conventional wisdom, so I try to spend some time there every day.

APL Orchard Stats

APL Orchard message statistics. The bottom contains the all-time total number of participants and messages.

To the New APLers

The Orchard is sometimes a challenging environment for a Dyalog CTO. Many of the bright young minds with computer science backgrounds that arrive here are quick to latch on to the fact that many things about APL go against the grain of what they were taught at university, and that a few of the design decisions made during the 55 years since the first APL implementation were questionable and could do with rationalisation. There is a constant barrage of complaints that we are not working hard to “fix” the things that they consider to be wrong about APL. This is, of course, how things should be when youthful enthusiasm meets us “fossils”; I fully understand many of the points made and actually agree with quite a few of them.

It is sometimes hard work having to repeatedly defend why Dyalog does not do things like just fix index origin at 0 and give the customers a few years to refactor their code. The implication is often that we are incompetent, and occasionally there is the insinuation that we are driven by questionable commercial motives like trying to get rich by locking the customers in and then doing a minimum of work.

Evolution, not Revolution

In my opinion, a large part of the value of our product rests on the fact that our customers can do things like read new legislation that affects the way their code needs to compute something, write code to deal with it before lunch, and then expect the code to keep working without changes until the law changes again, which might be in four years (depending on which country you live in), or in a few decades, after the current programmers have retired. Meanwhile, the code might need to move from the mainframe via UNIX and DOS to Microsoft Windows or macOS and then on to Linux in the cloud, and whatever comes after that.

APLers actually do enjoy refactoring code *if* it is in order to meet new challenges in the markets that they serve with products based on APL or compute better results. Our duty is to make this easy for them. On the other hand, there is no business value in refactoring code because we decided to change the way the language works, due to theoretical considerations – and it is our duty to protect them from that.

Breaking changes to Dyalog APL are quite simply not an option, at least not if users have to take immediate action. When we accidentally make a breaking change despite our best efforts not to, there is much wailing and gnashing of teeth, and we have to drop everything to fix it. That doesn’t mean we can’t make significant changes to how things work, but we must take an evolutionary approach, where the old way continues to work until everyone who needs it has moved on, possibly controlled by (sometimes undocumented) switches, like the one that forces APL to continue to give a DOMAIN ERROR on ¯1*0.5 to protect legacy code that relies on trapping the error in certain financial calculations.

Over the past decade, we have been able to move almost all of our user base from a 32-bit product with a fixed “alphabet” of 256 characters to using 64-bit APL with full Unicode support, in many cases without requiring any changes to application code. The stragglers are mostly major clients with support contracts that justify the additional cost of continuing to support the “legacy” version of the product until they are eventually able to move on. Once the Raspberry Pi stabilises as a 64-bit platform, we may finally be able to completely retire 32-bit Dyalog APL. I estimate that the “Classic” (non-Unicode) version probably needs to exist for another 5-10 years.

The next big challenges for us are to ensure that code can be moved to macOS and Linux (and “the cloud”) with a minimum of changes, and that both new and existing users have ways to integrate both new and legacy APL code nicely with modern source code management systems and continuous integration pipelines.

Existing Customers First

Our first priority at Dyalog is to support our existing clients and make sure that they remain competitive in their respective marketplaces.

Some contributors to the APL Orchard have suggested that this prevents us from being able to attract new users, which they believe can only happen if we quickly fix some of the “warts” in the language, or add some even more powerful language constructs.

First of all: not a single person who I consider to be a real prospect for writing substantial new application code in APL has brought up a single one of these issues as a reason why they might not want to adopt APL.

Secondly: if we do not first ensure our own financial stability, we will not have the resources to perform evolution, let alone revolution. For example, our investment in the creation of the APL Orchard, the APL Wiki, and the creation of new training materials for potential new users of APL, all depend on this.

As they say in the pre-flight briefing: put on your own oxygen mask before helping others.

Third: it is my very strong conviction that making breaking changes would violate the trust that exists between Dyalog and its customers, and I would personally consider it to be unethical. This may sound like a radical position, but it is one that I would expect to resonate with the younger generation. My personal motivation for working on software stems from witnessing the value that is created when people use software that I have helped create. Once someone starts using “my” product (and they continue to indicate that it is important to them by paying for support), I have an obligation to protect them from harm, if it is in my power.

Conclusion

APL is Fun
Old timers: If you are looking for some interesting discussions about current and future array languages, to help some newcomers, and maybe learn some new tricks yourself, I suggest that you head over to the APL Orchard and dive in, it is a lot of fun! You might want to check out apl.wiki/APL_Orchard, which has some helpful hints and tips, before you start.

New folks: I hope that this explanation of some of the philosophy that drives our work at Dyalog helps to explain some of the decisions that we make and that you find to be puzzling. I fully recognise that it is hard to imagine that a software company might have customer relationships that started before you were born 😊.

There are a couple of other topics that I should possibly also explain my views on, such as the dilemma of whether APL is a Notation or a Programming Language, and why Dyalog APL is not open source, but this post is very long already. Depending on the response I get to this post, I may return to that in the future.

Highlights of the 2020 APL Problem Solving Competition – Phase I

We received some excellent competition entries this year. Once again, thank you to all those who participated, congratulations to this year’s winners, and thank you to the Grand Prize Winner Andrii Makukha for his presentation at this year’s user meeting.

This post contains some suggested Phase I solutions along with some comments from the judges. Before each solution there is a brief summary of the problem and a link to the full problem on the practice problems site; you can also download the full set of problem descriptions as a PDF.

This page contains spoilers. The suggested solutions are not the only correct solutions, and may not necessarily be best practice depending on your application or most optimal in terms of performance, but they do solve the problems in some particularly interesting and elegant ways. If you’d like to try to solve the problems yourself before looking at these example solutions, you can use problems.tryapl.org to check your solutions.

1: Let’s Split

The first problem was to write a function splitting a vector right argument into a nested vector of vectors according to a signed integer left argument. For example:

      ¯3 (your_function) 'DyalogAPL'
┌──────┬───┐
│Dyalog│APL│
└──────┴───┘

Most entrants successfully solved this problem using dyadic take and drop .

{c←⍺+(⍺<0)×≢⍵ ⋄ (c↑⍵)(c↓⍵)}

It was common to use the left argument as-is with and , and swap the two parts of the result using ⌽⍣condition or condition⌽array, but the solution above avoids the swap by computing appropriate arguments to and .

Try it now with TryAPL

2: Character Building

In problem 2, we asked participants to partition a vector of UTF-8 character encodings, similarly to 'UTF-8'∘⎕UCS¨, without using ⎕UCS. For example:

      (your_function) 68 194 165 226 141 186 226 140 138 240 159 148 178 57   
┌──┬───────┬───────────┬───────────┬───────────────┬──┐
│68│194 165│226 141 186│226 140 138│240 159 148 178│57│
└──┴───────┴───────────┴───────────┴───────────────┴──┘

Eight participants submitted this (or a variation thereof):

{⍵⊂⍨1≠128 192⍸⍵}

Instead of doing multiple comparisons, this neatly uses interval index to check which range the argument is in. It then uses partitioned-enclose to create partitions beginning where code points are either below 128 or above 192.

Try it now with TryAPL

3: Excel-lent Columns

Problem 3 was simply to convert Microsoft Excel-style column letters to an integer. For example:

      (your_function) 'APL'
1104

Thirty-five participants submitted variations on this:

{26⊥⎕A⍳⍵}

While simple at first glance, it is actually quite involved because ⎕A⍳⍵ can give 26 (for Z) which isn’t a valid digit in base-26. However, decode handles out-of-bounds digits by carrying.

Try it now with TryAPL

4: Take a Leap

The task for problem 4 was to write a function to verify which of an array of integer years are leap years. For example:

      (your_function) 1900+10 10⍴⍳100
0 0 0 1 0 0 0 1 0 0
0 1 0 0 0 1 0 0 0 1
0 0 0 1 0 0 0 1 0 0
0 1 0 0 0 1 0 0 0 1
0 0 0 1 0 0 0 1 0 0
0 1 0 0 0 1 0 0 0 1
0 0 0 1 0 0 0 1 0 0
0 1 0 0 0 1 0 0 0 1
0 0 0 1 0 0 0 1 0 0
0 1 0 0 0 1 0 0 0 1

We had eight solutions like this one:

{0≠.=4 100 400∘.|⍵}

At first, it generates a 3-element vector showing whether the argument is divisible by 4, 100 or 400.

      0=4 100 400∘.|1900
1 1 0

The cleverness then is that ≠⌿ is used to express the logic of the leap-year algorithm. From Wikipedia:

if (year is not divisible by 4) then (it is a common year)
else if (year is not divisible by 100) then (it is a leap year)
else if (year is not divisible by 400) then (it is a common year)
else (it is a leap year)

We can check this with all possible length-3 boolean arguments:

      2⊥⍣¯1⍳¯1+2*3
0 0 0 1 1 1 1
0 1 1 0 0 1 1
1 0 1 0 1 0 1
      ≠⌿2⊥⍣¯1⍳¯1+2*3
1 1 0 1 0 0 1

Consider each case in turn:
1. Leap year, return 1
2. Can never occur
3. Not a leap year, return 0
4. Can never occur
5. Can never occur
6. Can never occur
7. Leap year, return 1

It is good because it uses no explicit loops and keeps intermediate values flat (no nesting). The solution leverages that each leap year rule is an exception to the previous one, and this particular formulation employs an unusual inner product ≠.= (equivalent to {≠/⍺=⍵} for vector arguments) to compute the parity of the divisibilities.

Try it now with TryAPL

5: Stepping in the Proper Direction

Problem 5 was to create a list generator somewhat similar to iota . However, this list generator takes a 2-element integer right argument and returns a list starting from the first integer and either increasing or decreasing in steps of 1 until the last integer inclusively. For example:

      (your_function) 4 ¯3
4 3 2 1 0 ¯1 ¯2 ¯3

Only one person had this exact solution, though many solutions were not too far off:

{(⊃⍵)-0,(××⍳∘|)-/⍵}

This dfn contains a 3-train or fork. Having seen contestants use the following format before, we feel compelled to provide you with a commented version of the above:

{
               -/⍵   ⍝ The length of the result is 1 more than the difference
           ⍳∘|)      ⍝ Integers up to the absolute difference
          ×          ⍝ times
        (×           ⍝ The sign of the difference
      0,             ⍝ Make the range inclusive
     -               ⍝ Use arithmetic to compute the correct result
 (⊃⍵)                ⍝ From the first value
                  }

Alternatively:

{
 (⊃⍵)                ⍝ From the first value
     -               ⍝ to
      0,             ⍝ inclusively
        (×           ⍝ The sign of...
          ×          ⍝ times
           ⍳∘|)      ⍝ Integers in the range of...
               -/⍵   ⍝ The difference
                  }    

This one excels in only computing necessary values once, and cleverly adjusts the generated values to rise or fall as needed, using the sign of the difference between the beginning and end points of the target range.

Try it now with TryAPL

6: Move to the Front

The task for problem 6 was to move all elements in the right argument vector equal to the left argument scalar to the start of that vector. For example:

      'a' (your_function) 'dyalog apl for all'
aaadylog pl for ll

Only one participant found this train, though two others submitted dfns using the same idea:

∩⍨,~⍨

Instead of computing indices or selecting elements, this simply employs two set functions, intersection and without ~. The asymmetry of intersection, namely that it preserves duplicates from its left argument, is here used to great advantage.

Try it now with TryAPL

7: See You in a Bit

Problem 7 involved writing a function to compare set bits in the base-2 representations of its integer arguments. For example:

      2 (your_function) 7   ⍝ is 2 in 7 (1+2+4)?
1

Eleven solutions used this method:

∧/(≤/2⊥⍣¯1,)

Indeed, the problem is about finding particular set bits in a binary number, hence the 2⊥⍣¯1. The overall function is a 2-train or atop, where the right-hand function is itself a 3-train or fork.

We can break it down as follows:

∧/             ⍝ Are all of (0 if any are *not*)
  (≤/          ⍝ Set left bits also set in right
     2⊥⍣¯1     ⍝ in The base-2 representation of
	      ,)   ⍝ The left and right arguments?

The function less than or equal to only returns 0 where a left bit is not found in the right argument:

      5((⊢,⍥⊂⍪⍤(≤/))2⊥⍣¯1,)9   ⍝ Just a fancy way of visualising the intermediate and final result
┌───┬─┐
│0 1│1│
│1 0│0│
│0 0│1│
│1 1│1│
└───┴─┘

This is pretty impressive, as it both demonstrates array orientation (in treating both arguments together) and uses Dyalog APL’s fancy inverse operator ⍣¯1 to use as many bits as necessary, while keeping the two representations aligned.

Try it now with TryAPL

8: Zigzag Numbers

The solution to problem 8 returns a 1 if its integer argument’s digits consecutively rise and fall throughout. For example, 12121 is a zigzag number, but 1221 is not.

We saw a handful of solutions of this type:

∧/0>2×/2-/10∘⊥⍣¯1

We can decompose it like so:

∧/                  ⍝ Are all
  0>                ⍝ Negative for...
    2×/             ⍝ Consecutively different signs of...
       2-/          ⍝ The pairwise difference of...
          10∘⊥⍣¯1   ⍝ The digits of the input?

It constitutes a good example of how the pattern in trains often is a natural one (this is actually an 8-train), and also shows off two uses of pairwise application 2f/ to compute the pairwise difference (the sign of which indicates the direction from digit to digit) and then the pairwise product (which due to the rules for multiplication of signed numbers indicates if a change has happened or not).

Try it now with TryAPL

9: Rise and Fall

Problem 9 involved writing a function to verify if a numeric vector has two properties:

  • The elements increase or stay the same until the “apex” (highest value) is reached
  • After the apex, any remaining values decrease or remain the same

For example:

      (your_solution)¨(1 2 2 3 1)(1 2 3 2 1)(1 3 2 3 1)
1 1 0

Actually, nobody had this exact solution, however, a handful came very close:

⊢≡⌈\⌊∘⌽⌈\∘⌽

Instead of trying to analyse the numbers, it does a running maximum from the left and from the right. If the minimum of those matches the original numbers, then we have exactly one peak.

⊢             ⍝ The input vector
 ≡            ⍝ matches
  ⌈\          ⍝ The max-scan from the left (msl)
    ⌊∘⌽       ⍝ The lower of msl and ⌽msr
       ⌈\∘⌽   ⍝ The max-scan from the right (msr)

We can visualise ⌊∘⌽ by stacking its arguments on top of one another:

      1 3 5,[0.5]⌽2 5 4
1 3 5
4 5 2
      ⌊⌿1 3 5,[0.5]⌽2 5 4
1 3 2
      1 3 5⌊∘⌽2 5 4
1 3 2

When used with the two max-scans, we can see how this solution works.

      (⌈\,[0.5]∘⌽⌈\∘⌽)1 3 2  
1 3 3  
3 3 2  
      (⌈\,[0.5]∘⌽⌈\∘⌽)1 0 2  
1 1 2  
2 2 2

Try it now with TryAPL

10: Stacking It Up

The task for problem 10 was to format a nested vector of simple arrays as if displayed using {⎕←⍵}¨, and then to return the formatted character matrix ({⎕←⍵}¨ simply returns its argument). For example:

      (your_function) (3 3⍴⍳9)(↑'Adam' 'Michael')(⍳10) '*'(5 5⍴⍳25)
1 2 3               
4 5 6               
7 8 9               
Adam                
Michael             
1 2 3 4 5 6 7 8 9 10
*                   
 1  2  3  4  5      
 6  7  8  9 10      
11 12 13 14 15      
16 17 18 19 20      
21 22 23 24 25

We had a couple of entries like this:

{¯1↓∊(⍕¨⍵),¨⎕UCS 13}

This was a tricky problem, especially as the automated testing didn’t include the 'a'1 test case, and many didn’t catch that one. Whilst most people wrote complicated code to get matrices for the element arrays, two participants thought outside the box, and simply joined the arrays with newlines after converting them to text.

Try it now with TryAPL

Conclusion

As always, not only are we deeply impressed by the ingenuity and cleverness of your submissions, but we also continue to be amazed by the number of people successfully solving most, if not all, of the problems in Phase I.

If you’d like to be notified when next year’s competition launches, go to dyalogaplcompetition.com and submit your email address.

Enhanced Debugging with Function Keys – Evaluate selection

See also Enhanced Debugging with Function Keys.

When tracing through a complex dfn and reaching a guard (condition:result), I am often wary of tracing into that line because if the condition evaluates to 1 then the current function I’m tracing through will terminate and return the result, leading to me losing situational awareness. Normally, I’d select the condition expression, copy it, move to the session and execute the expression, so I can predict what will happen next. Can we automate this? Yes we can.

Now, I usually prefer the Windows IDE for my daily development, but this is actually a case where RIDE has neat feature that’s missing from the IDE (but if you keep reading, I’ll show you how to achieve a similar effect in the IDE). In RIDE, go to Edit ⇒ Preferences ⇒ Shortcuts (or simply click ⌨︎ at the right end of the language bar), then type the name of a function key you want to use for this purpose, followed by a space, for example “F6 ” for . You’ll see exactly one entry in the listing. In the input field, write “<VAL>” (without quotes):

I defined a simple function to test it with, and traced into that:

      ⎕VR⎕FX'f←{' '⍺∧⍵:''both''' '⍺∨⍵:''either''' '''neither''' '}'
     ∇ f←{
[1]        ⍺∧⍵:'both'
[2]        ⍺∨⍵:'either'
[3]        'neither'
[4]    }
     ∇ 
      f

Tracing into f
Upon reaching a guard, I select the condition:
Selecting the condition
And Press :
Pressing F6
Voilà!

Cool, but how about the IDE?

Right, the Windows IDE doesn’t support the VAL command code, but we can easily emulate it by combining multiple command codes and assigning them to an F-key using ⎕PFKEY.

What we need to do is:

  1. Copy the current selection
  2. Jump to the session
  3. Paste
  4. Execute
  5. Jump back again

Options ⇒ Configure… ⇒ Keyboard Shortcuts ⇒ Description gives that the command codes for “Copy”, “JumP between current window and session window”, and “Paste” are CP, JP, and PT. We use ER (you can find all but JP using the ]KeyPress user command too) to press . Here we go:

      'CP' 'JP' 'PT' 'ER' 'JP' ⎕PFKEY 6
┌──┬──┬──┬──┬──┐
│CP│JP│PT│ER│JP│
└──┴──┴──┴──┴──┘

Keep it so!

RIDE keeps its setting, but of course, I wouldn’t want to be bothered with setting this up for every IDE session. So here’s a trick to set up F-keys (or anything else for that matter). When Dyalog APL starts up, it will look for MyUCMDs\setup.dyalog in your Documents folder ($HOME/MyUCMDs/setup.dyalog on non-Windows). If this file contains a function named Setup, it will be run whenever APL starts:

      ∇Setup
[1]  '<F6> is: ','CP' 'JP' 'PT' 'ER' 'JP' ⎕PFKEY 6
[2]  ∇
      (⊂⎕NR'Setup')⎕NPUT'C:\Users\Adam.DYALOG\Documents\MyUCMDs\setup.dyalog'

And now, when I start APL:
Upon start

Dyalog ’20 – Recordings Now Available

We are happy to announce that the full set of recordings from Dyalog ’20 online is now available. So if you missed the all or any of the talks, or would like to revisit one of the presentations, head over to https://dyalog.tv/Dyalog20!

It was disappointing not to be enjoying Portuguese food and drink with you all in Olhão. On the other hand, it was wonderful to be able to share our plans and user stories with so many people who would not normally be able to travel to one of our user meetings. According to the statistics, we had about twice the usual number of attendees, and Dyalog ’20 may have been the largest gathering of APL users in the last quarter century!

We learned that we need to invest in better microphones and find better solutions for chat both during and between the presentations, but in general we feel that the online format worked so well that we are making plans to run similar events in the future, even if international travel restrictions should ease and we are able to meet many of you face to face in Portugal this coming October. We are still thinking about the details, but it is likely that we will host an online meeting each spring with a focus on new users of Dyalog APL, while the autumn (fall) meeting will continue to provide experienced users with the usual “deep dive”.

We are also planning to offer workshops and other training sessions at other times of the year, and continue the regular series of webinars. Travel restrictions are helping to accelerate our plans to provide a steadily increasing quantity of online material. If there is sufficient interest, I am willing to expand my talk on Docker containers into a half-day “Bring Your Own Application” workshop early in 2021. If you would like to attend this workshop, or you have ideas for other topics for webinars, workshops or talks at future user meetings, please write to usermeeting@dyalog.com and tell us about it!