GSoC Midterm Blog!

Hi! it's the end of the first phase of GSoC and It's Midterm evaluation time :D. It has been a great time, with a good amount of frustrating one, and a great learning journey so far. The community have been very friendly and helpful to me. I wan't to thank my mentors and the RTEMS contributors: Gedare Bloom, Alan Cudmore, Hesham Almatary, Chris John, Joel Sherrill, and all others.

In this blog, I will be summarizing of things that I have been learning and the patches that I have submitted in the last few weeks during the coding period.

Renode implementation to test Kendryte K210 in rtems-test

This is the main patch that I have been working on in this first period of GSoC. The idea was to be able to test a BSP with rtems-test that is using renode in the background. So, the expectation is that we can run something like this:

$ rtems-test --rtems-bsp=bsp-name-renode ~/build-dir

Alan, one of my mentors, said that I can take a look at the new BSP that he has ported into RTEMS. It's the Kendryte K210, and he has a very good example repository of using renode-test framework for testing the Kendryte K210 RTEMS BSP implementation. Therefore, in my first implementation I chose to work in making Kendryte K210 being able to be tested using rtems-test framework.

The first piece of puzzle to understand is: how does rtems-test actually work? It's a challenging part because new implementation of testing framework -- I believe -- is not something that the RTEMS community would expect to happen often. So there is no complete walkthrough about it, it's different from something like adding a new rtems-source-builder config which has a very good documentation on how to do that. But, there is a good picture on the documentation that sort of show how does it work under the hood.

It uses the stdout of the simulator, and capture the output to see if it matches with the certain testing logic that is implemented in the test suites. Since I have been tinkering with renode for quite some time in the community bonding period, I know that Renode does not produce it's executable output from the stdout. It has its own window where the output is shown namely the staticAnalyzer. My next question is: how do I get the output of the executables then?

After reading the Renode's documentation, I found a section named "UART integration" that tells us different ways of getting the UART output -- that should be on the staticAnalyzer -- for our personal usage. It says that there are 2 ways of doing so: using UART pty terminal or using the socket terminal. The UART pty approach seems quite concerning because it states that:

This feature is available on Linux/macOS only.

Because RTEMS target Windows, macOS, and several Linux distribution to be supported, I decided that it's probably best to use the socket terminal. After discussing it with my mentors via email and discord, I decided to use it for the first attempt of the Renode implementation for rtems-test.

To be able to do so, I must know how the internals of rtems-test works. Spesifically, how does it get the stdout output from the simulator. This is where the python debugger comes in handy. There is a requirement before writing GSoC proposal that potential students should be able to compile and run the rtems executables. Because of that, I have the SPARC tool suite installed and I can use the rtems-test to test it.

$ rtems-test --rtems-bsp=erc32-sis build/sparc/erc32

Using that command, I started my python DAP (debugging analyzer protocol) session and just step in to the function and taking some notes of where potentially the simulator process is started. I found that in the exact line that does that, I also figured out one of the feature that rtems-tools has which is the console.

1def _dir_execute(self, data, total, index, rexe, bsp_arch, bsp):
2    self.process = tester.rt.exe.exe(bsp_arch, bsp, trace = self.exe_trace('exe'))
3    if not self.in_error:
4        if self.console:
5            self.console.open()
6        if not self.opts.dry_run():
7            self.process.open(data,
8                              ignore_exit_code = self.defined('exe_ignore_ret'),
9                              output = self.capture,
10                              console = self.capture_console,
11                              timeout = (int(self.expand('%{timeout}')),
12                                         int(self.expand('%{max_test_period}')),
13                                         self._timeout,
14                                         self._test_too_long))
15        if self.console:
16            self.console.close()

You can see that in the 4th and 15th line, there is an additional logic so that we can read the output of the simulator using a "console". It is said in the docs that The RTEMS Tester directly supports the telnet protcol as a console and can interface to the ser1net server. So I thought to myself, maybe I can use this console to read the output of the Renode's socket terminal. After knowing about the console, I started to create the renode script to run the socket server and I have my first implementation of renode working for the hello.exe test.

The main problem is, this test is waiting the console timeout to stop the console from reading the output and move on. This is a problem because there is no way of telling renode to stop executing after the console done reading all the output. This caused the rtems-test to only be able to test a single test exe and then we have a renode process that is unkilled hanging around. What we can do instead, is to terminate the renode after the test is finished by using the line hook integration on Renode. But even after doing so, there is a new error that the console failed to connect to the Renode socket terminal because it dies earlier before the console process get the chance to be spinned.

Confused, I decided to research another viable approach that I can use for the project. This is where I figured out that I can access python's socket library on the UART line hook of renode. After that, I got the idea of sending each line of the renode's output using the socket library instead of using the socket terminal from renode. Huh, how is that a benefit from using the previous approach? Well, after diggin deeper into the rtems-tools source code, I found that we can implement our own file handle for the _readthread of rtems-test (supplied by the argument). If I can implement a custom class that has a similar method as the file handle, I can implement my own custom logic so that it reads the socket message input from renode and kill the process after the test is completed. I know this is very hacky and arguable far from best practice, but Gedare (my mentor) said that its fine to have an initial implementation that is "ugly". As there might be suggestions from the community on how to do things better after submitting the patch.

Therefore, I spend my time to write the implementation. After a while, I realized that there are too many edge cases I need to cover because of how bad the idea was. To mention a few: I need to also make sure I can know the simulator process is Renode, handle a panic so we don't have a socket server process just hanging like that, how to make Renode wait for the socket server to start before sending the message, etc.

At this moment I decided to see if it is really not possible to output the renode's uart to stdout. Fortunately, after filing an issue to Renode's Github repository, turns out I can use a logAnalyzer to replace the staticAnalyzer so that it outputs the result on the stdout. The only problem was: there are some additional data from the logs I don't need but prepended by Renode. I tried to use sed to trim the command and it seems like rtems-test does not support piping operation. I give some time to help with that and sent my second patch on supporting pipe operation for rtems-test. After that, I also managed to have my working implementation of Renode for testing Kendryte K210 and send the patch to the community to be reviewed.

Building Renode via RSB

One of the feedback from the Renode's patch is that there should be a build config in the RTEMS Source Builder (RSB) to build Renode. And that is exactly what I tried to do after having the Renode's patch being reviewed. First I try to follow the example of adding a new configuration on the RTEMS user docs. The problem is, I can't get it built in the first place and there is no sign of error whatsoever. At this moment, Gedare told me to have a daily email of my progress update on this problem followed by the log file attached to the email.

After knowing the problem and fixing it, I get it to download the Renode release package. It still has some error, but its better than no error. Not longer after that, I managed to have it sorted out. I have sent the implementation to the community and it has gotten a feedback which I fixed in the v2 patch that I have just sent. It is tested in Ubuntu and I will try to test it in Windows and FreeBSD. I tried to setup Windows dev environment for RTEMS but can't seem to get it right. I will do this in paralel with other tasks I aim to do next.

What's next?

Phew, that was quite a blog (longest I have written so far, what a shame). In the next half of GSoC period, I will be working in implementing Renode for testing 2 other BSP: leon3 and sihive hifive unleashed. In paralel, I will also setup other RTEMS supported host dev environment on my computer so I can test the previous RSB implementation just to make sure. Other than that, I will try to discuss it further with my mentors.

Thank you for reading! Cheers!