There has been a total of 1211 commits since the last blog post, so we are probably overdue another update.
Unfortunately there has not been many screenshots until fairly recently as I continued to dive deeper down into the rabbit holes I mentioned in the previous post and ended up completely rewriting the core of the emulator so that we have a boot process which matches similarly to the real system. This means that in decaf-emu we start by booting IOS, and then IOS MCP process boots the Cafe Kernel which in turn boots the Loader process to load the CafeOS userland. I then went on to rework the CafeOS userland such that one day we could support multiple processes, as the Wii U can have both a foreground and background process running (like when a game opens the Internet Browser), as well as a few other system ones such as the Loader or the Home Button Menu. During this process, which took over a year, I was not even able to compile my emulator let alone run a game in it.
2017-05-08
At the end of the previous post I mentioned one of my goals was to create a latte shader assembler, turns out I started work on it the next day.
2017-05-10
And 2 days later, the initial commit of it!
github: 7c8466cc197b84be3a7598da8bed8bccc27c40d2
2017-05-12
I then went on to experiment with the idea of implementing something like dolphin's FifoCI for automated testing of graphics. Unfortunately our PM4 (the communication protocol with the AMD R600 series GPUs) replay mechanism is not yet mature enough to make this tool useful so it was shelved for now. But you can see from this screenshot I had the replay dumping each frame to images, which when the tool is ready would be used to compare against previous images and automatically detect changes to notify us of regressions.
github: 5a954424f49b33ac74889d88981e52e25192099c
2017-05 to 2017-07
Nothing notable happened here, just many small bug fixes and improvements to code quality.
2017-08 onwards
This is when I seriously started work on the huge rewrite which goes on to take up a year of my life. The first step was to completely emulate the whole IOS operating system, in the previous blog I talked about doing IOS but that was only implementing IOS devices which coreinit could ioctl with. Unfortunately because of this there are not many screenshots to show between here and when I finally finished this effort.
2017-09-23
Shoutout to Nintendo for always leaving us behind some useful debug information.
2017-09-24
You discover some fun things when reversing the internals, like the fact that the Wii U apparently supports using ~ in paths for your "home directory". Although I cannot remember where ~ actually points to.
2017-10-03
I found where the IOS_MCP process initialises the filesystem, this was useful.
2017-10-06
This is from the Loader, this is the list of binaries of which their text/code sections are loaded into a shared memory area which all processes will use.
2017-10-10
Here you can see that the MCP process in IOS is responsible for powering on / off the PPC processor.
2017-10-11
Taking notes whilst trying to figure out the full path to get into a userland process. Note I cannot vouch for the accuracy of this as I don't know if these were my final notes or not, this is just a screenshot taken at a moment in time.
2017-10-11
Sometimes IDA generates fun graphs.
2018-05-11
Shoutout to one of my personal heroes.
2018-05-22
I was also working on improving wut (my Wii U Toolchain for writing homebrew) alongside my work on decaf, especially as I had spent so much time reversing the real Loader I had much more knowledge of what was possible. I discovered that there was a custom relocation type which was perfect for translating R_PPC_REL32 relocations in the .elf files output by gcc into .rpx files compatible with the Wii U.
2018-06-13
This is a homebrew made with wut using the official swkbd.rpl
2018-08-02
Reversing and coding an implementation identical to the official loader was hard work and at time of writing this blog it stands at 8,469 lines of code over 45 files.
2018-08-16
Finally able to load games again, over a year after the huge rewrite begun. I love this screenshot because it represents the result of a full year of reverse engineering and programming where I could not even compile my code, let alone succesfully load a game. It also shows something I think is pretty cool, you can see see in the Memory window all our internal HLE functions, I rewrote how our HLE implementation interacts with the userland processes so that we now actually generate a .rpl file at runtime for each HLE library with its function and data exports which are then loaded by the Loader process as it would with a real .rpl. I thought this was cool, but I imagine the concept would require much more explaining to fully appreciate.
2018-08-27
Fixing a few bugs here and there in the huge rewrite and games start working again.
2018-08-31
That "filesystem bug" mentioned in the previous blog was actually a bug in the jit with the rlwinm, many many many thanks to Kinnay for finding this. It was slowly driving me crazy as I could not figure out the issue myself. This went on to make many games work again, including all Unity engine games such as this very important historical piece:
2018-09-11
IDA is normally more useful than this. }}}}}}} promise }}}}}}}}}}}}}}}}}}}}}.
2018-09-16
Emulating an emulator.
2018-09-25
Mario Kart 8 is getting in game again.
2018-09-30
I got slightly distracted and wrote a small console output library for the Wii U which supports all the things one would expect from a console. Except I never got around to adding it to wut, maybe one day. My original intention was to use it to write a fancy exception handler for wut which provided more useful information and could allow homebrew to exit gracefully upon encountering an error.
ramblings
To others this may not have been the most interesting year, afterall this was the year of people proclaiming "decaf is dead" whilst I quietly slaved away slowly marching onwards to achieve my own personal goals. I think some people forget what these complex long-term programming projects mean to their authors, although I can only speak for myself. decaf-emu fulfills a role of artistic expression in my life and is a long term project to focus my ambitions and desires for continuous learning, exploration and self improvement. I learnt so much about the internals of the Wii U so this was probably my favourite year yet. There was some great refactoring and rewriting of the core internals of the emulator, along with many bug fixes and improvements, such that I can say I am feeling more proud of what we have achieved. Of course we still have a long way to go, but we made some amazing steps in the right direction.
Check out part 5 and part 6, for lots of images from the recent Vulkan backend.