SPDX standardizes how source code conveys its copyright and licensing
information. See https://spdx.github.io/spdx-spec/1-rationale/ . SPDX
tags are adopted in many large projects, including things like the Linux
kernel.
This changes FileSystemProxy::Open to return a file descriptor wrapper
that will ensure the FD is closed when it goes out of scope.
By using such a wrapper we make it more difficult to forget to close
file descriptors.
This fixes a leak in ReadBootContent. I should have added such a class
from the beginning... In practice, I don't think this would have caused
any obvious issue because ReadBootContent is only called after an IOS
relaunch -- which clears all FDs -- and most titles do not get close
to the FD limit.
This makes it more convenient to emulate timings for IPC commands that
perform internal IOS <-> IOS IPC, for example ES relying on FS
for filesystem access.
According to hwtests, older versions of IOS are slower at performing
various filesystem operations:
https://docs.google.com/spreadsheets/d/1OKo9IUuKCrniz4m0kYIaMP_qFtOCmAzHZ_zAmobvBcc/edit
(courtesy of JMC)
A quick glance at IOS9 reveals that older versions of IOS have a
simplistic implementation of memcpy that does not optimize large copies
by copying 16 bytes or 32 bytes per chunk, which makes cached reads
and writes noticeably slower -- the difference was significant enough
that the OoT speedrunning community noticed that IOS9 (the IOS that
is used for the OoT VC title) was slower.
Instead of constructing IPCCommandResult with static member functions
in the Device class, we can just add the relevant constructors to the
reply struct itself. Makes more sense than putting it in Device
when the struct is used in the kernel code and doesn't use any Device
specific members...
This commit also changes the IPC command handlers to return an optional
IPCCommandResult rather than an IPCCommandResult. This removes the need
for a separate boolean that indicates whether the "result" is actually
a reply, and also avoids the need to set dummy result values and ticks.
It also makes it really obvious which commands can result in no reply
being generated.
Finally, this commit renames IPCCommandResult to IPCReply since the
struct is now only used for actual replies. This new name is less
verbose in my opinion.
The diff is quite large since this touches every command handler, but
the only functional change is that I fixed EnqueueIPCReply to
take a s64 for cycles_in_future to match IPCReply.
Now that the ES class (now called ESDevice) and the ES namespace do
not conflict anymore, "IOS::" can be dropped in a lot of cases.
This also removes "IOS::HLE::" for code that is already in that
namespace. Some of those names used to be explicitly qualified
only for historical reasons.
There are no functional changes.
Some of the device names can be ambiguous and require fully or partly
qualifying the name (e.g. IOS::HLE::FS::) in a somewhat verbose way.
Additionally, insufficiently qualified names are prone to breaking.
Consider the example of IOS::HLE::FS:: (namespace) and
IOS::HLE::Device::FS (class). If we use FS::Foo in a file that doesn't
know about the class, everything will work fine. However, as soon as
Device::FS is declared via a header include or even just forward
declared, that code will cease to compile because FS:: now resolves
to Device::FS if FS::Foo was used in the Device namespace.
It also leads to having to write IOS::ES:: to access ES types and
utilities even for code that is already under the IOS namespace.
The fix for this is simple: rename the device classes and give them
a "device" suffix in their names if the existing ones may be ambiguous.
This makes it clear whether we're referring to the device class or to
something else.
This is not any longer to type, considering it lets us get rid of the
Device namespace, which is now wholly unnecessary.
There are no functional changes in this commit.
A future commit will fix unnecessarily qualified names.
Nintendo's official title installation code and ES both only look at
content IDs but we should probably check for content hashes in addition
to checking for IDs for at least two reasons:
1. Some of the installed contents could be corrupted -- this cannot be
easily detected without checking hashes.
2. Some mod distributors do not bother to update content IDs, which
means that installing updates from the UI would not actually
update the installed game. This is confusing for users.
To keep the existing semantic (for IOS especially), the new content
hash checks are opt-in for callers of GetStoredContentsFromTMD.
This commit changes WiiUtils's WAD installation logic to enable
the content hash checks.
On some platforms (like Windows), the temporary file must be closed
before it can be renamed.
I guess nobody noticed this for so long because (1) the FS code has a
failsafe for missing FST entries (because existing users do not have
a FST), and most games do not care about file metadata;
(2) the write failures can only be seen in the logs.
Because we don't want this to break, I have turned the ERROR_LOGs into
PanicAlerts.
This fixes CreateFullPath to not create directories when it is known
that they already exist, instead of calling CreateDirectory anyway
and checking if the error is AlreadyExists. (That doesn't work
now that we have an accurate implementation of CreateDirectory
that performs permission checks before checking for existence.)
I'm not sure what I was thinking when I wrote that function.
Also adds some tests for CreateFullPath.
Now that all FS functions that create new inodes are properly
implemented, we can make GetMetadata actually return correct file
metadata rather than giving fixed information. The hack for the DQX
installer can also be removed now since our ES and FS keep track of
caller UID/GIDs now.
With the CreateFile/CreateDirectory fix and this commit, we can
finally return correct results in ReadDirectory and the sorting
hack -- whose purpose was to prevent certain versions of the
System Menu from crashing -- can be removed too.
Previously, the FS root directory would get created as a side
effect of calling CreateDirectory during boot (since the
implementation was sloppy and used File::CreateFullDir).
Since CreateDirectory no longer does that, it is necessary to ensure
that the FS root directory does exist by creating it explicitly.
Some official titles rely on implementation details of Nintendo's
FS sysmodule and will not work properly if those are changed.
Notably, some games and older versions of the System Menu appear
to be relying on the order of files returned by FS::ReadDirectory
and will either fail to find their save data (for Bolt) or
outright crash (for the System Menu).
Some titles also actually expect filesystem metadata to be correct.
One title that has been confirmed to do this is DQX, which generates
paths based on the GID of files within its own title directory.
While it is easy to make workarounds for these issues -- and in fact
we already do have some for the sysmenu and DQX, having hacks
is obviously nonideal and adding yet another hack would be required
to fix Bolt -- one that would be even uglier.
Furthermore, while it is currently unknown whether any official
title cares about permissions, the lack of FS metadata means that
we are unable to implement them if that turns out to be desirable
or necessary.
By adding a FST, we can implement things correctly and solve all
those problems without hacks.
Apart from DQX, the sysmenu and Bolt, this changeset also fixes
the Photo Channel complaining about corrupted system files
on the initial launch.
This first commit adds the basic structures and functions that
are necessary to load, save, query and update our version of the FST.
For simplicity, a binary format that is inspired from Nintendo's FST
structure was chosen for serialization. It is not expected to ever
receive an update.
PS: an update on the NAND image backend:
A long time ago I had planned to add another FS backend which would
be using a NAND image/blob as the storage. While I have already
written an implementation that has been tested, solves all the
aforementioned issues and more, produces images that are fully
compatible with IOS's FS driver, I feel like NAND images raise too
many issues: savestate sizes, code complexity and maintenance cost.
Since many fixes and additions that are part of that implementation
(e.g. FS timings, utility structures, FST) have already been merged
or will be submitted as part of this changeset, I will likely not
submit the branch.
Previously the logging was a in a little bit of a disarray. Some things
were in namespaces, and other things were not.
Given this code will feature a bit of restructuring during the
transition over to fmt, this is a good time to unify it under a single
namespace and also remove functions and types from the global namespace.
Now, all functions and types are under the Common::Log namespace. The
only outliers being, of course, the preprocessor macros.
This could happen with savestate loads, permission issues, or use by other processes.
Prior to this Dolphin assumed any existing file could be opened and crashes from invalid variant access.
Failing to open a file during savestate load will likely still crash but at least the user will know why.
As suggested here: https://dolp.in/pr7059#pullrequestreview-125401778
More descriptive than having a std::tuple of FS::Mode, and lets us
give names to known triplets of modes (like in ES). Functions that
only forward mode arguments are slightly less verbose now too.
The existing backend did not handle cases where the target exists
correctly.
This is a bug that has been around forever but was only recently
exposed when ES started to use our FS code.
Also adds some unit tests to make sure this won't get broken again.
Creating a file then opening it in read write mode is a pretty common
operation. This commit adds a helper function that makes it easier
to read and clearer.
